Showing posts with label Open Source GIS. Show all posts
Showing posts with label Open Source GIS. Show all posts

Wednesday, August 24, 2016

Automatic PostgreSQL tuning in Ansible

Once upon a time, there was a tool called pgtune, which I heavily used for setting reasonable defaults for automatically deployed PostgreSQL instances.

Unfortunately, this tool is not maintained for a long time, doesn't support new PostgreSQL versions and finally it was removed from Debian and all it's derivates.

I understand, that proper database configuration is result of deep knowledge of your data and queries, but in current world of automated deployment and containers, it is much more better to configure your PostgreSQL instance with some computed values based on current environment, then leaving default values designed for some very low memory system.

Therefore I have started to search Internet for some pgtune alternative and found great, new web based pgtune replacement developed by Alexey Vasiliev (le0pard).

Since I am using Ansible for my systems orchestration, I have decided to rewrite it's computing logic as a new Ansible postgresql_tune module.

Module requires following information to compute results:
  • PostgreSQL version
  • deployment type
  • total memory
  • maximal number or connections
  • path to resulting PostgreSQL file
  • path to resulting sysctl file

Example usage:

- name: Install PostgreSQL configuration
  postgresql_tune:
    db_version: "{{ POSTGRESQL-VERSION }}"
    db_type: dw
    total_memory: "{{ ansible_memtotal_mb }}MB"
    max_connections: "{{ POSTGRESQL-CONNECTIONS }}"
    postgresql_file: "{{ pgsql_config_dir }}/99-postgresql-tune.conf"
    sysctl_file: "/etc/sysctl.d/99-postgresql-tune.conf"
  notify:
    - service postgresql restart
    - sysctl postgresql reload
  become: yes


Source code is here and any feedback is very much appreciated.

Monday, February 29, 2016

New print from QGIS templates in GIS.lab Web


During implementation of print for our new generation of GIS.lab Web, our core developer Marcel came with very interesting proposal of interactive work with map content inside of print template produced by QGIS, which I haven't seen anywhere else.

The main idea is, that once print is activated in GIS.lab Web, it will download raw print output from QGIS Server using GetPrint request and will allow interactive visualization of map content directly in this template. You can zoom, pan and rotate map and see exactly how the result will look like. To get the better idea, see video below.

Print template in QGIS Print Composer
 Print template in GIS.lab Web

Video presentation of print in GIS.lab Web

Marcel and Rasto, thanks for very good work !

Thursday, February 4, 2016

GIS.lab Web - New Generation QGIS Web and Mobile Interface

Last year, GIS.lab team has decided to start a development of brand new version of our GIS web publishing interface known as GIS.lab Web. Our goal is to have a web application built on top of modern technologies, with very modern UI, optimized for mobile devices. Also, we want to build a hybrid Android application from the same code base, but with native UI interface. All of this standing on the shoulders of our favourite QGIS desktop and server software.

We have realized very soon, that if we would separate this project from the core GIS.lab system, we can produce generally usable QGIS web interface, usable with or without GIS.lab infrastructure. Maybe, even replace current version of QGIS Web Client.

This week, first alpha version of GIS.lab Web was born.

For a quick look, see the screenshots below,





GIS.lab Web



watch our short screencast,

GIS.lab Web


or try it yourself.

Thanks to all helping this happen. Looking forward for any feedback.

GIS.lab Web on GitHub - https://github.com/gislab-npo/gislab-web

Thursday, July 24, 2014

FOSS4G Europe - hope we ALL meet together again

For me, it was very memorable week in Bremen. I came very excited and happy to home, to my family.

But, there is also one thing which makes me feel very bad.
At the same time, when all of us where enjoying this event, there was at least one person, speaking nearly the same language as me, who was probably little bit scared of how the return to home would look like. The home destroyed by madness and war.

Somehow I feel, that without a peace, nothing makes a sense. I hope, next year, we ALL meet together again and everybody will be returning to home with a smile on the face.
Tatiana, Ukraine, all peaceful people under the sun, god bless You !

Friday, January 31, 2014

High performance processing services in GIS.lab

This week I have attended GIS Ostrava conference in Czech republic with basic talk about GIS.lab. After my speech, I was asked to answer a question about missing Web Processing Service (WPS), namely PyWPS, in GIS.lab. Here is my short answer.

Currently, GIS.lab server acts as conventional server. It provides boot service, file, geo-database or chat server. From the first sight, it would be nice to implement also some WPS service there.
Since I have realized great and mostly free and unused power of bunch of modern multi-core client machines connected to GIS.lab network (which is increased by each connected machine), I have started thinking differently. I have started thinking about distributing load of all CPU intensive services also to all client machines in terms of load balancing. Modern multi-core machine wouldn't be hurt at all, if one of it's cores would be shortly occupied by computing task requested by some other member of GIS.lab network.

For better imagination, I would describe my idea on WMS service provided by QGIS Mapserver on which GIS.lab WebGIS application relies.
Currently, QGIS Mapserver is installed only on GIS.lab server and thus it can provide only limited number of concurrent requests. Moreover, too many concurrent WebGIS users could cause server overload and could slow down also all other server provided services.
Since all resources served by WebGIS are shared to all members of GIS.lab network and all members of network are running same operating system, it seems reasonable to implement load balancing service on top of QGIS Mapserver, which will redirect WMS requests randomly to all network members. In such way, we will improve situation drastically (disk load could be next bottleneck).
All we need, is to have all required server software installed on all machines in GIS.lab network. I am not in favor to install server software to all client machines. I am thinking about creating some processing software image (maybe LXC container) on server which will be mounted to all client machines and used in some chroot like environment.

This is simplified description of my idea, how to create high performance computing network from random office computers in a few minutes. Any comments ?

PS: Thanks for all listening my talk. Jachyme, once I will have this load balancing system, I will surely implement PyWPS on top of it.

Tuesday, January 21, 2014

Waiting for GIS.lab 0.3

GIS.lab 0.3 is going to be released in a few days. Here is a list of some interesting new features.

VirtualBox client support

Since 0.3, it will be possible to boot regular GIS.lab client session in VirtualBox and run it in full screen mode or in windowed mode along with applications of your favorite operating system (Linux, Windows, Mac).


Built-in IRC server

To allow effective communication between all GIS.lab users without a need of Internet connection, version 0.3 will provide built-in IRC server and prepared #gislab chat room.



Launching empty WebGIS project

Until now, WebGIS project was always based on some existing QGIS project. It means, that you had to create at least some basic skeleton of your project in QGIS, before it was possible to publish it on web.
Since 0.3, it will be possible to launch empty WebGIS project by visiting URL http://web.gis.lab in web browser or via applications menu GIS.lab > Empty WebGIS project. Empty WebGIS project will be automatically loaded with OSM and Google base maps and it will be prepared to start capturing initial data for your new project.


Data capturing and sharing in WebGIS

By my opinion, best new feature of 0.3 are improved data capturing tools, which allow to draw points, lines and polygons on top of WebGIS layers. All drawings are saved forever in GIS.lab database under unique ID and can be shared along with whole map via Permalink.

 

Simplified publishing of QGIS project in WebGIS

In 0.3 we have created QGIS plugin, which allows very simple publishing of created QGIS project to WebGIS. It provides possibility to choose desired base layers (OSM, Google) and automatically detects visible layers, zoom extent and used list of scales and more.


Detailed changelog can be seen here.

Thursday, January 2, 2014

QGIS development support in GIS.lab

One of a goals of GIS.lab project is to provide clean, uniform and predictable environment which can be easy used for GIS software development and testing. Support for GIS development is optional and can be enabled in configuration file before client image is installed.
GIS.lab project heavily relies on features provided by QGIS, therefore I have decided to put together some tools to automatize building and installation of this software from Git source code. Now we have two scripts which should be run on client machine.

  • gislab-dev-qgis-install - download latest QGIS source code and make install
  • gislab-dev-qgis - run QGIS installed from source code
These scripts are very simple. There is a lot of room for improvements. Any ideas ?

For more information see our wiki page.

Wednesday, January 1, 2014

GIS.lab runs on your Windows and Mac too

Since I have started my work on GIS.lab, I am receiving quite positive responses. In general, it seems that people like this idea, but in many cases they are not able to simply loose their favorite working environment by rebooting their machine to GIS.lab.  There is a clear demand to keep user's current Windows or Mac OS and run GIS.lab inside with all its benefits. In this post, I would like to explain possibilities of running both server and client sides on various operating systems.

GIS.lab server runs in virtual machine which successfully works without any difference on Linux, Windows or Mac. It is really up to your preferences which platform you will choose.

GIS.lab client can be launched as physical client machine where you will say goodbye to a system you are used to do your work until now, configure your machine to boot from LAN and enjoy a new GIS.lab environment. This option is, and always will be preferred, because it provides best performance.

In case you can't offer loosing your current system, there is already a possibility to launch GIS.lab client inside of your favorite Windows or Mac system in virtual client mode. This mode will boot the same client environment from LAN inside of VirtualBox machine and will keep your system. You can use GIS.lab client in a windowed mode, similar as it would be standard application or in fullscreen mode for best user experience.



For more informations how to install and use virtual client mode see our wiki page
Please, visit also our new Quick Start manual where you will find detailed step-by-step instructions how to install GIS.lab and create your first GIS project.


 

Thursday, November 28, 2013

GIS.lab 0.2 released

I have released GIS.lab version 0.2. This is a first major version which completes major ideas of this project. On this occasion I have also created simple web site.

Changes from version 0.1

  • renaming project to 'GIS.lab'
  • adding possibility to configure GIS.lab via configuration files (config.cfg, config-user.cfg)
  • adding possibility to restrict client machines by MAC address
  • refactoring of file sharing directory structure
  • adding possibility to mount external network storage on client machines
  • adding example data and QGIS project file
  • adding 'Conky' widget to client desktop, removing session manager from panel
  • adding support for custom projections configuration (added S-JTSK Krovak East/North).
  • adding WebGIS application for automatic QGIS project publication on web (formerly known as WMS Viewer)
  • adding additional software like Gimp, Inkscape, Google Earth ...
  • documentation update

Release notes

Sometimes after shutting down GIS.lab server ($ vagrant halt) it is not possible to launch it again ($ vagrant up). The reason is that GRUB boot loader is endlessly waiting for manual input.
To solve this problem shut down server machine using VirtualBox Manager GUI and uncomment line 'v.gui = true' in 'Vagrantfile' file. Start GIS.lab server again ($ vagrant up) and press ENTER in server's GUI console to boot.

Tuesday, November 19, 2013

New data capturing and collaboration tools in GIS.lab web application

GIS.lab WMS Viewer web application allows automatic publishing of user saved QGIS projects on web.
Few days ago we have expanded its possibilities with nice point capturing, GeoJSON export tool and possibility to share these data via permalink to other users.

Look how easy is to collaborate with other users and apply captured data back in QGIS.


Capture data and share map via URL
Export data to GeoJSON
Load data in QGIS

Sunday, November 10, 2013

How long it takes You ?

How long it takes You to deploy complete solution of GIS office including operating systems and software of central server providing geodatabase and web server, centrally managed 8 GIS editor's desktop computers and create three nice GIS projects on three different computers and publish them on web ?

Few days ago I have bought new laptop (Lenovo E330 with Inter i5 and 8 GB RAM).  I have installed it with Xubuntu 12.04 and encrypted LVM and made a test to accomplish this task with a help of GIS.lab (GIS.lab is an Open Source solution to create fully equipped, easy-to-use, pre-configured, centrally managed portable and unbreakable GIS LAN). 

The results where impressive:

  • installation of GIS.lab software dependencies from internet (Git, VirtualBox, Vagrant) - 4 minutes
  • downloading base Vagrant image - 5 minutes
  • GIS.lab installation ($ vagrant up ) - 20 minutes !
  • connecting my laptop to my company's LAN and booting GIS.lab on 8 computers from LAN - 7 minutes
  • creating three nice GIS projects in QGIS (data where prepared before) and publishing them on web using GIS.lab WMS Viewer - 12 minutes !
  • TOTAL: 48 minutes


Here are the results:



Do You know something which can faster ? Do You want to try it ? Any questions and comments are welcome.

Thursday, November 7, 2013

GIS.lab WMS Viewer improvements

WMS Viewer web application is one of the killer features of GIS.lab server (see my earlier post). It allows automatic publication of user created QGIS projects without any need of configuration. My colleague Marcel has spent a time with adding some nice features.

At first he added nice toolbar containing basic tools like pan, zoom and measures.


Than he improved application with nice feature info tool and attribute table. Currently it is only available when QGIS development version is used on server, because it depends on GetFeatureInfo output in GML format which was added just recently. You can install it manually from my PPA.

$ vagrant ssh
$ sudo su -
$ sed -i "s/qgis2/qgis-master/" /etc/apt/sources.list
$ apt-get update
$ apt-get install qgis qgis-common qgis-providers qgis-providers-common \
qgis-mapserver --no-install-recommends
$ service apache2 restart



Besides these features we have improved overall layers loading performance by grouping all overlay layer in one GetMap request. It seems that it helped a lot.

You can try these new features in master Git branch.  

Saturday, October 26, 2013

GIS.lab - Rapid deployment of complete Open Source GIS LAN

One thing which keeps me excited these days is my new project called GIS.lab which began as a tool for simple, unbreakable, always-ready-and-clean deployment of multiple machines running software for Geographic Information Systems - QGIS. Shortly after finishing first workable version I realized big potential of this technology and started adding new features.

What is GIS.lab

GIS.lab is a solution to create fully equipped, easy-to-use, pre-configured, centrally managed, unbreakable and portable local area network (LAN) solution offering complete office and GIS software. It is configured to work out-of-box, without any need of configuration or other behind a scene knowledge, allowing users to keep high focus on their task. To achieve this goals it uses modern virtualization techniques in conjunction with automatic server provisioning and management.


 How it works

GIS.lab consists from one host machine running Linux, Windows or Mac operating system and any number of client computers with nearly no requirements - no operating system, even no hard disk is required (but no problem if they exists). All these computers must be connected with gigabit cables and switch. 

Host machine has two software requirements - virtualization software VirtualBox or VMware Workstation (or VMware Fusion on Mac) and Vagrant software for automatic server provisioning and management.

After installing required software on host machine and downloading GIS.lab sources only single one command is needed to install complete GIS LAN solution with out-of-box running central operating system and users management, services as file sharing, PostGIS database, map server and unlimited number client machines ready for immediate GIS work.

That magic command is called 'vagrant up' and orders to automatically provision GIS.lab server inside of virtual machine on the host computer with all mentioned services. It also installs key technology - Linux Terminal Server (LTSP), which enables to boot Xubuntu fat-client images on client machines from LAN. Great advantage of fat-client setup is that it enables to use all client machine resources, it is very friendly to server resources and provides real desktop experience without any thin-client glitches.

Where to use

This solution offers very large number of possible deployment scenarios from effective deployment of computers in brand new office or rapid provisioning of portable lab in the field to school lab installations where students can gain various knowledge from Linux systems principles to GIS development.
Central management, automatic deployment and unbreakability of client machines can save huge amount of obtaining and maintenance costs.


Future plans

My main goal is to provide complete set of services and Open Source tools for work and software development in GIS. Whole system will be provided as all-in-one solution with out-of-box running services capable to run with minimal maintenance requirements.
Very nice example of such services is included WMS Viewer application which is automatically publishing every user saved QGIS project and it is running without any additional configuration.


Also, I want to create fine tuned client desktop environments providing intuitive user interface for various GIS software.


Very valuable improvement will be addition of large amount of tutorials and how-to documents for day-to-day GIS tasks which is planed to develop with collaboration of Department of Geography and Regional Development at University of Prešov.


Development state

Project is currently in early development stage, constantly changing and adding features, but it is pretty much usable. It is now possible to deploy client machines as needed, perform various tasks in QGIS, collaborate on data editing with other users via PostGIS database, share files and publish data with WMS Viewer application and more.
I have released version 0.1, but until 0.2 is out I recommend using current master version from Git, because it is containing many new features.

Project released under GNU GPL version 3 and it is hosted on GitHub where You can find more detailed instructions. Project page is https://github.com/imincik/gis-lab .

I really appreciate all early testers, contributors, ideas, comments and patches.