In our previous blog post we highlighted the GeoServer Code Sprint 2017 taking place at the of this month. We are all looking forward to GeoSolutions hosting us in beautiful Tuscany and have lots of work to do.
One of the secrets (and this comes as no surprise) to having a successful code sprint is being prepared. With this year’s REST API migration from restlet to spring model-view-controller we want to have all technical decisions made, and examples for the developers to work from, prior to any boots hitting the ground in Italy.
But before we get into the details …
Code Sprint Sponsors
We would like to thank our sprint sponsors - we are honoured that so many organizations have stepped up world wide to fund this activity.
Gaia3D is a professional software company in the field of geospatial information and Earth science technology. We would like to thank Gaia3D for their gold sponsorship.
Insurance Australia Group (IAG) is our second gold sponsor. This is a great example of open source being used, and supported, by an engineering team. Thanks to Hugh Saalmans and the Location Engineering team at IAG for your support.
Boundless is once again sponsoring the GeoServer team. Boundless provides a commercially supported open source GIS platform for desktop, server, mobile and cloud. Thanks to Quinn Scripter and the Boundless suite team for their gold sponsorship.
How 2 Map is pleased to support this year’s event with a bronze sponsorship.
I am overjoyed FOSSGIS (German local OSGeo chapter) is supporting us with a bronze sponsorship. This sponsorship means a lot to us as the local chapter program focuses on users and developers; taking the time to support our project directly is a kind gesture.
Sponsorship Still Needed
While we have a couple of verbal commitments to sponsor - we are still $1500 USD off the pace. If your organization has some capacity to financially support this activity we would dearly love your support.
This is an official OSGeo activity; any excess money is returned to the foundation to help the next open source sprint. OSGeo sponsorship is cumulative. Check their website for details on how your helping out the geoserver team can be further recognized.
Update: Since this post was published we are happy to announce new sponsor(s).
Thanks to Caroline Chanlon and the team at Atol Conseils et Développements for bronze sponsorship.
Getting Ready for REST
In this week’s GeoServer meeting we had a chance to sit down and plan out the steps needed to get ready.
The majority of prep will go into performing the restlet to spring mvc migration for a sample REST API end point to produce a “code example” for developers to follow. We have selected the rest/styles endpoint as one of the easier examples:
Preflight check: Before we start we want to have a good baseline of the current REST API responses. We would like to double check that each endpoint has a JUnit test case that checks the response against a reference file. Most of our tests just count the number of elements, or drill into the content to look for a specific value. The goal is to use these reference files as a quick “regression test” when performing the migration.
Migrate rest/styles from StyleResource (restlet) to StyleController (spring): This should be a bit of fun, part of why spring model-view-controller was selected. Our goal is to have one Controller per end-point, and configure the controller using annotations directly in the Java file. This ends up being quite readable with variable names being taken directly out of the URL path. It is also easier to follow since you do not have to keep switching between XML and Java files to figure out what is going on. It is important that the example is “picture perfect” as it will be used as a template by the developers over the course of the sprint, and will be an example of the level of quality we expect during the activity.
Create StyleInfo bindings (using XStream for xml and json generation): The above method returns a StyleInfo data structure, our current restlet solutions publishes each “resource” using the XStream library. We think we can adapt our XStream work for use in spring model-view-controller by configuring a binding for StyleInfo and implementing in using XStream. _This approach is the key reason we are confident in this migration being a success; existing clients that depend on exactly the same output from GeoServer - should get exactly the same output. _
StyleController path management: There is some work to configure each controller, while we have the option of doing some custom logic inside each controller we would like to keep this to a minimum. _This step is the small bit of applicationContext.xml configuration work we need to do for each controller, we expect it to be less work then reslet given the use of annotations. _
Reference Documentation Generation: We are looking into a tool called swagger for documentation generation. Our current reference documentation only lists each end-point (and does not provide information on the request and response expected - leaving users to read the examples or try out the api in an ad-hoc fashion). See screen snap below, our initial experience is positive, but the amount of work required is intimidating.
Updated examples for cURL and Python: We would like to rewrite our examples in a more orderly fashion to make sure both XML and JSON sample requests and responses are provided. Ideally we will inline the “reference files” from the JUnit regression test in step 1 to ensure that the documentation is both accurate and up to date.
You can see a pretty even split in our priorities between performing the migration, and updating the documentation. We believe both of these goals need to be met for success.
Next stop Tuscany
Although this blog post focuses on the sponsorship/planning/logistics side of setting up a code sprint there is one group without whom this event could not happen - our sprint participants and in-kind sponsors (providing a venue & staff).
For more information:
This is a special year because the Geoserver Brazilian Community celebrates 10 years. It all started in 2007 with an GeoServer’ Course at III ENUM (event focused on MapServer) in Brasilia, federal capital, and since then the community has approximately 600 members and more than 5,000 messages exchanged during those 10 years.
During this time, GeoServer has been widely disseminated both in education, government and corporate. There are reports that large banking institutions, telecommunications and information technology companies in addition to governmental sectors such as IBGE (Brazilian Institute of Geography and Statistics), Embrapa (Brazilian Agricultural Research Corporation), IBAMA (Brazilian Institute of Environment and Natural Resources Renewables), among others, has used GeoServer in its various projects.
However, the most notable among his many successes was the establishment of GeoServer as the official map server of INDE (Spatial Data Infrastructure), in 2008 by then President Lula.
We would like to thank everyone who has contributed to the growth of GeoServer in Brazil, whether writing articles, giving lectures or simply encouraging the use of GeoServer within its institutions. A special thanks to Andrea Aime who implemented the projection used by Brazil.
This is a beta release of GeoServer made in conjunction with GeoTools 16-beta.
We want to encourage people to test the release thoughly and report back any issue found. With no further delay, let’s see what’s new, that is, what is there to test!
YSLD module graduated to supported land
The YSLD styling language has been graduated to supported land, becoming an official extension. YSLD is a YAML based language which closely matches the stucture of SLD, and the internal data model that GeoServer’s renderer uses. Here is an example from the YSLD Cook Book:
<span class="l-Scalar-Plain">title</span><span class="p-Indicator">:</span> <span class="s">'YSLD</span> <span class="s">Cook</span> <span class="s">Book:</span> <span class="s">Simple</span> <span class="s">Line'</span> <span class="l-Scalar-Plain">feature-styles</span><span class="p-Indicator">:</span> <span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">name</span> <span class="l-Scalar-Plain">rules</span><span class="p-Indicator">:</span> <span class="p-Indicator">-</span> <span class="l-Scalar-Plain">symbolizers</span><span class="p-Indicator">:</span> <span class="p-Indicator">-</span> <span class="l-Scalar-Plain">line</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">stroke-color</span><span class="p-Indicator">:</span> <span class="s">'#000000'</span> <span class="l-Scalar-Plain">stroke-width</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">3 </span>
Compared to SLD it sports a similar structure, but in a more compact and readable syntax. Here the same example can be expressed in a more compact format:
<span class="l-Scalar-Plain">line</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain"> stroke-color</span><span class="p-Indicator">:</span> <span class="s">'#000000'</span> <span class="l-Scalar-Plain"> stroke-width</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">3</span>
Similarly to SLD and CSS, the user guide contains both a reference and a cookbook to get you started. There is also a styling workshop with both CSS and YSLD examples (including an important example for converting your SLD styles to YSLD).
Vector tiles graduate to extension
The vector tiles community module graduated to extension. The module allows to generate vector tiles out of WMS and WMTS requests, using SLD to filter the contents of the tile at the requested scale. Supported vector tiles formats are MVT (MapBox Vector Tile), GeoJSON vector tiles, and TopoJSON vector tiles.
This allow for much more compact data transfers, reduced tile caches thanks to overzooming support, and allows the client to control styling. Here is an example of the client rendering a world map as a vector tile, and highlighting a specific area by attribute:
Hop on the documentation to get more information about this extension, along with a couple of presentations highlighting vector tile usage.
Underlying labels in SLD
A new vendor option allow to underline labels in SLD (and CSS). Just add:
Opaque Container Layer Group Mode
A new layer group mode has been added, called “Opaque Container”. It’s similar to “single” in that you cannot see the layers contained in it, but it also prevents its layers from showing up at the root level of the capabilities document. This new mode is targeted to building “base maps” that the client is not allowed to take apart into their components.
You can read more about layer group modes in the GeoServer user guide.
Layer Group Security
The build-in data security can now secure layer groups too, for both global and workspace specific groups. “Tree” mode groups (named tree, container tree, opaque container) will extend the security rules applied to them to the layers contained.
Improved loading and OGC request times for large installations
GeoServer has historically had issues with long startup times on installation having several thousands layers. A group of recent improvements significantly sped up these times making it possible to handle tends of thousands of layers with no particular headaches.
For reference, some tests were run on a Core i7 860, 16GB of memory, 2TB hybrid drive (spinning disk plus SSD cache), on two different data directories:
“Many states”: 1 workspace, 1 store, 10k layers, 10k cached layers
“Large”: 1001 workspaces, 11000 stores (a mix of shapefiles, postgis, directory of shapefile, single tiff, arcgrid, mosaics), 42000 layers and 42000 associated tile layers
Startup times have been measured in both “cold” and “warm” mode, “cold” means the operating system file system cache contains none of the configuration files and the startup has to actually read everything from the disk, “warm” means the data dir is fully cached in memory instead:
Many states, cold startup: 30s
Many states, warm startup: 21s
Large, cold startup: 107s
Large, warm startup: 45s
As you can see, worst case is loading in excess of 40 thousands layers in less than a minute and a half!
For reference, “Large” startup times with JDBCConfig are 290s for cold startup, and 120s for warm startup. This is mostly due to cached layers loading, if your configuration has none the startup time will be of around 20-30 seconds instead (no matter how many layers are configured, because JDBCConfig loads layers on a as-needed basis at runtime… for a cost, see below).
It is also interesting to compare the times needed to run a GetMap against one of the “topp:states” in the “many states” data directory.
|2.10.1 + JDCBConfig||68||117|
Improved lookup of EPSG codes
Ever been annoyed by some shapefile with a .prj declaration that GeoServer does not understand? If so, try this new release, we pulled some extra heuristics and smarts to figure out the equivalent EPSG code of thousands more cases. This will hopefully reduce guesswork when configuring new vector layers, not to mention significantly speeding up importing large directories of shapefiles in the “Importer” extension.
Other assorted improvements
There are many improvements to look at in the release notes, cherry picking a few here:
New REST call allowing a users to modify his/her password (no UI for it yet, sorry)
Support JSON encoded exceptions for WFS 2.0 too (was already available for 1.0 and 1.1)
Drastically reduced output size for WMS PDF output format, when using a graphic fill based on repeated marks or SVG symbols
Improved scalability when using “advanced projection handling” thanks to a new implementation of SoftValueHashMap (used as a cache by the referencing subsystem)
Better transparency handling of NODATA and ROI (Region Of Interest)
Fixes in computation of global disk quota (if you find you have a negative number, upgrade, wipe out the quota database, and restart)
Test, test, test!
Now that you know about all the goodies, please go, download and test your favorite ones. Let us know how it went!
About GeoServer 2.11
GeoServer 2.11 is scheduled for March 2017 release. This puts GeoServer back on our six month “time boxed” release schedule.
GeoServer 2.10.2 is the latest stable release of GeoServer and is recommended for production deployment. This release is made in conjunction with GeoTools 16.2 and GeoWebCache 1.10.2. Thanks to all contributors.
New Features and Improvements:
Promote YSLD module to extension
Add jetty-servlets.jar file to default jetty to help users set up CORS
Handle non error http codes (from HttpCodeException) as normal response
Reducing output size of PDF with graphic fills by using tiling patterns
Disk quota usage page no longer shows negative bytes free (if you already have negative values in your quota DB, follow this instructions)
New style editor can now detect and fix old GeoServer CSS styles (which were marked as SLD)
Fixes to json output for layergroup
Can not display table based pages in Turkish locale
WMS 1.3.0 GetCapabilities response doesn’t validate against the schema when using LayerGroups
Editing style and moving it into workspace loses the edits
Style Edit Page: Apply then Submit causes WicketRuntimeException
Force transformation of ROI and NoData to transparent/bg color in direct raster rendering path
|And more! For more information on this release check the release notes (2.10.2||2.10.1||2.10.0||2.10-RC1||2.10-beta||2.10-M0 )|
About GeoServer 2.10
Articles, docs, blog posts and presentations:
The YSLD extension added, with extensive documentation
State of GeoServer 2016 (slideshare)
The style editor has been refreshed with the best ideas from the css extension
The styling workshop has been updated for foss4g 2016 and now includes both CSS and YSLD examples.
QGIS SLD export improvements (GeoSolutions)
A new community module to backup/restore and restore GeoServer configuration
A resource browser is available allowing remote management of styles, icons and fonts (needs building from sources).
A new WMTS multidimensional domain discovery community module for discovering patches of data in scattered data sets
GeoSolutions has offered to host the GeoServer team for a Java 2017 code sprint to look at updating, fixing and documenting the the GeoServer REST API. The GeoServer REST API is used to remotely manage a GeoServer instance and has proven highly successful for automation, integration with other applications, with libraries for java and python remote management.
The code sprint is dedicated to:
Migrating from the restlet library to Spring MVC library. As an early adopter GeoServer selected the restlet library as best of breed at the time. It has not aged well, and Spring MVC represents a supported annotation based solution that is familiar to more developers.
Although popular the REST API has not attracted a lot of investment, leading it to have the highest bug count of any of our GeoServer modules! This sprint would like to directly reduce this bug count, and indirectly reduce this bug count by introducing more developers to this area of the codebase.
The REST API also has the greatest number of requests for documentation and examples. This code sprint will update the documentation for each area of the REST API as as it is migrated, and look at some of the solutions for the automated collection of examples requests.
The GeoServer team has previously planned and executed a highly successful code sprint. We would like to once again ask for your support and participation in 2017.
The code sprint is planned for a week in March in the glamorous GeoSolutions headquarters of Viareggio, Lucca. Thanks to GeoSolutions for providing a venue, space is limited to 10-15 people so hit the wiki to sign up if you are interested.
A note on the timing: We were unable to join the Daytona Code Sprint 2017 as it is scheduled too close to the GeoServer 2.11 code freeze. GeoSolution’s offer to host in Europe will reduce travel costs allowing us to run the event with minimal sponsorship.
Participation and Sponsorship
We have the following sponsorship levels available:
We are reaching out to international and local sponsors. Contributions will be put towards travel costs for overseas sprinters who would be otherwise unable to attend. Any surplus at the end of the event will be turned over to OSGeo or used for a future code sprints.
For more details on participation, sponsorship or budget for the event please see the Java 2017 Code Sprint 2017 on the OSGeo wiki.