Exposing to Geo Search

Introduction

Google's new Geo Search (formerly named KML search but it's been expanded) is a great way to get data on the geospatial . We want to make it so GeoServers that people stand up are automatically searchable from Google Maps and Google Earth. So if you configure a layer in GeoServer, filling out all the appropriate metadata, then it should be exposed to geosearch.

Feature level crawling

Introduction

The key to making every feature in GeoServer automatically exposed on Geo Search is known URLs. One thought is to make it so vector based super overlays are the way that crawling is done. This is nice, since the crawling is given a natural hierarchy, which is probably the one we want. The downside is that we don't yet know the best methods for vector super overlays, and where features are in the hierarchy may change. So for now we should make the concerns of crawling and display orthogonal. This document will focus on the problem of exposing data to Geo Search.

Canonical urls

Our first aim should be to provide every feature with a canonical url. This should be stable over time. I like how atom does it with its 'self' relative url:

<link rel='self' type='application/atom+xml' href='http://www.google.com/base/feeds/snippets/9757221540783527947'/>

I believe we should include such a URL in most all our output. KML gives us a bit of a hook with it's atom:link. They seem to use it to credit the URL where it came from, but we can embed it to be the canonical location of each feature. Ideally this lets the search engine know that it's crawled it before, which should be helpful for when we get complex hierarchies of regions. We will want to return full features there, but we should have the rel link to notify the crawler that it may have already been crawled.

This will also let us return pages of kml that have many features in them. The crawler should hopefully become smart enough to index those as individual entities, instead of as a 'page'.

Root document

We should have a root document that is similar to a WFS Capabilities document, but is built to be crawled. It should be accessible in several different formats, just like FeatureServer. It should ideally also include all relevant meta information that's in the other capabilities documents. The default should probably be html, so it's readable by people. It can have a number of links to the various formats, and also links to represent itself as other formats:

<link rel='alternate' type='text/html' href='http://geoserver.org/geoserver/features.html'>
<link rel='alternate' type='applicationatomsvc+xml' href='http://geoserver.org/geoserver/features.atom'>
<link rel='alternate' type='application/vnd.google-earth.kml+xml' href='http://geoserver.org/geoserver/features.kml'>
<link rel='alternate' type='text/json' href='http://geoserver.org/geoserver/features.json'>

Each representation of the root document should then in turn link down to the individual featureTypes/layers. These should link to the same features as the format it is served in. Those in turn can have rel='alternate' links to alternate format representations, if possible.

featureTypes/layers

Each featureType/layer should have its own url:

http://geoserver.org/geoserver/features/roads http://geoserver.org/geoserver/features/parks

These would also be available as the formats above

http://geoserver.org/geoserver/features/roads.html http://geoserver.org/geoserver/features/roads.kml http://geoserver.org/geoserver/features/roads.atom http://geoserver.org/geoserver/features/roads.json

(For further thoughts on what the roads.html page should look like, see REST Overview Page)

kml, atom and json should all just link to the first maybe 100 features, with links to the following 100, ect. Each feature listed in that list of 100 should have a link to its canonical url, ideally with an atom:link rel='self' link if the format supports it.

The top level features resource should also support the WFS key value pair parameters:

http://geoserver.org/geoserver/features/roads?maxfeatures=10 http://geoserver.org/geoserver/features/roads?maxfeatures=10&format=kml http://geoserver.org/geoserver/features/roads?maxfeatures=10&startindex=11 http://geoserver.org/geoserver/features/roads?srsname=EPSG:900913 http://geoserver.org/geoserver/features/roads?propertyname=numlanes,geom http://geoserver.org/geoserver/features/roads?featureversion=345 http://geoserver.org/geoserver/features/roads?sortby=numlanes http://geoserver.org/geoserver/features/roads?bbox=0,0,10,10 http://geoserver.org/geoserver/features/roads?filter=population>8000000

Note that filters are CQL filters, not the normal ogc xml filters. Also note that 'format' is used instead of 'outputformat'. The rest should be similar to WFS calls. These should be able to be combined in any way. (do we want to consider sld's here? It'd be useful for kml format). Maybe for just this interface we could also consider like png output, since it would just ask geoserver to render the result. We should also probably alias start-index and max-results to startindex and maxfeatures, to have it in line with gdata apis.

Features

With WFS we already have stable identifiers, they're just not listed as stable urls. This should be easy to do:

http://geoserver.org/geoserver/features/roads/1
and of course http://geoserver.org/geoserver/features/roads/1.kml http://geoserver.org/geoserver/features/roads/1.atom http://geoserver.org/geoserver/features/roads/1.json

This is the exact equivalent of the wfs call: http://geoserver.org/wfs?request=GetFeature&featureid=roads.1

First steps

This page presents far more than we need to get started. All that we really need is kml, and we don't need any querying except maxfeatures and startindex. And relative links in the KML to the next set of features.

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://earth.google.com/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom">&gt;
	<Folder>
                <!-- This is taken from the WMS 'title'.  If we want something shorter we can do it from the 'name' -->
		<name>Spearfish data, served by The Open Planning Project</name>

		<!-- WMS 'contact organization'.  Could be 'contact person'? -->
		<atom:author>The Open Planning Project</atom:author>

		<!-- From online resource of WMS.  Could have this link to full KML dataset? -->
		<atom:link>http://topp.openplans.org</atom:link>

		<!-- For GeoServer this is meta information about the service, and whose standing it up.  Like it's
		     the address of the atom:author.  It looks like for KML it more means the location of the data? -->
		<address>349 W. 12th Street, New York, NY 10014, United States</address>

		<!-- from contact info in GeoServer -->
		<phoneNumber>212-219-6000</phoneNumber>
		
		<!-- From the 'abstract' section, can just do first lines -->
		<Snippet>This server is stood up to experiment with versioned editing, feature portrayal service, kml-ows5, 
and geosearch crawling.  Enjoy!</Snippet>
		
		<open>1</open>
		<LookAt id="RestrictedAreas_lookAt">
			<longitude>-103.7552528062868</longitude>
			<latitude>44.43278309383453</latitude>
			<altitude>0</altitude>
			<range>22324.91751352843</range>
			<tilt>0</tilt>
			<heading>0</heading>
		</LookAt>
		<NetworkLink>
			<name>Restricted Areas</name>
			<open>1</open>
		
			<!-- From abstract for FeatureType -->
			<Snippet>This shows the restricted sections in the area of Spearfish.  Unless you are 
heavily armed you should not go there</Snippet>
			<Url>
				<href>http://geo.openplans.org:8080/geoserver/features/restricted?start-index=1&max-results=100&format=kml</href>
			</Url>
		</NetworkLink>
		<NetworkLink>
			<name>Archeology Sites</name>
			<open>1</open>
                        <Snippet>This dataset comes from the famed GRASS spearfish data set. 
 Which is probably living on a bit past its time, but so it goes.  This shows archeology sites around spearfish.</Snippet>
			<LookAt id="ArchSites_lookAt">
				<longitude>-103.7552528062868</longitude>
				<latitude>44.43278309383453</latitude>
				<altitude>0</altitude>
				<range>22324.91751352843</range>
				<tilt>0</tilt>
				<heading>0</heading>
			</LookAt>
			<Url>
			<href>http://geo.openplans.org:8080/geoserver/features/archsites?start-index=1&max-results=100&format=kml</href>
			</Url>
		</NetworkLink>
	</Folder>
</kml>

notes: I think we could handle topp:restricted as topp/restricted. So you have /geoserver/features/restricted, which is a shortcut for /geoserver/features/topp/restricted. Whatever the default namespace is can be used without the extra /topp/ thing... We also might consider having two network links for each, one to be crawled, one to be viewed? The other thing we could do is have two top level KML documents, one to be crawled and one to be viewed. We also will have /geoserver/features/archsites.kml be an alias for the start-index=1&max-results=100, so we could consider just having those nicer versions listed as the url.

The result of following the archsites link would be as follows:

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://earth.google.com/kml/2.2"       
     xmlns:atom="http://www.w3.org/2005/Atom"
     xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/'>

   
   <Document>
      <name>opengeo:archsites 1 to 100</name>
      <atom:link rel='self' type='application/vnd.google-earth.kml+xml' href='http://geo.openplans.org:8080/geoserver/features/archsites?start-index=101&amp;max-results=100&amp;format=kml'/>
      <atom:link rel='next' type='application/vnd.google-earth.kml+xml' href='http://geo.openplans.org:8080/geoserver/features/archsites?start-index=1&amp;max-results=100&amp;format=kml'/>
      <atom:author>The Open Planning Project</atom:author>

      <!-- not sure if this is a good idea, it's the one thing not using an already existing KML construct, but it is taken from gdata apis -->
      <openSearch:totalResults>354</openSearch:totalResults>
      <openSearch:startIndex>1</openSearch:startIndex>
      <openSearch:itemsPerPage>100</openSearch:itemsPerPage>

      <Style id="GeoServerStylearchsites.1">
         <IconStyle>
            <Icon/>
         </IconStyle>
         <IconStyle>
            <color>ffffffff</color>
            <colorMode>normal</colorMode>
            <Icon>
               <href>http://maps.google.com/mapfiles/kml/pal4/icon25.png</href>
            </Icon>
         </IconStyle>
      </Style>
      <Placemark id="http://geo.openplans.org:8080/geoserver/features/archsites/1">
      <atom:link rel='self' type='application/vnd.google-earth.kml+xml' href='http://geo.openplans.org:8080/geoserver/features/archsites/1'/>
         <name>Signature Rock</name>
         <description>Signature Rock Description</description>
         <LookAt>
            <longitude>-103.82681673469851</longitude>
            <latitude>44.381622551256754</latitude>
            <range>700</range>
            <tilt>10.0</tilt>
            <heading>10.0</heading>
         </LookAt>
         <styleUrl>#GeoServerStylearchsites.1</styleUrl>
         <Point>
            <coordinates decimal="." cs="," ts=" ">-103.82681673,44.38162255</coordinates>
         </Point>
      </Placemark>



      <Style id="GeoServerStylearchsites.25">
         <IconStyle>
            <Icon/>
         </IconStyle>
         <IconStyle>
            <color>ffffffff</color>
            <colorMode>normal</colorMode>
            <Icon>
               <href>http://maps.google.com/mapfiles/kml/pal4/icon25.png</href>
            </Icon>
         </IconStyle>
      </Style>
     <Placemark id="http://geo.openplans.org:8080/geoserver/features/archsites/100">
      <atom:link rel='self' type='application/vnd.google-earth.kml+xml' href='http://geo.openplans.org:8080/geoserver/features/archsites/100' />

         <name>arch name</name>
         <description>arch description</description>
         <LookAt>
            <longitude>-103.71202919990733</longitude>
            <latitude>44.41414639584103</latitude>
            <range>700</range>
            <tilt>10.0</tilt>
            <heading>10.0</heading>
         </LookAt>
         <styleUrl>#GeoServerStylearchsites.25</styleUrl>
         <Point>
            <coordinates decimal="." cs="," ts=" ">-103.7120292,44.4141464</coordinates>
         </Point>
      </Placemark>
   </Document>
</kml>



It would be cool if there was some sort of NetworkLink that we could put for the 'next' set of features, but I couldn't find a way to do that without the google earth client attempting to follow it and resolve it. That would probably be fairly disastrous with huge, huge datasets, though we could try it out. But it'd be nice if there was a link at the end of the dataset that let users hit 'next'. Also something to think about is how to go from one of individual kml markers to the full set - could do a link in the description perhaps, that users would actively click on.

Actually, looks like we can do that really nicely with something like this:

<description>arch description 
<br/><a href='http://geo.openplans.org:8080/geoserver/wms/reflect?layers=opengeo:archsites&amp;format=kml'> 
click for full dataset</a></description>

That gives a little link to click on for each placemark, that gets really nicely turned in to a network link. What we'll have to do is figure out a way to append that to 'description' outputs automatically, at least for the links that get crawled. Easiest would be a param like links=true, or maybe one of the format_options we use for KML? Perhaps could have a nicer link like 'part of archsets dataset'.

Sitemap

(the above is probably much more the way we want to do things, just keeping this around for its info on sitemaps, as we still may want to autogenerate those, ect.)

For a first we don't want to pollute the geo search index with tons of data, so we'll just put a KML sitemap at the base of GeoServer that talks about the layers and provides links to the WMS for them (and perhaps super-overlays?). There is a google maps api [blog post| http://googlemapsapi.blogspot.com/2007/01/get-more-traffic-to-your-maps-api-site.html] on this:

Publishing a KML Sitemap involves a few easy steps:

  • Identify those features on your maps site which can be displayed as KML features.
  • Convert those features into KML equivalents and publish them within one or more KML files.
  • Create a sitemap.xml file that identifies links to all of your KML files.
  • Place the sitemap.xml in the root directory of your site.

Sitemap details

http://www.google.com/apis/maps/sitemap.html has some good information on setting up a kml sitemap. To do it with kml it's pretty basic, I think we just need to put the following type of thing at the root:

  <?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url>      
      <loc>http://www.example.com/example1.kml</loc>
    </url> 
    <url> 
      <loc>http://www.example.com/example2.kml</loc>
    </url> 
    <url> 
      <loc>http://www.example.com/example3.kml</loc>
    </url>
  </urlset> 

And then I believe we just need to do a KML file for each of our layers. Would be nice if we could somehow automatically exclude the demo files. Perhaps we need a 'publish' param that would expose the KML - and maybe even auto-generate a link from a wiki? And when someone hit the publish button we could do more checks to make sure they've filled out all their appropriate metadata?

KML file details

I talked with the Geo Search PM, and we sketched out a rough version of what should be in the site for a GeoServer instance. Unfortunately I lost the paper that I had the notes on, but I think I can remember most. But the first pass would be to just do a KML file that contains meta information and links to other places.

  • A KML polygon placemark of the bounding box for the dataset
  • Name: The title of the featureType (could fall back on name if its not a nice title? Better yet we should have our default title generation in featureTypes use the namespace and not append 'Type'
  • Address: Use the contact info stuff that we gather in the web admin and use in WMS
  • Phone Number: Same as address
  • Snippet: Start of the Abstract? Keywords?
  • description: The abstract for the layer, perhaps a preview png (can make a WMS call of the right bounds). And then consider at least one of a link to the superoverlay, or a link to the full WMS? We can make a network link to the WMS... Superoverlays look better, but are also a bigger performance hit, at least until we get JTileCache (after which they perform better than normal WMS)
  • LookAt: Derive in same way we do in WMS results?
  • Metadata: A describeFeatureType response? Or a link to the fgdc or iso metadata document if it's listed?
Added by Chris Holmes, last edited by Chris Holmes on Oct 21, 2008  (view change)
View Attachments (0) Info