Perrygeo

Creating UTFGrids directly from a polygon datasource

2012-08-20T00:00:00-07:00

We’ve begun to rely on the interactivity provided by UTFGrids in many of our recent web maps. (Quick recap: UTFGrids are “invisible” map tiles that allow direct interactivity with feature attributes without querying the server.) Earlier this year, I created the initial OpenLayers UTFGrid support and was glad to see it accepted into OpenLayer 2.12 (with some enhancements).

With the client-side javascript support in place, the only missing piece in the workflow was to create the UTFGrid .json files. We had expirimented with several alternate UTFGrid renderers but Mapnik’s rendering was by far the fastest and produced the best results. Using Tilemill was a convenient way to leverage the Mapnik UTFGrid renderer but it came at the cost of a somewhat circuitious and manual workflow:

Load the data up into Tilemill,
Configure interactivity fields
Export to .mbtiles
Convert to .json files

What we really needed was a script to take a polygon shapefile and render the UTFGrids directly to files. Mapnik would provide the rendering while the Global Map Tiles python module would provide the logic for going back and forth between geographic coordinates and tile grid coordinates. From there it’s just a matter of determining the extent of the data set and, for a specified set of zoom levels, looping through and using Mapnik to render the UTFGrid to a .json file in Z/X/Y.json directory structure.

Get `create-utfgrids` on github

If we have a mercator polygon shapefile of ecoregions and want to render UTFGrids for zoom levels 3 through 5 using the dom_desc and div_desc attributes, we could use a command like

$ ./create_utfgrids.py test_data/bailey_merc.shp 3 5 ecoregions -f dom_desc,div_desc

WARNING:
This script assumes a polygon shapefile in spherical mercator projection.
If any of these assumptions are not true, don't count on the results!
 * Processing Zoom Level 3
 * Processing Zoom Level 4
 * Processing Zoom Level 5

and inspect the output (e.g. zoom level 5, X=20, Y=18)

$ cat ecoregions/5/20/18.json | python -mjson.tool
{
    "data": {
        "192": {
            "div_desc": "RAINFOREST REGIME MOUNTAINS", 
            "dom_desc": "HUMID TROPICAL DOMAIN"
        }, 
...
    "grid": [
        "  !!!!!!!!!#####$%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%", 
...

Some caveats:

This currently only works for polygon datasets in a Web Mercator projection.
It’s only tested with shapefiles as it assumes a single-layer datasource at the moment. Full OGR Datasource support would not be too difficult to add for PostGIS, etc.
It assumes a top-origin tile scheme (as do OSM and Google Maps). Supporting TMS bottom-origin schemes in the future should be straightforward.
Requires OGR and Mapnik >= 2.0 with python bindings. Finding windows binaries for the required version of Mapnik may be difficult so using OSX/Linux is recommended at this time.

Many thanks to Dane Springmeyer for his help on UTFGrid related matters and and to Klokan Petr Přidal for his MapTiler docs

Introducing the Madrona framework

2012-07-11T00:00:00-07:00

Madrona: A software framework for effective place-based decision making

My work at Ecotrust mainly revolves around creating web-based spatial analysis tools - software to bring data-driven science to the place-based descision making process. This began several years ago when I joined the MarineMap team. Since working with Ecotrust, we’ve taken the MarineMap software far beyond it’s original niche. What was once a specific tool for marine protected area planning has now become a powerful framework for all sorts of web-based spatial tools in the realms of marine, forestry, conservation planning, aquatic habitat restoration, etc. So, in a sense, Madrona is a recognition of that evolution.

From the official Madrona release announcement from the Ecotrust blog post:

Over the last year we’ve distilled the best ideas from our most successful tools into a suite of software building blocks that can be mixed and matched to create cutting-edge software for decision support and spatial planning at any scale. These building blocks are already at the heart of our work and now we’re ready to share them with you.

So what is Madrona from a developer’s perspective?

A set of python django apps that provide models, views and templates for representing spatial features and solving problems specific to spatial decision tools.
A RESTful API for accessing spatial features
A collection of javascript libraries (based on JQuery) to provide a web-based interface to the API.

In short, we think its a great platform for spatial tools and we want to open it up to the wider developer audience. Ecotrust already has many madrona-based apps in the wild (with many more in development) but we’re hoping to get other folks using (and contributing to) the Madrona framework in the future.

I know this post is short on technical details but there will more to come … for now, check out the technology page for an overview or the developer’s page to dive in.

Migrating from Wordpress to Jekyll

2012-04-28T00:00:00-07:00

I just switched this blog from an ancient version of wordpress running on a VPS to a static-file jekyll bootstrap site (hosted by github). Let me know if you experience any wierdness on the site or feeds. I’ve taken good measures to make sure links don’t break (old URLS should get a 301 permanent redirect to blog.perrygeo.net) but let me know if you get any 404s.

So why do it?

Having a PHP-MySQL app running on a VPS just to serve up a bunch of blog posts seemed excessive. I don’t have the desire to maintain that sort of infrastructure for a simple blog!
Wordpress’ editing and admin interface suck. I prefer vim and bash.
Markdown is a great language for quickly banging out blog posts.
Static files just make sense for what is basically static content.
Github pages provides the hosting for me and even handles CNAMEs for DNS.
Managing revisions with git.

The conversion process

It was not an entirely smooth transition, most of which can be traced directly to dumb decisions on my part. I won’t recount the entire process (there are plenty of guides on internets) but I’ll outline the major steps here:

Export the wordpress blog to an xml file. I has to use xmllint to clean it up a bit.
Set up a disqus account and import my wordpress file. Disqus will handle all the comments which are the only dynamic content on the page.
Use exitwp.py to convert the xml to jekyll markdown files. This worked OK. Not great. Tags and formatting did not come through as expected and I had to wrestle the script a bit. Tables were destroyed and some iframes (youtube links) were lost.
Forked Jekyll Bootstrap and brought in my posts.
Started tweaking of css and markdown to get formatting right. Still have a ways to go on this front - let me know if there is any content you’d like me to restore faster than others.
Had to write a little web service to redirect posts; the old blog stupidly used the default wordpress URLS like /wordpress/?p=4 which needed to go to /2010/01/01/blah
My images were all over the place; some I had in wordpress uploads, others on various servers, some were absolute links, others relative. Gathering them all in one place and using some sed-fu to get the paths right was essential.
Retagged some posts - still working on tags.
Set up Google Analytics to track usage.

I think that’s about it. There are still some big formatting problems on older posts (mostly due to the fact that I used blockquotes for code). And tables are still destroyed. I’ll be working on cleaning these up as I go along.

Overall impression of Jekyll-Bootstrap and hosting with Github pages? Awesome. I would highly recomend it to anyone starting a new blog or converting a smaller/better-behaved wordpress site. It is so much better than having to deal with PHP and MySQL (hopefully the last time I’ll ever see them!). But the conversion was a bit tricky and took way more of my Friday and Saturday than I’d like to admit. I would not want to do that again… But I’m glad did.

What do you think of the new digs?

Working with mbtiles in python

2012-03-25T11:42:48-07:00

python-mbtiles. Check it out.

I’ve been working a bit with Tilemill lately and love the Carto css styling, iteractivity through UTFGrids and being able to export the whole deal as a single mbtiles sqlite database. But when it comes to working with the mbtiles databases, I’ve found both Tilestache and Tilestream to be fairly limiting:

Tilestache serves images but does not (yet) serve up UTFGrids directly from mbtiles while Tilestream hardcodes a “grid()” JSONP callback around the returned json data making it fairly specific to Wax client libraries.

So I went down two paths, first trying to export all the tiles out of mbtiles to json and png files (for those times when you just want to serve static files), then trying to write a simple server that would do dynamic jsonp callbacks. Turns out that in the process, I was able to abstract a lot of the python< ->sqlite interaction into some generic python classes.

Thus python-mbtiles was born. It provides a simple mbtiles web server, a conversion script, and some python classes to work with. No frills, no anything really at this point. More an experiment gone right that might be useful to someone out there in GeoPython land. Enjoy and let me know if you have any ideas!

Average Aspect

2012-03-18T07:26:32-07:00

Ever try to figure out what the average aspect of an area is? i.e.

What direction does this hillside face?

Let’s say we want to determine the average elevation of an area based on a raster DEM. Just take the arithmetic mean of all the elevation cells contained in the area - a simple zonal statistics problem.

Turns out that aspect is not quite as straightforward. True, we can easily use gdaldem to create an aspect map.

gdaldem aspect elevation.tif aspect.tif

This gives a raster with values in degrees: 0 is north, 90 is east, 180 is south, etc… but note that 360 is north as well. We’re dealing with angular units, not linear units.

For example, take a nearly North facing hillside; the left edge is facing slightly NW (350 degrees) while the right edge faces slighty NE (10 degrees).

The arithmetic mean of the aspect values = (350+350+10+10)/4 = 180°. Due south? That’s entirely wrong! It doesn’t take into account the angular units. For that we need to create grids representing the sin and cos of the aspect.

Luckily you can use the handy gdal_calc.py utility that comes with recent versions of gdal. This allows you to apply numpy’s trigonometric functions to a raster…

gdal_calc.py -A aspect.tif --calc "cos(radians(A))" --format "GTiff" --outfile cos_aspect.tif  
gdal_calc.py -A aspect.tif --calc "sin(radians(A))" --format "GTiff" --outfile sin_aspect.tif

Now we can look at the sum of the cos/sin grid cells for our area and take the arctangent according to this python code

import math
avg_aspect_rad = math.atan2(sum(cos_cells), sum(sin_cells))
avg_aspect_deg = math.degrees(avg_aspect_rad)
print avg_aspect_deg

In our example avg_aspect_deg comes out to an aspect of 0 degrees (due north) which is exactly what we’d expect.

Thanks to Dan Patterson for his forum post which clued me into this approach.

UTFGrids with OpenLayers and Tilestache

2012-02-24T13:11:19-08:00

A while back, the Development Seed team developed the UTFGrid spec to provide

a standard, scalable way of encoding data for hundreds or thousands of features alongside your map tiles.

The basics

In more detail, the UTFGrids are invisible “ASCII Art” and attribute data embedded in json. They sit “behind” your map tiles (they are not rendered visually) and allows quick attribute lookups without going back to the server. This allows a high degree of real-time map interactivity in an HTML web map - something that has typically been the strong point of plugin-based maps.

So take this tile image…

and it’s corresponding “utfgrid” …

          !######$$$$%%% %%%% % 
          !#######$$$$%%%    %%%
         !!#####   $$$%%%    %%%
         !######  $$$$%%% %% %%%
        !!!####  $$$$$%%%%  %%%%
      ! !###### $$$$$$%%%%%%%%%%
     ! !!#####  $$$$$$$%%%%%%%%%
    !!!!!####   $$$$$$%%%%%%%%%%
    !!!!!####   $$$$$$%%%%%%%%%%
    !!!!!####   $$$$$%%%%%%%%%%%
    !!!!!#####% $$   %%%%%%%%%%%
    !!!!!### #      %%%%%%%%%%%%
    !!! #####   ''''%%%%%%%%%%%%
     !   ###      ('%%%%%%%%%%%%
       ) ### #  ( ((%%%%%%%%%%%%
      ))  ##   (((((%%%%%%%%%%%%
      ))  #    ****(+%%%%%%%%%%%
       )        %**++++%%%%%%%%%
       , , ------*+++++%%%%%%%%%
.     ,,,,,------+++++++%%%%%%%%
..  /,,,,,,------++++++%%%%%%%%%
.  //,,,,,,------000++000%%%%%%%
  211,,,,,33------00000000%%%%%%
 2221,,,,33333---00000000000%%%%
222222,,,,3635550000000000000%%%
222222,,,,6665777008900000000%%%
22222::66666777788889900000 %%%%
22222:;;;;%%=7%8888890  0   %%%%
22222;;;; ==??%%888888  00 %%%%%
222222 ;;  =??%%%8888       %%%%
222     ;;   ?A>>@@@          B%
CCC      ;;   DEE@@@          BB

You can see how each character corresponds with a country. The character’s code is used as a lookup key to retrieve the data associated with that feature (which is also included in the json tile).

If you want to dig in, check out the mapbox demo.

The Server side

I’m going to assume you have Tilestache and Mapnik 2+ already installed (if not, you should!). The steps to configuring your server for UTFGrids are fairly simple..

First, set up mapnik xml file pointing to your data source.

<?xml version="1.0"?>

<!-- An ultra simple Mapnik stylesheet -->

<!DOCTYPE Map [
<!ENTITY google_mercator "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over">
]>

<Map srs="&google_mercator;">
    <Style name="style">
        <Rule>
            <PolygonSymbolizer>
                <CssParameter name="gamma">.65</CssParameter>
                <CssParameter name="fill">green</CssParameter>
                <CssParameter name="fill-opacity">0.5</CssParameter>
            </PolygonSymbolizer>
            <LineSymbolizer>
                <CssParameter name="stroke">#666</CssParameter>
                <CssParameter name="stroke-width">0.3</CssParameter>
            </LineSymbolizer>
        </Rule>
    </Style>
    <Layer name="layer" srs="&google_mercator;">
        <StyleName>style</StyleName>
        <Datasource>
            <Parameter name="type">shape</Parameter>
            <Parameter name="file">sample_data/world_merc.shp</Parameter>
        </Datasource>
    </Layer>
</Map>

Next, set up tilestache configuration file

{
"cache": {
           "name": "Disk",
           "path": "/tmp/stache"
},
"layers": {
    "world":
    {
        "provider": {"name": "mapnik", "mapfile": "style.xml"}
    },
    "world_utfgrid":
    {
        "provider":
        {
        "class": "TileStache.Goodies.Providers.MapnikGrid:Provider",
        "kwargs":
        {
            "mapfile": "style.xml", 
            "fields":["NAME", "POP2005"],
            "layer_index": 0,
            "scale": 4
        }
    }
  }
}

Finally, you’re ready to run the tilestache server…

tilestache-server.py -c your.cfg -i localhost -p 7890

Now you should be serving utfgrids to http://localhost:7890/world_utfgrid/

The Client side

Now we need something to consume the UTFGrid tiles and interact with them in an HTML/JS environment. The original client implementation of UTFGrid support is provided by Wax which sits atop mapping clients like Modest Maps and Leaflet. Wax is very slick and easy to use but doesn’t work so well for more complex arrangements or with OpenLayers-based maps.

Rather than clog up Wax with the complex UTFGrid use cases that we envisioned, we decided to implement a UTFGrid client in native OpenLayers. Hence my project for the OSGEO code sprint was born.

The result was a new OpenLayers Layer which loads up the json “tiles” behind the scenes…

        var grid_layer = new OpenLayers.Layer.UTFGrid( 
            'Invisible UTFGrid Layer', 
            "./utfgrid/world_utfgrid/${z}/${x}/${y}.json"
        );
        map.addLayer(grid_layer);

and an OpenLayers Control that handles how the mouse events interact with the grid. In this example, as the mouse moves over the map, a custom callback if fired off which updates a div with some attribute information.

       var callback = function(attributes) {
            if (attributes) {
                var msg  = "<strong>In 2005, " + attributes.NAME 
                    msg += " had a population of " + attributes.POP2005 + " people.</strong>";
                var element = OpenLayers.Util.getElement('attrsdiv');
                element.innerHTML = msg;
                return true;
            } else {
                this.element.innerHTML = '';
                return false; 
            }
        }

        var control = new OpenLayers.Control.UTFGrid({
            'handlerMode': 'move',
            'callback': callback
        });
        map.addControl(control);

Overall the design goal was to decouple the loading/tiling of the UTFGrids from the interactivity/control. I think this works out nicely and, while a bit more cumbersome than the method used by Wax, it is more flexible and integrates well with existing OpenLayers apps.

You can see them in action on the examples pages:

* Demonstrating the use of different event handlers (click, hover, move)

* Demonstrating multiple interactivity layers (the interactivity layer need not visible in the map tiles!)

And feel free to check out the code at my github fork for the code.

What do you think? Let me know…

Optimizing KML for hierarchical polygon data

2011-05-18T13:34:40-07:00

For all the benefits of KML, it is decidedly a step backwards for handling large vector datasets. Most KML clients, including the cannonical Google Earth application, experience debilitating slow-down when viewing a couple dozen MB of vector data - datasets that I could easily open on a Pentium 4 in ArcView 3.2 10 years ago!

The unfortunate reality is that optimizing the performance of KML datasets is conflated with the structure of the data and is thus the responsibility of the data publisher. The wisdom of combining styling, performance-related structure, organizational structure, geometry and attributes into a single file format may be questionable, but KML has become the defacto geographic markup language due to it’s other benefits.

Anyways, back to performance enhancements on big vector datasets… The concept of “regionation” is used by several KML software to improve performance. From the Google LatLong Blog:

You can think of Regionation as a hierarchical subdivision of points or tiles, which shows less detail from afar, and more detail as you zoom in to the globe. This dynamic loading creates clearer visualizations by minimizing clutter, while simultaneously speeding up the rendering process.

In most implementations, there is a generic strategy for determining this hierarchy based on attributes or geometry size (in the case of vectors) or by a tile system. Neither is ideal when you want to preserve the vector nature of the data, split it into small, easily-loadable files and determine it’s view based on the natural hierarchy that is built into the data structure.

Specifically I am thinking about watersheds here - the US Hydrologic Units. Hydrologic units are watershed boundaries that are organized in a nested hierarchy; higher levels contain smaller watersheds that are contained within a single watershed from a “parent” level. The unique identifiers (hydrologic unit codes or HUCs) are rather ingenious as well; Each level is represented by 2 digits and are concatenated to form a single identifier that can be used to determine it’s “parent”. For example:

Level 4 HUCs
e.g. 17090011

Level 5 HUCs
e.g. 1709001104

Level 6 HUCs
e.g. 170900110403

Instead of fabricating a hierarchy of features, why not just use this natural hierarchy to structure the KML documents?

Or as KML markup:

    <placemark>
        <name>17090009</name>
        <styleurl>#HUC_8-default</styleurl>
        <polygon><outerboundaryis><linearring><coordinates>...
        </coordinates></linearring></outerboundaryis></polygon>   
    </placemark>
    
    <networklink>
    <name>17090009_children</name>
    <region>
      <latlonaltbox>
        <west>-123.001645628</west>
        <south>44.8300083641</south>
        <east>-122.203351254</east>
        <north>45.298653051</north>
      </latlonaltbox>
      <lod>
        <minlodpixels>256</minlodpixels>
        <maxlodpixels>1600</maxlodpixels>
      </lod>
    </region>
    <link>
      <href>./17090009_children.kml</href>
      <viewrefreshmode>onRegion</viewrefreshmode>
    </link>
    </networklink>

The advantages to this design are that you don’t have to break the geometries up to fit into a square tiling pattern, data loads and renders in a logical pattern and there will always be 100 or less (usually far less) placemarks per file due to the design of the HUC data structure. File sizes stay low, network links load quickly and request/rendering occurs only when they come into view. For this example dataset totaling 300M of shapefiles, there are several hundred resulting kmz files without any repeated features and all less than ~ 150K each. In essence, it achieves optimal performance by its very design.

Here’s a video of it in action:

iframecontent

This was all done with a fairly “hackish” python script. I’ll continue to refine it as needed for this particular application but, at this time, it’s not intended to be a reusable tool - if you want to use it, be prepared to dig through the source code and get your hands dirty. The same concept could theoretically be applied to any spatially-hierarchical vector data (think geographic boundaries … country > state > county > city).

Um - nice “review” of QGIS

2010-12-20T11:25:09-08:00

RJ Zimmer at American Surveyor magazine did what he described as a comparison of several free GIS application entitled ”Something for Nothing”

First of all, the title bugs me. The idea that the sole benefit of free software is simply cost savings is pretty naive. It disregards openness, community support, ability to transfer knowledge, freedom from restrictive licensing, etc. But I can live with the title.

I can also live with his decision to include only a single open-source GIS application alongside 3 closed-but-gratis applications. He doesn’t claim that it’s a comprehensive review despite the fact that the ecosystem of Free GIS is far more diverse.

But I can’t accept his treatment of Quantum GIS:

I did not fully test Quantum GIS. I did download and install it but the software was too complicated to use “right out of the box”, and I did not have time to learn to use it.

The feature comparison chart includes mainly ”?” in the QGIS column.

OK we get it - your deadline hit before you could bother to learn one of the applications you were supposedly reviewing. One even wonders why he included QGIS the review at all. This is nothing short of irresponsible reporting. When people post stuff like this, it really rubs me the wrong way - now a whole audience of users have a inaccurate view of QGIS and entire free GIS ecosystem thanks to his slacker journalism.

kmltree

2010-06-09T06:27:55-07:00

When the MarineMap team started delving into the Google Earth plugin, it was apparent that it supported the display and rendering of KML files almost as well as the Google Earth desktop application. The missing piece of functionality was the nice tree-style legend that is provided with the desktop app. The plugin lets you add KML for display but gives you no HTML interface to work with it. For simple apps, you can just roll your own html/js form. But that quickly becomes unmanageable if you’re adding KML dynamically and need to create a tree-style legend for any arbitrary KML document.

Enter kmltree.

kmltree is a javascript tree widget that can be used in conjunction with the Google Earth API. It replicates the functionality of the Google Earth desktop client, and is fast, extensible, and stable for use in advanced web applications. It’s built utilizing the earth-api-utility-library and jQuery.

Any arbitrary KML can be parsed and represented in a tree-style legend right in the web browser. Try it out.

Kmltree is the brainchild of Chad Burt who developed it as part of the marinemap codebase but had the foresight to realize that this would be useful to a much wider audience and abstracted it into its own javascript library. If you’re building a web mapping application with the Google Earth API, give it a shot!

MarineMap wins award for Environmental Conflict Resolution

2010-05-27T15:36:07-07:00

For the last year or so, I’ve had the pleasure of working with the MarineMap Consortium. We just learned yesterday that the U.S. Institute for Environmental Conflict Resolution awarded MarineMap the “Innovation in Technology and Environmental Conflict Resolution”.

I joined the team after the launch of the South Coast of California site which was already widely recognized as a successful decision-support tool for marine spatial planning. We’ve since been working on version 2 of the MarineMap tool which is deployed currently for the North Coast of California in support of their Marine Life Protection Act (MLPA) process.

It’s been a tremendous challenge to bring a new version of the software to life and have it meet and exceed the standards set by its predecessor. It has also been tremendously rewarding and having our work recognized at this level is a great honor. It’s nice to know that the tools we’ve developed have been so helpful and instrumental in the marine planning process along the coast of California. Looking forward, I see MarineMap growing beyond a tool for a specific purpose (supporting the MLPA Initiative) to a robust framework for developing web-based spatial planning tools for all sorts of environmental applications, both marine and terrestrial. And this award confirms that we are already heading in the right direction. Very exciting news!

Exploring Geometry

2010-05-06T18:35:54-07:00

I don’t know how I let this gem slip past my radar for so long. It was only via a post by Dr. JTS himself (aka Martin Davis) that I saw a screenshot of JTS TestBuilder and decided to check it out.

I was actually just talking with someone about a tool that could provide simple visualization of WKT geometries; JTS Test Builder does that and much more.

You can input geometries (graphically or by well-known text) and compare two geometries based on spatial predicates:

Do overlay analyses with the two geometries. Note that you can see the result as WKT below.

And there are a host of other spatial operations to generate geometries using buffers…

… convex hulls …

This app provides a very nice and user-friendly way to quickly and simply explore and test geometric operations. To try it out, download JTS and unzip the contents somewhere. If you’re on windows, the .bat file is provided. If you’re running anything else, you have to cook up a shell script that will set up the environment and run JTS TestBuilder:

JTS_HOME=/usr/share/java/jts-1.11
CP=$CLASSPATH
for i in $JTS_HOME/lib/*.jar; do CP=$i:$CP; done
java -Xmx256m -cp $CP com.vividsolutions.jtstest.testbuilder.JTSTestBuilder $*

Distributed

2010-03-31T14:06:11-07:00

I’ve been playing around with some distributed version control systems (DVCS) to replace svn.

First, the why: I’ll leave the details up to Joel in his excellent HgInit tutorial. Its mercurial-specific but the general concepts apply to any DVCS. The takeaway message for any project with > 1 developer is this:

Mercurial ed: DVCS separates the act of committing new code from the act of inflicting it on everybody else.

Next, the implementation: I’m using git to work on another project (Golden Cheetah) and its been a tough learning curve. Git is no doubt the most powerful DVCS out there. You can do magical things with it like combine commits and mess with history trees. And you can also screw things up pretty badly if you misinterpret the esotric docs for some non-intuitive piece of the workflow.

I just tried mercurial this morning - hg seems to fit my mind well. There is less power but the workflow is very clear and intuitive. And there are docs written for people who don’t want to do an in-depth study of their version control software. It stays out of the way.

Long story short, I’m going to use mercurial/hg for my new projects. Ah what the heck my old/ongoing projects as well. My googlecode repository has been converted over to Mercurial. Svn will stick around but wont be updated.

Lazy raster processing with GDAL VRTs

2010-02-18T09:16:43-08:00

No, not lazy as in REST :-) … Lazy as in ”Lazy evaluation”:

In computer programming, lazy evaluation is the technique of delaying a computation until the result is required.

Take an example raster processing workflow to go from a bunch of tiled, latlong, GeoTiff digital elevation models to a single shaded relief GeoTiff in projected space:

Merge the tiles together
Reproject the merged DEM (using bilinear or cubic interpolation)
Generate the hillshade from the merged DEM

Simple enough to do with GDAL tools on the command line. Here’s the typical, process-as-you-go implementation:

gdal_merge.py -of GTiff -o srtm_merged.tif srtm_12_*.tif 
gdalwarp -t_srs epsg:3310 -r bilinear -of GTiff srtm_merged.tif srtm_merged_3310.tif 
gdaldem hillshade srtm_merged_3310.tif srtm_merged_3310_shade.tif -of GTiff

Alternately, we can simulate lazy evaluation by using GDAL Virtual Rasters (VRT) to perform the intermediate steps, only outputting the GeoTiff as the final step.

gdalbuildvrt srtm_merged.vrt srtm_12_0*.tif
gdalwarp -t_srs epsg:3310 -r bilinear -of VRT srtm_merged.vrt srtm_merged_3310.vrt 
gdaldem hillshade srtm_merged_3310.vrt srtm_merged_3310_shade2.tif -of GTiff

So what’s the advantage to doing it the VRT way? They both produce exactly the same output raster. Lets compare:

	Process-As-You-Go	"Lazy" VRTs
Merge (#1) time	3.1 sec	0.05 sec
Warp (#2) time	7.3 sec	0.10 sec
Hillshade (#3) time	10.5 sec	19.75 sec
Total processing time	20.9 sec	19.9 sec
Intermediate files	2 tifs	2 vrts
Intermediate file size	261 MB	0.005 MB

The Lazy VRT method delays all the computationally-intensive processing until it is actually required. The intermediate files, instead of containing the raw raster output of the actual computation, are XML files which contain the instructions to get the desired output. This allows GDAL to do all the processing in one step (the final step #3). The total processing time is not significantly different between the two methods but in terms of the productivity of the GIS analyst, the VRT method is superior. Imagine working with datasets 1000x this size with many more steps - having to type the command, wait 2 hours, type the next, etc. would be a waste of human resources versus assembling the instructions into vrts then hitting the final processing step when you leave the office for a long weekend.

Additionaly, the VRT method produces only small intermediate xml files instead of having a potentially huge data management nightmare of shuffling around GB (or TB) of intermediate outputs! Plus those xml files serve as an excellent piece of metadata which describe the exact processing steps which you can refer to later or adapt to different datasets.

So next time you have a multi-step raster workflow, use the GDAL VRTs to your full advantage - you’ll save yourself time and disk space by being lazy.

Peaksware licensing revisted …

2009-12-16T18:33:42-08:00

I had previously bitched and moaned about the licensing restrictions on the TrainingPeaks WKO+ software. Truth be told, the reason I was so put off by their crappy licensing scheme was that my cycling training relied so heavily on their software. It was not perfect but it was the best tool available. I’ve since discovered Golden Cheetah which is a viable open-source alternative but it still lags behind WKO+ in many critical features.

Now, fresh in time for the 2010 training season, Peaksware has released a new version 3.0 of WKO+ which, amongst many UI and functionality improvements, has made considerable progress on the licensing front.

We know, our licensing has been a challenge to deal with for our customers in the past, but we’ve always tried to be as helpful as possible getting you back up and running after a hard drive crash or new computer. To remedy this, we’re pleased to announce an all new flexible licensing system. First, with every purchase we now allow you to install WKO+ 3.0 on up to two computers; second, we’ve built an online activation/deactivation system so you are free to move your active licenses from machine to machine. Are you leaving on a 2 week trip? Just de-activate your home computer, activate your laptop, and you’re on your way. When you get home, de-actiavate your laptop, re-activate your desktop and you’re all set.

It ain’t open source (there is still a place in this world for proprietary software if they can push the boundaries and innovate) but the sensitivity to the licensing issue just may have restored my faith in their company.

Nice examples of ESRIs geoprocessing python module (9.3)

2009-08-10T07:37:51-07:00

Just thought I’d point out a great presentation about the “new” 9.3 geoprocessing (gp) python module from ESRI.

Ghislain Prince and Elizabeth Flanary do a great job of introduction by examples. The latest gp module is much more pythonic and these examples show how to leverage that to its full advantage. If you try to do this with older gp versions, the code would make most pythonistas cringe. This latest version returns objects and lists, use real booleans, and uses true objects instead of funky string parameters. Basic OO stuff for most python libraries but a big improvement for gp.

Here’s the powerpoint presentation. Thanks to Jamey Rosen for the tip!

Peaksware licensing hell

2009-06-23T14:29:36-07:00

I’ve been using Peaksware’s WKO+, a cycling and running training tool to manage data from heart rate monitors, GPS units, power meters, etc. Its a powerful tool with a clunky UI but I’ve gotten used to it.

You pay $100 for a “personal” license. Not a big deal to me since they basically have a monopoly on this software niche. I first installed it on my work computer to test the data from my daily bike commute. Cool it works. Then I went to install it at home since that’s where I’ll be using it. Works ok. I proceed to gather all my fitness data into their proprietary binary format.

Fast forward a few months. I’m reformatting the hard drive on the laptop and want to move all my data and software to my desktop. But installing WKO+ is giving me a headache (“Error: Too many installations”). The registration process takes a hardware fingerprint and your must active it via the web to get a registration code. However, hidden withing their EULA, is a term which dissallows the transfer of license to another computer other than the one to which it was originally installed. The second installation was just an allowance they make to allow for “hard drive crashes” and such.

Since neither of those machines would be available to me, certainly there would be a way to transfer it? After several progressively more desperate communications with Matt Allen at peaksware support, he informed me that there was no way they would transfer the license (the non-transfer clause IS in the EULA after all). I would need to purchase another license simply because I switched computers!

Here is my response:

Basically what you are telling me is that I can no longer use WKO+ without paying again. I get to use the software for a few months and you revoke my right to use it because I buy a new computer! I am a paying customer, trying to be totally legit here, willing to support your business in exchange for a license to use your software and you insist on screwing me over. Brilliant.

This is one of the most unprofessional and idiotic stances I have ever seen from a software company. Your intention appears to be to screw over your paying customers and milk as much cash from them as possible - you might want to rethink that business model unless you want to loose customers! I will never endorse, recommend or purchase another product or service from peaksware nor will any of my family, friends, teammates or readers once the word gets out about your disrespectful policies.

There are numerous typical situations where a new copy of the software would need to be installed including:

Hard drive failure

Operating system upgrades

New computer purchases

Extended traveling and touring (installing onto a laptop or netbook)

Now I fully understand why your policy is one license per computer. It makes perfect sense. I have seen plenty of other software with a similar licensing model. But they also allow to uninstall the software and re-register it on another computer due to these circumstances. There is simply no technological reason why you could not implement a licensing structure that allowed the user more freedom to transfer licenses while still preventing piracy. As it stands, your licensing model treats paying customers like criminals if they happen to run across any one of the above situations.

So, to sum it up - your foolish license policy has lost you one customer and many future ones.

Good riddance.

So if you want to support a company that treats its paying customers like criminals because they get a new computer, go right ahead and support Peaksware. But anyone who expects to use software that they pay for even if they happen to buy a new computer should steer clear.

The real kicker is that all that work is locked away in their proprietary file format simply because of their draconian licensing. This is the real take home lesson to all software users (not just fitness geeks): If you lock your data away in a proprietary format and are beholden to a single company in order to access it, they can and will screw you. Always insist on open data formats, even if using proprietary software. Oh and always read the EULA carefully before clicking OK!

Reading XFS partition from Windows

2009-06-21T08:09:30-07:00

When I was setting up my linux system a few years ago, I did some research into filesystems and determined that the XFS file system, being particularly proficient in dealing with large files, would be ideal for my home directory. And it was. But the one factor I didn’t consider was portability. Turns out that there is basically no support for XFS in windows.

So how do you access your files from Windows if they are on an XFS partition? I had just shy of 1 TB of data to transfer so using my other linux box and transferring across the network would have taken forever. The solution I came up with is a bit convoluted but it has some real advantages:

1) Install Sun’s VirtualBox. 2) Download an iso for your favorite linux distribution (mine being Ubuntu 9.04) 3) Create a virtual machine from the linux iso 4) Install the VBOxGuestAdditions in the linux virtual machine. 5) Create a Share folder on the windows host and register it with the virtual machine. This will allow you to transfer files from the guest (linux) to the host(windows) You may have to manually mount the drive in the linux guest:

mount -t vboxsf share_name /mnt/share_name

6) Using the windows host cmd line, create a vmdk from the physical drive that your XFS partition resides on. In this case, PhysicalDrive1 corresponds to the second SATA connector. This will allow your guest OS to talk directly with the drive:

cd C:\Program Files\Sun\xVM VirtualBox
VBoxManage.exe internalcommands createrawvmdk 
  -filename "C:\Documents and Settings\perry\.VirtualBox\HardDisks\Physical1.vmdk" 
  -rawdisk \\.\PhysicalDrive1 -register

Once completed, you should see: _

RAW host disk access VMDK file 
C:\Documents and Settings\perry\.VirtualBox\HardDisks\Physical1.vmdk created successfully.

7) Make sure to add the physical drive to your list of hard drives in the linux guest options. Restart the linux guest virtual machine and your XFS partition should already be mounted. Now you can begin transfering files between your XFS partition and the shared folder on the windows host.

Whew. Lots of hassle for a simple file transfer, right! But the side benefit is that now you have a fully functional linux virtual machine with a shared folder set up to the windows host. Very useful - even when you must run windows, it helps to have a linux VM standing by!

IronPython (2.6) and ArcGIS - ready for prime time!!

2009-06-16T14:43:08-07:00

Not sure why this didn’t occur to me before I wrote that last post but I tried the “pythonic” version of the code under the IronPython 2.6 Beta 1 release and it works!

lyr = Carto.LayerFileClass()
lyr.Open('C:\\test.lyr')
print lyr.Filename

Works perfectly now. So IronPython 2.6 promises to be a viable option for extending ArcGIS. My enthusiasm has been renewed.

IronPython and ArcGIS - not quite ready for prime time

2009-06-16T13:25:46-07:00

Occasionally I find myself in the C#/.NET world in order to write code using ESRI ArcObjects. Today I was toying with the idea of automating the creation of ESRI Layer files (a file which defines the cartographic styling of a dataset). Of course they are in an undocumented binary file format, inaccessible to anything but ESRI software. So I pop open Visual Studio ….

I feel a nagging unease every time I type a set of curly braces. And VB just makes me insane. I prefer, of course, to use python. Luckily there is IronPython which runs on .NET - which means I could theoretically use it to interact with ArcGIS.

I only found a single working example of using ArcObjects through IronPython. But it looked promising enough to close Visual Studio and give it a go.

The first nagging problem is an IronPython-specific one. Relatively minor annoyance but you have to add the reference to a .NET assembly (library) before you can load it.

import clr
clr.AddReference('ESRI.ArcGIS.System')
clr.AddReference('ESRI.ArcGIS.Carto')
from ESRI.ArcGIS import esriSystem
from ESRI.ArcGIS import Carto

Now there is the issue of grabbing an ESRI license. A little verbose IMO but it could easily be encapsulated in a helper function to clean things up.

aoc = esriSystem.AoInitializeClass()
res = esriSystem.IAoInitialize.IsProductCodeAvailable(aoc, 
         esriSystem.esriLicenseProductCode.esriLicenseProductCodeArcView)
if res == esriSystem.esriLicenseStatus.esriLicenseAvailable:
    esriSystem.IAoInitialize.Initialize(aoc, 
      esriSystem.esriLicenseProductCode.esriLicenseProductCodeArcView)

Now that we’ve satisfied the demands of our proprietary license overlords, we can proceed with the real work .. in this case I just want to open an existing Layer file and see if the resulting object knows it’s own file path. Really simple, right?

lyr = Carto.LayerFileClass()
if "Open" in dir(lyr): print "The Layer object has an Open method but...."
lyr.Open('C:\\test.lyr')
print lyr.Filename

The Layer object has an Open method but....
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
AttributeError: 'GenericComObject' object has no attribute 'Open'</module></stdin>

Hrm. Looks like we’ve run across bug 1506 which doesn’t allow access to the properties and methods of a given instance - instead your have to work through the functions provided by the implementation. Grr…

Carto.ILayerFile.Open(lyr, 'C:\\test.lyr')
print Carto.ILayerFile.Filename.GetValue(lyr)

That is unwieldy, ugly and unpythonic. What’s the point of object oriented programming if you can’t access the methods and properties of an object directly? Since all ArcObjects applications are based on extending COM interfaces, this would be a major pain in any non-trivial application. Basically, until these .NET-accessible COM objects can be treated in a pythonic way, I don’t see any compelling reason to pursue IronPython and ArcGIS integration. Looks like its back to C# for the moment … (/me take a deep sigh and opens Visual Studio) … unless of course anyone has some brilliant solution to share!!

The GPS told me to do it

2009-06-12T13:44:18-07:00

Another disastrous consequence of inaccurate spatial information… Not only can you accidentally tag your neighbor as a criminal, now it appears that sloppy spatial data has lead to the wrong house getting demolished.

I’ve asked it before but its worth repeating … with all the recent advances in spatial data publishing, where are the advances in metadata and data quality assurance? How do you know where the data comes from, what’s been done to it and by whom? What is the intended use of the data? For the vast majority of the data being shoved out onto the web, these bits of metadata are sorely lacking.

Of course this case is more a matter of one person’s sheer stupidity; I’m not sure any caveats in the metadata would have stopped the wrecking ball!

The magic bullet

2009-03-25T13:38:16-07:00

Never fear, OGR is here. The magic bullet for fixing corrupted shapefiles is, 90% of the time, accomplished by using ogr2ogr to convert the shapefile to another shapefile.

ogr2ogr -f "ESRI Shapefile"  shiny_new_clean_dataset.shp corrupted_dataset.shp corrupted_dataset

OGR’s internal data model cleans it up and the output is a fresh shiny new shapefile that works without hassle.

TV cycling coverage is dead

2009-02-19T12:56:12-08:00

Real-time spatial application developers take note…

I’ve been following the Tour of California this week (looking forward to the Solvang Time Trial this Friday) and have been disappointed with the TV coverage on Versus. Its not that the coverage is bad, its just that long-distance endurance sports don’t lend themselves to the traditional 2 announcers and 1 camera format. There are multiple groups of riders and so much spatial information to keep track of if one really wants to understand the dynamics of a cycling event.

Maybe I’ve just been spoiled by the Amgen Tour Tracker. It is a crowning example of a spatially-aware real-time web application.

It provides two cameras of live coverage, live commentary with interviews, chat, summary updates, gps tracking of riders shown on both an elevation profile and a yahoo-based aerial map, “gps+” location prediction, race standings, time checks, etc. Far more information than any TV coverage without resorting to information overload.

Stimulus watch

2009-02-12T10:12:23-08:00

Last time I posted on this blog, Hillary and Obama were still battling it out for the Democratic nomination. Now Barack Obama is our president with an uphill battle to save the economy. So yeah, it’s been a while. I haven’t been doing too much innovative Geo-related stuff lately, hence the lack of blog posts. I’ll try to pick up the pace a bit, even if I have to resort to fluff pieces like this one…

Well, it looks like the economic stimulus bill is going to pass. The bill doesn’t actually specify the projects that will be funded; the money will be allocated to cities and some federal grant agencies. The mayors have already proposed thousands of “shovel-ready” projects that might get a green light depending on how much funding the city gets.

There’s a great site, stimuluswatch.org, that allows the public to review these proposals. Good to know where our tax dollars are headed!

There are several GIS proposals ranging from projects with specific, well-defined (and measurable) objectives to the nebulous “Give us $500,000 to upgrade our cities’ GIS program”. It will be interesting to see which ones pan out, which ones produce results and which ones are just a pure waste of taxpayer dollars.

P.S. If you’d like to see where most of my time and energy is going these days, it’s training for the US National Cup mountain bike race series. My cycling exploits are available for all who are inclined to read them.

R is for Radiohead

2008-07-15T15:01:27-07:00

Radiohead realeased their video for House of Cards yesterday. Besides being a big radiohead fan, I was also loving the LIDAR technologybehind the video.

If you want to check it out yourself, there are code samples on the site as well as access to the raw data. The csv files have four columns (x, y, z, and intensity). For me the quickest way to visualize the data was through R and it’s OpenGL interface called rgl (which is a wonderful high-level 3D data visualization environment).

Assuming you have R installed, rgl is a simple add on through the CRAN repositories:

install.packages("rgl")

Then you need to load the library, load the csv, scale the intensity values from 0 to 1. Then it’s a simple rgl.points command to get an interactive 3D rendering:

library(rgl)
d < - read.csv("C:/temp/radiohead/22.csv", header=FALSE)

# scale intensity values from 0 to 1
d$int <- d[,4] / 255

# rgl.points(x,y,z,size=__,color=__)
# note y value is inverted
# color is a grayscale rgb based on intensity
rgl.points(d[,1],d[,2]*-1,d[,3], size=3, color=rgb(d$int,d$int,d$int))

That’s all it takes to render Thom Yorke in all his 3D digital glory:

Geospatial Reddit - 2 weeks later

2008-06-12T08:42:36-07:00

So, despite frustrations with getting submitted URLs to appear, Geospatial Reddit is still puttering along. Not exactly a vibrant community yet but there are currently 133 subscribers. If you’re subscribed, take a minute to submit your favorite URLs. If you haven’t subscribed, check it out.

I thought 133 subscribers was a decent number until I found that the Bacon subreddit has over 500. Apparently the world would rather discuss their greasy breakfast food than maps.

Jabref - Open Source Alternative to EndNote

2008-06-08T09:03:16-07:00

For those of you that use EndNote to keep track of your bibliographies/references , there is an alternative. JabRef. I find the UI to be very intuitive and it has a range of customizable import/export formats. JabRef uses the BibTex format as it’s native file format so, of course, it integrates very well with LaTeX.

One of the neat features is the ability to create custom bibliographies in HTML, complete with javascript-based search capabilities. Here’s my reference list which I’ll be slowly adding to as I convert all my old text-based and EndNote reference lists over.

Posting to Geospatial Reddit

2008-05-28T12:20:50-07:00

Some folks have had trouble submitting links so I figured I should post a bit more detail on that. To get articles to show up on the geospatial reddit (not the main reddit), go to http://reddit.com/r/geospatial/submit or click the “Submit a Link” button on the right - from the geospatial page. When you’re submitting the url, you should see “submit to geospatial” as the page header.

I know at least 2 of us have been successful at posting. If this doesn’t work for you, please let me know and I’ll try and figure it out.

Geospatial Reddit - A democratic solution to geo blog overload?

2008-05-28T07:54:37-07:00

All the great GIS news/blog aggregators out there (planetgs, slashgeo, etc) are moderator driven - a few people act as the gatekeepers and inevitably have to decide what information is useful. This is not the ideal way to do things.

There’s a more democratic and distributed way to spread the role - it’s called reddit. More specifically, Geospatial Reddit. For those unfamiliar with reddit (or similar sites like digg), the idea is simple: users submit stories and users vote on stories. The most popular ones rise to the top and, theoretically, the best articles magically appear on the front page. Much like democracy itself, there are flaws in the theory but its the best thing we’ve got.

Geospatial Reddit is public so sign up, submit your favorite stories and vote. Lets see if we can make this work.

So you want to learn to learn about kriging …

2008-05-25T11:52:38-07:00

Guides like Tomislav Hengl’s Practical Guide to Geostatistical Mapping of Environmental Variables and Rossiter’s Introduction to applied geostatistics do an excellent job of providing a grounded, relatively easy to understand, introduction to geostatical prediction and kriging.

But if you’re an experience learner (like me) you don’t absorb the mathematics fully without doing something with the knowledge; Seeing it in action brings the concepts to life. Unfortunately most geostats/kriging software is either too complex for exploratory learning (not enough immediate feedback) or too simplistic (making too many assumptions, disallowing access to the nitty-gritty details). Either way, you’re bound to produce output with fundamental flaws because you’re not aware of the finer details of variogram modelling. I speak from exerience!

Luckily Dennis J. J. Walvoort of the Wageningen University & Research Center saw the same problem and created an nifty learning to to explore varigoram models and spatial predictions using ordinary kriging - EZ-Kriging. No degree in math or statistical theory required. Just drag the points around, play with the parameters and alter the underlying data as a table and see the results immediately.

Its nothing more than a simulation so don’t expect to load your own datasets or produce any meaningful output with it. But it truly excels as a learning tool to understand the core concepts behind kriging and is a great complement to Hengl and Rossiter’s work. With that knowledge you can do the real deal in Surfer, R, ILWIS or your geostats software of choice.

EDIT: One complaint about this EZ-Kriging that I have: it doesn’t show the observed sample variogram cloud overlayed on the variogram model. Oh well still a nice tool.

EDIT2: It’s a windows .exe but it runs smoothly under wine in linux.

Ubuntu as a GIS workstation (updated for Hardy Heron)

2008-05-14T10:36:29-07:00

As a followup to my previous post on turning Ubuntu Gutsy into a GIS workstation, Here are the revised instructions for Ubuntu 8.04 (The Hardy Heron).

Note that there are a few additonal apps and changes in here:

* Postgis

* Mapnik

* New version of QGIS installed via repository

* OpenStreetMap tools (JOSM and osm2pgsql)

* Geotiff utilities

* Some nice python spatial libs (shapely, owslib, geopy and pyproj)

Run the following as root on your new Hardy installation, answer a few configuration questions and you’ll be ready to go.

echo 'deb http://ppa.launchpad.net/qgis/ubuntu hardy main' >> /etc/apt/sources.list

apt-get update

apt-get -y --force-yes install grass mapserver-bin \
gdal-bin cgi-mapserver python-qt4 python-sip4 python-gdal \
python-mapscript gmt gmt-coastline-data r-recommended gpsbabel \
shapelib qgis qgis-plugin-grass python-setuptools \
python-mapnik mapnik-plugins mapnik-utils osm2pgsql josm postgresql-8.3-postgis \
python-dev build-essential libgdal-dev geotiff-bin sun-java6-jre

easy_install shapely geopy owslib pyproj

EDIT: If you’re looking for more up to date packages for geos, gdal, etc, try adding deb http://les-ejk.cz/ubuntu/ hardy multiverse to your /etc/apt/sources.list

Hike of Doom #2- OGC KML

2008-04-21T23:06:28-07:00

In commemoration of the OGC approval of KML as an open standard to share geographic content over the web, I’d like to share our recent “Hike of Doom #2” (kml provided by Mark Dotson).

The first weekend to hit 90 degrees, my friends and I travel inland to dive and swim in the Santa Ynez river. It is billed as a “30 minute” hike to our favorite watering hole. It becomes much more than that.

Of course the road leading up to the trailhead is closed due to construction so we have 3 miles of hiking on pavement just to get to the former trailhead- the Red Rocks parking lot.

Then the fun begins. A decent rainy season and some dam releases make for high flows and we’ve got half-dozen major river crossings to contend with. The recent fires added a good deal of organic matter to the river and the algae has bloomed accordingly. It is a wet, hot, rocky and slimy hike.

We make it to the swimming hole and enjoy the day. We dive, laugh, have a few beers.

The sun sets and the fun _ really _ begins.

Klaus, the Bavarian cyclist whom we’d met at the swimming hole, met up with us just after my girlfriend, Joselyne, sprained her ankle on a rock. Her ankle hadn’t started to swell yet but I could tell, drawing from my basketball injuries from the past, that she was not putting weight on it any time soon. We fashioned crutches from some driftwood. We met up with some turkey hunters (dressed in more camouflage more effective than most military uniforms) who helped us out by providing us some ankle wrap.

David and Andy began the trek back to the car to get help. The rest of us could either go back via the river bed , a rocky and treacherous endeavor given the setting sun, or head up to the main road and get some help. We decided on the main road and Shaun took off to alert the others to our plans. The main fire road was a trek in the _ opposite_ direction - longer, more elevation changes but smooth enough for a bike or truck and more accessible to vehicles.

I carried Jos, over my shoulder fireman-style and/or piggy-back, over the river crossings. On the flats, Mark and I pushed Jos on Klaus’ bike.

We pushed on up the trail until we reached the main road. Klaus, after drinking the last of our beer, biked up to the dam keeper’s residence at Gibraltar Dam while Christina, Sarah, Mark, Jos and I continued up the trail. A half-hour later, Klaus and the dam keeper arrived in a pickup and drove the rest of us back to the Red Rock “parking lot”.

But the construction and rebar on the causeway meant there was no way to cross with a normal vehicle so we went by foot. Jos got back on Klaus’ bike and we pushed.

Luckily the slight downhill grade allowed her to glide back for a good portion, graciously sparing Mark and I from permanent back injury.

Meanwhile the away team had gotten some semblance of cellular reception and attempted to call the authorities. The goal was to get a ranger truck to drive out to get us or at least unlock the gate to meet us half way at the Red Rock parking lot. The authorities response was fantastic if not a bit overzealous. By the time we had gotten within a 1/4 mile of our car, we spotted helicopters. Then a firetruck. Then an ambulance. Joselyne was coasting by on Klaus’ bike and they didn’t even stop for her on the first pass! Apparently expecting to rescue a mangled body from the wilderness, the EMTs were somewhat disappointed at the less challenging situation they faced - a girl, coasting down the road on a bike with a sprained ankle.

We were back in the car, on the road before dark and got home in time for pizza.

So what did we learn from this? Well as a Boy Scout, I am ashamed to say I wasn’t prepared. A well prepped emergency kit would have helped a lot. At least we had an LED headlamp. Some rope would have gone a long way towards making a stretcher. An instant-ice-pack, ankle wrap and some ibuprofen would have been handy. We were wet and the mercury was falling quickly; some emergency shelter and clothing would have assuaged my concerns about the nighttime chill.

But this was offset by the generosity of the many people we met for the first time - The hunters who lent us their medical supplies, the dam keeper who got up from his Sunday dinner to make sure we got back safely, the EMTs who put tremendous resources into organizing a military-scale search party, Klauss who so generously stuck with us and shared with us his bike, his wisdom and his company. Without their help and our group of friends, the story might have a less happy ending.

Never underestimate the power of human kindness, generosity and cooperation! And never believe me when I say it’s a short hike.

A quick Cython introduction

2008-04-19T10:30:49-07:00

I love python for its beautiful code and practicality. But it’s not going to win a pure speed race with most languages. Most people think of speed and ease-of-use as polar opposites - probably because they remember the pain of writing C. Cython tries to eliminate that duality and lets you have python syntax with C data types and functions - the best of both worlds. Keeping in mind that I’m by no means an expert at this, here are my notes based on my first real experiment with Cython:

EDIT: Based on some feedback I’ve received there seems to be some confusion - Cython is for generating C extensions to Python not standalone programs. The whole point is to speed up an existing python app one function at a time. No rewriting the whole application in C or Lisp. No writing C extensions by hand. Just an easy way to get C speed and C data types into your slow python functions.

So lets say we want to make this function faster. It is the “great circle” calculation, a quick spherical trig problem to calculate distance along the earth’s surface between two points:

p1.py

import math

def great_circle(lon1,lat1,lon2,lat2):
    radius = 3956 #miles
    x = math.pi/180.0

    a = (90.0-lat1)*(x)
    b = (90.0-lat2)*(x)
    theta = (lon2-lon1)*(x)
    c = math.acos((math.cos(a)*math.cos(b)) +
                  (math.sin(a)*math.sin(b)*math.cos(theta)))
    return radius*c

Lets try it out and time it over 1/2 million function calls:

import timeit  

lon1, lat1, lon2, lat2 = -72.345, 34.323, -61.823, 54.826
num = 500000

t = timeit.Timer("p1.great_circle(%f,%f,%f,%f)" % (lon1,lat1,lon2,lat2), 
                       "import p1")
print "Pure python function", t.timeit(num), "sec"

About 2.2 seconds. Too slow!

Lets try a quick rewrite in Cython and see if that makes a difference: c1.pyx

import math

def great_circle(float lon1,float lat1,float lon2,float lat2):
    cdef float radius = 3956.0 
    cdef float pi = 3.14159265
    cdef float x = pi/180.0
    cdef float a,b,theta,c

    a = (90.0-lat1)*(x)
    b = (90.0-lat2)*(x)
    theta = (lon2-lon1)*(x)
    c = math.acos((math.cos(a)*math.cos(b)) + (math.sin(a)*math.sin(b)*math.cos(theta)))
    return radius*c

Notice that we still import math - cython lets you mix and match python and C data types to some extent. The conversion is handled automatically though not without cost. In this example all we’ve done is define a python function, declare its input parameters to be floats, and declare a static C float data type for all the variables. It still uses the python math module to do the calcs.

Now we need to convert this to C code and compile the python extension. The best way to do this is through a setup.py distutils script. But we’ll do it the manual way for now to see what’s happening:

# this will create a c1.c file - the C source code to build a python extension
cython c1.pyx

# Compile the object file   
gcc -c -fPIC -I/usr/include/python2.5/ c1.c

# Link it into a shared library
gcc -shared c1.o -o c1.so

Now you should have a c1.so (or .dll) file which can be imported in python. Lets give it a run:

    t = timeit.Timer("c1.great_circle(%f,%f,%f,%f)" % (lon1,lat1,lon2,lat2), 
                     "import c1")
    print "Cython function (still using python math)", t.timeit(num), "sec"

About 1.8 seconds. Not the kind of speedup we were hoping for but its a start. The bottleneck must be in the usage of the python math module. Lets use the C standard library trig functions instead:

c2.pyx

cdef extern from "math.h":
    float cosf(float theta)
    float sinf(float theta)
    float acosf(float theta)

def great_circle(float lon1,float lat1,float lon2,float lat2):
    cdef float radius = 3956.0 
    cdef float pi = 3.14159265
    cdef float x = pi/180.0
    cdef float a,b,theta,c

    a = (90.0-lat1)*(x)
    b = (90.0-lat2)*(x)
    theta = (lon2-lon1)*(x)
    c = acosf((cosf(a)*cosf(b)) + (sinf(a)*sinf(b)*cosf(theta)))
    return radius*c

Instead of importing the math module, we use cdef extern which uses the C function declarations from the specified include header (in this case math.h from the C standard library). We’ve replaced the calls to some of the expensive python functions and are ready to build the new shared library and re-test:

    t = timeit.Timer("c2.great_circle(%f,%f,%f,%f)" % (lon1,lat1,lon2,lat2), 
                     "import c2")
    print "Cython function (using trig function from math.h)", t.timeit(num), "sec"

Now that’s a bit more like it. ** 0.4 seconds ** - a 5x speed increase over the pure python function. What else can we do to speed things up? Well c2.great_circle() is still a python function which means that calling it incurs the overhead of the python API, constructing the argument tuple, etc. If we could write it as a pure C function, we might be able to speed things up a bit.

c3.pyx

cdef extern from "math.h":
    float cosf(float theta)
    float sinf(float theta)
    float acosf(float theta)

cdef float _great_circle(float lon1,float lat1,float lon2,float lat2):
    cdef float radius = 3956.0 
    cdef float pi = 3.14159265
    cdef float x = pi/180.0
    cdef float a,b,theta,c

    a = (90.0-lat1)*(x)
    b = (90.0-lat2)*(x)
    theta = (lon2-lon1)*(x)
    c = acosf((cosf(a)*cosf(b)) + (sinf(a)*sinf(b)*cosf(theta)))
    return radius*c

def great_circle(float lon1,float lat1,float lon2,float lat2,int num):
    cdef int i
    cdef float x
    for i from 0 < = i < num:
        x = _great_circle(lon1,lat1,lon2,lat2)
    return x

Notice that we still have a python function wrapper (_def_) which takes an extra argument, num. The looping is done inside this function with for i from 0 < = i < num: instead of the more pythonic but slower for i in range(num):`. The actual work is done in a C function (_cdef_) which returns float type. This runs in **0.2 seconds** - a 10x speed boost over the original python function.

Just to confirm that we’re doing things optimally, lets write a little app in pure C and time it:

#include <math .h>
#include <stdio .h>
#define NUM 500000

float great_circle(float lon1, float lat1, float lon2, float lat2){
    float radius = 3956.0;
    float pi = 3.14159265;
    float x = pi/180.0;
    float a,b,theta,c;

    a = (90.0-lat1)*(x);
    b = (90.0-lat2)*(x);
    theta = (lon2-lon1)*(x);
    c = acos((cos(a)*cos(b)) + (sin(a)*sin(b)*cos(theta)));
    return radius*c;
}

int main() {
    int i;
    float x;
    for (i=0; i < = NUM; i++) 
        x = great_circle(-72.345, 34.323, -61.823, 54.826);
    printf("%f", x);
}

Now compile it with gcc -lm -o ctest ctest.c and test it with time ./ctest… about 0.2 seconds as well. This gives me confidence that my Cython extension is at least as efficient as my C code (which probably isn’t saying much as my C skills are weak).

Some cases will be more or less optimal for cython depending on how much looping, number-crunching and python-function-calling are slowing you down. In some cases people have reported 100 to 1000x speed boosts. For other tasks it might not be so helpful. Before going crazy rewriting your python code in Cython, keep this in mind:

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.” – Donald Knuth

In other words, write your program in python first and see if it works alright. Most of the time it will… some times it will bog down. Use a profiler to find the slow functions and re-implement them in cython and you should see a quick return on investment.

Links: WorldMill - a python module by Sean Gillies which uses Cython to provide a fast, clean python interface to the libgdal library for handling vector geospatial data.

Writing Fast Pyrex code (Pyrex is the predecessor of Cython with similar goals and syntax)

Spatial data in SQLite

2008-04-15T20:43:09-07:00

Slashgeo pointed me to a very interesting set of projects - SpatiaLite and VirtualShape. They provide a spatial data engine for the sqlite database. Think of it as the PostGIS of SQLite. It looks like this extends sqlite’s spatial capabilities far beyond the sqlite OGR driver.

SpatiaLite provides many of the basic OGC Simple Features functions - transforming geometries between projections, spatial operations of bounding boxes, and some basic functions to disect, analyze and export geometries.

VirtualShape provides the really neat ability to access a shapefile using the SpatiaLite/SQlite interface without having to import a copy - it reads directly off the shapefile by exposing the shapefile and its attributes as a “virtual table”. I can think of a million uses for this. For example, lets say you have a shapefile of US counties and the number of voter in the 2004 election as an attribute in the dbf. You want to find the total voter count in each state:

$ ls -1 counties.*
counties.dbf
counties.prj
counties.shp
counties.shx
$ sqlite3 test.db
sqlite> .load 'SpatiaLite.so'
sqlite> .load 'VirtualShape.so'
sqlite> CREATE virtual table virtual_counties using VirtualShape(counties);
sqlite> select sum(voters) as total_voters, state_name 
            from virtual_counties 
            group by state_name 
            order by total_voters desc;
9830550.0|California
7563055.0|Florida
7346779.0|Texas
...

Now this is fairly straightforward non-spatial SQL but the ability to run it against a shapfile without having to export to an intermediate data format is a very valuable tool.

Links:

When to use SQlite.
A video presentation by Richard Hipp (the author of sqlite).

Shell history - Why not?

2008-04-11T12:29:08-07:00

What an odd meme .. I don’t know why but I expected some more interesting results. I guess the majority of the commands I use are pretty pedestrian.

history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
163 vi
48 screen
29 python
28 ls
17 cp
17 cd
9 sqlite3
6 rm
5 sudo
4 htop

Working hard for some REST

2008-04-02T22:26:05-07:00

I don’t spend much time with web programming these days but I decided to give web.py (the minimalist python web framework) a shot and, while I was at it, try implementing a simple REST api.

First of all, web.py is truly everything it claims to be - small, light and easy to deploy behind lighttpd. It gives you a ton of flexibility to implement anything however you want - which is a plus or minus depending on how you look at it. I liked the inifinte flexibility but I can see alot of refactoring taking place and features needing to be implemented just to match the functionality built into a more structured framework like Django.

Back to the REST side of things. So I created a url-mapping to my “resources” or “nouns” and used the HTTP verbs (POST,GET,PUT,DELETE) to supply the interface. This was a joy to do in web.py which made it easy.

urls = ("/thing/(\d+)", "thing")
...
class thing:
    def GET(self, thingid):
        # select query and render to template
        ....
    def POST(self, thingid):
        # insert query and redirect to /thing/thingid
        ....
    def DELETE(self, thingid):
        # delete query
        ....
    def PUT(self, thingid):
        # use cgi args to run update query on specified thing
        ....

The hard part came when I realized that HTML forms do not implement DELETE or PUT methods! 2 of the 4 cornerstone HTTP verbs are not implemented in HTML forms?

Surely this can be accomplished with a top-notch AJAX library. I tried Prototype.js and it appears that the PUT and DELETE methods are simply tunneled over POST with an extra arg attached and the server side has to handle it accordingly. So I ended up just using a straight XMLHttpRequest which works but has it’s own problems.

How are you supposed to call PUT or DELETE through a web page? Is XMLHttpRequest the only way? What about browsers without javascript?

Upcoming books

2008-03-12T15:48:53-07:00

There are two new books coming out this summer which fill a valuable niche in the open-source GIS bookshelf:

* Applied Spatial Data Analysis with R

* Desktop GIS: Mapping the Planet with Open Source

These are both written by some of the top developers within their respective topics and I’m really looking forward to reading them.

Google Earth and the tilt sensor joystick on the X61s

2008-02-17T11:13:34-08:00

The X61s is one bad-ass machine. Besides the great performance, battery life and solid engineering, there are other hidden gems. Like the tilt sensors that were designed to protect the hard drive in case of a drop can also be used to detect the laptops motion under more normal circumstances.

There are some interesting applications that use some simple statistics to determine when the machine is “tapped” or julted to left or right. You can then assign actions to unique combinations of taps.

These applications all use the sysfs interface to the sensors (_ cat /sys/bus/platform/devices/hdaps/position _ will show your position in the x and y axis). But the sensors also provide a joystick interface that allow you to tilt the laptop along the two horizontal axes to control any number of applications. Including Google Earth.

1. Install tp_smapi

2. Test the sensors by running hdaps-gl , a simple OpenGL app showing the real-time tilt of your thinkpad.

3. Run jscal to calibrate the joystick. You’ll need to install the “joystick” package for this. The command is: jscal -c /dev/input/js0 After which you should keep your laptop level for a few seconds. Then, when prompted, tilt left, center, right, back (towards you), center, then forward.

4. Now fire up Google Earth. Open the Options menu, go to Navigation and select Enable Contoller.

5. You should now be able to zoom around by tilting the laptop. The keyboard shortcuts really help when you’re in this mode (Ctl-Up/Down to zoom, Shift-Up/Down to tilt, Shift-Left/Right to pivot).

There’s also a neat Perl-script technique to control a web-based google map which has some cool potential for an openlayers based system.

Since most Apple laptops have a similar sensor, you should be able to get the same thing going on your Macbook. Try it out..its alot more fun that using the mouse!

The shiny new X61s

2008-02-16T09:00:14-08:00

My HP laptop was nearing 5 years old. It had held up extremely well but most modern software taxed it to the absolute limits (just having firefox open with a flash ad in one tab was enough to send the system load through the roof). So I decided to try something new.

I was looking for something in the ultra-portable range. I tried out the OLPC and looked seriously at the Asus eeepc for a while. But they were far too difficult for me to type on. Ergonomics were extremely important and the only ultraportable that consistently rated high in that department was the IBM/Lenovo thinkpads. The X61s was appealing with its low voltage core2 duo and 2GB of RAM. All that in a small package about 3 lbs and about an inch thick.

So the X61s arrived and I figured I’d give it a try with the “stock” software. It was my first experience with Vista and I gave it my best shot. After about 1/2 hour of excessive clicking, boggy performance and pop-up windows, I shrunk the ntfs partition and installed Ubuntu Hardy Heron Alpha 4.

Sound, wireless with WPA, Compiz with 3D; the major things that normally plague a linux laptop install worked right out of the box. On the other hand, I’m running into a few bugs in nautilus (this is is alpha software after all), I can’t get bluetooth working, suspending to ram works but is a little buggy (have to restart some services manually) and I had to edit a few config files and compile a kernel module to utilize all the bells and whistles provided by the hardware. But it is still more fun than using Vista.

One thing that really shines on this machine is the battery. I got the 8-cell extended life battery and used some powertop tweaks cut my power consumption and was able to get the wattage down in the 10 to 15 watt range depending on usage patterns. No wonder it is energy star compliant! With that kind of wattage and battery capacity, I’m easily getting about 6 to 7 full hours of battery life.

Some tips if you’re setting up Linux on your X61s:

First and foremost, read thinkwiki. There you’ll find 95% of your answers. But to summarize my experience:
Upgrade your BIOS first (this is a good reason to keep your Vista partion around since Lenovo ships some handy update utils for windows).
Install the tp_smapi kernel module with HDAPS support. This will enable Linux to access the hard drive sensors for disk protection, motion sensing and the joystick interface
The big blue “Thinkvantage” button doesn’t work out of the box. I’m not sure what it should do but its a nicely placed button so don’t let it go to waste.
Tweak the power consumption. For the impatient, just install powertop and follow the instructions .. it will tell you what processes are waking your CPU and how to stop them. Also check out Less Watts - a full resource for tweaking linux power consumption.
Configure your trackpoint pointer and buttons. This involves setting up you xorg.conf file to emulate a middle scroll wheel as well as tweaking the speed and sensitivity of the pointer. BTW - if you’ve never tried a pointer, give it a shot … I’ve found it much more comfortable than a touchpad.
Laptop-mode , a set of kernel and userspace tools to manage hard-drive power consumption, can be handy. It can also be deadly to your disk if configured incorrectly. Basically it aggressively spins down the disk after short periods of inactivity to save power. Inevitably an application will try to hit the disk again and it will spin right back up. This leads to an unreasonably high amount of load cycles (100 per hour) and the drive can only handle a finite amount before failure (~600,000). You can configure it for more sane behavior but do your research before you enable laptop-mode! And check out smartctl to monitor the disks health.
If, after you unsuspend the machine, your screen is way too dark, try Ctl-Alt-F1 followed by Ctl-Alt-F7. There are some other hacks involving acpi configuration or grub kernel options but none of them have worked for me yet.

Human Impacts on the Global Marine Ecosystem

2008-02-15T09:06:11-08:00

We did it!

As some of you may know in 2005 through 2006, I was part of a research team1 , led by Ben Halpern at NCEAS, developing a global model of human impacts on the marine ecosystem. We created or compiled 17 high-resolution global datasets of human-induced threats (land-based pollutants, fishing, shipping, climate change, etc.) and 20 ocean habitat datasets. These were combined to create an impact index which models the cumulative level of human-induced stress on our oceans.

The results were published today in Science magazine and presented yesterday at the AAAS Annual Meeting. To summarize, we found that the entire ocean is affected and 40% is heavily impacted. It is not all bad news as there are many areas of relatively low impact which could provide examples for ecosystem restoration and opportunities for conservation. The global map is the first of its kind and will help clarify and quantify our cumulative impacts on the ocean and allow us to focus efforts geographically. The model is not perfect and can’t really be used to make decisions at a very localized scale but, given the available globally-consistent, reasonably-high-resolution data for all the various ocean threats and habitats, this is the best effort to date. The model itself is relatively simple with a very clear methodology which will allow scientists to tweak the parameters and add better data as it becomes available. For those of you interested in the GIS modeling end, NCEAS has a great summary of the data used in the model. Most of the data are available as raster data products or KML.

The media has picked up on the story with NPR, MSNBC, The Washington Post, USA Today and National Geographic covering it (to name a few). I especially recommend the NPR site as it has a great animation and an audio segment.

So congratulations to everyone who made this happen!

_ 1 Benjamin S. Halpern, Shaun Walbridge, Kimberly A. Selkoe, Carrie V. Kappel, Fiorenza Micheli, Caterina D’Agrosa, John F. Bruno, Kenneth S. Casey, Colin Ebert, Helen E. Fox, Rod Fujita, Dennis Heinemann, Hunter S. Lenihan, Elizabeth M.P. Madin, Matthew T. Perry, Elizabeth R. Selig, Mark Spalding, Robert Steneck, Reg Watson (2008). A global map of human impact on marine ecosystems. Science, vol. 319 _

EDIT:

Some additional articles:

* New York Times

* BBC Video on YouTube

Why is the command line a dying art?

2008-02-02T20:50:19-08:00

Sadly, a lot of GIS folks have never come into contact with a command line interface (CLI) . I’ve met even experienced computer users who, when faced with a command-line prompt, experience some autonomous nervous system lock up that causes their eyes to glaze over and prevents any knowledge from entering their brain from that moment forward. The all-Windows, all-GUI mentality of the current GIS market leaders just doesn’t expose you to it (if you remember working with coverages at the ESRI Arc/Info command line, you official qualify as an “old-timer”). And the DOS command line is virtually invisible to XP and vista users. Linux users are more CLI aware but this is even becoming less important as distros such as ubuntu GUI-ify everything.

So why the fear of the command line? Why is it assumed to be more “complicated” than a graphical user interface (GUI)? I have found that, in some cases, the opposite is true … there is something reassuringly simple about typing something and getting a response back. It feels like you are in direct control of the computer. Which, indeed, you always are. The computers always do exactly what you tell them, whether you are in a GUI or a CLI. But GUIs attempt to abstract away the details so that you don’t need to know exactly what you’re telling the computer to do. This nice fluffy feeling comes at the cost of many important factors.

The benefits of the command line interface

Automation

If you had, for instance, monitoring data coming in in a hourly basis and needed to process the data, would you want to be on call 24 hours a day to click a few buttons. Of course not. Write a command that performs the job and schedule it to execute at some regular interval. (I wonder if those guys on LOST ever thought to just set up a cron job to enter the numbers in the hatch?)

Repeatability

Whenever I show someone a CLI-based method for solving their problem, they almost immediately say (or at least imply) that the typing is too much trouble. Consider this command to convert a .tif image to ERDAS .img (HFA) format:

cd /data/images
gdal_translate -of HFA aerial.tif aerial.tif.img

You might ask, “Why not just use a GUI, click a button or two, and get your output”. Sure. Now do that for 2,000 tif images. With a CLI you only have to type a few extra lines.

cd /data/images
for i in *.tif; do 
  gdal_translate -of HFA $i $i.img;
done

Documentability

There is nothing more important to a GIS Analyst than documenting his/her work! We live by metadata and methods write-ups. Now picture an intense 5 hour work session … everything needed to get out by 2pm. You’re done and now it’s time to document your procedure and methods. With the CLI, you copy and paste your commands from the terminal or simply look at your command history which will show exactly what you did and how. You can store this in a text file and come back to it months later and be able to re-run the procedure.

With the GUI, you have to remember and describe every click, every sub-menu, every option, every action taken to arrive at the answer. Often this requires verbose description, screenshots, etc. None of which is recorded in any history file of course. And of course, when the client inevitably comes back the next day with modifications, none of it is repeatable in any automated fashion with a GUI.

Accessibility

It’s just plain text with a CLI. You can print it out and study it on the bus. You can email the whole process to co-workers. You can use a concurrent versioning system to keep track of changes to scripted procedures. You can transfer massive amounts of knowledge without having to sit down and go through everything step-by-step, click-by-click in a visual interface.

Accuracy

Far too often, GUI designers make over-reaching assumptions about how things should work. The idea is often that the user should not need to know anything more than the absolute minumum. To use a car analogy, the driver turns the key, presses the pedal and steers but does not need to know what goes on under the hood. This works most of the time. But the law of leaky abstractions usually takes hold and something inevitably breaks or performs differently than expected. Since the CLI does not hold your hand (it executes the exact command you give it) it more accurately mimics the actual physical interaction with the computer and is much more useful in debugging and investigating complex problems.

So basically, don’t make the mistake of thinking that a pretty window will always contain the magic button to get the job done. In many cases, a command line is much more efficient, even essential. If you don’t know how to effectively work in a command-line environment, do yourself a huge favor and learn.

Oh and I’d be remiss if I didn’t mention Neal Stephenson’s book on the subject … a bit technically outdated but a great quick read on why command lines are still very relevant in the face of increasingly sophisticated graphical interfaces.

Impervious surface deliniation with GRASS

2008-01-26T11:47:32-08:00

Watersheds with lots of roads, buildings, parking lots, rock surfaces, compacted dirt, etc tend to prevent inflitration and cause rapid runoff in response to rainfall. This poses a number of challenges for managing stormwater and water quality. Not surprisingly, the percentage of hydrologically impervious surface in a given watershed is an important factor in many hydrologic models. Using standard aerial photography and GRASS, it’s a relatively simple process to create an impervious surface map using supervised classification.

First find an aerial photo. I grabbed a NAIP image from CASIL but you might want to try using OpenAerialMap. The red, green and blue visible bands are usually sufficient for differentiating between impervious and pervious land use types… For distinguishing different types of vegetation you might want to use a multispectral imagery source with non-visible bands (ie near infrared) but this is usually lower resolution (eg. 30 meter pixels of Landsat) or much more expensive.

Next we jump into GRASS and import our image into a new location:

r.in.gdal -e input=naip.img output=naip location=impervious

Exit and log back into your new location. If you look at the imported rasters, you’ll see three rasters, not one. Each band (R, G and B) gets imported separately.

GRASS 6.3.cvs (impervious):~/>  g.list rast
raster files available in mapset permanent:
naip.1 naip.2 naip.3

We need to indicate that these rasters form a logical group

i.group group=naip2 subgroup=naip2 input=naip.3@PERMANENT,naip.2@PERMANENT,naip.1@PERMANENT
i.target -c group=naip2

At any time you can list the rasters in a given group/subgroup to confirm.

i.group -l -g group=naip2 subgroup=naip2

Now the real heart of the process. We need to define “training areas” which are polygons around representative land use types. I used QGIS to load the aerial photo and create a new polygon layer with an integer attribute field called vegnum. I digitized a few rocks, paved areas, rooftops and dirt roads to represent the impervious areas to which I assigned vegnum=1. Then I selected some grasslands, forests, lakes and chaparral and assigned 2 as the vegnum. The next step is to load the polygon data into GRASS and rasterize it (_in retrospect it would have just been easier to create the grass vector layer from scratch in QGIS to avoid the import step_). Note that the vegnum field is specified as the raster value column.

v.in.ogr -o dsn=./training/train1_utm/train1_utm.shp output=train1 layer=train1_utm min_area=0.0001 type=boundary snap=-1
v.to.rast input=train1 output=train1 use=attr column=vegnum type=point,line,area layer=1 value=1 rows=4096

Next we use i.gensig to generate a spectral signature (the statistical profile; mean and covariance matrix of the input pixels) for the training areas.

i.gensig trainingmap=train1 group=naip2 subgroup=naip2 signaturefile=naip2_train1.sig

Now that we have a signature of impervious vs. non-impervious surfaces, we can use the maximum likelihood method to classify each pixel into the highest probability category.

i.maxlik group=naip2 subgroup=naip2 sigfile=naip2_train1.sig class=imperv

You might notice a slight speckled, noisy appearance due to things like shadows, reflections or imperfect training areas. Usually these small 1-pixel deviations are not interesting enough to keep so we can smooth out the image taking the mode (most comon) cell in a 3x3 window.

r.neighbors input=imperv output=imperv_mode method=mode size=3

And here are the results… calculating imperviousness will most likely be an iterative process so be prepared to evaluate the output, tweak the training areas and rerun the process a few times. Once you’re happy with the results, you can use zonal statistics with a tool like starspan to find the percent imperviousness of your watersheds or other regions.

Mirone: A GUI for GDAL and GMT

2008-01-06T01:37:42-08:00

In the why-haven’t-I-ever-heard-of-this department:

Mirone is a Windows MATLAB-based framework tool that allows the display and manipulation of a large number of grid formats through its interface with the GDAL library. Its main purpose is to provide users with an easy-to-use graphical interface to the more commonly used programs of the GMT package.

There is also a version that does not depend on MATLAB which is what I decided to try. This is a great package; easy to install, very usable, lots of high-end raster functionality, and a good sense of humor…

Considering GMT and GDAL can be a bit challenging and unfamiliar for a typical windows user, Mirone is a huge step forward.

Among some of the functionality that is an absolute pleasure to work with compared to some other software packages: surface profiles, image-flipping, DEM derivatives, color-ramping, contouring, histograms, kernel filtering… And that’s just scratching the surface. I highly recommend checking it out.

More on Google Charts and a python interface

2007-12-19T22:39:29-08:00

Well it’s been almost a full two weeks since google charts API came out. A really nice service but it’s only going to be useful with a high-level programming API. Enter PyGoogleChart .. a python interface to generate google chart urls.

Taking one of my previous example datasets, here’s the 10-second howto:

<blockquote>from pygooglechart import SimpleLineChart
chart = SimpleLineChart(400,200)
data = [32.5,35.2,39.9,40.8,43.9,48.2,50.5,51.9,53.1,55.9,60.7,64.4]
chart.add_data(data)
url = chart.get_url()
print url
</blockquote>

which gives us:

http://chart.apis.google.com/chart?cht=lc&chs;=400x200&chd;=t:32.5,35.2,39.9,40.8,43.9,48.2,50.5,51.9,53.1,55.9,60.7,64.4

and our chart image:

Geologist vs. Engineer

2007-12-12T13:45:07-08:00

Uncylopedia, the self-proclaimed encyclopedia “full of misinformation and utter lies”, has a hillarious article about Geologists. I especially like the ”Geologist-Engineer Controversy” which, having worked with both geologists and engineers extensively, is a pretty accurate portrayal of their respective approaches.

Geology, being an art as much as a science, has always baffled and worried engineers, hence the engineers’ defensive weapons of pocket protectors, slide rules, black socks, and eventually computers.

A related joke:

A geologist and engineer walk into a job interview. They are each asked a simple math question : ‘What is 2 times 2?’. The engineer replies, ‘It’s 4.00000’. The geologist replies, ‘Ah.. it’s about 4’

Quick way to publish a point shapefile to html

2007-12-10T20:14:14-08:00

There are better ways to put data on the web but my latest little project wasn’t about the best way but the quickest way to get some spatial data into the hands of those unfortunate souls who don’t have GIS software. The goals were pretty simple:

* Take a single point shapefile (or other OGR readable vector data source)

* Convert it into html/js that would use one of the web mapping APIs to display the points and all their attributes.

* The output had to be a standalone, self-contained html file that could be emailed. No server side anything required.

I came up with a quick python hack to do the job (source code). Mapstraction, with it’s goal of providing a common javascript API for a number of map providers, seemed like an obvious choice. The python portion of the code reads the shapefile using OGR (you will need the python-gdal bindings, see FWTools) and constructs the html/js. All the javascript is sourced to external URLs so there is no software dependency except for a working network connection.

This allows for a single command:

shp2Mapstraction.py bearboxes.shp bearboxes.html Yahoo

which produces an html file providing a Yahoo maps interface to the data; in this case the point location of all the bear boxes (food storage lockers to keep your stuff separated from the bears) in the Sierra Nevada.

Currently it just supports Microsoft Virtual Earth and Yahoo. I had to bypass Google because their key system is restricted by URL. And the mapstraction-to-openlayers connection wasn’t working too well though I haven’t really investigated.

Anyways, it provides a quick and easy way to deliver spatial data to anyone with a browser and internet connection.

Google Charts - their latest web service

2007-12-06T23:57:01-08:00

Google Charts is a web based API for generating charts/graphs. It supports alot of the common types of graphics including line, pie, bar, scatter plots and Venn diagrams. I’ve relied on a bunch of other server-side graph generators (owtchart, jpgraph, sparklines, matplotlib, etc) but this looks like it might be a contender.

Still there is no higher-level programming API yet … but give it a few days (interface with numpy anyone?). ExileJedi blog listssome other potential disadvantages:

* You are limited to 50,000 queries per user per day, which may pose some scalability concerns if you plan to build something big on this.
* You have to be careful about the number of data points you submit in your request as you can quickly exceed the allowable URL length, and furthermore you might end up with illegibly smooshed-together data points due to the scale of your output.
* There's always the "OMG Google will absorb all our data and become sentient, turn evil, and unleash an army of death robots on us all, run for your lives!" paranoia, but that's really just silly talk.
EDIT: It appears this service only support GET requests. On one hand you’re adding new data so you should be POSTing it, right? On the other hand, you’re asking to GET a graphical representation of a set of numerical values. What would a “restful” version of a web graphing API look like? Maybe some of the REST gurus can clear that up.

Take the larger view of GIS

2007-12-05T11:04:51-08:00

It’s interesting to see the passionate responses to Joe Francia’s article claiming that neogeography is != GIS. One one side there is a small group of folks bashing neogeography and claiming the superiority of “GIS”. On the other side there is the attitude claiming that some “revolution” has occurred which has supplanted traditional geographic techniques. You’d think there was a cold war going on! Both memes are as wrong as they are arrogant.

I have always defined GIS as

Geographic Information System: The integration of hardware, software, procedures and people to manage the collection, creation, analysis, synthesis, sharing and visualization of spatial information.

Neogeography easily fits that bill. So does Enterprise IT. So does Desktop mapping. So does Geostatistics. Geodesy. Web Mapping. Remote Sensing. LBS mobile technologies. Cartography. Surveying. Spatial Analysis and Modeling. Database management. Sensor webs. GPS… These disciplines are all a small piece of the larger puzzle that is GIS (whether their staunch adherents will admit to it or not!).

The key word in this controversial acronym is System. In order for any organization to implement a successful GIS, they must figure out a) which technologies will work for them and b) how to integrate them into a coherent whole. All of these aspects of GIS have something to offer so it’s important not to get stuck in a rut with blinders on. This goes for all “sides” of this ridiculous “neogeo vs GIS” argument.

For the cartographers in the house…

2007-12-04T20:04:18-08:00

Here’s another one for the blogrolls:

http://strangemaps.wordpress.com/

Privacy, Location Technology and Bad Journalism

2007-11-20T00:12:50-08:00

The Ventura Star has run an article about privacy issues and modern geolocation technology.

As important as this topic is, John Moore (the author) is clearly uninformed. This is a horrible piece of journalism. Moore mixes the potential negative effects of various technology such as RFID, cellular communication, sensor networks, nanotech, community data collection efforts, navigation systems, and GPS into one chilling, over-simplified and baseless viewpoint. Instead of reporting the details of Michael Goodchild’s talk at Ventura College, he treated us to his own paranoid, incoherent vision of the future of technology. Moore’s entire premise is based on the fact that:

“GPS is a system that basically allows you to know where you are anywhere in the world within one meter”

That much is true. He uses this fact to extrapolate the conclusion that GPS allows some nefarious force to monitor your groceries, cell phone calls, and indeed your every movement.

GPS recieves satellite signals translates those signals into a location. It takes an entirely different technology to transmit these locations to some third party. I guarantee you that none of my gps tracks have gotten into anyones hands without my consent (come on John Moore, prove me otherwise).

The title speaks volumes to his ignorance:

“Where are you in life? If you don’t know, others using GPS devices do”

Suggesting that other people with GPS can a) track my movements or b) be tracked by me , shows a complete lack of understanding of the technology. Sure there are privacy dangers. But those dangers must be presented clearly and concisely by someone with half a clue, not this paranoid bullshit journalism. This article would not even pass as a high school essay.

Looking for LIDAR services

2007-11-12T18:04:40-08:00

I’m looking for a LIDAR specialist to fly some sensors around San Diego. Ideally we would need someone who could collect both LIDAR data and digital aerial photography (high-res but only visible spectrum), process the data (generate bare-earth DEMs and georeferenced aerials) and deliver it in a GIS-compatible format. This is in response to the recent fires related to erosion control.. with rainy season coming we’d be on a tight schedule.

Does anyone have any suggestions of good companies who could provide this service? Please feel free to recommend your own services if you think it would be a good fit.

You can also contact me directly at perrygeo+lidar at gmail.com

Poetics of Cartography

2007-10-20T10:40:56-07:00

In case you missed the fantastic Chicago public radio program last night on This American Life, the NPR-syndicated show did an entire program on “mapping”. It goes well beyond the idea of simply mapping our physical infrastructure and really opens up the idea of mapping to the widest possible definition; using all our senses to create a multi-dimensional representation of our world. Within the vast experience of life, mapping is described as the abstract process of summarizing and synthesizing a singular slice of that experience.

The show is availableas a stream and is really worth a listen this weekend.

P.S. The title of this post comes directly from a quote by Denis Wood, the author of The Power Of Maps and geographer who is mapping some non-conventional aspects of his neighborhood in Raleigh, North Carolina. The first and arguably most interesting portion of the show from a geographer’s standpoint.

Turning Ubuntu into a GIS workstation

2007-10-20T10:31:13-07:00

It just keeps getting easier and easier to get a fully functional open source GIS workstation up and running thanks to Ubuntu. The following instructions will take your vanilla installation of Ubuntu 7.10 and add the following top-notch desktop GIS applications:

* Postgresql/PostGIS : a relational database with vector spatial data handling

* GRASS : A full blown GIS analysis toolset

* Quantum GIS: A user-friendly graphical GIS application

* GDAL, Proj, Geos : Libraries and utilities for processing spatial data

* Mapserver : web mapping program and utilites

* Python bindings for QGIS, mapserver and GDAL

* GPSBabel : for converting between various GPS formats

* R : a high-end statistics package with spatial capabilities

* GMT : the Generic Mapping Tools for automated high-quality map output

While this is not a comprehensive list of open source GIS software, these packages cover most of my needs. If you want to live on the bleeding edge and have to have the absolute latest versions, you’ll be better off installing these from source. But for those of us that want a stable and highly functional GIS workstation with minimal fuss, this is the way to go:

1. Go to _ System > Administration > Software Sources _ and make sure the universe and multiverse repositories are turned on. Close the window and the list of available software packages will be refreshed.

2. Open up a terminal (ie the command line) via _ Applications > Accessories > Terminal_ and type the following:

sudo apt-get -y install qgis grass qgis-plugin-grass mapserver-bin gdal-bin cgi-mapserver \

 python-qt4 python-sip4 python-gdal python-mapscript gmt gmt-coastline-data \
 r-recommended gpsbabel shapelib libgdal1-1.4.0-grass

The sudo part indicates that the command will be run as the administrator user, _ apt-get -y install_ is the command telling it to install the list of packages and answer yes to any questions that pop up.

3. There is one package that is worth upgrading to the latest and greatest - Quantum GIS. The latest version (0.9) is due out very shortly and has the ability to write plugins using the python programming language. A big plus!

Download the latest build from http://qgis.org/uploadfiles/testbuilds/qgis0.9.0.debs_ubuntu_gutsy.tar.gz and extract it ( right-click > Extract Here ). In the directory you’ll see 4 .deb files, only 3 of which you’ll need unless you plan on doing any development work.

Double click libqgis1_0.9.0_i386.deb and you’ll get a message saying an older version is available from directly from ubuntu. We already know this so just close and ignore it. Click Install Package and wait for it to complete then close out.

Repeat for qgis_0.9.0_i386.deb and qgis-plugin-grass_0.9.0_i386.deb (in that order).

And there we have it, about 15 minutes depending on your internet speed and you’ve installed a high-end GIS workstation built completely on free and open source software.

Update to QGIS Geocoding plugin

2007-10-19T15:00:35-07:00

With the release of QGIS 0.9 imminent , I decided to install in on Windows XP and noticed that the geocoding pluginwas failing… sure enough I had hardcoded linux temporary directories. So I reworked the python code to determine the temp dir in a more cross-platform way (using tempfile.gettempdir() ) and it works fine.

The update can be downloaded here.

Assuming you’ve installed qgis in the standard location, just unzip this into C:\Program Files\Quantum GIS\python\plugins (windows) or /usr/share/qgis/python/plugins (Linux) and you should be good to go. Note that you’ll have to create the “plugins” directory if it doesn’t exist.

CTech software goes multithreaded

2007-10-12T16:20:17-07:00

CTech has announced that the next version of it’s flagship software package, EVS (Environmental Visualization System), will take full advantage of multiple processors.

My experience with EVS is mostly in the realm of 3-dimensional kriging and geostatistics. Given the amount of data crunching involved, it’s always been sluggish when dealing with a non-trivial amount of data. Nothing is more frustrating that seeing one of your CPU cores cranking away while the others sit idle! But some users are reporting that the new multithreaded modules get nearly linear performance increases when adding more processing cores.

CTech is certainly not the first scientific/geostats application to go parallel. But it is the first program that I personally use on a regular basis that will take advantage of a multi-processor system. I hope this marks the beginning of an industry trend in that direction.

Autodesk open sources coordinate system software

2007-09-25T09:07:55-07:00

Not very often do I see open source mentioned on the front page of my Google Finance page (let alone Geospatial Open Source). But here it is.. the announcement was made at FOSS4G2007 that autodesk will be open sourcing part of it’s coordinate system and map projection technology.

So what motivation does Autodesk (or any other company) have to open source it’s technology? An important line from Lisa Campbell, vice president, Autodesk Geospatial:

“Our intent to contribute again to the open source community is a reflection of our customers’ desire for faster innovation, more frequent product releases, and lower total cost of ownership.”

Parallel python and GIS

2007-09-18T21:23:51-07:00

Let’s face it - processing speeds aren’t going to be increasing according to Moore’s Law anymore; Instead of faster CPUs, we’ll be getting more of them. The future of programming, it seems to me, lies in the ability to leverage multiple processors. In other words, we have to write parallel code. Until I read Seans’ post, I was unware that there was a viable python solution. I had been growing quite dissillusioned by python’s dreaded Global Interpreter Lock which confines python to a single processing core. I’ve even started learning Erlang to leverage SMP processing (until I realized that Erlang and it’s standard libraries are virtually useless for anything that needs to handle geospatial data).

So I gave Parallel Python (pp) a shot. Since Sean also offered up a bounty for the first GIS application that used pp, I thought it might be a good time to try ;-)

A good candidate for parallel processing is any application that has to crunch away on lists/arrays of data and whose individual members be handled independently (see pmap in Erlang). I have been working on an application to smooth linework using bezier curves. It’s not quite polished yet but the image below shows the before and after

… but bezier curves aren’t quite the subject of this post. Let’s just say the algorithm takes some time to compute (if you’re using a high density of verticies) and can be handled one LineString feature at a time. This makes it a prime candidate for parallelization.

Given a list of input LineStrings, I could process them the sequential way:

<blockquote>smooth_lines = []
for line in lines:
    smooth_lines.append( calcBezierFromLine( line, num_bezpts, beztype, t) )</blockquote>

Or use pp to start up a “job server” which doles the tasks out to as many “workers”. A busy worker utilizes a single processing core so a good rule of thumb would be to start up as many workers as you have CPU cores:

<blockquote>numworkers = 2 # dual-core machine
job_server = pp.Server(numworkers, ppservers=ppservers)
smooth_lines = []
jobs = [(line, job_server.submit(calcBezierFromLine, (line, num_bezpts, beztype, t), \
                             (computeBezier, getPointOnCubicBezier), ("numpy",) ))  for line in lines]
for input, job in jobs:
    smooth_lines.append( job() )</blockquote>

Theoretically the parellized version should run twice as fast as the sequential version on my core2 duo machine. And reality was pretty darn close to that:

<blockquote>$ time python bezier_smooth_pp.py 2
Shapefile contains 1114 lines
Starting pp with 2 workers
Completed 1114 new lines with 8 additional verticies for each line segment along a cubic bezier curve

real    0m10.908s
...

$ time python bezier_smooth_pp.py 1
Shapefile contains 1114 lines
Starting pp with 1 workers
Completed 1114 new lines with 8 additional verticies for each line segment along a cubic bezier curve

real    0m20.007s
...
</blockquote>

Just think of the possibilities. In the forseeable future, the average computer might have 8+ cores to work with. This could mean that your app will move 8x faster if you parallize the code (assuming there are no IO or bandwidth bottlenecks). I’d love to test it out on a system with more than 2 processing cores but, unfortunately, I don’t have access to any beowulf clusters, Sun UltraSparc servers, or 8-core Xeon Mac Pros. This is what I really need to complete my research ;-) So if anyone want to donate to the cause, send me an email!

And to answer Sean’s bounty, I don’t consider this an actual application (yet) but I hope it can spur some interest and move things in that direction. But if you feel the need to send me some New Belgium swag (or one of the machines listed above), feel free ;-)

The world turned right-side up

2007-09-05T14:45:25-07:00

I’ve been working alot in Surfer these days; an excellent geostats and surface mapping package. I was very happy to find that GDAL read it’s .grd binary format until I noticed the output from gdalinfo:

C:\Workspace\Temp\interpolation>gdalinfo svpce_5.grd

Driver: GS7BG/Golden Software 7 Binary Grid (.grd) Files: svpce_5.grd Size is 555, 339 Coordinate System is ' Origin = (383371.000000000000000,3764907.000000000000000) Pixel Size = (0.500000000000000,0.500000000000000) Corner Coordinates: **Upper **Left ( 383371.000, **3764907.000**) **Lower **Left ( 383371.000, **3765076.500**) Upper Right ( 383648.500, 3764907.000) Lower Right ( 383648.500, 3765076.500) Center ( 383509.750, 3764991.750) Band 1 Block=555x1 Type=Float64, ColorInterp=Undefined NoData Value=1.70141e+038

Notice that upper Y value is south of the lower Y value! Basically the raster lines order is reversed (bottom-to-top instead of the normal raster orientation of top-to-bottom). I’ve also experienced the same issue with some NetCDF files so I thought it would be good to have a generic solution to the problem.

So I hacked up the gdal_merge.py script (distributed with gdal, fwtools, etc) and created a raster flip script that will invert the image along the y axis and retain the georeferencing and metadata. The resulting flip_raster.py script seems to work pretty well though it is far from tested.

Here’s an example:

The standard gdal_translate method (which doesn’t account for the inverted coordinate space):

gdal_translate -of GTiff krig1.grd krig1_translate.tif

And the flipped raster method:

flip_raster.py -o krig1_flip.tif -of GTiff krig1.grd

And we’re good. gdalinfo confirms that we have the same extents, pixel sizes, metadata, etc as the original dataset.

Mapserver vs Mapnik revisited

2007-09-04T20:12:02-07:00

A while ago, I was enamored with mapnik’s image quality despite it’s limitations compared to the vast configurability of the mapserver mapfile. Now that mapserver uses the AGG rendering library, it might not be necessary to compromise configurability in order to get beautiful linework. I just installed the recent beta of mapserver 5.0 and the image quality is very crisp… but this comes at the expense of rendering speed.

All the times below are the average of ten runs using a full global view of a simplified shapefile of country borders.

mapserver (gd) : 0.082 sec , 18kb

OUTPUTFORMAT

NAME “GD_JPEG” DRIVER “GD/JPEG” MIMETYPE “image/jpeg” IMAGEMODE RGB EXTENSION “jpg” END

shp2img -m test.map -o mapserver_gd_test.jpg

mapserver (agg) : 0.188 sec , 16kb

IMAGEQUALITY 80 OUTPUTFORMAT NAME ‘AGG_JPEG’ DRIVER AGG/JPEG IMAGEMODE RGB END

Note that if we bump up imagequality to 90% to (roughly) match the mapnik image, the rendering time and size increase a bit (.201 sec, 25kb)

shp2img -m test.map -o mapserver_agg_test.jpg

mapnik (agg) : 0.282 sec, 23kb python test_mapnik.py

Running this through the python interpreter is likely to interfere with the speed of the results so these times may not be very comparable to shp2img.

Using these preliminary results, it looks like mapserver 5.0 with AGG rendering is roughly equal to mapnik based on a balance of quality/speed/image size. But since I’d prefer to use mapfiles over the undocumented mapnik xml format any day, I think I’ll stick with my beloved mapserver. Kudos to the mapserver developers for raising the bar once again.

Performance testing rasters with mapserver

2007-09-04T00:19:42-07:00

There’s been some good talk on the mapserver list (thanks to Gregor’s diligent testing) about performance related to serving up raster imagery.

First off, comparisons of image formats.Then a look at some TIFF optimization techniques like overviews (similar to “pyramids” in ESRI land) and internal tiling to boost rendering speed.

Most of the conclusions are not all that staggering:

* TIFF is fastest but takes up more space compared to ECW and JPEG2000.

* Overviews speed up TIFFs tremendously when zoomed out (ie when mapserver would otherwise have to perform some heavy downsampling)

* Internal tiles in GeoTIFF format give a boost when zoomed in (only the necessary tiles are read from disk)

* The TIFF comparison was run on two setups; a monsterous 8-core, RAID-5 equipped beast and a low-memory virtual machine on low-end PC hardware. The TIFF optimizations are very noticeable on the lesser machine but almost completely negligible on the high-end machine.

Both tiling and overviews are useful, but only on machines with resource

shortages, such as slow disks or a lack of spare RAM for caching.

Nothing earth-shattering (these techniques are often mentioned as best practices) but is very nice to see some hard numbers to back it up. Plus the verbose test logs provide a good example for a newbie trying to implement them. Good stuff Gregor!

Mapping the Undesirable

2007-08-28T22:35:21-07:00

While by no means a new phenomenon, Vision 20/20 is offering a service allowing you to see a map of the registered sex offenders in your area. WorldChanging, one of my favorite blogs on emerging technologies, has a great article discussing the issues surrounding mapping of sex offenders.

Is this sort of service, based on powerful networked technologies – and one being sold on the basis of fear – an appropriate use of the technology? Where is the data being sourced from? How are the people inputting it being supervised? And what rights to privacy and presumptions of innocence are the people it tracks entitled to?

These are good points, but even more disturbing to me as a citizen and a GIS professional, is that these maps use geocoding services that are not nearly accurate enough for the scale at which they are being viewed. Even in suburban areas, using linear-referenced geocoding techniques can still yield errors of 100s of meters! The margin of error in the geocoding engine alone is enough to place the sex offender icon directly on an innocent citizens’ home.

For instance, which of the homes in the map below is the residence of a sex offender? Does the ambiguity bother you? Would it matter more if you were the innocent person living next door?

For maps with this much social weight, I think that a bit more diligence is due to ensure that this data is as accurate as it needs to be!

Zaca Lake Fire Map

2007-08-03T15:19:29-07:00

Ah the joys of living in southern california. The Zaca Lake fire has been burning since July 4th and recently flared up again with a shift in winds which is blowing ash and a very ominous plume of smoke all over downtown santa barbara. While it’s still burning in the wilderness areas north of town, the Paradise Road area along the Santa Ynez river has been evacuated. Check it out on google maps.

The Santa Barbara News Press is reporting the fire has reached 39,000 acres and has cost $43 million thus far to contain. The county supervisors are likely to declare a state of emergency and there is already a health warning in effect. So much for my bike ride this afternoon…

Desktop vs Web UI

2007-06-11T17:37:04-07:00

This might be a dup story for some but I thought it was interesting enough to post nonetheless:

Jeff Atwood wrote an interesting piece about Desktop vs Web UI that is directly relevant to mapping : Who Killed the Desktop Application?. He compares the usability of Microsoft Streets and Trips with Google Maps and concludes

All the innovation in user interface seems to be taking place on the web, and desktop applications just aren’t keeping up.

OGR and matplotlib examples

2007-06-10T19:49:17-07:00

Jose Gomez-Dans posted a great example of using OGR, Postgis and Matplotlib with Python - OGR, Python y Matplotlib (Spanish only).

FDO, GDAL/OGR and FME ?

2007-05-31T16:40:30-07:00

FDO, GDAL and FME all seem to operate in roughly the same domain - Providing a data model, API and tools to translate between spatial data formats. Does anyone know of any good write-ups comparing/contrasting the features of these three libraries?

QGIS Geocoding plugin

2007-05-28T10:11:23-07:00

A few weeks back, I decided to take the plunge and learn the python bindings for QGIS 0.9. My first experiment was to implement a geocoder plugin. What started mostly as a learning experiment turned into something that might actually be useful!

The idea was to use web services to do all the actual geocoding work (the hard part!) and the delimited text provider to load the results into qgis. Right now it’s built on top of the Yahoo geocoder which is, IMO, the best out there.. very flexible about the input format. The geopy module is used to interact with the geocoding services so it could potentially support other engines such as geocoder.us, virtual earth, google, etc.

The user interface is very straightforward; enter list of addresses/placenames seperated by a line break, pick an output file and go. To be legitimate, you should also sign up for a yahoo api key, though the ‘YahooDemo’ key will work ok for testing purposes.

Here’s the install process (assuming you already have python, pyqt4, qgis 0.9, qgis bindings, etc. set up):

svn checkout http://perrygeo.googlecode.com/svn/trunk/qgis/geocode cd geocode emacs Makefile # change install directory if needed sudo make install

This is just a rough cut and it’s my first attempt at using the qgis and qt apis so there are probably many things that could be improved upon. Ideally this plugin could:

* Parse text files as input

* Allow for a choice of geocoding engine

* ???

Feedback (and patches) welcome ;-)

Python gpsd bindings

2007-05-27T11:31:48-07:00

If you want to get a linux/unix machine talking to your GPS unit, most likely you’ll be using gpsd. There are many great apps that build off of gpsd such as kismet and gpsdrive.

Installing gpsd on debian/ubuntu systems is as simple as

sudo apt-get install gpsd gpsd-clients

You should be able to connect your gps via serial port and start a gpsd server

sudo gpsd /dev/ttyS0

The gpsd server reads NMEA sentences from the gps unit and is accessed on port 2947. You can test if everything is working by running a pre-built gpsd client such as xgps.

This is very useful for situations where you need lower-level access to the gps data; for logging your position to a postgres database for example. The debian packages (and most others I’m assuming) come with gps.py, a python interface to gpsd allowing you to pull your lat/long from the gps in real time. This opens the door for all sorts of neat real-time gps apps.

import gps, os, time

session = gps.gps()

while 1:
    os.system('clear')
    session.query('admosy') 
    # a = altitude, d = date/time, m=mode,  
    # o=postion/fix, s=status, y=satellites

    print
    print ' GPS reading'
    print '----------------------------------------'
    print 'latitude    ' , session.fix.latitude
    print 'longitude   ' , session.fix.longitude
    print 'time utc    ' , session.utc, session.fix.time
    print 'altitude    ' , session.fix.altitude
    print 'eph         ' , session.fix.eph
    print 'epv         ' , session.fix.epv
    print 'ept         ' , session.fix.ept
    print 'speed       ' , session.fix.speed
    print 'climb       ' , session.fix.climb
    
    print
    print ' Satellites (total of', len(session.satellites) , ' in view)'
    for i in session.satellites:
        print '\t', i

    time.sleep(3)

… which gives you a simple readout to the terminal every 3 seconds.

Obviously there are much more interesting applications for this ( logging data to postgis, displaying real-time tracking data in QGIS via a python plugin, etc). But this is a good start for any python based app.

Sparklines in python

2007-05-19T15:56:07-07:00

Edward Tufte, the outspoken guru of data visualization, has long been an advocate of clear and concise (almost minimalist) graphical representations of data. He’s got a lot of great ideas relevant to cartography (my cartography course at Humboldt State used his book “The Visual Display of Quantitative Information” as our text).

One of the coolest ideas are “sparklines” which he describes as “data-intense, design-simple, word-sized graphics”. Instead of standalone charts that are often placed on their own and separate from the text that discusses them, sparklines are meant to be placed in-line with the text and provide memorable, simple and contextually-relevant data to support the surrounding text. For example:

The US National Debt as a percentage of GDP increased during the Reagan and Bush presidencies but dropped off slightly during the Clinton administration .

Now of course I had to figure out how to produce these in python. Theres a great cgi application, written in python by Joe Gregorio, that does sparklines. I needed something that was abstracted away from the CGI framework, more of a proper python module. Replacing all the CGI-specific code was straightforward and I came up with a standalone sparkline python module (View / Download the Source Code. ) The only dependencies are python and the python imaging library.

In the minimalist spirt of sparklines, the interface was kept simple. First you create a list of data values then simply pass the list to one of the sparkline generators:

import spark

a = 32.5,35.2,39.9,40.8,43.9,48.2,50.5,51.9,53.1,55.9,60.7,64.4 spark.sparkline_smooth(a).show()

Or if you prefer a more discrete, bar-graph-style instead of a smooth line:

spark.sparkline_discrete(a).show()

There’s plenty of room for configuration. For example, in the national debt example above I wanted to keep the y axis at the same scale (instead of the default min-max scaling) and make each step 6 pixels wide:

spark.sparkline_smooth(a, dmin=30,dmax=70, step=6).show()

How does this relate to cartography? GIS typically takes a snapshot representation of earth, frozen in time. Since sparklines seem particularly good at representing change-over-time, it could be an interesting way to add a time dimension to a 2-D map. For example, instead of just displaying country polygons with labels, you could place a sparkline right under the label showing the population changes over the last century. It seems like it would be an ideal way to embed alot of useful information into a small map.

Anyone know of any good examples?

Blessed Unrest - Paul Hawken’s presentation

2007-05-14T21:29:28-07:00

I got the chance to see Paul Hawken speak tonight in Santa Barbara. I knew him best as the author of Natural Capitalism which provided a great roadmap for integrating ecologically sustainable practices with the business world. This talk was based on his recent book - Blessed Unrest - How the Largest Movement in the World Came into Being and Why No One Saw It Coming.

The basis of this book is simple: that organically-developed, bottom-up, non-hierarchical organizations (which number in the millions according to his research) are now leading the world in many diverse areas of service. He describes these environmental and social justice organizations as the “immune system” of our societies; our response to destructive and corrupt habits perpetrated by those in power who are willing to compromise our future for short-term gain.

One thing that struck me about the subject was the importance of sharing information and ideas (as opposed to spreading an ideology). I thought one of the most interesting stories of the night was his description of how the meme of non-violent civil disobedience evolved… from Emerson, to Thoreau, to Ghandi, to Rosa Parks to Martin Luther King, Jr. At each turn of the story, there was someone (often unnamed but vitally important) who turned on each of these people to the ideas of those who came before.

Paul was eager to point out the role of technology in this inter-connected mesh of grassroots community organizations. He mentioned open-source software a few times and even gave a shout out to Ruby on Rails (which I gather was the backbone for his WiserEarth.org site focussed on connecting these diverse organizations).

It was a careful mix of optimism and pessimism; Paul was careful in noting the many severe challenges we’ve been handed but was confident that this bottom-up mesh of interconnected citizens can form a community strong enough to withstand anything that comes it’s way. In the end, his message was about doing what you love, connecting with others and standing up for your values. Sounds like good advice to me.

Cleaning up CAD data with postgis

2007-05-14T11:58:28-07:00

Don’t you just love getting CAD data into GIS! I received a .dwg file with study areas delineated as polylines which we needed as polygons for analysis purposes. And it wasn’t just one polyline surrounding each study area … there were hundreds of little line segments which outlined a couple dozen areas (what was this CAD tech thinking?) . Luckily each segment had a name to associate it with the proper area.

I found that ArcMap’s tools for doing this are painfully inadequate so I turned to postgis. After converting the dataset to a shapefile, the solution was simple:

shp2pgsql “study_areas.shp” areas | psql -d gisdata

pgsql2shp -f “study_areas_poly.shp” gisdata \ “SELECT BuildArea(collect(the_geom)) AS the_geom, name FROM areas GROUP by name”

Viola… a new shapefile with my proper polygons instead of CAD chicken scratch.

Worldwind Java - Jython example

2007-05-13T14:32:53-07:00

The worldwind java sdk has finally been released. It’s a neat SDK, well organized, easy to bring into Eclipse with some good examples to start hacking away.

The only problem is the examples are written in Java ;-) . If braces make you cringe but you still want to work with all the excellent Java libraries out there, you’ll want to take a look at Jython. Taking the AWT1Up.java code and porting a subset of the functionality to Jython was surprisingly easy and yielded much more readable code in my opinion. And the ability to manipulate objects at the interactive prompt is just so sweet.

View the Source Code

Setup is not too terrible:

1. Get a Java JDK (I’m using sun java 6)

2. Download and install Jython 2.2b2

3. Download and unzip the worldwind java sdk (ex: /opt/wwj )

4. Set your LD_LIBRARY_PATH variable to /opt/wwj

5. Set your CLASSPATH variable to /opt/wwj/worldwind.jar

6. Run jython wwjdemo.py_

One thing that is a bit disappointing with the WorldWind SDK in general is the lack of support for rendering common formats. Maybe I missed something but I couldn’t get gpx or georss feeds working properly. It is version 0.2 so I expect support for GeoRSS and GPX to improve and for GML, KML, GeoJSON, Shapefiles, Rasters, WMS, etc to be included eventually.

Anyone else out there started playing with Jython / Worldwind yet?

Back on the train

2007-05-13T13:46:30-07:00

I’d like to have some interesting excuse as to why I haven’t posted since last July. But I don’t.

I’ve since left my postion at NCEAS, started a new job at Geosyntec and have been keeping busy with life, love and the pursuit of happiness. Oh and GIS of course.

Anyway, I expect to be posting on a much more regular basis from here on (unless I get distracted again ;-) ).

The reliability of web services

2006-07-24T21:24:52-07:00

A few months back I posted a link to my ten favorite Web Mapping Services. The post included live links directly to the WMS servers. At first I questioned this move as locally hosted images would be far more reliable. But I thought it would be a neat experiment to see the downtime of each site. So I checked it daily just out of curiosity…

Well with today’s apparent disappearance of the NASA JPL site, all but one of my WMS layers mentioned have been down for at least a significant portion of a day. (The only one that’s been consitently up has been http://mesonet.agron.iastate.edu) .

This echos back to what I was complaining about with the whole USGS National Map debacle. The bottom line is that whenever we rely heavily on a web service to deliver essential data, we are risking the integrity of the end product. The chain is only as strong as it’s weakest link and, unfortunately as the USGS and NASA have shown, those links can and will fail completely from time to time.

Converting Shapefiles (and more) to KML

2006-07-14T18:27:03-07:00

A while back I wrote about converting KML files into a shapefile for use with GIS apps other than GoogleEarth. I got a ton of emails and site traffic from people looking to go the opposite direction; getting their GIS data into KML.

There are, of course, a couple of utilities already implemented: ArcMap-based extensions including KML Home Companion and Arc2Earth, a nice MapWindow app called Shape2Earth, and the open source WMS Geoserver all support KML output.

Not to be left behind, GDAL/OGR now supports KML output. Oddly enough it does not yet read KML. But hand it any OGR-readable vector dataset and it can be converted into KML. It currently doesn’t offer as much control over the output as the above options but is quicker to implement, works with a wide variety of input formats and can be easily scripted.

This functionality is in CVS only at the moment but should be included in the next release. If you can’t wait and don’t feel like compiling from cvs source, try the 1.0.5 version of FWTools (for Windows and Linux).

The conversion process is pretty straightforward. For example, the following will convert a shapefile (sbpoints.shp) to KML (mypoints.kml).

ogr2ogr -f KML mypoints.kml sbpoints.shp sbpoints

The KML format flys in the face of the GIS mantra stating that content should be seperate from styling. Since styling information is purposefully absent from most standard vector formats, it makes for pretty bland KML output. The attributes just get dumped out into one big text block and there is no classification or styling control.

But in terms of getting your data into Google Earth quickly (esp. point data), the OGR method looks promising.

Wardriving with Ubuntu Linux and Google Earth

2006-07-03T22:24:56-07:00

Wardriving is fun. Going around the neighborhood and mapping all the wireless networks may be nothing more than a geeky hobby but it can sure teach you alot. And viewing the results in Google Earth is icing on the cake.

I’ve used NetStumbler on windows and this works great but since my computers at home are now nearly Microsoft-free, I had to relearn the process on Linux. It breaks down into a few easy steps:

1. Install the drivers for you wireless card. On my HP laptop with a Broadcom card, I followed the instructions on the ubuntu forums which worked great with one exception: the driver link on that page doesn’t have a valid md5 sum so you can download it from this url instead

2. Install gpsd. This is the software that talks to your gps unit and is available in the ubuntu packages through apt. The one hitch is that I had to set up my Magellan GPS unit up for the correct baud rate and NMEA output. Once installed, I connected the GPS unit via a serial port, turned it on and ran _ gpsd /dev/ttyS0 _ to start the gpsd server.

3. Install kismet, the wireless packet sniffer. The version in the ubuntu repository is not recent enough to support my Broadcom driver so I had to download the latest source and compile it with the standard _ configure, make, sudo make install . Then I had to edit the /usr/local/etc/kismet.conf to reflect my system configuration; I changed thesuiduser_, source and logtemplate variables. Once configured, you can start it with the command sudo kismet.

4. Now drive/bike/walk around for a bit with your laptop and gps unit. When you’re done, shutdown kismet and you’ll have a bunch of fresh logfiles to work with.

5. The main kismet log is an xml file containing all the info on the available wireless networks including their SSID, their encryption sheme, transfer rater and their geographic position via gpsd. I worked up a small python script, kismet2kml.py (based on a blog entry at jkx@Home), to parse the logfile into a KML file for use with Google Earth. It could certainly use some tweaking but it’s a start. To run it, give it the kismet logfile and pipe the output to a kml file:
_ kismet2kml.py kismet-log-Jul-03-2006-1.xml > wardrive.kml _

6. Now fire up Google Earth (Linux version now available!) and load your KML file.

Also, as James Fee points out, posting your data as KML files means that the data can be integrated into a growing number of kml-ready apps including google maps (just upload the kml and point your browser to http://maps.google.com/maps?q=http://your.server/wardrive.kml).

Another neat application I’ve found for dealing with kismet logs is the kismet2gpx script for converting the kismet gps tracklog into gpx. Since most gps units have pretty tight limitations on the length of stored tracks, logging them to your laptop with kismet could be an effective way of creating detailed tracks on very long trips.

Mapserver Include

2006-06-25T15:29:13-07:00

If you mange even a small number of Mapserver sites, eventually you notice that you use a number of identical layers in multiple mapfiles. The way this is typically done is to copy and paste the LAYER definition into each mapfile. But inevitably you’ll need to change the styling or the data source and you have to manually go through each mapfile to sync the changes. Wouldn’t it be nice to define the layer in a single file and use it in many mapfiles?

While Mapserver has no concept of an “include”, the C preprocessor (cpp) does. This is mentioned on the Mapserver list every time the subject of includes comes up. Still I have yet to find an actual example so I thought I’d share my notes on how I accomplish a mapserver include:

1. Create your mapfile as usual but leave out any LAYER definitions that you wish to share amongst mapfiles. Instead use something like :

include “landsat.layer”

2. The C Preprocessor doesn’t deal well with ”#” which is the mapfile’s chosen comment charachter. Instead replace with “##” to indicate a comment

3. Save this pseudo-mapfile as mymap.template

4. Create a file in the same directory called landsat.layer with the LAYER block.

5. Run the template through the preprocessor to generate the real mapfile :

cpp -P -C -o mymap.map mymap.template

The next step would be to script the preprocessing of all your mapfiles so that changing a layer definition in multiple mapfiles was as simple as changing the .layer file and running the script.

Some thoughts on Where 2.0

2006-06-15T00:00:26-07:00

Oh man, it’s a long drive from San Jose back to Santa Barbara! Anyways, just got back from where 2.0 and want to throw out my quick summary of the event.

* There was alot of talk about all things open; open data, open source and open standards. There was lots of buzz around the open street map project, osgeo applications like grass, ossim, gdal, mapbender, etc., and tons of discussion of WMS, WFS and other relevant standards. This is great as I think all three will be the cornerstone of the spatial industry in the near future.

But, as I’ve mentioned before, people throw the word “open” around so much that it begins to loose meaning. From alot of conversations I had, I found many people were confused about the differences. Some folks seemed to think that the osgeo foundation was a data repository for open data (it may soon be! .. but not quite yet) and also that osgeo was an open standards organization trying to “compete” with the OGC. But that is what an event like this is for; to reach out and communicate, clarify and bridge the gaps between communities.

Of course I had to laugh as I heard a couple dozen people refer to Google Maps as an “open source” application…. it’s proprietary source code using proprietary data through a proprietary data transfer mechanism. It may be “free” as in beer but that’s about the extent of it’s openness.

* Social Data: using location technology as the basis for sharing personal experiences and social networking was a powerful theme at Where 2.0. It ran the gammut from tagging locations to writing personal travelogs to mobile location-based games to virtual worlds to mobile apps that would could differentiate stangers vs aquantainces in range of your bluetooth device.

* Security and privacy: There are implications to the web/where2.0 mindframe. Publishing your location and personal information in real time through the web and mobile devices brings up some frightening security and privacy issues. Who owns the data? What licenses are your personal data distributed under? Do you need others permission to post their photos or locations? Who decides what is acceptable and what gets taken down? How is spam dealt with? Only two speakers were brave enough to fully address these issues head on and the panel had some good discussion on these topics. Kudos to them.

* Bringing location technology to the masses: This was repeated by a few speakers; that in order to be successful in spatial technologies you need to bring your service to the masses. Certainly if you’re trying to compete in the social networking space, this is true. But in general GIS and spatial tech has application that are far beyond the interests of the vast majority of people.. emergency management, infrastructure, environmental, real estate, etc.

The mantra that spatial data and services must appeal to a wide audience is analogous to saying that family cars are the only successful type of motorized vehicle. In terms of numbers, they may be a majority. But in terms of utility, there is a reason that construction companies pay hundreds of thousands of dollars for heavy industrial machinery.. because trying to haul tons of earth and debris with a Toyota Camry just doesn’t work. Likewise there is a similar reason most municipalities don’t use a Google Mashup to manage their parcel data.. it simply doesn’t work. So what is appropriate for mass consumption may have little applicabilty to business/government/industry/research. And vice versa.

* Mobile Applications: So much potential here and some really cool innovations in geotagging content. Really, for the first time, I got a sense that these personal devices could become a means for creating a vast database of socially relevant information. But the lack of security and privacy safegaurds along with the domination of the cellular networks and the heterogenous environment of mobile platforms, I still view most of this as pie-in-the-sky.

* Some new discoveries:

* metacarta: A text parsing engine with a public API to extract geo info from plain text! 

    
* gutenkarte: An application of the above to classic works of literature.

    
* open layers: A javascript application with a slick UI and simple API for displaying WMS and WFS

    
* open street map: A fantastic project focussing on collaborative development of a public street database 

    
* mapstraction: A javascript layer on top of the 'Big 3' Mapping APIs that allows yoiu to switch seamlessly between the service providers.

    
* Google Earth & Sketchup: GE for linux!!! Wooo-hooo!! There was also a sweet demo of creating 3D drawings in Sketchup and placing them in GE. Very slick.

    
* Google Maps: Now with kml support! Just try http://maps.google.com/?q=http://path.to.your.kml 

    
* Mapguide: I am embarrased to say I have never tried out Autodesk's open source offering but the demo was sweet.. a very high powered GIS for a web app. And the Autodesk folks were about the nicest group of guys you could meet.

    
* ArcGIS/Server 9.2: Author a map in ArcMap. Save as .mxd. Drop into web server. Instant kml and wms server! 

    
* And while not new to me, there were alot of good overviews of some of my favorite software packages like OSSIM, GRASS, GDAL, Geoserver and World Wind (Java version coming this fall!!).

* Finally, the prize for most interesting talk goes to Chris Spurgeon who spoke about the best geohacks of the last 3000 years. Long before computers, Chris showed how Eratosthenes measured the diameter of the earth, how the Polypenesian’s used the stars as an advanced navigation system, how the post-renaissance world rediscovered stars as a the key to navigation. And in more recent times he showed how Harry Beck reinvented the cartography of transportation with the London subway maps and how the VOR transmitters created highways in the featureless sky. This presentation really put current innovations in location technologies into perspective.

OK sorry about the lack of links but it’s too late in the evening for that. Hope you enjoyed my rundown and I’m sure I’ll have more to say after I get some sleep!

Animating the Blue Marble

2006-06-09T20:06:07-07:00

A while back I posted my technique for creating an animated gif out of a time series of maps. While this may have been the pinnacle of web animation circa 1997, the animated gif just didn’t quite seem hip enough for this day and age.

Today I found a more modern example. This WorldKit interface, built with Flash, shows the seasonal progression of snow and land cover changes courtesy of the next generation Blue marble images. Complete with time slider, image fading and full animation controls, this interface really shines at providing an interactive experience rather than a passive visual display.

HostGIS Linux 3.6 Released

2006-06-03T15:15:31-07:00

Though probably not as big of a news item as this week’s release of Ubuntu Dapper, there’s another Linux release that might be of interest to us GIS folk:

Built off of a Slackware base (one of the oldest, most stable linux distros), HostGIS Linux aims to be a “minimal yet complete” distribution specifically built with GIS in mind. It is first and foremost a server platform; it does not include any window system at all. If you’re looking for desktop GIS applications out-of-box, it might not be the best for you.

But for a GIS server, it comes with most of the open source stack preinstalled and configured. This latest release has a few changes and version upgrades for most of the components.

* PHP, Python and Perl Mapscript

* GDAL/OGR with PHP, Python and perl bindings

* Postgresql 8.1 with PostGIS 1.1

* drivers for many extra formats including jpeg2000 and ecw

* Apache web server with Mapserver CGI

The primary motivation for creating HGL was to speed up the installation of new gis-enabled servers. Gregor Mosheh, the head programmer for HostGIS, has done an excellent job pretty much single-handedly putting this together. ( In full disclosure, I do consulting work for HostGIS, though I wasn’t really involved in the creation of HostGIS Linux. )

The setup is your standard text-based install and is a piece of cake if you’ve ever installed Linux before. When you’re through, you have the good ole’ black and white text console staring at you. Not very interesting… But the really satisfying part is to fire up a web browser after the install and be able to point it to a working webGIS application. Anyone who has spent the time to set up the mapserver stack and its seemingly infinite dependencies can appreciate the amount of work this saves!

If you’re not into learning a new distro, there is always the FGS linux installer which will set up a similar software stack on pretty much any linux.

And for Desktop GIS, many linux distros have a selection of GIS apps in their package repositories (You’ll want to certainly grab GRASS, GDAL and QGIS) . FWTools can be a good option on both Linux and Windows to get you up and running quickly. Finally there are a number of other more desktop-oriented distros for GIS including Knoppix GIS and GeoLivre, both of which run as a live-cd so you can check it out before you install.

Anyways, back to sum up HostGIS Linux:

If you need to set up a GIS server with minimal fuss and you have some experience with Linux, you might like to try it out. It will save lots of time.

If you’re a GIS user who needs a graphic windows environment to do GIS work on the Desktop, HostGIS Linux will not really make you happy out-of-the-box. Of course, since HGL is slackware based, you can use the slackware package management system to build an impressive Desktop system. But if you don’t need to run a server or really care about having the latest versions, Ubuntu comes with a solid desktop environment and packages for alot of good GIS apps.

More on Mapnik WMS

2006-05-18T15:38:49-07:00

One of my initial complaints about the Mapnik WMS server was that it would not accept any parameters that were not in the OGC WMS spec. Some WMS clients will tag on extra parameters for various reasons and the OGC supports this in relation to vendor-specific parameters. The fix was pretty simple;in mapnik/ogcserver/common.py you can simply comment out

    #for paramname in params.keys():
    #    if paramname not in self.SERVICE_PARAMS[requestname].keys():
    #        raise OGCException('Unknown request parameter "%s".' % paramname)
to get the desired effect.

There was also the question of speed and how it compared to other WMS servers such as Mapserver. Since I already had both a Mapnik and Mapserver WMS set up using the exact same data source, styled in the same fashion, it was pretty simple to write a quick python script that would smack each WMS server with a given number of back-to-back WMS GetMap requests:
#!/usr/bin/env python
import urllib

server = sys.argv[1]
hits = int(sys.argv[2])

if server == 'mapnik':
    url = "http://localhost/fcgi-bin/wms?VERSION=1.1.1&REQUEST;=GetMap&SERVICE;=WMS&LAYERS;=world_borders&SRS;=EPSG:4326&BBOX;=-4.313249999999993,20.803500000000003,59.58675000000002,52.75350000000002&WIDTH;=800&HEIGHT;=400&FORMAT;=image/png&STYLES;=&TRANSPARENT;=TRUE&UNIQUEID;="
elif server == 'mapserver':
    url = "http://localhost/cgi-bin/mapserv?map=/home/perrygeo/mapfiles/world.map&VERSION;=1.1.1&REQUEST;=GetMap&SERVICE;=WMS&LAYERS;=worldborders&SRS;=EPSG:4326&BBOX;=-4.313249999999993,20.803500000000003,59.58675000000002,52.75350000000002&WIDTH;=800&HEIGHT;=400&FORMAT;=image/png&STYLES;=&TRANSPARENT;=TRUE&UNIQUEID;="
    
for i in range(0,hits):
    urllib.urlretrieve(url)
Then just run the script from the command line, specifying the server and number of hits, and wrap it in the time command. Here are the results:

Pretty close. Mapserver was just slightly faster in every case. Now this is just a preliminary test and it would be interested to see a comparison:

* With larger datasets and more complex styling including classification and text labelling

* With data from other sources such as postgis where the connection overhead might be significant

* With Mapserver running as a fastcgi

* With concurrent requests as opposed to back-to-back requests

Overall though, my opinion of Mapnik WMS remains high and I’d love to put it in production use in the near future. Stay tuned…

Mapnik WMS Server

2006-05-17T23:37:30-07:00

A few months ago, Mapnik came onto my radar and I was immediately impressed with the beautiful cartography. But, until recently, it was just a C++ libary with some python bindings that could be used to programmatically build nice map images from shapfiles, geotiffs or postgis layers. There were no common interfaces such as WMS to access mapnik… until last month. Jean Francois Doyon recently added a prototype WMS interface to Mapnik. It runs as a fastcgi script under apache. It is still a bit rough around the edges but the result is well worth a little extra setup effort.

I set up Mapnik as a WMS server recently and would like to share my process and results. This tutorial assumes you already have python, postgresql/postgis, proj4, python imaging library and apache2 already running. The examples are for Ubuntu Dapper Drake.. they may work well on other versions of Ubuntu and Debian but for other nixes (and certainly windows) many things may need to be tweaked.

First off, we have to install the base mapnik libs. These depend on the boost python bindings and the whole compile process is very simple (if a bit slow) in Ubuntu:

sudo apt-get install \
 libboost-python1.33.1 libboost-python-dev \
 libboost-regex1.33.1 libboost-regex-dev \
 libboost-serialization-dev \
 libboost-signals1.33.1 libboost-signals-dev \
 libboost-thread1.33.1 libboost-thread-dev \
 libboost-program-options1.33.1 libboost-program-options-dev \
 libboost-filesystem1.33.1 libboost-filesystem-dev \
 libboost-iostreams1.33.1 libboost-iostreams-dev
cd ~/src
svn checkout svn://svn.berlios.de/mapnik/trunk mapnik
cd mapnik
python scons/scons.py PYTHON=/usr/bin/python PGSQL_INCLUDES=/usr/local/include/postgresql \
  PGSQL_LIBS=/usr/local/lib/postgresql BOOST_INCLUDES=/usr/include/boost BOOST_LIBS=/usr/lib
sudo python scons/scons.py install PYTHON=/usr/bin/python PGSQL_INCLUDES=/usr/local/include/postgresql \
  PGSQL_LIBS=/usr/local/lib/postgresql BOOST_INCLUDES=/usr/include/boost BOOST_LIBS=/usr/lib
sudo ldconfig

Now we have to set up some additional libs in order to run the WMS:

cd ~/src
wget http://easynews.dl.sourceforge.net/sourceforge/jonpy/jonpy-0.06.tar.gz
tar -xzvf jonpy-0.06.tar.gz
cd jonpy-0.06/
sudo python setup.py install
# copy the ogcserver stuff into its own dir
mkdir /opt/mapnik; cd /opt/mapnik
cp ~/src/mapnik/utils/ogcserver/* .
Now you’ll want to edit the ogcserver.conf file and change the following lines. The module is essentially the name of a python file (minus the .py extension) that we’ll create later. The height and width just cutoff the maximum possible image size that can be requested.
	module=worldMapFactory
	maxheight=2048
	maxwidth=2048
Create our “map factory” module defining data sources, styles, etc.( worldMapFactory.py ). Most of this configuration is explained in the mapnik docs and well-commented examples. One thing to note is that the shapefile must be specified without the .shp extension :
from mapnik.ogcserver.WMS import BaseWMSFactory
from mapnik import *

class WMSFactory(BaseWMSFactory):

        def __init__(self):
                BaseWMSFactory.__init__(self)
                sty = Style()

		rl = Rule()
		rl.symbols.append(PolygonSymbolizer(Color(248,216,136)))
		rl.symbols.append(LineSymbolizer(Color(0,0,0),1))
                sty.rules.append( rl )

		self.register_style('style1', sty)

                lyr = Layer(name='world_borders', type='shape', \
                            file='/opt/data/world_borders/world_borders')
			
                lyr.styles.append('style1')
                self.register_layer(lyr)
                self.finalize() 
Now we need to set up apache2 to handle fastcgi:
sudo apt-get install libapache2-mod-fcgid
sudo a2enmod fcgid
… and add some config lines to the apache config files, usually /etc/apache/httpd.conf but, in the case of this Ubuntu install, /etc/apache2/sites-enabled/default :
        ScriptAlias /fcgi-bin/ /usr/lib/fcgi-bin/
        < Directory "/usr/lib/fcgi-bin" >
                AllowOverride All
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
                SetHandler fastcgi-script
        < Directory>
Create the fast-cgi directory refered to by apache
sudo mkdir /usr/lib/fcgi-bin
Now create the actual server script as /usr/lib/fcgi-bin/wms
#!/usr/bin/env python

# Your mapnik dir containing the map factory 
# must be in the python path!

import sys
sys.path.append('/opt/mapnik')

from mapnik.ogcserver.cgiserver import Handler
import jon.fcgi as fcgi

class WMSHandler(Handler):
    configpath = '/opt/mapnik/ogcserver.conf'

fcgi.Server({fcgi.FCGI_RESPONDER: WMSHandler}).run()
Finally restart the apache server
sudo /etc/init.d/apache2 force-reload
Now you can access it with a WMS request like so:
http://localhost/fcgi-bin/wms?VERSION=1.1.1&REQUEST;=GetMap&LAYERS;=world_borders&
FORMAT=image/png&SRS;=EPSG:4326&STYLES;=&BBOX;=-81.54375,-58.3125,-59.04375,-47.0625&
EXCEPTIONS=application/vnd.ogc.se_inimage&width;=600&height;=300
Compare the linework with a comparable WMS service with UMN Mapserver on the backend. I’ll let the results speak for themselves…

Even if it’s map rendering is smooth, Mapnik’s WMS server is still a bit rough around the edges:

* It does not support GetFeatureInfo requests

* The server has trouble with extra parameters. For instance some WMS clients like mapbuilder like to tag on an extra ‘UNIQUEID’ parameter to the URL and this causes an unnecessary error with mapnik’s WMS server.

* Mapnik intself does not support reprojection

* It only supports shapefiles, geotiffs and postgis layers.

The readme.txt file in docs/ogcserver/ directory of the recent mapnik SVN checkout has a full list of known features and caveats so refer to them for the complete story.

But, all in all, I am very impressed with the quality of the Mapnik WMS server. I figured that, since Mapnik’s goal has been high-quality cartographic output, speed would be sacrificed but I didn’t notice any significant lag; on the contrary I think it was actually about on-par with Mapserver running as a CGI. If it was any slower, I didn’ t notice it immediately. But then again it was only working with a relatively small shapefile and I was the only user. I’d like to do more rigourous stress tests on the Mapnik WMS to see how it compares to Mapserver and Geoserver under varying loads with greater volumes of data.

Educational ways to waste some time

2006-05-12T20:09:46-07:00

It’s always great to find fun internet-based games that actually challenge you in “real world” skills. (And no, working on your wizard’s Ether Flame spell in EverQuest is NOT a real world skill). After all, if you going to waste some time, it might as well be educational, right. Can you tell that my mother is a school teacher? Happy Mothers day!

Anyways, these might be old news to some folks but I’ve found two fun games that will keep your brain fresh.

First, there is GeoSense. This is a fanstastic interactive game that pits users one-on-one in a timed geography quiz. You’re given a city and country and you have 10 seconds to click the map. The player with the best combination of speed and accuracy wins. Given American youth’s horrible knowledge of geography, this site could be really helpful. I would recommend it to children of all ages if it weren’t for the chatroom being infested with pubescent teen sex fiends. Just go use use myspace or something…

Secondly, for you Python programmers out there, there is the Python Challenge, a surprisingly challenging and mind-boggling course of puzzles that can be solved with Python. Actually some people have solved them with UNIX shell commands, perl or ruby, but many of the hints are python specific. They require a good dose of logic, persistence, knowledge of python libraries and a knack for finding patterns. Basically your goal is, given a minimal set of hints to find and process the data that will lead you to the next URL. I’m on level 9 right now and, well, I’m not going to admit to anyone how long it took to get there. Addictively challenging…

Thats it for now. Have fun.

The impact of urban areas on CO2 emmissions

2006-05-06T19:51:49-07:00

Increases in atmoshperic carbon dioxide (CO2) due to vehicle emmisions are considered one of the most important human-induced factors of climate change. Conventional wisdom would say that urban areas, with their huge populations, dense road networks and congested freeways, are the biggest offenders. This is true to some extent. But, viewed from a different perspective, the per-capita CO2 emmissions for these urban areas can be considerably less than surrounding rural and suburban areas.

Travelmatter.org has posted a series of maps comparing these two conflicting views. Here’s a sample from Chicago that demonstrates the sharp dichotomy; both entirely accurate but different ways of analyzing the same data:

In every case, the total CO2 emmissions are much greater in dense urban areas. But, per-capita, the urban areas have much lower emmissions, sometimes dramatically lower. This second view indicates, as WorldChanging points out, that living in denser neighborhoods can reduce your climate impact. It makes sense that living closer to the places you need to go on a daily basis and having more access to public transportation would reduce the emmissions impact. Maybe cities are “greener” than most of us percieve them to be?

USGS Seamless is back

2006-05-05T09:17:01-07:00

Two weeks after I first noticed something had gone awry with the USGS Seamless site, they appear to have fixed their server issues. As of this morning, the interactive data viewer and download interface is fully functionaly as far as I can tell.

Now be gentle on their server. Rumour has it, if you download more than 3 DEMs at a time, the server might go down for another 2 weeks! Just kidding… everything seems to be working fine. Download away….

What’s going on with seamless.usgs.gov ?

2006-04-25T09:10:59-07:00

Since April 21, I have not been able to view or extract any data from the USGS Seamless site, ostensibly the central distribution center for the US National Map. The site has been changing rapidly from day to day ever since and it seems that changes are underway so at least we know someone is working on it.. or hacking it to pieces. The last day or two they appear to have given up and are just redirecting people to gisdata.usgs.gov which, of course, has no mention of the outage on the home page.

When I develop an internet application, even if it’s only used by a few people, I usually seperate the development version from the stable, live version to minimize any downtime. And if you absolutely can’t keep the app running, at least put a big banner on the page indicating that the system is down so people (like me) don’t waste half an hour trying to figure out what they’re doing wrong. Is this too much to ask of the USGS? They are supposed to be the official portal for accessing our nation’s spatial data, right? And we’re not talking about a small server hiccup here, it has been down since at least April 21st with no public indication that problems are occuring on the site.

I just recieved an email this morning from the USGS web mapping admin. The emphasis is mine:

We apologize for any issues you may have experienced lately. The Seamless server, and all related map services will be unavailable for at least the next few days. During this time, the sites may still appear to be functioning. Some may ask for a password, and others may not show up at all. Normally our status messages are posted at http://seamless.usgs.gov. However, since this server has been affected by this outage, users are being re-directed to http://gisdata.usgs.net. We are in the process of posting a message here as well, which you will be able to monitor for any updates. We are estimating that the site will be available again by Monday May 1st 2006. Our team is working diligently to have this service available as soon as possible. We appreciate you patience during this time.

I really shouldn’t be surprised that a government agency botched it so badly; that seems to be the norm here in the US. But I’ve really come to rely on the seamless site for alot of data and it seems that 10 days of downtime for the sole distributor of our seamless national spatial data archive is a bit… amateur.

The distinction between open source and open standards

2006-04-23T12:23:21-07:00

Time and time again I see open source and open standards mentioned in the same sentence. While I’m a strong proponet of both, it is a bit disheartening to see how closely intertwined the two concepts are in the eyes of many GIS folks.

Open source refers to software distributed with a license that allows access to view and modify the source code. There are also some other criteria but unrestricted access to the source code is the key component.

Open standards refers to software-neutral specifications, usually developed collaboratively, to accomplish a technical goal. In the GIS world, this typically means OpenGIS specifications for sharing data across a network (WMS/ WFS/ WCS), data formats (GML), or for working with spatial data in a relational database (Simple Features Spec for SQL). We could arguably include pseudo-open specifications for data such as shapfiles and KML.

Open source applications do not always conform to open standards. Standards-compliant software does not necessarily have to be open source. So why are the two often mentioned in the same breath as though they were synonymous? Perhaps open source software is perceived as being “ahead” of other types of software in terms of adoption of standards; and maybe that’s true. But there are many proprietary software companies that have devoted alot of effort towards making their software communicate via open standards and their efforts should not go unnoticed (ESRI and Cadcorp just to name the two I’m familiar with).

The promise of open standards is that anyone can develop and use compliant applications that can easily interoperate regardless of the chosen software package. While that promise is far from being fully realized, associating open standards with a particular type of software will not get us any closer.

Update: Or maybe we are getting close… check out Geoff Ziess’ post on the OGC interoperability demonstration in Tampa. Ten vendors interoperating and sharing data in real time… this is what it’s all about.

Animating Static Maps - The Geologic Evolution of North America

2006-04-11T18:24:53-07:00

The Cartography blog recently talked about a series of excellent Paleogeographic maps developed by Dr. Ron Blakey at Northern Arizona University. Ever since I first studied geology, I had dreamed of an atlas that would clearly and visually demostrate how our current land masses came to be. This time series of maps focuses on North America and the geologic events that shaped have shaped it for the last 500 million years. Truly fascinating and excellent work. I encourage everyone to check out the site and read a little about it as well as the narrative by Geoff Manaugh .

Now it occured to me that a time series of maps lends itself very well to an animated sequence. While I am no graphic artist, I have done a few projects in the past that required stiching together a time-series of maps into an animated gif. The process is fairly simple:

1. Download or create each map you want to include in the series. For best results, all maps should have the same size and extents.

2. Rename the images in alpha-numeric order (001.jpg, 002.jpg…. 045.jpg)

3. Install ImageMagick - a collection of efficient command line tools for image processing. It supports almost every common image format available these days.

4. run the convert command to create the animated gif:

convert -geometry 500x483 -delay 200 -loop 0 *.jpg mymovie.gif
The geometry is simply the WIDTHxHEIGHT dimensions of the output image (it helps if this is proportional to the original image dimensions).

The delay parameter specifies how many hundreths of a second delay occurs between each frame.

The loop parameter, when set to zero, indicates the gif will loop infinitely.

The .jpg, if your operating environment supports wildcards, will take each of the jpg images in the current directory and stich them into an animated gif named mymovie.gif

Viola! An animated movie from a series of static maps. In the case of the Paleogeologic maps, there were 41 maps which produced a sizable animated gif (about 7.5 MB). You can check out the results here. I could watch this play for hours!! Really fascinating stuff.. many thanks to Dr. Ron Blakey for putting this project together.

LIDAR data processing with open source tools

2006-04-01T13:41:59-08:00

LIDAR data is certainly a hot technology these days. LIght Detection And Ranging data can be used to create extremely detailed terrain models but there are lots of barriers to using LIDAR data effectively. USGS Center for LIDAR Information Coordination and Knowledge was put in place to “_facilitate data access, user coordination and education of lidar remote sensing for scientific needs_”.

Beyond the sheer size of the datasets and the knowledge and hardware required to process them, software is a big issue. In the realm of open-source GIS tools, there are many applications (GRASS being the most prominent) for dealing with elevation point data and processing it into more meaningful products such as elevation DEMs and contours.

Usually the data comes as simple ASCII text files and the x,y and z values are easily extracted from such a file. But take a look at the USGS data distribution site and you’ll notice some of the datasets are distributed as LAS binary files. It makes sense to store such massive datasets in binary so I started looking for some LAS conversion tools.So after some searching, I found a bunch of proprietary products for working with LAS but no open source tools. Luckily, the format is well documented thanks to the efforts by the ASPRS to make it an open specification.

So dusting off my notes about parsing binary files in python, I set out to create a python module for extracting LIDAR data from LAS files. The LAS format contains a header which needs to be parsed first in order to read the point cloud. Once you have the header info, you can scan your way through the dataset to pick out the x,y,z values.

Here’s an example of the python interface that will read the first 10,000 points into a 2D shapefile with the elevation as a attribute in the dbf:

import pylas
infile = 'sanand000001.las'
outfile = 'lidar.shp'
header = pylas.parseHeader(infile)
pylas.createShp(outfile, header, numpts=10000, rand=False)

The issue I struggled with is the sheer size of these datasets. A USGS quarter quad can contain 10 million points which is an excessive number of points to create, say, a 10 meter DEM over such a small area. Clearly there was a need to extract a subset of this dataset but just taking the points sequentially gives you a subset of the total area. So, by default, pylas randomly scans the data to pull the number of specified points so that the point cloud could cover the entire area (at a much lower point density). Without numpts specified, it will randomly select 1/2000th of the total number.

So the simplified interface to make a more manageable lidar shapefile would be:

header = pylas.parseHeader(infile)
pylas.createShp(outfile, header)

Once the shapefile is created, you can bring it into GRASS to do the processing to generate DEMs, contours and other derived elevation products:

v.in.ogr dsn=lidar.shp layer=lidar output=lidar
g.region vect=lidar
g.region res=10
v.surf.rst input=lidar elev=lidar_dem zcolumn=elev

# Launch the interactive 3D viewer
nviz lidar_dem

Of course the method I just described is very simplistic and does not even come close to utilizing the full potential of the LIDAR point cloud, but it’s a start.

The pylas.py module can be downloaded here. The code has worked for me on the few datasets I’ve tested it with but it should certainly be considered a rough-cut, alpha product. There is much room for improvement and, of course, if you have any suggestions or contributions, please get in touch.

WMS layers: My Top Ten

2006-03-26T20:25:26-08:00

Web Mapping Services (WMS) are not always my prefered option for accessing data; relying on a remote server to generate a pretty picture of the data is hardly a substitute for having the raw data in hand. But for many cases, I just need a decent looking basemap image and don’t want to download gigabytes of data, especially if that data is updated frequently.

Software like GeoServer and Mapserver are making it easier to publish data via WMS and the number of WMS servers is surely growing… but how do you find them? There is no central registry for WMS servers but efforts like the refractions research ogc survey, mapdex and a few google tricks are making it easier to find data distributed via WMS. After many hours digging through WMS services to find the ones that suite my mapping needs, I’ve come across a number of gems that I use time and time again. Hopefully this will inspire some others to share their secret stash of WMS servers!

(Update: Anything Geospatial has a great link to a well-organized WMS server list for public use. Nice. )

You should be able to provide the online resource URL to your favorite WMS client software (my personal choice is openjump) and the client should display the list of layers available from that service.

If you’re contructing WMS URLs “by hand” or in a browser, you can do a capabilities request (the online resource URL with service=WMS?request=GetCapabilities appended to it) which will return an XML document describing the available layers, image formats, projections,etc. Take a look at the image src for any of the thumbnails below to see how the map request is constructed.

1. TerraServer Digital Raster Graphic (DRG): USGS Topo Quads ** Online Resource URL ** : _ http://terraservice.net/ogcmap.ashx? _

** Layer Name ** : DRG

2. TerraServer Digital Ortho Photo Quads (DRG): Black and white aerial photos for the US ** Online Resource URL ** : _ http://terraservice.net/ogcmap.ashx? _

** Layer Name ** : DOQ

3. NASA Landsat Imagery The Landsat mosaic is available in fase color (default) or in natural color (style=visual) as shown below.

** Online Resource URL ** : http://onearth.jpl.nasa.gov/wms.cgi?

** Layer Name ** : globalmosaic_

4. 45-minute Weather Radar Images (NEXRAD Base Reflectivity). Since this is a dynamic data source, the image below may look really boring (ie blank) if there’s no storms over the Continental US.

** Online Resource URL ** : http://mesonet.agron.iastate.edu/cgi-bin/wms/nexrad/n0r.cgi?

** Layer Name ** : nexrad-n0r-m45m

5. USGS National Landcover The 30-meter natial landcover dataset. USGS is nice enough to provide a legend, of course.

** Online Resource URL ** : http://gisdata.usgs.net/servlet/com.esri.wms.Esrimap?ServiceName=USGSWMS_NLCD&_

** Layer Name ** : USNLCD_

6. USGS National Elevation - Shaded Relief ** Online Resource URL ** : http://gisdata.usgs.net:80/servlet/com.esri.wms.Esrimap?servicename=USGSWMS_NED&_

** Layer Name ** : USNED_Shaded_Relief_

7. USGS Reference Maps ** Online Resource URL ** : http://gisdata.usgs.net:80/servlet/com.esri.wms.Esrimap?servicename=USGSWMS_REF&_

** Layer Names ** : States,County,Roads,RouteNumbers,Streams,Federal_Lands_

8. Life Mapper Besides the standard WMS paramters, some services can take extra parameters in order to render a map. In this excellent service, LifeMapper requires that you provide the species name and it will render maps of known species locations and modelled distributions. Here’s an example of the distribution of Black Bear (ie. Ursus americanus) over central california

** Online Resource URL ** : http://www.lifemapper.org/Services/WMS/?ScientificName=Ursus%20americanus&

** Layer Names ** : Species Distribution Models,Political Boundaries,Species Data Points

9. MODIS Daily Satellite Imagery ** Online Resource URL ** : http://wms.jpl.nasa.gov/wms.cgi?

** Layer Names ** : dailyterra, daily_aqua_

Terra Aqua

10. SRTMPlus 90 Meter DEM The image below doesn’t make for a very good basemap OR a very good DEM for analytical purposes since all the values are scaled to an 8-bit color depth. However, JPL also offers this layer as an integer (16bit) GeoTIFF (Use format=image/geotiff and styles=shortint_), so this can be a valuable way to quickly grab a DEM for a given region. ** Online Resource URL ** : http://wms.jpl.nasa.gov/wms.cgi?

** Layer Names ** : srtmplus

If you’d like to view these layers interactively, here’s a mapserver application which “cascades” the above WMS layers through a single interface. If you’re interested in setting up these layers in a mapserver application, check out the WMS Mapfile for some examples.

StarSpan for vector-on-raster analysis

2006-02-17T21:07:44-08:00

It’s amazing how many excellent open source GIS applications are out there just waiting to be discovered. I’ve been working with open source GIS for over 3 years now and I still find new and interesting software on a regular basis. The latest “Why haven’t I heard of this before?” discovery came from the GRASS mailing list discussion on StarSpan, a tool developed at University of California at Davis “_designed to bridge the raster and vector worlds of spatial analysis using fast algorithms for pixel level extraction from geometry features_”.

Our research project for the Ecosystem Based Management group at UCSB is in need of this exact tool in order to extract raster statistics based on a vector watersheds layer. ArcGIS and GRASS both have some of the capabilities we need through the Zonal_Statistics and v.rast.stats functions respectively. However they have their limitations and neither really handles categorical raster summaries by polygon. StarSpan looks like a more efficient option in terms of speed, scriptability and capabilities.

Installation is very smooth. It requires a recent version of GDAL (>= 1.2.6) and GEOS (>= 2.1.2). Once the dependencies are met, compilation on a nix system is as easy as configure, make, make install (There are also Windows binaries available). There is a single command line interface for all the functionality and StarSpan is able to handle all GDAL rasters and OGR vectors.

For classified rasters such as a land cover raster, we’d like to get the number of pixels for each landcover class by watershed. StarSpan creates a nice, normalized csv with three columns; The vector feature id, the raster value, and the number of pixels. There will be up to (number of features X number of classes) rows.

starspan –vector watershed.shp –raster landcover.tif –count-by-class landcover_by_watershed.csv

In order to find the percentage of a given raster class for each watershed, you can bring the csv into a relational database and do a quick SQL query. Here’s an example of finding the percentage of cropland (class value is 12) for each watershed:

SELECT t.fid AS fid, (t.count::numeric / s.total::numeric) * 100 AS percentage_cropland
FROM landcover_by_watershed t,
               (SELECT fid, sum(count) AS total 
                FROM lancover_by_watershed 
                GROUP BY fid) as s 
WHERE t.fid = s.fid
AND t.class = 12;

Which gives us…

 fid |      percentage_cropland
-----+------------------------------------------------
   1 | 28.571428571428571429
   2 | 71.428571428571428571
   3 | 36.363636363636363636
   4 | 63.636363636363636364

For continuous surfaces such as elevations and slopes, we’ll need to get quantitative statistics of those rasters by watershed. StarSpan can easily generate averages, mode, standard deviation, min and max:

starspan –vector watershed.shp –raster slope.tif –stats slope_stats.csv avg mode stdev min max

Which outputs a csv with one row per feature identified by feature id and each stat as a column:

FID,numPixels,avg_Band1,mode_Band1,stdev_Band1,min_Band1,max_Band1
1,25921,34.694822,38.917000,14.491952,0.347465,66.241035
2,21755,7.965552,0.000000,5.484245,0.000000,42.017155
...

While I can confirm that these small test cases work very quickly and give us pretty much the exact outputs we need, it will be interesting to see how well it stacks up to ArcGIS and GRASS when it comes to cranking out the big datasets. We’ll likely try all three methods and I’ll make sure to post the results.

Oh and the comparison between StarSpan and GRASS may become at moot point in the future since there is talk about integrating it with the GRASS project. While a GRASS module would be nice, not everyone has GRASS installed so I would hope the stand-alone version is still maintained since it can deal with pretty much any vector or raster data source.

Forest Service plans largest land sale in decades

2006-02-13T11:14:27-08:00

The Seattle Times is reporting some details on President Bush’s “Secure Rural Schools Initiative” which involves the largest US Forest Service land sales in decades in order to pay for rural school and roads. Some 309,421 acres will be up for sale which amounts to only 0.16 % of the 190 million acres managed by the Forest Service. Most of the parcels are isolated areas bordering private land.

Details and some limited maps of the initiative can be found here as well as a listing of forest service land that are potentially eligible for sale.

No doubt environmentalists, developers and timber companies will be scrutinizing these pieces of land in the coming months. More details and maps should be available around Feb 28th. Since the Forest Service is required to request public input on the sales, it would be nice if they could provide a GIS version of the maps for download… A public web GIS would inform the process immensely. I’ll be keeping an eye out for some detailed GIS data. Let me know if you know of a good source.

GDAL-based DEM utilities

2006-02-08T00:53:53-08:00

These DEM tools have been incorporated into GDAL. The code referenced on this page is no longer maintained and I'd highly recommend using gdaldem instead.

A few months ago, I began looking for some efficient command-line tools to analyze and visualize DEMs. I typically use GRASS for such tasks but GRASS only works with it’s native raster format. Sure you can import/export to common formats but that’s not as efficient as a single command line tool that could work with the native DEM format, run on systems without GRASS installed and provide easy scriptablity.

Not having found anything that fit the bill, I decided to port some of the common GRASS DEM modules to C++ using the GDAL libraries. For someone with very little experience with C++, this was surprisingly not that difficult though I learned quite alot along the way. The result: 3 command line utilities to generate hillshades, slope and aspect maps and 1 excellent utility contributed by Paul Surgeon to apply color ramping to a DEM.

Installation

Requirements

I built these utilities on Ubuntu Linux. I admittedly have no idea how to compile them on Windows but some folks have confirmed that the hillshade code compiles under VC++. So to get these running under Linux (and presumably other nixes), there are very minimal requirements:

1. GDAL shared libraries

2. GNU C++ Compiler

Download

Get the current source and unzip it. EDIT: This code is now avaible through my SVN repository : http://perrygeo.googlecode.com/svn/trunk/demtools/,

Compiling

Alas there is no makefile but installation should be fairly painless. To compile the source code under linux, the following commands should take care of it:

g++ hillshade.cpp -lgdal -o hillshade
g++ color-relief.cxx -lgdal -o color-relief
g++ aspect.cpp -lgdal -o aspect
g++ slope.cpp -lgdal -o slope

The four binaries can then be placed wherever your local binaries reside (typically /usr/local/bin)

Examples

The original DEM

In this particular example the input DEM is a GeoTIFF but these utilities can use any GDAL-supported raster source.

Slope

This command will take a DEM raster and output a 32-bit GeoTiff with slope values. You have the option of specifying the type of slope value you want: degrees or percent slope. In cases where the horizontal units differ from the vertical units, you can also supply a scaling factor.

slope dem.tif slope.tif

Aspect

This command outputs a 32-bit GeoTiff with values between 0 and 360 representing the azimuth of the terrain.

aspect dem.tif aspect.tif

Hillshade

This command outputs an 8-bit GeoTiff with a nice shaded relief effect. It’s very useful for visualizing the terrain. You can optionally specify the azimuth and altitude of the light source, a vertical exaggeration factor and a scaling factor to account for differences between vertical and horizontal units.

hillshade dem.tif shade.tif

Color ramps

After I posted the hillshade utility to the gdal-dev mailing list, there was some discussion about creating color relief maps to supplement the hillshades. Paul Surgeon took up the challenge and created a gdal-based C++ utility to colorize DEMs (or any other single band raster data sources for that matter). The technique is simple and powerful; by using a text-based color configuration file, you can create any range of color ramps for your data.

color-relief dem.tif scale.txt colordem.tif

Where scale.txt is a text file containting 4 columns per line, the elevation value and the corresponding RGB values:

3500   255 255 255
2500   235 220 175
1500   190 185 135
700    240 250 150
0      50  180  50
-32768 200 230 255

The colors between the given elevation values are blended smoothly and the result is a nice colorized DEM:

Color Shaded Relief (blending hillshade and colorized DEM)

There are two ways I’ve come up with to blend the hillshade and the colorized DEM:

1. Using GIMP or Photoshop, open both images, copy the shaded relief, paste on top of the color DEM and adjust the opacity in the layers dialog.

2. If you’re publishing to the web with Mapserver, just stack the two images in your mapfile and set the TRANSPARENCY for the hillshade to a value between 30 and 70 depending on your preference

Though both methods work nicely, neither is really ideal since they don’t generate a georeferenced tiff. You can get around this in the GIMP method by creating a world file (.tfw) for the output tiff. It might be nice, in the future, to do this step programatically but for now…

Let me know if you’ve got any suggestions or comments. The technique for all of these utilities is a simple 3x3 moving window so this code might serve as a good template to develop other raster processing utilities… let me know what you come up with!

First thoughts on the Open Source Geospatial Foundation

2006-02-04T18:56:53-08:00

Well after a long and productive day in Chicago, the 25 attendees (and a few dozen more from IRC) were able to establish a solid plan for the foundation. Gary Sherman at Spatial Galaxy has a good overview of the meeting outcome and has set up a very helpful IRC log of the meeting and the focus group discussions (Go Gary!). Tyler Mitchell has posted some photos of the meeting. I attended via IRC and phone for only a few hours so my understanding of the entire meeting is limited but I’ll add a few thoughts on what went down.

First of all, the name was decided early on to be the “_Open Source Geospatial Foundation_”. IMO, this name fits very well. Now that Autodesk’s open source contribution has been rebranded from Mapserver Enterprise to MapGuide Open Source, I am glad to see the final chapter in the whole naming debacle!

I was also very interested in the funding discussion. The general consesus seemed to be that the foundation would generate income through sponsorships. The benefits to being a sponsor/supporter of OSGF include official recognition and the obvious PR value in addition to being able to direct your funds to a particular project. It would work something like this: 2/3 of your donation could be directed to a particular software project while 1/3 would go to the foundation itself. Of the 2/3 going to the project, the Project Steering Comittee (PSC) would decide how to best allocate those funds. There was brief discussion of doing some sort of “bounty” system that would allow sponsors to fund a particular feature but this was generally thought to be a bad idea since there are so many aspects of software development that are not “sexy” enough to generate income… like cleaning up and optimizing code, bug fixes, etc. By allowing the PSC to allocate the funds, the focus can be on a solid code base and careful feature additions. Of course those who want to fund specific features can still contract directly with the developers.

One of the ironies of the initial foundation’s project membership is that Mapserver (the project that was the center of the original Mapserver Foundation) is not yet a member! While this may seem strange at first, the reasoning is so that the Mapserver community can vote on the matter. Other community-based efforts such as QGIS are likely waiting to hear from their users as well. Once the official statements from the OSGF are released, I suspect there will be a vote from these communities (and others) to decide whether they should join.

The criteria for projects to join the foundation was not entirely clear but it appears that they will be based on the commonalities of the initial projects. Requirements such as licensing and open standards are still foggy but will likely be written in such a way that they don’t conflict with any of the initial projects.. a sort of reverse engineering of the criteria if you will.

There were many interesting discussions as far as the implementation of the foundation web presence, the legal protections that would be provided by the foundation, the expected costs of running the foundation, promotion and the structure of the governing board. I’ll wait until the official announcement to see how these issues were resolved.

Overall it was an exciting and historic day for open source GIS. Many thanks to all the attendees and IRC participants for all the interesting and productive discussions. The future of the foundation is looking very bright and I look forward to seeing where it’s heading in the coming months…

One quick update: Schuyler Erle, who deserves an extra round of thanks for his amazing efforts to keep us IRC attendees informed of the meeting, has some great first-hand insights on the OSGF.

OK another quick update: The official foundation website will eventually reside at www.osgeo.org.

Mexico-US Border Crossing Maps

2006-01-24T16:42:10-08:00

Thousands of Mexicans come to the United States every year and, besides the legal troubles of border crossing, they face a tough journey across the desert in order to reach their destination. They have very little information to go on and many die from dehydration as they attempt to find their way through the vast deserts of the american southwest.

A faith-based organization called Humane Borders is trying to help the situation. They have produced a number of maps documenting town locations, roads, water stations, walking distances, cell phone towers and even places where other immigrants have died along the way. This was made possible in part by GIS software donated by ESRI.

I heard today on CNN with Lou Dobbs that the Mexican government is now printing and distributing these maps to citizens. Though the maps will clearly state “Don’t Do It! It’s Hard! There’s Not Enough Water!”, critics are saying the maps aid criminals and will enourage illegal aliens to cross the border. Others have pointed out that, from an economic standpoint, this may benefit the US border patrol since so much of their budget is devoted to aiding sick and injured imigrants and properly taking care of the dead. Humane Borders is hoping to make people aware of the risks so that they can either choose not to go or be better prepared should they decide to cross.

In any case, it is an interesting example of how geographic information is still so important (and controversial) in our society.

Geocoding an address list to shapefile

2006-01-20T18:50:39-08:00

Most commercial software comes with fairly elaborate geocoding engines and there are nice geocoding services on the web that can do one-at-a-time geocoding but the recent post at Spatially Adjusted pointed out a great free resource for batch geocoding named, conveniently enough, Batch Geocode. Just give it a list of tab or pipe delimited addresses and it outputs a table with your original data plus a lat/long for every row.

I have been working on a python script to convert text files into point shapefiles and thought this would be a great chance to put it to work. The only dependency is a recent version of python with the ogr module (see FWTools for an easy to install package for windows or linux).

First, I take a list of cities and feed it to batchgeocode.com (a very nice feature is that the yahoo geocoder, on which batchgeocode is based, does not require street level addresses):

City|State Santa Barbara|CA Arcata|CA New Milford|CT Blacksburg|VA

After running the geocoder, I get back a table with lat/longs:

City|State|lat|long|precision
Santa Barbara|CA|34.419769|-119.696747|city
Arcata|CA|40.866261|-124.081673|city
New Milford|CT|41.576599|-73.408821|city
Blacksburg|VA|37.229359|-80.413963|city

Copy and paste that into a text file and add a second header row that defines the data type for each column. It would be possible to autodetect the column types but there are cases where a string of numeric digits should be kept as a string (for instance the zipcode 06776 would become 6776 if it was read as an integer).The possible column types are string, integer,real, x and y with x and y representing the coordinates.

City|State|lat|long|precision
string|string|y|x|string
Santa Barbara|CA|34.419769|-119.696747|city
Arcata|CA|40.866261|-124.081673|city
New Milford|CT|41.576599|-73.408821|city
Blacksburg|VA|37.229359|-80.413963|city

Now run the txt2shp.py utility. The input and output parameters are self-explanatory and the d parameter defines the string used as a delimiter. Notice that the syntax follows the GRASS standard of parameter=value:

txt2shp.py input=cities.txt output=cities.shp d='|'

And now you’ve got a shapefile of the geocoded cities!

The txt2shp.py script can be downloaded here. Try it out and let me know how it’s working for you.

Update: In order to generate a .prj file for your output shapefile, you can use the epsg_tr.py utility if you know the EPSG code. Batch Geocoder returns everything in lat/long (presumably with a WGS84 datum?) so you can use EPSG code 4326:

epsg_tr.py -wkt 4326 > cities.prj

Tissot Indicatrix - Examining the distortion of map projections

2005-12-11T23:53:56-08:00

The Tissot Indicatrix is a valuable tool for showing the distortions caused by map projections. It is essentially a series of imaginary polygons that represent perfect circles of equal area on a 3D globe. When projected onto a 2D map, their shape, size and/or angles will be distorted accordingly allowing you to quickly assess the projection’s accuracy for a given part of the globe.

I’ve seen great Tissot diagrams in text books but I wanted to create the indicatrix as a polygon dataset so that I could project and overlay it with other data in a GIS. To do this I wrote a python script using the OGR libraries, which I will revist in a minute. But first the visually interesting part:

Here is a world countries shapefile overlaid with the Tissot circles in geographic (unprojected lat-long) coordinates:

Next I reprojected the datasets to the Mercator projection using ogr2ogr:

ogr2ogr -t_srs "+proj=merc" countries_merc.shp countries_simpl.shp countries_simpl
ogr2ogr -t_srs "+proj=merc" tissot_merc.shp tissot.shp tissot

Note that the angles are perfectly preserved (the trademark feature of the Mercator projection) but the size is badly distorted.

Now lets try Lambert Azimuthal Equal Area (in this case the US National Atlas standard projection - EPSG code 2163).

ogr2ogr -t_srs "epsg:2163" countries_lambert.shp countries_simpl.shp countries_simpl
ogr2ogr -t_srs "epsg:2163" tissot_lambert.shp tissot.shp tissot

This is a great projection for preserving area but get outside the center and shapes become badly distorted:

The best way to experiment with this is to bring the tissot.shp file into ArcMap (or another program that supports on-the-fly projection) and play with it in real time. The distortions of every projection just leap off the screen…

OK, now for the geeky part. Here’s the python/OGR script used to create the tissot shapefile. The basic process is to lay out a grid of points across the globe in latlong, loop through the points and reproject each one to an orthographic projection centered directly on the point, buffer it, then reproject to latlong. The end result is a latlong shapefile representing circles of equal area on a globe.

 #!/usr/bin/env python
 # Tissot Circles
 # Represent perfect circles of equal area on a globe
 # but will appear distorted in ANY 2d projection.
 # Used to show the size, shape and directional distortion
 # by Matthew T. Perry
 # 12/10/2005
 
 import ogr
 import os
 import osr
 
 output = 'tissot.shp'
 debug = False
 
 # Create the Shapefile
 driver = ogr.GetDriverByName('ESRI Shapefile')
 if os.path.exists(output):
         driver.DeleteDataSource(output)
 ds = driver.CreateDataSource(output)
 layer = ds.CreateLayer(output, geom_type=ogr.wkbPolygon)
 
 # Set up spatial reference systems
 latlong = osr.SpatialReference()
 ortho = osr.SpatialReference()
 latlong.ImportFromProj4('+proj=latlong')
 
 # For each grid point, reproject to ortho centered on itself,
 # buffer by 640,000 meters, reproject back to latlong,
 # and output the latlong ellipse to shapefile
 for x in range(-165,180,30):
     for y in range (-60,90,30):
         f= ogr.Feature(feature_def=layer.GetLayerDefn())
         wkt = 'POINT(%f %f)' % (x, y)
         p = ogr.CreateGeometryFromWkt(wkt)
         p.AssignSpatialReference(latlong)
         proj = '+proj=ortho +lon_0=%f +lat_0=%f' % (x,y)
         ortho.ImportFromProj4(proj)
         p.TransformTo(ortho)
         b = p.Buffer(640000)
         b.AssignSpatialReference(ortho)
         b.TransformTo(latlong)
         f.SetGeometryDirectly(b)
         layer.CreateFeature(f)
         f.Destroy()
 
 ds.Destroy()

KML to Shapefile Scripting

2005-12-11T22:26:56-08:00

Christian Spanring has been doing some great work with Google Earth’s KML data format. The latest offering is a fairly robust XSLT stylesheet for transforming KML into GML.

In the article, he mentions ogr2ogr as a method to convert GML to shapefiles so I immediately had to try it out! I came up with a simple bash script, kml2shp.sh, that provides a quick command-line interface:

kml2shp.sh input.kml output.shp

Here’s the step-by-step:

1. Make sure you have xsltproc (the command-line xslt processor) and OGR installed.

2. Copy the xslt stylesheet to /usr/local/share/kml2gml/

3. Create the kml2shp.sh script below (make sure to change the paths to reflect your system, chmod +x it, etc)

!/bin/bash

if $# -ne 2; then echo “usage: kml2shp.sh input.kml output.shp” exit fi

echo “Processing KML file” sed ‘s/ xmlns="http:\/\/earth.google.com\/kml\/2.0"//’ $1 > /tmp/temp.kml xsltproc -o /tmp/temp.gml /usr/local/share/kml2gml/kml2gml.xsl /tmp/temp.kml

echo “Creating new Shapefile” ogr2ogr $2 /tmp/temp.gml myFeature

echo “Cleaning up temp files” rm /tmp/temp.gml rm /tmp/temp.kml

echo “New shapefile has been created:” echo $2

Now as far as I can tell, the XSLT is fairly robust although I’ve only tested it on a few datasets. The wrapper script, however, could use alot of work. Type and error checking would be nice for starters and a better method to remove the xml namespace might be necessary. This is really meant as a starting point.

One potential problem with this technique is that you will most likely get a 3D shapefile (x, y AND z coordinates). Many applications can handle 3D shapefiles but some (QGIS, others?) cannot at the present time. Once the geometry type is known, one could always specify the ogr2ogr “-nlt” parameter to force 2D output. But that’s all for now… let me know if anyone has any suggestions on improving this technique.

The new blog

2005-12-03T15:16:08-08:00

Well I finally got around to installing some real blogging software. SimplePHP Blog was just not cutting it and WordPress looks like a healthy option. So far I’ve been really impressed! Let me know if you have any troubles accessing it…

Processing S57 soundings

2005-12-03T14:25:41-08:00

NOAA Electronic Navigational Charts (ENC) contain (among many other things) depth soundings that can be processed into raster bathymetry grids. The ENC files are available as a huge torrent from geotorrent.org (http://geotorrent.org/details.php?id=58).

Download this torrent and check readme.txt to find the chart of interest:

Port Hueneme to Santa Barbara|5|2005-10-03|2005-10-03|US5CA65M

First check out the gdal documentation for s57 files at http://www.gdal.org/ogr/drv_s57.html.

Change to the US5CA65M directory and you’ll see a .000 file (and maybe .001, .002 etc). Run ogrinfo on the .000 file and you’ll see ~ 61 layers, one of which (“SOUNDG”) represents the soundings. Let’s start by examining the soundings layer:

ogrinfo -summary US5CA65M.000 SOUNDG

We see that there are 43 “features” but since the features are multipoints, there are actually thousands of soundings. The multipoints are 3D so If we convert to a shapefile with ogr2ogr’s default settings we loose the 3rd dimension. To solve this, we need to append “25D” to the layer type. Furthermore, the multipoint geometry confuses some applications so we want to split it into a layer with simple 3D point geometries. Luckily there is a SPLIT_MULITPOINT option that must be specified as an environment variable:

export OGR_S57_OPTIONS=”RETURN_PRIMITIVES=ON,RETURN_LINKAGES=ON,LNAM_REFS=ON,SPLIT_MULTIPOINT=ON,ADD_SOUNDG_DEPTH=ON”

ogr2ogr -nlt POINT25d test3.shp US5CA65M.000 SOUNDG

Now we get ~ 3000 3D points with the depth added as an attribute for good measure.

Now bring these into grass and create a raster:

v.in.ogr -zo dsn=test3.shp output=soundg layer=test3 v.info soundg g.region vect=soundg nsres=0.001 ewres=0.001 v.surf.rst input=soundg elev=bathy layer=0 r.info bathy

since depths actually show up as positive elevations, we want to multiply the grid by -1

r.mapcalc sb_bathy=bathy-1

And of course we want to make some nice shaded relief and contour maps for viewing with QGIS:

r.shaded.relief map=sb_bathy shadedmap=sb_shade altitude=45 azimuth=315

r.contour input=sb_bathy output=sb_contour step=5 qgis &

From the screenshot, we see the pits and spikes from potential outliers so we might want to go back and adjust the tension and smoothing on the raster creation (the v.surf.rst command).