Thiiink: Ideas, Imagination, and Innovation in GIS

Within ArcInfo the Overlay toolset contains  a rich set of tools i.e. Erase, Identity, Intersect, Spatial Join, Symmetrical Difference, Union, and Update.  But if you only have an ArcView license (aka the majority of GISers out there) then you only get Intersect (two inputs only), Union (two inputs only), and Spatial Join.

So what do you do if you only have ArcView?  Well thankfully it isn’t very difficult to use the already existing capabilities of ArcGIS and a little bit of Python programming to roll your own.

Intersect and Union are the easiest ones to overcome, all you need to do is know how to split a multi-value list and loop over the results (yeah you might not be able to use the ranks option…but how many people actually use that and don’t have AI already).

Then all you need to do is get back to basics.  For example, a symmetric difference is essentially a UNION minus an INTERSECT.  Erase is a little trickier but still doable via some additional select by’s and some light post processing.

Identity, well the tricky part here is the “keep relationships” option.  Personally I never use that option because of the way the attributes get renamed.  So with this one simplifying step an identity can be created chaining the existing tools and again some minor post processing.

Update is an interesting one but still relatively straightforward – often wondered why this one existed in the first place since it is remarkably similar (but not exactly the same) as an erase and append.

Anyways, in the span of a morning we were able to write the full suite of Overlay tools so that even at an ArcView license level you can get a reasonable hand drawn facsimile of the functionality available with ArcInfo.

if you too would like these and other tools available and would like to learn how to make these tools yourself then sign up for the Advanced version of our Geoprocessing with Python course via BrainCandiii.

Continuing from the last post — nice progress this week made on a the AI only tools in the Features Toolset.  This is turning out to be super useful!  

Yesterday we knocked out another 5 tools from the Data Management Toolbox that were limited or non-existent in ArcView and ArcEditor, ahem, ArcGIS for Desktop Basic and ArcGIS for Desktop Professional.  While we were at it we added in some additional functionality to the tools, you know, just to increase the number of pieces of flair.

When I first cut my teeth in GIS I was introduced at the ArcView GIS 3.x days but after some time learned about Arc/Info Workstation.  Ever since then I’ve been conscious of the differences in functionality between the products and even more so since everything was blended together in ArcGIS Desktop.

Although most of us in the company usually use AI (Professional for those early adopters of 10.1) for our normal work, our development mantra has been to assume nothing and code for AV wherever possible.  Of course this is not so much of a hard/fast rule as it is just guidance and practice so sometimes we forget that most people do not have AI (Professional) and oops!, in slips an AI only tool.  Of course there are some other tools that just act differently when not using AI, for example, union and intersect only allow for two inputs.  We lump these into the same category as AI only since we really want the same behavior in our code.

Something we have chatted about internally for some time now has been writing some reasonably hand drawn facsimiles of the the AI only tools and including them inside of pygp.  Not necessarily exposing new toolboxes or tool interfaces just so that when an AI only tool is called from within our code base that the license level is checked and if AI is not available then use the alternative code.

Easy enough to identify the tools that need to be addressed – this is published – and even easier to identify those that should be addressed first (i.e. the ones we bump up against the most often).

Today seemed like as good a day as any to start cracking on this idea.  Churned out three tools from the Analysis toolbox and have stubbed out the code for five more from the Data Management toolbox.  Should make for a very sidebar project to start the year off right.

Been some really nice activity on our BrainCandiii sweet of training products as of late.  Modularizing, adding new content, and getting ready for ArcGIS 10.  Overall quite nice! 

Data as a Deliverable (DaaD) is something we’ve been promoting for years; now we have a cool acronym.

The very first alignment sheet generator software we wrote was way back in the days of ArcView GIS 3.x using Avenue and some tiff libraries for performing image rotation.  When ArcGIS 8.x came around we did an overhaul of the application to take advantage of an event driven environment where much more of the data could be live fed into a rotated data frame and labeling (and overposting) was dynamic.  It definitely made for an improvement on the overall application and usage.

After porting to 9.x and maintaining the application for the last few years as ArcGIS minor versions and service packs were released we decided to drop the entire code base instead of porting to 10.  Risky?  Yup, you bet, but how else will you get step change improvements if you keep dragging legacy code around?

First thing we did was do survey of the exposed and desired functionality of ArcGIS 10 with all the new mapping modules and interaction with maps and layers and data driven pages.  Let’s just say that the amount of functionality exposed to Python was sufficient (not perfect) to allow us to move forward.

Next step, wrap up all the functionality we needed into our pygp python package, things like offsets and angles for on-pipe data (blue)

and the ability to read PODS and APDM data directly onto the map layout (orange). 

pygp has become our storehouse of unique and useful functionality, things we should only have to figure out once and use many times, it is also the place where we choose to expose functionality as would like it to be exposed for Python development as opposed to the command-line style of arcpy.  Wrapping the Python functionality has a very nice side benefit too – you really get to know the functionality well and quickly.

After that we thought about all the new capability we wanted in the application with the main item being the ability to draw graphs of data (e.g. pipeline integrity data) along the pipeline route (green). 

This turned out to be a very sweet add to the application and it wasn’t more than a few weeks later that one of our clients was using this for generating alignment sheets for a regulatory inquiry showing things like rupture pressure ratio, hydrostatic test pressure overlaid with MAOP, defect density, close interval survey and cathodic protection test station readings.  All displaying very crisply on the alignment sheet and all performed using Python.

Another item we wanted and subsequently implemented was the ability to configure the alignment sheet generator solely using an Excel table to define the data connections and types and ArcGIS layout tools for making the template (we are thinking about driving that from an Excel configuration table too).  This was a super add to the application since it makes for a much quicker way to add data to your map document and to ensure that all the settings of the layers are consistent and follow the desired labeling and cartographic settings.

About the only thing we have left to do now is get the documentation updated and this application is ready to ship.  And while we did originally make this product for the pipeline industry the new configuration and graph capabilities make it very suitable for transmission lines and other linear corridor assets.

Working through some design items today and kept bumping up against locking issues in the geodatabase used for testing – so strange.  Looked into it only to realize that I had inadvertently created a Personal Geodatabase.  Gah!  Back to File Geodatabases now, all good.  Sigh.

We’ve been super busy with a series of sprints, ports, refactors, migrations, and new product development. All I can say is thanks to all those involved and man, that was fun.

For a number of years we had a couple of products used for working inside of a formal development environment where we could easily repoint/repath data sources in layers, map templates, and map documents from one environment to another. With a improving stability of ArcGIS 10 we decided it was time to see if the newly exposed capabilities (via Python that is) for updating data sources layers and map documents could offer an alternative to "dot not".

The short answer is that we were able to accomplish what we needed to accomplish with respect to product migration. The long answer is that there was a large portion of capability we had to build, improve, fix, work around, or expose for ourselves.

Especially disheartening was the apparent lack of testing on Esri’s part of the new functionality against SDE.  Actually if you look at the forums you’ll see a boat load of comments about performance against SDE (lack of caching and making multiple connects).

Despite the issues we bumped up against we now have a very nice tool that is configuration driven (can be read off of an Excel file with specific columns) and handles a large number of migration paths (workspace type change, folder changes, hyperlink base path, from file based to SDE, from SDE to file based, schema to schema, and so forth). 

Release v0.3 went out last week and v0.4 (includes the skip file options shown above) will go out this week and also includes another tool for reading a table containing the files to be worked on by the Repoint and Repath engine.