Filed under: General
If it weren’t for plane rides I ‘d probably cancel most of my magazine subscriptions, no shortage of time spent in the air these days and the solitude of air travel, as odd as that it may sound, certainly does give me the opportunity to read.
This month in Alberta Venture they had an interesting article “Taxes and Duties”, the topic of taxes is never that exciting but the stance of this article covered definitely spanned some interesting ground. Janet Keeping’s comments are brilliant and refreshing, so good to see that there are people in the tax world that are above-board and actually understand the value of taxes.
My opinion on the topic is simple and ideal — looking for loopholes to avoid taxes is a poor substitute for contributions to the community either directly or via taxes. Greed is not the same as ambition.
In decision support for linear assets the most familiar result is the optimal alignment but this is far from the only useful output from a rather rigorous process. In many respects, the optimal alignment is more misleading than useful since it only provides a glimpse of what is without painting a full picture of the situation – like reading an executive summary without regard for the underlying body of knowledge. What then in the decision making process can provide us with additional insights? Cost corridors!
Before we plow into talking about cost corridors let’s remind ourselves that least cost analysis is merely a numerical technique for solving a complex system. With this now fresh in our minds we can talk a little more abstractly, numerical techniques used in solving systems operate in the realm of a solution space. Solution space is not constrained to two, three, or four dimensions but rather the dimensions are as numerous as the number of variables, constraints, and interaction rules that comprise the system.
As mentioned, the most common or expected result of a least cost analysis is the optimal path, this represents just the minimum solution, and, for those who have studied approaches for solving systems, we know that this is just one solution found within the solution set. Observing the solution set is where cost corridors come into play. The unfortunate thing is that even though solution space can handle multi-dimensions we mere mortals (okay, more to do with technical limitations) need to determine the projection of the solution set onto the practical plane of a GIS graphical display. The cost corridor is the intersection of the solution set with combined cost surface.
In earlier posts we detailed how the least cost analysis process works when all that is needed is the optimal route, quite simply:
- establish a start point
- accumulate cost weighted distance from the start
- establish an end point, and
- trace the steepest path from the end point to the start point.
The process for creating cost corridors capitalizes on the optimal route process quite heavily and is implemented as:
- establish a start point
- accumulate cost weighted distance from the start
- establish an end point
- accumulate cost weighted distance from the end
- combine start and end accumulated cost weighted distance
- find the minimum value within the combined accumulated cost weighted distance, and
- using a cutoff percentage query the combined accumulated cost weighted distance for values between the cutoff and the minimum.
Check out this for some more information on this process and its usefulness.
Filed under: Python
Another great get away to recharge and read and read and read and read and … yup, always lots to stay on top of in our ever changing dynamic world. Had some real fun taking in the “new” features of C# 3.0 and the .NET framework 3.5, some really nice things included there like LINQ and lambda but other than that, just more CLR chatter and hype around how the new VS makes coding sooo much easier. Probably a good thing considering how user expectations continue to heighten more rapidly than quality code can be created and pushed into the cloud.
Most would know that I’m into Python fairly heavy so I figured to bridge some of the “new” .NET features like LINQ and lambda back into their corresponding cohorts inside Python. Although not exactly the same thing, it does for my purposes cover off the majority of items that we handle in our code development, the LINQ as is pertains to SQL is YAORM (yet another object relational modeler). In Python we’ve had for quite some time a set of wonderful modules that provide this exact capability, granted they are not part of the core development but none the less they are beneficial packages if you’re doing anything more than one or two simple SQL statements. The modules include some favorites like SQL Object and SQLAlchemy – if you like the idea of being efficient for your clients then you want to look at these and some of the plug-ins too.
Onto lamdba’s, well this isn’t a new thing for Python peeps, been around forever. Of course you always need to ask yourself “is there ever a chance that this function will be used elsewhere?” which normally equates to “Yes” but if not then lambda away! Here is one of my favorite ways to use a lambda, it is in the key kwarg for sort or sorted. This approach I find really handy for being able to capture the sorting logic directly inline in the code which for me really helps in readability!
import random
import operator
lowerBound, upperBound = 0, 1000
values = [tuple([random.randint(lowerBound, upperBound) for i in range(10)]) for i in range(1000)]
sorted(values, key=lambda v: sum(v[:2])**2 + sum(v[2:5]) + reduce(operator.mul, v[5:], 1.))
What this code does is generate a list of tuples containing values and then the sort is based on a combination (like a weighting) of the tuple components. Way cool and super fast.
Well enough on that for now, nice to see that C# is beginning to catch-up with where Python has been for quite some time…YATTOS (Yet another testament to open source).
Filed under: Python
Recently we finished off a very aggressive sprint to release a new set of tools for one of our clients. Much fun had and the delivery and roll-out went very smooth – great teamwork all around and excellent interaction with the client internal IT department.
One item that crept up late in the development cycle (not that we didn’t know about it, just that we hadn’t thought through the full set of implications) was localization. This really hadn’t been an issue in previous releases but with adding new functionality and capability related to dates and times, well, of course it was now an issue.
The application needed to work regardless of the personal preferences each person had set for their date/time format. Hmmm, will that require a bunch of string manipulation and testing and checks, or maybe, just maybe the good folks behind the Python language have already crossed that bridge…and, of course, they had.
A word that comes up often when talking about Python and implementation is elegance, and this time was no exception. Simple solution to what could have been a nightmare coding effort in the final days. Hanging off of the datetime (and time) modules is strftime and one of its format strings does exactly what we needed, for example, the following interactive code shows how a datetime is return in my system (local) format – way cool!
>> import datetime
>> t = datetime.datetime.today()
>> print t
2009-03-26 08:25:29.160000
>> t.strftime(‘%x’)
‘03/26/09′
Where this really becomes a benefit is when working with an ESRI cursor object, and you’re wondering “How the heck should my date be specified so that it ends up in the database correctly?”, next time just wrap it with some %x and let the magic happen.
Filed under: GIS
I’ve been looking lately at Bluetooth technology, you know, how it works, what makes sense (i.e. not real time sound) for its use, range and a few other things. What got me on this topic is the seemingly small selection of Bluetooth mouse and keyboards — the computer peripherals that should be the mainstay of Bluetooth products.
In all that searching I have found a nice headset from Plantronics Pulsar 590A, and another from Motorola S9-HD — both are rechargeable and the range on the Plantronics is phenomenal. Favorite mouse has got to be the Logitech 470 with a close second being the Microsoft Notebook Mouse 5000 — battery life seems to be a tad bit better with the 470. Finally for keyboard/mouse combinations the Logitech DiNovo is slick, and the Microsoft Wireless Entertainment Desktop 7000/8000 are nice options too (although the mouse is a little heavy in the Microsoft bundle).
Not a Bluetooth keyboard, but check out this new one I picked up today today.
Filed under: GIS
Continuing on from a post back in January where we talked about a really nice database engine for road warrior development. The question, when posed to our internal development team, was “what about spatial?” — a very relevant question given the line of work we are in.
So in my travels (this time just meaning a cruise around the web), I stumbled on a really cool set of open source code that extends SQLite with spatial capabilities. The package is called SpatiaLite (http://www.gaia-gis.it/spatialite/) and offers up the very useful blend of SQLite relational database and GEOS library spatial operations, the result is a lightweight spatial database with built-in spatial operators.
If you check out the website you’ll also see something else that is fairly exciting – it’s called VirtualShape and it is a driver for SQLite that enables SQLite databases to access the data held in shapefiles. Wow! This might be the closest thing we’ve had to the live spatial operations we could easily perform using Avenue and theme definitions (and if you can remember writing something like [shape].contains(…) then please drop me an email).
Filed under: Python
Kudos to anyone who has tried to code on plane or in an airport without noise canceling earphones. Not only is it way too noisy but the lighting can really mess with your eyes after a short time. My main issue when trying to accomplish something while in transit is finding enough space to open up my laptop fully, which means my viewing angle is far less than optimal.
On my last journey to the south I took some time to read up on colors and fonts used in IDE’s. There are a bunch of good sites out there with solid suggestions on programming fonts. For me I initially used Courier New (come on, who hasn’t accepted the defaults at one time or another) but soon switched over to Consolas when changed over to Vista — enjoyed Consolas for a while but then fell in love with Bitstream Vera Sans Mono after a stint working on one of my Ubuntu laptops.
For most (but not all) of our Python development we use Komodo IDE, this IDE allows full control (although the granularity of control could be better with respect to the categories of language Element Types) over the fonts and colors used. Here is the first scheme that we used (and some still use) in our shop when developing:
This one worked for quite some time for me but it was too harsh on my eyes when working on my laptop screen (using 1920×1200 resolution) but was OK when on my external display (24" Widescreen LCD). Fiddling with the various settings I’ve come up with these two schemes that really help reduce eye strain for me on my laptop screen and external display, even when surrounding light and viewing angles are sub-optimal.
This last one is the scheme that I’m presently using, it seems to work great on the plane — a subtle change is that the operators are bold, subtle but makes a difference.
Filed under: GIS
A new year and some new eats, trying to cut out those heavy snacks that make you feel bloated — remarkably similar feeling as working with some enterprise level databases that are poorly tuned…At one point in my career it was uber important for to be on top of all the latest and greatest technology available; that was in the days when web-time was being defined and it certainly made for some interesting reading, but the relevance was soon lost — the world of business does not move at the speed of technology change and rightly so, change is expensive (but can also be a good investment when properly handled).
Even though we have fully wired environment and hooked-in computing environment that allows us to work remotely, the convenience of this is soon lost when you spend a lot of time sitting on planes or are unable to get a reliable signal that’s strong enough to handle the data throughput. For example, working a pet project a few weeks back while on the road saw that nearly 95% of the time spent waiting was for data transmission across the skinny wire 3000km away.
This got me looking for an alternative…something that would be lightweight, easy to code against, worked with Python, and for which the code was easily transferable (i.e. SQL DML and DDL are very rigid to ANSI-SQL) by only needing to switch a connect string.
First path was to look at an ORM, SQLAlchemy is not a database engine but it does offer the convenience of providing a layer of abstraction between your code and the database. Although SQLAlchemy is an awesome package, it was overkill for what was really needed. Next thing, was looking at SQLite — wow, what a slick database engine and does it work with Python, for sure, built-in for 2.5 and available for 2.4. Granted SQLite does not support all the capabilities of Oracle or SQL Server, it is remarkable close in its syntax and capabilities and far better than MS Access with respect to syntax and has a very tiny memory footprint.
In the data model GIS stream of posts we’ve only addressed the mechanics of the data model, up to now the assumption was that we were converting an empty schema (well, not exactly empty, it has tables and lookup values) from relational into spatial. Today we’ll get into how to handle the case where there is spatial data stored in a relational database as (these are taken from here):
- Coordinate pairs stored as numbers with associated projection/datum code
- Listing of coordinate pairs as with an associated sort order identifier
- Coordinate Collections having different spatial references defined
Cases 1, 3, and 3 are very common in the PPDM data model, and Cases 2 and 3 are fairly common in PODS (not many standalone points in PODS that aren’t part of a line or actually an event).
Let’s treat Cases 1 and 2 as virtually the same thing — one or more coordinate pair that needs to be translated into geometry. In one case we have points and in the other case we typically have lines (seismic lines and well directional surveys are great examples).
In a pure ESRI implementation we’d look towards the Point Object and the Array Object of the geoprocessing framework to supply us what we need for generating geometry. In a less restrictive world (but still ESRI) like ArcObjects the we can use a Point or Polyline and make them Z and M aware as required. In a more open world we gravitate towards Well Known Text (WKT) and Well Known Binary (WKT) or maybe even the geometry formats of osgeo.ogr. We’ll stick with WKT for now since it is easier on the eyes and is handled by all the major RDMBSs in their spatial add-ins.
Since we are primarily concerned with points and lines the WKT forms we need to be familiar with are POINT(X Y) and LINESTRING(X Y Z M, X Y Z M…X Y Z M, X Y Z M). It should be obvious how to proceed from this point but just in case it isn’t here’s how we proceed (in Python code and using PPDM):
curs.execute(“SELECT UWI, LONGITUDE, LATITUDE FROM WELL”)
wktResults = [(r[0], ‘POINT(%3.9f %3.9f)’%r[1:]) for r in curs.fetchall()]
And that simple piece of code handles the points for us, how you proceed from there depends on your desired approach, maybe write the results to a file for bulk load, or maybe use a Python compliant DB API to load the results straight into a spatially enabled table. The approach for lines is much the same, all you need to do is query the coordinates and then serialize the coordinates into a string and wrap that in the LINESTRING keyword.
That’s really the easy part, but we haven’t done anything to handle spatial references on the coordinate pairs. The easy case is where all the coordinates in a database are in the same coordinate reference system, of course this is rarely the case for PPDM instances (international exploration is one of the main reasons why) but predominantly the case in PODS instances. So what to do? Let’s at least assume that each coordinate pair is assigned (wrongly or rightly) an ESPG CRS value. That’s a great start but we also need to know the target spatial reference and, equally as important, the transformation method. Once we have all that information we’re in a position to begin thinking about how to handle.
That’s enough for one post, more on this later…
Dexterity of the hands and mind, and words per minute, are these part of the skills that were one time considered laughable by the elders but now are manifesting themselves as vital skills in the ability to compete in today’s knowledge market?
Once again, I’m at the airport, wired into the hotspot, setting up my Linux machine, writing emails, browsing a website, and biting the sides of my cheeks to keep from laughing. The guy next to me is definitely old school, probably even had a secretary at one point in his life. The markings on his bag, laptop, and login splash screen show that he is in the game of engineering consulting (with a rather large public engineering company). Engineering consulting is that wonderful world where a client gets dinged for the inexperience and inefficiencies of the staff, where timelines are guidelines, and delivery rhymes with “change orders”, time is rounded to the 5 minute interval, and charge codes are needed before anything can get done (including make a photocopy).
So why am I laughing, well given this person’s age I would expect him to be a person with seniority and experience (which means tres cher). Yet watching him type with just one finger on just one hand — come on — what era are we living in? Yes, knowledge and experience are important in the consulting world, but isn’t the ability to communicate (aka write) equally important, especially in the realm where things must be documented and explained? Wouldn’t you expect…one second, he just sped up, now at 15 characters per minute, nope, I’m wrong, that was just some pounding on the backspace key…that you (as a client especially) wouldn’t be slammed with invoices that simply cover off the inability for someone to type?
Gotta shutter down, boarding time for my flight, I’ll get back to more technical things in the coming weeks.
