Boxes for expenditure data is ordered as General public services, Defence, Public order and safety, Economic affairs, Environment protection, Housing and community amenities, Health, Recreation culture and religion, Education, Social protection.

1. (top) Cows grazing near UK’s expenditure for “Housing and community amenities”.

2. Player is standing in front of government expenditure for Malta looking at Germany:

3. Overview of Germany, France, Italy and UK:

Future ideas include rendering budget deficits as holes in the ground and making it possible to enter boxes to see subdivisions. Data for Greece should be rendered in lava I guess.

This hack was made possible by using the pymclevel library by David Rio Vierra.

Source data: Eurostat – General government expenditure by function (COFOG) 2010.

TL;DR See if it is possible to publish your open data as file dumps instead of building advanced API:s that force entrepreneurs to integrate their apps with your infrastructure.

A fictional background

It was supposed to be a regular day for John at the server facility at the government weather agency. But when he came in to work that morning his colleague Mike was in a panic. -”Look! We are in the middle of a DDoS attack. The API-server is flooded and the database server is on it’s knees. The meteorologists can not work” . John started looking at server logs. Between 7 and 8 a.m. there was a sharp increase in traffic. Loads of API calls were made from a lot of different IP:s. Then, all of a sudden server load decreased and everything was back to normal.

What happened?

A year ago the weather agency had started to make their data available as part of the agency’s open government initiative. They were in a rush at the time and had decided to create an API for their weather data by setting up an internet-facing API-server. The API design had tried to take into consideration potential use cases that entrepreneurs may have but it was hard to know what people wanted. They had settled on three generic API calls.

Fast forward a year and they discover that one entrepreneur had built a very successful mobile app used by several hundred thousand users. Every morning it wakes you up by announcing today’s weather. To get the data necessary the app developer had to make two API calls which all of the installed apps did every morning to wake up their owners. That promptly crashed the API server which wasn’t designed to cope with the load.

Let’s call this the Direct integration API model.

The Direct Integration API Model

The direct integration API model has several drawbacks:

1. Since the API:s [2] are designed for direct integration this is what developers did. The government agency is now (unknowingly) a critical component in making sure the apps are working.
2. By forwarding API calls to the one and only database server internal applications are affected when there is a high API load [1].
3. High load from one application will impact all other applications using the same API infrastructure.
4. API:s were designed from hypothetical use case scenarios forcing applications to make multiple API calls to get the data required.

Even if the data was offloaded to a separate database server to isolate external load from internal systems developers still need to rely on the capacity of the API .

A better way to publish open government data

An alternative to the direct integration API model is to publish data dumps in files. “Boring!” may be the initial reaction from developers but they will thank you later. In this model data from the database is exported, transformed to an open readable format [1] (e.g. CSV), properly named and stored on the web server [2]. This means entrepreneurs can get all your data, load it into their own system and design their API according to their use case. Also, high loads will hit their own infrastructure without affecting other apps.

As an added bonus it is very simple to publish data dumps on a web server.

If files and URL:s are named consistently it is easy for entrepreneurs to pick up data over time (e.g. http://data.example.com/weather/country/2012-03-01.csv). An alternative is to create an API that is designed for mirroring of data and its changes (e.g. event sourcing over Atom).

These two models gives us some background for design considerations for publishing open data.

Design considerations

1. Do you really need an API? API projects can become expensive and typically compete with other IT projects that may have a higher priority. Also, designing an API involves making decisions on API use cases. Do you know how users will use your data? Will your API design prevent users from making efficient use of your data in their applications? What is your plan to cope with load?

2. Make it easy for entrepreneurs to keep a local copy of your data up to date. By providing wisely named data dumps it is simple to keep a local database up to date. More advanced scenarios include using existing protocols and technology for low latency distribution (e.g. pubsubhubbub).

3. Isolate internal systems from the effects of external data publishing.

4. Make sure you can change your technology without breaking URLs. People are building software that depends on your URL:s. Don’t force them to rewrite their software just because you are switching to a new platform. Early warning signs is the existence of platform-specific fragments like “aspx”, “jsp” in your URLs. Get rid of those.

There are of course other things to take into consideration such as semantic descriptions but that is a matter for a later post.

Hopefully this will save you both money and time as file exports may be a lot cheaper than creating API:s. What do you think?

Also see: Publishing Open Government Data by W3C

In the next blog post I will look at cases where an API makes a lot of sense (real time data, collaborative processes) .

Click to see entire document (PDF).

1. Libraries for using in your own web project.
2. Client side applications that generate a standalone visualization.
3. Hosted applications to which you upload your data and create visualizations in the cloud.

1. Visualization libraries for using in your own web project

A general recommendation is to stick with libraries that work on as many client platforms as possible. This means staying away from libraries that require Flash and Silverlight. The following libraries use javascript and SVG with various fallbacks and should work well on modern phones and tablets.

JavaScript InfoVis Toolkit (JIT)

The toolkit implements advanced features of information visualization like TreeMaps, an adapted visualization of trees based on the SpaceTree, a focus+context technique to plot Hyperbolic Trees, a radial layout of trees with advanced animations called RGraph and other visualizations.

License: BSD

gRaphaël

Makes it possible to create interactive charts on your website. Primarily traditional charts. It is a plugin for the Raphaël graphics library. Fast.

License: MIT

D3.js

D3 allows you to bind arbitrary data to a Document Object Model (DOM), and then apply data-driven transformations to the document. Rather than provide a monolithic system with all the features anyone may ever need, D3 solves only the crux of the problem: efficient manipulation of documents based on data.

License: BSD

PhiloGL

A WebGL Framework for advanced data visualization, creative coding and game development. It can be used to create applications that are focused on performance. PhiloGL also provides a rich module system covering Program and Shader management, IO, XHR, JSONP, Web Worker management, Effects and Tweening, and much more.

License: MIT

Polymaps

Javascript library to add interactive overlays to maps. In addition to standard image tiles, Polymaps supports vector tiles that are rendered with SVG.

License: BSD

2. Client side applications that generate a standalone visualization

Graphviz

A collection of graph layout programs. Takes descriptions of graphs in a simple text language, and make diagrams in useful formats, such as images and SVG for web pages, PDF or Postscript for inclusion in other documents; or display in an interactive graph browser.

License: The Eclipse Public License (EPL)

Gephi

Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs. Runs on Windows, Linux and Mac OS X. A large number of plugins provide additional functionality.

License: Dual - CDDL or GNU GPLv3

Tableau Public

Stand alone visualization tool for the Windows platform.

License: Commercial (but free download)

3. Hosted visualization applications

IBM Many eyes

A platform to create and discuss visualizations. All the visualizations on Many Eyes have an attached discussion forum. As you explore a visualization, you may find a view that you’d like to talk about or share. If you post a comment, your “view” is saved along with your comment so others can see what you’re seeing. Visualizations can be included on your own website.

Google Public Data Explorer

Allows you to upload your own data in the Dataset Publishing Language (DSPL) format and create visualizations similar to those available in Google Spreadsheet.

Google Spreadsheet

Often overlooked, Google Spreadsheet has some powerful tools for data visualization including the ability to create motion bubble charts similar to those made famous by Hans Rosling. See the quick guide to motions charts at Gapminder.

Did I miss your favorite tool?

Please submit them in the comments below.

When Intentions Go Public – Does Social Reality Widen the Intention-Behavior Gap? by Peter M. Gollwitzer et al.

On the management of software projects:

Programming management will continue to deserve its current poor reputation for cost and schedule effectiveness until such time as a more complete understanding of the program design process is achieved.

We build systems like the Wright brothers built airplanes — build the whole thing, push it off the cliff, let it crash, and start over again.

Production of large software has become a scare item for management. By reputation it is often an unprofitable morass, costly and unending. This reputation is perhaps deserved.

On the distinction between design and production:

Software production takes us from the result of the design to the program to be executed in the computer. The distinction between design and production is essentially a practical one, imposed by the need for a division of the labor. In fact, there is no essential difference between design and production [...]

The most deadly thing in software is the concept, which almost universally seems to be followed, that you are going to specify what you are going to do, and then do it. And that is where most of our troubles come from.

On dealing with user requirements:

Users are interested in systems requirements and buy systems in that way. But that implies that they are able to say what they want. Most of the users aren’t able to.

We should have feedback from users early in the design process.

On programmer performance (compare with the essay Great Hackers by Paul Graham):

I have a question on the huge range of variability of programmer performance. Are similar ranges found in other engineering areas? [..] The variation range in programming is in fact greater than in other fields.

On the production of software:

We are starting gradually, and building up. My motto is ‘do something small, useful, now.

Large systems must evolve, and cannot be produced all at one time. You must have an initial small core system that works really well.

On testing software:

System testing should be automated as well. A collection of executable programs should be produced and maintained to exercise all parts of the system. [...] As an output of a test validation run, each test should list the modules it has exercised, and as well, should list the interfaces and tables it has tested. It is important to document success, as well as failure.

A lot of people are thinking about how we can reduce household energy use these days. Changing household energy use can be a challenge because to most people there is no working feedback loop available. Energy use is invisible. My own feedback loop consists of getting a monthly invoice displaying the cost I had for the previous month (e.g. “278 SEK”). That is a lousy feedback loop if you want to change behavior. I can not even see how much energy I used. Not that it matters because that information is useless without knowing what would be a reasonable level to compare with. And getting the information once a month does not provide a feedback loop for me to see the effect of my immediate behavior.

A basic model of a feedback loop for a system looks like this:

Based on a desired state (a reference value) a controller operates on a system and adjust control behavior based on the output from a sensor to make the system align the output to the desired state. For this to work there are a couple of things required:

1. We must know what the desired state is. Without it we don’t know if the output from the system is good or bad.
2. We must know the system output. Without the output we can’t calculate the measured error between output and the desired state.
3. The controller must be able to provide system input to control the system and its parts in a predictable way.
4. The delay between the actions of the controller and the sensor feedback needs to be short so that the result of different actions are visible.

Applying this basic model to the typical feedback loop for household energy use we get:

The “system” in this model would be your household. It consists of several things that use energy (refrigerator, TV, other appliances…). You control it by operating various appliances, setting the refrigerator temperature and making purchasing decisions for new hardware that affects energy use. You get the energy bill once a month with the amount of energy used.

As you can see there are several problems. I don’t know the desired energy level. Is my household below or above the average for similar households? If I switch off the TV instead of keeping it on standby, would it matter? Will setting a higher temperature in the refrigerator impact my energy use? Since I can’t see the immediate effects of my behavior there are no drivers to change it.

Real-time feedback and knowledge about the desired state

A first step to  influence behavior would be to provide real-time feedback and information about a desired state. That would make it possible for you to try various control actions and see what implications they have on the system. There are a lot of third party solutions to get real-time feedback e.g. Wattson, TED-5000 and so on, but from what I have seen there are very few options to get information about a desired state. Getting more data about desired state can in part be accomplished by sharing information about your household and its energy use with others.

But, to be able to really affect behavior we need to increase the resolution of the parts of the system (at least sections or types of energy use in the household) by using multiple sensors and comparing the output with multiple desired states. This may quickly become complicated for an individual and a reasonable next step is to automate the controller function (you) by allowing automated control actions on parts of the system (e.g. adjusting your refrigerator temperature based on data about its contents, model, and similar household data).

I guess we have some work to do before we can get a better feedback loop in place. What’s in it for the energy companies? Will we have to install our own real-time meters?

(Thank you to Daniel Brolund who provided interesting insights into control theory last week).

1. The benefits appear when a vocabulary is used.
2. An important factor is how a vocabulary and its parts can be published, discovered and referenced.
3. There needs to be one and only one vocabulary artifact.
4. The vocabulary artifact must be machine processable to allow aggregation and automatic generation of other artifacts.

Background

The need for interoperability arise in scenarios with many contributing actors. In the old days this was solved by hiring a (large) consulting organization that (hopefully) had experience from glueing together many products from their vast product portfolio. Everyone involved was happy if it worked. Being able to swap out a part of this system or connect it to a third party that had a different technical platform was considered a risk.

Today more work is being done but in my experience there are still issues in keeping it all together. Domain experts create terminology in Word documents and architects draw UML class diagrams. They all talk about the domain but each type of artifact does not describe the domain sufficiently. In a different room developers merge domain concepts with coding practices to form software that may express fragments of a domain.

Little work is done to define vocabularies in a way that worked both for software developers and for domain experts. Typically you would find a Word document with a flat terminology that was ignored by the software developers, or, you could find class diagrams in UML or software documentation that was impossible for domain experts to understand. And when you look at the code or interfaces of the system there are no references to the correct item in a vocabulary.

How can we make them work together?

Requirements and challenges

I like to think of semantic interoperability as a state where the same cognitive processes kicks in for two persons studying a specific artefact from the domain they are working in.

In order to work with this in a structured way we need to simplify it. One way to look at interoperability can be found in the draft of the European Interoperability Framework version 2. It introduces four interoperability levels. If we remove the European cross border services context they can be described in more general terms:

1. Technical interoperability: issues involved in linking computer systems and services together (e.g. transport and serialization of data).
2. Semantic interoperability: the meaning of information specified in a way understood by all parties (e.g. definitions, relations and structure of terms used to describe data).
3. Organizational interoperability: coordination of processes in the context where data is used/transformed. (e.g. shared definitions of the roles, responsibilities and interactions of/between participants).
4. Legal interoperability: shared interpretation and understanding of laws regulating information exchange and cooperation (e.g. can we transfer information about people? What about privacy?).

There are other models that could be used to discuss levels of interoperability (e.g. LCIM – Levels of Conceptual Interoperability Model) but let’s stick with the EIF 2 model above.

If you have a scenario with only two parties everything is simple. You could just do the minimum amount of work to make sure systems talk to each other and that people share a common image of the various levels of interoperability. Enabling more parties to participate requires more work though.

If you have a scenario with many parties (where some may not yet exist) you have to create a framework describing how interoperability should work by harmonizing a set of solutions at each level of interoperability. In a dictatorship this is easy. Just evaluate a few solutions, choose one and make everyone use it. In other contexts it is more complicated.

If we consider the EIF 2 model above as a set of requirements we should also make sure that the solutions we propose:

1. scale down: If the parties (organizations, departments etc) vary greatly in size (as they will if the number is sufficiently big) some may consider that the solution you propose in a framework is unnecessary complex. They are right. In order to maximize the number of parties and processes that can participate in an interoperability scenario you will have to make sure the proposed solutions scale down.
2. are technology neutral: Given a large number of organizations you will find almost every technology platform you can imagine. There will also be a number of ways to send and receive information. Some like to force a specific set of transport mechanisms as part of a interoperability framework but this will quickly put you on a path where you can’t make use of new standards and technologies. Forcing participating organizations to use a specific tool or platform will fail.
3. give a reasonable amount of flexibility: This may require the framework to contain partly overlapping standards.
4. facilitate an emerging semantic interoperability: The idea of having a central organization defining everything only works in a dictatorship. We should enable the decentralisation of participating in vocabulary work.
5. are discoverable and referencable: If the artefacts that are created are difficult to find for a newcomer you can give up now. By lowering the threshold to participate more parties can be involved and more information can be reused.

This proposal is aimed at semantic interoperability only.

A lightweight semantic interoperability framework proposal

1. The Web Ontology Language (OWL) is used to express vocabularies

The primary reason for selecting OWL is not the features it has to express domain concepts but rather the possibilities of publishing it and referencing it on the web that come bundled with it. Using the web as a publishing platform enables us to make use of already existing standards for referencing a vocabulary and its parts.

Although plagued by research projects of poor usability and lack of longevity there are several tools to create vocabularies in OWL. The framework does not specify the process for creating a vocabulary as that may hinder small organizations form participating.

2. Vocabularies are published properly on the web

Putting an OWL vocabulary on the web means that it can easily be discovered, referenced and transformed. Not only will it help your own organization but it will lower the threshold for others who want to use parts of it in their own work. As soon as that happens you have the first step towards semantic interoperability.

By “published properly” I mean that you should choose your URI for the vocabulary wisely as it will be the reference string used in software interfaces and other artifacts down the road. Many get web publishing wrong by scrapping one of the best features: long term referenceability, as they dump information tightly coupled to the publishing tool they are using. When you change the tool (e.g. a content management system) you may break references for everyone else. Don’t do that.

To make the vocabulary human readable (nobody wants to read OWL/RDF-schema) generate at least an HTML-version of the vocabulary for users pointing their browser to the URL of the vocabulary and a class hierarchy overview to help users understand relations between things.

3. Use a standardized way of referencing vocabularies in other artifacts

Although OWL may seem to be only for semantic web scenarios it can be used in other areas. By standardizing how references from WSDL and XML schema are made it is possible to keep an unbroken chain of references. SAWSDL provides a mechanism to reference e.g. a class in the vocabulary. This makes it possible for developers to refer to the vocabulary in a consistent way from XML schemas and WSDL files and provides a way to keep the vocabulary closely connected to API implementations.

One important area missing is how to point to vocabularies from code. Code is a design artifact for a system and as such it is important that it is possible to keep an unbroken reference chain to it too. But we’ll leave that open for now.

4. Aggregate all vocabularies on a central website

Although vocabularies can and should be published under a URI in control by the authoring organization they should be aggregated on a central searchable website. This is easy as they are machine readable and will serve two purposes:

1. It helps people discover other vocabularies before creating their own. This is an important step in creating an environment for an emerging semantic interoperability.
2. The aggregated material will become a tool to find areas where harmonization efforts should be focused. By querying the aggregated information it is possible to find vocabularies that define similar things. Multiple classes called “Person”? Get going.

5. Express provenance in the vocabulary

Provenance provides a foundation for assessing authenticity and enabling trust. By standardizing information about provenance you also facilitate interaction between parties involved in the domain.

What do you think?

I believe this could be a lightweight approach to work with semantic interoperability in a way that scales (both up and down) and that provides for an emerging semantic interoperability over time. I guess it could work for government scenarios on country scale as well as for a small organization.

The only thing I miss is a standardized way of referencing a vocabulary from code.

Please leave your feedback.

If you try “:let s = 0.5″ in MacVim and get an error you are affected. This happens when you have launched MacVim from Finder. In this case MacVim does not pick up all the locale settings properly for some reason. This includes the settings for the separator character in floating point numbers which may make all vim scripts with a dot separator (“0.05″) fail.

1. Install the FuzzyFinder script (including the l9 library).

2. In your .gvimrc add the following lines. This will deconnect the Apple-T from File->New tab and instead invoke FuzzyFinder with a wildcard for subdir search:

if has("gui_mac") || has("gui_macvim")
  macmenu &File.New\ Tab     key=<Nop>
  map <silent><D-t> :FufFile **/<CR>
endif

Now, start MacVim from terminal with “mvim” instead of double-clicking the icon in Finder and try Apple-T. FuzzyFinder should now work correctly. Here are some of the discussions for reference:

• Error in vim-l9
• FuzzyFinder and error E806: using Float as a String

On Sunday there were 12 teams presenting their apps. Some of the teams had used public transport data made available by the Viktoria Institute but many were scraping government websites for lack of proper API:s. In the 24 hours set aside for the competition the following apps were created:

• Slagkryssaren: Valkoll – an Android app to see election results where you are right now
• Slumptal: Lokalpolitik – geocoding city council docs to be able to present relevant decisions in your neighborhood.
• New and shiny things: @Myndighetskoll – a twitter bot that ansers questions about government agencies
• Popdevelop: Kollektivt.se – an API service and realtime display of buses in the city of Malmö
• Facebookpolitik: Svenska Riksdagen på Facebook – pushing government decisions and initiatives to Facebook
• First Blood: Engagemangsindex – visualization of government issues
• Boreda: Boreda @ Codemocracy – using public transport data as a selection criteria in finding properties for sale.
• Exjobbarna: CrimeTracker – a Windows mobile app to see crime stats
• Anyro: Tågtider till Android – an adroid app for train data
• Existic: Riksdagsdata i Ply – an extension to PLy to do drill-down in government data.
• Akvo Phone: Georapport – an Android app that geo-encodes and signs a picture that could be used in development aid projects or fix my street-scenarios.
• Kreativ Stuga: Edemokrat.se – An app that connects government documents in a way that makes it easy to navigate them.

More info at the official Codemocracy website. Many CC pictures are in the Codemocracy Flickr group.

I am already looking forward to the next event!