We’ve already written about why we wanted to map suppliers, factories and making facilities and also covered the way we’re gathering the data and how we’re storing it in Open Street Map.

Collecting the data is only half the story. What do we do with it once it’s gathered? How can people use it? How can we share it to make that easier?

We chose to use Open Street Map to store the data in part because that gives us access to a raft of existing tools and guides to use the information collected. Some of the things we’re mapping—particularly the trade counters, but also some of the factories and machine shops—will show up in the standard rendering on the Open Street Map website, but many of them are too specialist to be displayed (including all the data in Open Street Map on the standard map would make it far too crowded!)

That extra detail is still available, you just need to know how to find it.

Querying the Data

One of the easiest and most powerful tools for querying the OSM dataset is Overpass Turbo. It lets you run queries to retrieve different datasets, which can then be displayed on a map or returned as JSON-formatted text. You have to write the queries in the Overpass Query Language, but it’s not too complicated to get to grips with. There is a wizard built into the Overpass Turbo website to help with simple queries and for more complicated work the API/Lanugage Guide is a good place to start. We also created a few example queries to pull out the Indie Manufacturing data which we shared at the end of that blog post.

The Indie Manufacturing Map

To save people having to query the data themselves, we also built an Indie Manufacturing map page on this site. It took a couple of iterations to get to the map you see now which, although there are things we’d like to improve (we’ll return to that), was something we’re happy with as a starting point.

Given that indie.mcqn.com is built with Jekyll I started out with the jekyll-mapping plugin.

It was nice and easy to get up and running, but seems a bit more focused on linking to individual posts on the site which have an associated location. That’s to be expected, and wasn’t a deal-breaker—I extended the code to pull in the list of markers from the Jekyll data folder. However, that was what sent me looking for an alternate approach. If your dataset isn’t too big then this approach would be fine (and was for the current makerspace, machine shop, and factory sets); but there are a lot of trade counters in Open Street Map already, and just to cover the UK would mean processing half a MB of data each time the map was loaded, which made the map rather slow to render.

In looking for other solutions I came across a couple of helpful blog posts from Mappa Mercia (the West Midlands OSM community). They explain how to use uMap to build the map (http://www.openstreetmap.org/user/ybon/diary/23611 gives a good overview of uMap’s capabilities), and then how to tie it into Overpass Turbo to query the dataset on demand.

That would be the ideal solution: the data being displayed would always be up to date, and by only requesting the data for the area of map being viewed it would keep the load on the servers to a minimum. However, in practice the queries to Overpass Turbo take too long to complete, often resulting in error messages and a poorly populated map.

We ended up with a mixture of those two approaches.

For the smaller datasets—the makerspaces/fablabs/etc.; factories; and machine shops—I set up a script which runs periodically to query Overpass Turbo and download the latest information to a file on the Indie Manufacturing server. That’s used to populate those layers on the map (built with uMap), so the data loads quickly at the expense of being a bit stale at times; however, it doesn’t change quickly enough for an update frequency of every-few-hours to be a problem.

The trade counter data, which is much more widely used and therefore a much bigger dataset, is turned off by default (otherwise it would tend to throw an error message at the user when they arrived at the map). It is hooked up to query Overpass Turbo in real time, and just download the relevant data for the geographic area the user is looking at.

For bigger areas (anything beyond a single city, in practice) when it’s turned on the trade counter queries will tend to time out before they’ve gathered that data. If you only want to find the trade counters near to you, however, that’s usually a small enough data set that it will work. Not ideal, but we thought it was an acceptable compromise.

Which brings us to…

Making Things Better

How might we improve things? Both for the Indie Manufacturing map and for anyone else wanting to use maps to help their community.

Easier Submission of Data

By far the biggest challenge for the Indie Manufacturing map was getting people to contribute data. Everyone is busy and the time required (or perceived time required) to learn how to use a new system is too great a barrier to share what you know.

The New Cloud Atlas editor that Tim Waters built is a good example of how it’s possible to modify the standard OSM editor tools to suit a particular subset of mapping. However, that requires pretty good OSM developer chops to deploy at present. Making that easier would be good.

Even simpler could be an online form that lets someone find an existing item on OSM or easily add a new point to it, and then provide a set of useful tags - ideally hidden behind friendlier descriptions provided by the map-maker.

I think either of those approaches would need to find a way round the requirement for an OSM account, if they’re to achieve making it easy to contribute. That runs slap bang into the OSM community’s (completely valid) concerns about data provenance.

Maybe any of those submissions could include some sort of data source tag—which seems a well-established convention in OSM for bulk imports—so that consumers of the data would be able to make decisions on how much to trust the information.

It would also be useful to build more tools to help people with a foot in each of those communities. For example, here I’m both a member of the maker community wanting to map making facilities and the OSM community.

An ability to monitor all the changes which include a certain set of tags would let me keep an eye on what was being contributed, and then intervene and fix up the data or ask further questions if I spot anything amiss. I’m surprised that this doesn’t already exist, but I’ve failed to find it if it does.

If allowing semi-anonymous contributions is too risky for OSM then some sort of approval queue could help. That makes the list of curators of submissions a potential bottle-neck, but being able to run through submissions choosing from “approve”, “deny” and “push to ‘needs more detailed checking’ queue” could be good enough.

Better Maps

On the display side, uMap and Overpass Turbo are already 90% of the way there and it would be good to find ways to help those projects.

Any project looking to fund building a similar map system might find that spending some of the money on more kit for Overpass Turbo is a better investment (for all of us).

However, there are many situations where we don’t need right-up-to-the-minute data and occasional queries to Overpass Turbo which are then cached would reduce load on their servers with negligible impact on the viewers experience.

We’ve shown that for small datasets that’s already possible.

A caching service could be built to slurp in data from Overpass Turbo and then serve it out again to uMap using the same query syntax that’s used to connect the two of them together at present. That would allow the map-maker to insert the small caching server as a shim between the two services, reducing the load on Overpass Turbo and providing data more quickly to uMap as there’d be less of it to sift through in response to a query.

Even nicer would be if the remote data syntax used by uMap was extended to allow something like tile-boundary tags as well as the existing {south}, {west}, {north}, {east}. That would let the map-maker create a set of static files with the data in, much like tilesets are generated at present, thus removing the need to run a database or a particular web language on your server. You’d be able to run a script to query Overpass Turbo and generate a bunch of JSON files to then upload whenever you wanted to update the map. Obviously, more advanced users could set up a cron job on the server to similarly run the update there to automate things.

That would solve getting the data into uMap for it to show. There are a couple of improvements that would make the display even better.

uMap has a template system which lets you use some of that data imported from Open Street Map (or wherever) to populate the pop-up when the viewer clicks on a marker. We use that to show the name and website of the firm, plus some details of the materials they use and such-like from the OSM tags.

That works well, but results in text like “print_shop” rather than “print shop” as it’s just using the content of the OSM tag. If there was a way to conditionally include bits of templates (so we could add a heading of “Materials”, for example, but only if that tag was present) and also provide a set of mappings to let us define more user-friendly names, that would make the maps even slicker.

Finally, and this might be beyond a general solution like uMap (or at least, I’ve not got an easy suggestion of how to implement it…), maps such as the Indie Manufacturing map would be much improved if the viewer could easily filter the results shown.

Coarse-grained filtering can be done by putting the different sets of data onto different layers, as we’ve done with the factories vs. machine shops, etc. However, that doesn’t let the user drill into the data on a single layer. For the Indie Manufacturing map it would be really useful if you could limit the list of machine shops to just the ones offering laser-cutting, for example. That information is held in the tags, so you’d need some way to be able to tell the map interface which tags can be filtered and then choose values or provide a text box to enter the search terms.

The Open Street Map tools make all sorts of mapping activities lots easier. With the addition of a few more pieces it could form an even better toolkit for many more communities of interest.