Support Us

You are browsing the archive for Visualization.

120+ CKAN Portals in the Palm of Your Hand. Via the Open Data Companion (ODC)

Osahon Okungbowa - September 24, 2015 in Community, Data, News, Visualization

CKAN is a powerful open-source data portal platform which provides out-of-the-box tools that allow data producers to make data easily accessible and reusable by everyone. Making CKAN as Free and Open Source Software (FOSS) has been a key factor in helping grow the availability and accessibility of open data across the Internet.

The emergence of mobile devices and the mobile platform has led to a shift in the way people access and consume information. Popular consensus  and reports show that mobile device usage and time spent on mobile devices are rapidly increasing. This means that mobile devices are now one of the fastest and easiest means of accessing data and information. Yet, as of now, open data lacks a strong mobile presence. Read the rest of this entry →

Implementing VectorTiles Preview of Geodata on HDX

chadhendrix - September 16, 2015 in Extensions, Feature, Visualization

This post is modified version of a post on the HDX blog.  It is modified here to highlight information of most interest to the CKAN community.  You can see the original post here.

Humanitarian data is almost always inherently geographic. Even the data in a simple CSV file will generally correspond to some piece of geography: a country, a district, a town, a bridge, or a hospital, for example.

HDX has built on CKAN’s preview capabilities with the ability to preview large (up to 500MB) vector geographic datasets in a variety of formats.  Resources uploaded (or linked) to HDX with the format strings ‘geojson’, ‘zipped shapefile’, or ‘kml’ will trigger the creation of a geo preview. Here is an example showing administrative boundaries for Colombia:


image00

To minimize bandwidth in the interest of often poorly-connected field locations, we built the preview from vector tiles. This means that details are removed at small scales but will reappear as you zoom in.

The preview is created only for the first layer it encounters in a resource. If the resource contains multiple layers, the others will not show up. For those cases, you can create separate resources for each layer and they will be available in the preview. Multiple geometry types (polygon + line, for example) in kml or geojson are not yet supported.

Implementation

It’s a common problem in interactive mapping: to preview the whole geographic dataset, we would need to send all of the data to the browser, but that can require a long download or even crash the browser. The classic solution is to use a set of pre-rendered map tiles — static map images made for different zoom levels and cut into tiny pieces called tiles.  The browser has to load only a few of these pieces for any given view of the map. However, because they are just raster images, the user cannot interact with them in any advanced way.

We wanted to maintain interactivity with the data, eventually having hover effects or allowing users to customize styling, so we knew that we needed a different approach. We reached out to our friends at Geonode who pointed us to the recently developed Vector Tiles Specification.

The vector tile solution is a similar approach to traditional map tiles, but instead of creating static image tiles, it involves cutting the geodata layer into small tiles of vector data. Each zoom level receives a simplification (level of detail, or LoD) pass, which reduces the number of vertices displayed, similar to the way that 3D video games or simulators reduce the number of polygons in distant objects to improve performance. This means that for any given zoom level and location, the browser needs to download only the vertices necessary to fill the map.  You can learn more about how vector tiles work in this helpful FOSS4G NA talk from earlier this year.

Because vector tiles are a somewhat-new technology, there wasn’t any off-the-shelf framework to let us integrate them with our CKAN instance. Instead, we built a custom solution from several existing components (along with our own integration code):

Our architecture looks like this:

image03

The GISRestLayer orchestrates the entire process by notifying each component when there is a task to do. It then informs CKAN when the task is complete, and a dataset has a geo preview available.  It can take a minute or longer to generate the preview, so the asynchronous approach — managed through Redis Queue (RQ) — was essential to let our users continue to work while the process is running. A special HDX team member, Geodata Preview Bot, is used to make the changes to CKAN. This makes the nature of the activity on the dataset clear to our users.

Future development

This approach gives HDX a good foundation for adding new geodata features in the future. We will be conducting research to understand what users think is important to add next. Here are some initial new-feature ideas:

  • Automatically generate additional download formats so that every geodataset is available in zipped shapefile, GeoJSON, KML, etc.
  • Allow the contributing user to specify the order of the resources in the map legend (and therefore which one appears by default).
  • Allow users to preview multiple datasets on the same map for comparison.
  • Automatically apply different symbol colors to different resources in the same dataset.
  • Allow users to style the geographic data, changing colors and symbols.
  • Allow users to configure and embed maps of their data in their organization or crisis pages.
  • Provide OGC-compliant web services of contributed datasets (WFS, WMS, etc.).
  • Allow external geographic data services (WMS, WFS, etc) to be added to a map preview.
  • Make our vector tiles available as a web service.

If any of these enhancements sound useful or you have new ideas, send us an email at hdx.feedback@gmail.com. If you have geodata to share with the HDX community, start adding your data here.

We would like to say a special thanks to Jeffrey Johnson who pointed us toward the vector tiles solution and to the contributors of all the open source projects listed above! In addition to GISRestLayer, you’ll find the rest of our code here.

Presenting public finance just got easier

Tryggvi Björgvinsson - March 20, 2015 in Extensions, Feature, Featured, Releases, Visualization

mexico_ckan_openspending

CKAN 2.3 is out! The world-famous data handling software suite which powers data.gov, data.gov.uk and numerous other open data portals across the world has been significantly upgraded. How can this version open up new opportunities for existing and coming deployments? Read on.

One of the new features of this release is the ability to create extensions that get called before and after a new file is uploaded, updated, or deleted on a CKAN instance.

This may not sound like a major improvement  but it creates a lot of new opportunities. Now it’s possible to analyse the files (which are called resources in CKAN) and take them to new uses based on that analysis. To showcase how this works, Open Knowledge in collaboration with the Mexican government, the World Bank (via Partnership for Open Data), and the OpenSpending project have created a new CKAN extension which uses this new feature.

It’s actually two extensions. One, called ckanext-budgets listens for creation and updates of resources (i.e. files) in CKAN and when that happens the extension analyses the resource to see if it conforms to the data file part of the Budget Data Package specification. The budget data package specification is a relatively new specification for budget publications, designed for comparability, flexibility, and simplicity. It’s similar to data packages in that it provides metadata around simple tabular files, like a csv file. If the csv file (a resource in CKAN) conforms to the specification (i.e. the columns have the correct titles), then the extension automatically creates the Budget Data Package metadata based on the CKAN resource data and makes the complete Budget Data Package available.

It might sound very technical, but it really is very simple. You add or update a csv file resource in CKAN and it automatically checks if it contains budget data in order to publish it on a standardised form. In other words, CKAN can now automatically produce standardised budget resources which make integration with other systems a lot easier.

The second extension, called ckanext-openspending, shows how easy such an integration around standardised data is. The extension takes the published Budget Data Packages and automatically sends it to OpenSpending. From there OpenSpending does its own thing, analyses the data, aggregates it and makes it very easy to use for those who use OpenSpending’s visualisation library.

So thanks to a perhaps seemingly insignificant extension feature in CKAN 2.3, getting beautiful and understandable visualisations of budget spreadsheets is now only an upload to a CKAN instance away (and can only get easier as the two extensions improve).

To learn even more, see this report about the CKAN and OpenSpending integration efforts.

IVPR / OIC bringing the Weave visualization platform to CKAN

Rufus Pollock - June 5, 2014 in Community, Featured, Visualization

The CKAN Association is always looking to further enhance the visualization options available for the CKAN data management platform. We are therefore delighted to connect with the Institute for Visualization and Perception Research (IVPR) at the University of Massachusetts Lowell and the Open Indicators Consortium who have led the development of Weave (Web-based Analysis and Visualization Environment), a richly featured, open source data visualization platform for the web.

CKAN-WEAVE_demo

WEAVE running using CKAN

Weave is a state-of-the-art system which provides software tools that researchers, educators, analysts, trainers, students, and the general public can use to interactively analyze, visualize and disseminate remote, local or distributed data. Weave is the IVPR’s fifth-generation visualization system and incorporates 20 years of research and embedded patented algorithms. Previous generations of this software were desktop versions. Both the earlier versions and Weave were designed to solve complex problems in a variety of application areas including drug discovery, medicine, economics, and national security. The bulk of Weave’s source code is released under the GNU General Public License (GPLv3), and its core is released under the Mozilla Public License (MPLv2).

Weave_obesity2

To facilitate the connection between CKAN and Weave, IVPR recently developed an adaptor for Weave that facilitates bringing CKAN-indexed CSV or XLS files directly into Weave for creating visualizations. The Connecticut Data Collaborative, a founding OIC member, is currently building a site that will integrate CKAN and Weave.

UK_elections2

For more information about Weave you can:

  • Visit the Weave wiki which provides everything needed to install and explore Weave.
  • See the many OIC websites using Weave here
  • We will keep you posted on new developments as they arise. In the meantime, interested developers should join the Weave-users and Weave-dev google groups to connect with other developers working with Weave.