Support Us

You are browsing the archive for Releases.

Easy “one-click” deployment of CKAN 2.0 on VM Depot

Gavin Chait - June 13, 2013 in Deployments, News, Releases


CKAN 2.0 on VM Depot

CKAN 2.0 on VM Depot

Along with our release of CKAN 2.0, we have been building simple methods for faster deployment. Our intention is to get you up and running quickly and easily.

VM Depot is “a community-driven catalog of preconfigured operating systems, applications, and development stacks that can easily be deployed on Windows Azure.” You will find a straightforward deployment script and a step-by-step set of instructions.

We would like to thank the team at Microsoft Open Technologies which made this possible and we hope that you find this a simple way to develop test instances and custom extensions, as well as new stand-alone open data services.

We would love to hear about the sites you set up there, so please let us know and we can feature them on ckan.org.

U.S. government’s data portal Data.gov relaunched on CKAN

Irina Bolychevsky - May 23, 2013 in Deployments, News, Releases

Today, we are excited to announce that our work with the US Federal Government (data.gov) has gone live at catalog.data.gov! You can also read the announcement from the data.gov blog with their description of the new catalog.

Catalog.Data.gov

The Open Knowledge Foundation’s Services team, which deploys CKAN, have been working hard on a new unified catalog to replace the numerous previously existing catalogs of data.gov. All geospatial and raw data is federated into a single portal where data from different portals, sources and catalogs is displayed in a beautiful standardized user interface allowing users to search, filter and facet through thousands of datasets.

This is a key part of the U.S. meeting their newly announced Open Data Policy and marks data.gov’s first major step into open source. All the code is available on Github and data.gov plan to make their CKAN / Drupal set-up reusable for others as part of OGPL.

As one of the first major production sites to launch with the shiny new CKAN 2.0, data.gov takes advantage of the much improved information architecture, templating and distributed scalable authorization model. CKAN provides data.gov with a web interface for over 200 publishing organizations to manage their members, harvest sources and datasets – supporting requirements being outlined in Project Open Data. This means that agencies can maintain their data sources individually, schedule regular refreshes of the metadata into the central repository and manage an approval workflow.

There have been many additions to CKAN’s geospatial functionality, most notably a fast and elegant geospatial search:

Geospatial search filter

We have added robust support for harvesting FGDC and ISO 19139 documents from WAFs, single spatial documents, CSW endpoints, ArcGIS portals, Z39:50 sources, ESRI Geoportal Servers as well as other CKAN catalogs. This is available for re-use as part of our harvesting and spatial extensions.

Most importantly, this is a big move towards greater accessibility and engagement with re-users. Not only is metadata displayed through a browsable web interface (instead of XML files), there is now a comprehensive CKAN API with access to all web functionality including search queries and downloads which respects user and publisher permission settings. Users can preview the data in graphic previews as well as exploring Web Map Services, whilst the dataset page provides context, browsable tags, dataset extent, and maintainers.

Web Map Service

As data.gov invites users to get involved and provide feedback, we would also like to say that we are really excited about CKAN’s future. We have a very active mailing list, new documentation for installing CKAN and ways to contribute to the code for anyone wanting to join the CKAN community.

If you’re launching a CKAN portal soon or have one we don’t know about, let us know and we’ll make sure to add you to our wall of awesome!

Announcing CKAN 2.0

Mark Wainwright - May 13, 2013 in News, Releases

Today we are very happy and excited to announce the final release of CKAN 2.0. This is the most significant piece of CKAN news since the project began, and represents months of hectic work by the team and other contributors since before the release of version 1.8 last October, and of the 2.0 beta in February. Thank you to the many CKAN users for your patience – we think you’ll agree it’s been worth the wait.

[Screenshot: Front page]

CKAN 2.0 is a significant improvement on 1.x versions for data users, programmers, and publishers. Enormous thanks are due to the many users, data publishers, and others in the data community, who have submitted comments, code contributions and bug reports, and helped to get CKAN to where it is. Thanks also to OKF clients who have supported bespoke work in various areas that has become part of the core code. These include data.gov, the US government open data portal, which will be re-launched using CKAN 2.0 in a few weeks. Let’s look at the main changes in version 2.0. If you are in a hurry to see it in action, head on over to demo.ckan.org, where you can try it out.

Summary

CKAN 2.0 introduces a new sleek default design, and easier theming to build custom sites. It has a completely redesigned authorisation system enabling different departments or bodies to control their own workflow. It has more built-in previews, and publishers can add custom previews for their favourite file types. News feeds and activity streams enable users to keep up with changes or new datasets in areas of interest. A new version of the API enables other applications to have full access to all the capabilities of CKAN. And there are many other smaller changes and bug fixes.

Design and theming

The first thing that previous CKAN users notice will be the greatly improved page design. For the first time, CKAN’s look and feel has been carefully designed from the ground up by experienced professionals in web and information design. This has affected not only the visual appearance but many aspects of the information architecture, from the ‘breadcrumb trail’ navigation on each page, to the appearance and position of buttons and links to make their function as transparent as possible.

[Screenshot: dataset page]

Under the surface, an even more radical change has affected how pages are themed in CKAN. Themes are implemented using templates, and the old templating system has been replaced with the newer and more flexible Jinja2. This makes it much easier for developers to theme their CKAN instance to fit in with the overall theme or branding of their web presence.

Authorisation and workflow: introducing CKAN ‘Organizations’

Another major change affects how users are authorised to create, publish and update datasets. In CKAN 1.x, authorisation was granted to individual users for each dataset. This could be augmented with a ‘publisher mode’ to provide group-level access to datasets. A greatly expanded version of this mode, called ‘Organizations’, is now the default system of authorisation in CKAN. This is much more in line with how most CKAN sites are actually used.

[Screenshot: Organizations page]

Organizations make it possible for individual departments, bodies, groups, etc, to publish their own data in CKAN, and to have control over their own publishing workflow. Different users can have different roles within an Organization, with different authorisations. Linked to this is the possibility for each dataset to have different statuses, reflecting their progress through the workflow, and to be public or private. In the default set-up, Organization user roles include Members (who can read the Organization’s private datsets), Editors (who can add, edit and publish datasets) and Admins (who can add and change roles for users).

More previews

In addition to the existing image previews and table, graph and map previews for spreadsheet data, CKAN 2.0 includes previews for PDF files (shown below), HTML (in an iframe), and JSON. Additionally there is a new plugin extension point that makes it possible to add custom previews for different data types, as described in this recent blog post.

[Screenshot: PDF preview]

News feeds and activity streams

CKAN 2.0 provides users with ways to see when new data or changes are made in areas that they are interested in. Users can ‘follow’ datasets, Organizations, or groups (curated collections of datasets). A user’s personalised dashboard includes a news feed showing activity from the followed items – new datasets, revised metadata and changes or additions to dataset resources. If there are entries in your news feed since you last read it, a small flag shows the number of new items, and you can opt to receive notifications of them via e-mail.

Each dataset, Organization etc also has an ‘activity stream’, enabling users to see a summary of its recent history.

[Screenshot: News feed]

Programming with CKAN: meet version 3 of the API

CKAN’s powerful application programming interface (API) makes it possible for other machines and programs to automatically read, search and update datasets. CKAN’s API was previously designed according to REST principles. RESTful APIs are deservedly popular as a way to expose a clean interface to certain views on a collection of data. However, for CKAN we felt it would be better to give applications full access to CKAN’s own internal machinery.

A new version of the API – version 3 – trialled in beta in CKAN 1.8, replaced the REST design with remote procedure calls, enabling applications or programmers to call the same procedures as CKAN’s own code uses to implement its user interface. Anything that is possible via the user interface, and a good deal more, is therefore possible through the API. This proved popular and stable, and so, with minor tweaks, it is now the recommended API. Old versions of the API will continue to be provided for backward compatibility.

Documentation, documentation, documentation

CKAN comes with installation and administration documentation which we try to keep complete and up-to-date. The major changes in the rest of CKAN have thus required a similarly concerted effort on the documentation. It’s great when we hear that others have implemented their own installation of CKAN, something that’s been increasing lately, and we hope to see even more of this. The docs have therefore been overhauled for 2.0. CKAN is a large and complex system to deploy and work on improving the docs continues: version 2.1 will be another step forward. Where people do run into problems, help remains available as usual on the community mailing lists.

… And more

There are many other minor changes and bug fixes in CKAN 2.0. For a full list, see the CKAN changelog.

Installing

To install your own CKAN, or to upgrade an existing installation, you can install it as a package on Ubuntu 12.04 or do a source installation. Full installation and configuration instructions are at docs.ckan.org.

Try it out

You can try out the main features at demo.ckan.org. Please let us know what you think!

Translators needed!

Sean Hammond - March 11, 2013 in Releases

We’re preparing to release CKAN 2.0 in a few weeks time, and we need help with translations.

CKAN’s user interface is available in over 20 languages. The new version introduces over 450 new words and strings in the interface, and they all need translating before it’s released. If you speak one of the relevant languages, you can join that language team to help with the translation effort.

If you are a speaker of a language where CKAN isn’t available yet, this would also be a great time to start a translation project for it!

If you can help with translating the strings and keeping CKAN open to people across the world, please sign up here to contribute.

CKAN 2.0 beta has arrived!

Irina Bolychevsky - February 27, 2013 in News, Releases

Work on what has become the 2.0 release started back in June last year when we decided that a more user friendly cleaner simpler CKAN was the next step to raise the profile and usefulness of open data portals.

We went back to basics and redesigned the theme to focus on the core parts of CKAN: discoverability of data for end users and better data management tools for publishers.

Instructions to install or upgrade are below and you can go play around with it on beta.ckan.org – just create an account to try out all the funcitonality.

You’ll see a more engaging search page that highlights the use of facets and search result sorting as well as the keyword search; a redesigned dataset page showcasing resources, tags, activity streams and re-use of data (the “related” tab); a new multistep dataset registration process to guide publishers as well as a scalable authorization model for managing access rights to datasets (more info below under ‘Organisations’).

Highlights

Organisations

We’ve simplified the authorisation model for datasets. Previously a dataset was ‘owned’ by the user who added it to the portal – edit or admin access to the dataset had to be manually added one by one. This made it harder to distribute management of datasets for large portals with numerous publisher accounts for multiple publishing organizations (e.g. Departments or Agencies).

In 2.0, by default, datasets always belong to an organisation (e.g. Department of Health, Office of National Statistics etc) and access rights are automatically allocated depending on the organisation access level members have. So if your department has 1 admin, 5 editors (add/edit/delete datasets, but no member management access) and 10 members (can see org private datasets, member, editor, admin accounts, but not add or edit anything) then any dataset that one of the editors adds will be editable by any of the other 4 editors or the admin.

Screen Shot 2013-02-27 at 12.16.54

Activity Streams

Datasets, users and groups now generate activity streams that can be ‘followed’.

Group activity screenshot

Templating

We have switched our templating engine to use Jinja2, making it easier to customize CKAN. We’ve found this to be much more designer friendly and intuitive than Genshi. It’s also generally cleaner and renders faster. We’re also able to extend templates (for example by extensions) in a less invasive way than previously using the {% ckan_extends %} template tag.

Note: Jinja2 templates do not support the IGenshiStreamFilter extension point, so extensions should either extend templates or provide template helper functions via the ITemplateHelpers extension point.

Installing or upgrading

This first beta release is aimed mainly at developers, and is only available via source install. You can refer to the “Install from Source” instructions or the “Upgrading a source install” sections on the documentation depending on your needs.

In either case, make sure to point the repository to the 2.0 release branch (release-v2.0):

  • For first time source installs, in section 2b use:
pip install -e 'git+https://github.com/okfn/ckan.git@release-v2.0#egg=ckan'
  • For source install upgrades, in section 2 use:
git checkout release-v2.0

Note that this is a major version change and there are significant backwards incompatible changes, so if you are upgrading an existing instance make sure to backup your database before trying to upgrade it.

Our supported extensions are currently being upgraded, and the degree of support for CKAN 2.0b varies depending on the extension. If you are using custom extensions on your instance, it is likely that you will need some work to upgrade them to 2.0. We are working on new documentation explaining how to upgrade from 1.X to 2.0, but in meantime to test this beta release we recommend disabling all extensions and enabling them one by one to see which ones will require updating. In case of doubt, feel free to send an email to the mailing list or ask on the IRC channel.

As with all beta releases, you are bound to find bugs or things that need improvement. We are still working on the proper stable release, and collecting all pending issues on this milestone on GitHub Issues:

https://github.com/okfn/ckan/issues?milestone=1&state=open

We would love to hear any feedback and bug reports, feel free to send them to the mailing list or add an issue to GitHub.

What’s coming next

We’re now working on a dataset approval workflow for moving datasets from private to public state as well as support for bulk actions and harvesting management interfaces. We’ll also be adding tweaks and improvements as people use the beta and provide feedback, as well as activity stream support for organisations.

Check out the new-look publicdata.eu

Mark Wainwright - February 5, 2013 in Deployments, Feature, News, Releases


A beta version of CKAN 2.0 will be released soon, but meantime, a sneak preview has come online with a major upgrade of publicdata.eu. The site, run by the Open Knowledge Foundation as part of the LOD2 project, collects records together in one place from a number of data catalogues throughout the European Union. It is the first production site up and running the code that will become CKAN 2.0.

Personalisation

A significant new feature of CKAN 2.0 is its personalisation features, which allow you to ‘follow’ particular datasets, groups, and users. For example, logged-in users see a ‘Follow’ button when viewing a dataset. A dashboard with the activity and updates you’re interested in, as shown. Details of how the personalisation features work are likely to change, and feedback is welcome while we’re working on it.

[IMG: dashboard on publicdata.eu]

Linked data

Ivan Ermilov of the University of Leipzig, another LOD2 partner, has implemented a CSV to RDF converter that is integrated with publicdata.eu. This means that many datasets now have data available in RDF, the native format of Linked Data, in line with the whole aims of the LOD2 project. This addition means that the site does not only collect data from different European catalogues so that they can be searched together, but enhances them with RDF – opening up interesting possibilities for harvesting.

CKAN’s metadata records are themselves available in RDF and on publicdata.eu this is now exposed on the user interface, where each dataset page has buttons offering its metadata in RDF and JSON.

Interface

A major change under the surface is that in CKAN 2.0, the ‘templating’ system has changed, making it much easier to develop custom themes for CKAN-powered sites. publicdata.eu uses a brand-new theme using the new templating. Users might also notice improvements to the UI, with better signposting around the site setting the current page in context, as on the dataset page shown below. The biggest UI change in CKAN 2.0 is to the dataset creation process. Though this is not visible at publicdata.eu, curious readers may try it out at demo.ckan.org.

Each dataset page includes a link to the source from which it was harvested, and there are buttons for sharing the dataset on Google+, Facebook and Twitter.

[IMG: dataset page]

Upgrading the harvesters

Work is in progress on improving publicdata.eu’s harvesting system. This will be completed soon, enabling the harvesting to be brought up-to-date. Please bear with us while this happens and watch this space for more information!

CKAN 1.8 released

Adrià Mercader - October 22, 2012 in Feature, Releases

The CKAN team is pleased to announce the release of its new major
version, CKAN 1.8. Thanks to all the team and external contributors
who have made it possible!

This release includes new features like the ability to follow users
and datasets, and visualize their activity on a dashboard, and a
completely revamped version of the Datastore. The new version of
the Datastore is powered by PostgreSQL, which avoids having to install
extra dependencies and allows us to support more powerful features,
like a full SQL query interface. Users of the previous version of the
Datastore based on ElasticSearch can contact the mailing list for
details on how to migrate their data to the new version.

Apart from new features, we are working hard on internal refactorings
to make the CKAN code base more maintainable and provide a more solid
integration for extensions, making it easier to build them and upgrade
them between CKAN versions. Some of these changes have been included
in this release: have a look at the “API changes and deprecation”
section in the CHANGELOG to see if you need to update any existing
extensions.

We have also made available version 1.7.2, which contains important
bug fixes for those users that want to stay on the CKAN 1.7 line.

Please refer to the documentation for instructions on how to install
or upgrade CKAN, either via packages or from source, and feel
free to contact the mailing list or the IRC channel if you have any
questions.

v1.8 2012-10-19

Note: This version requires a requirements upgrade on source installations

Note: This version requires a database upgrade

Note: This version does not require a Solr schema upgrade

Major

  • New ‘follow’ feature that allows logged in users to follow other users or
    datasets (#2304)
  • New user dashboard that shows an activity stream of all the datasets and
    users you are following. Thanks to Sven R. Kunze for his work on this (#2305)
  • New version of the Datastore. It has been completely rewritten to use
    PostgreSQL as backend, it is more stable and fast and supports SQL queries
    (#2733)
  • Clean up and simplifyng of CKAN’s dependencies and source install
    instructions. Ubuntu 12.04 is now supported for source installs
    (#2428,
    #2592)
  • Big speed improvements when indexing datasets (#2788)
  • New action API reference docs, which individually document each function and
    its arguments and return values (#2345)
  • Updated translations, added Japanese and Korean translations

Minor

  • Add source install upgrade docs (#2757)
  • Mark more strings for translation (#2770)
  • Allow sort ordering of dataset listings on group pages
    (#2842)
  • Reenable simple search option (#2844)
  • Editing organization removes all datasets
    (#2845)
  • Accessibility enhancements on templates

Bug fixes

  • Fix for relative url being used when doing file upload to local storage
  • Various fixes on IGroupFrom (#2750)
  • Fix group dataset sort (#2722)
  • Fix adding existing datasets to organizations
    (#2843)
  • Fix 500 error in related controller
    (#2856)
  • Fix for non-open licenses appearing open
  • Editing organization removes all datasets
    (#2845)

API changes and deprecation

  • Template helper functions are now restricted by default. By default only
    those helper functions listed in lib.helpers.allowed_functions
    are available to templates. The full functions can still be made
    available by setting ckan.restrict_template_vars = false in your ini file.
    Only restricted functions will be allowed in future versions of CKAN.
  • Deprecated functions related to the old faceting data structure have
    been removed: helpers.py:facet_items(), facets.html:facet_sidebar(),
    facets.html:facet_list_items().
    Internal use of the old facets datastructure (attached to the context,
    c.facets) has been superseded by use of the improved facet data structure,
    c.search_facets. The old data structure is still available on c.facets,
    but is deprecated, and will be removed in future versions.
    (#2313)

v1.7.2 2012-10-19

Minor

  • Documentation enhancements regarding file uploads

Bug fixes

  • Fixes for licences i18n
  • Remove sensitive data from user dict
    (#2784)
  • Fix bug in feeds controller (#2869)
  • Show dataset author and maintainer names even if they have no emails
  • Fix URLs for some Amazon buckets
  • Other minor fixes

News from the CKAN team 8 Oct 2012

Mark Wainwright - October 9, 2012 in Feature, News, Releases

There have been some changes of plan and a lot of work behind the scenes since the last round-up of CKAN news back in July. Major areas of work include the new Datastore and significant overhauls to CKAN’s web user interface. The release of CKAN 1.8 has been delayed, in order that it can include the new Datastore. Some details are below.

New Datastore

CKAN’s Datastore provides structured storage and querying of data, and underpins CKAN’s previews as well as other custom-made applications to access and process data. In current releases of CKAN, the Datastore is built on elasticsearch, an open-source search engine which was quick to build and is good for providing full-text search of a data resource. However, elasticsearch is limited in what it can do, and we have wanted for a while to replace it with a full database-backed system. We’ve now had a chance to spend some time on this – David has built a new Datastore based on PostgreSQL, and Dominik has been adding improvements and working on the migrations.

The new Datastore will have a full SQL search API, enabling more complicated queries, including queries that join data across different data tables. (We’ve also found some reliability problems with elasticsearch which it will hopefully solve.)

Version 2.0

The new Datastore is one of several major changes in progress to the code base. As a result work has started on a code branch for version 2.0 to reflect the fact that it will be a significant change. The changes include the new Jinja templating system (and a resulting change to all the default templates), which will not cause much visible change for users, but will make life much easier for anyone wanting to run their own customised version of CKAN. Legacy support for the old templates will be included, so that instances with existing customisations can keep them until they switch to the new system.

New UI

CKAN’s default user interface hasn’t changed significantly in 5 years and sadly needed attention. That changed dramatically for the better recently when the team gained a front end developer – first Aron and now JohnM – who have worked with Toby on improving the UI. The results can be seen on the CKAN demo site. They will be integrated into CKAN’s default UI in version 2.0.

Organizations and authorization

CKAN 2.0 will also include a new access authorization system. At its heart will be Organizations, a new feature to enable a smoother workflow for data publishers. An Organization will be a collection of datasets and users, so that a dataset can only be changed by users in the relevant Organization. (For a community instance like the DataHub, all users will belong to a default Organization.)

Where’s 1.8?

The release of CKAN 1.8 has been delayed, in order that it can include the new Datastore. To anyone who’s been waiting eagerly for 1.8, apologies! On reflection, we decided that for existing sites running 1.7.1 to upgrade now, only to have to do another upgrade shortly when the new DataStore is added, would be an unnecessary effort for little gain. The good news is that the Datatore has now been added to the 1.8 branch and will be undergoing testing over the next couple of weeks. An added complication is that sites using the old elasticsearch Datastore will need a way to migrate to the new one.

Comings and goings

Our first front end developer Aron left recently, and John Martin has joined the team to replace him. Dominik, a university student from Potsdam who joined us as an intern for the summer, will be going back to college soon, but we hope to be seeing more of him! We also said farewell recently to Ross, who made a great contribution during his time on the team as a developer, and Toby too is imminently off to pastures new. All the best to them from the CKAN team.

Translators needed!

Sean Hammond - July 31, 2012 in Releases

We’re preparing to release CKAN 1.8 in about two weeks’ time. A few parts of the interface are new, and so some words and sentences need translating from English to a number of other languages before the release.

If you can translate into any of these languages, please help! You can contribute translations, and help keep CKAN open to people across the world, by signing up here.

If there is currently no translation of CKAN into your language, and you’d like to start or contribute to one, you can do that too by signing up. If you have any problems get in touch via the ckan-dev list.

CKAN 1.7.1 released

Adrià Mercader - June 20, 2012 in News, Releases

The CKAN team is pleased to announce the release of version 1.7.1.
This is bug fix release, and does not introduce any backwards
incompatible change with version 1.7 (It is our intention to do the same
with all forthcoming point point or minor releases).

The new version is available on PyPI, the apt repository and of course
via source install. See the administration guide for full installation
instructions.

For those users that deploy CKAN via package install, we have updated
the upgrading instructions to make the process clearer, whether
upgrading from another major version or the same major version.

As usual, feel free to email the dev list or pop by on #ckan at
irc.freenode.net if you need any help.

A full list of changes is below.

v1.7.1 2012-06-20

Minor:

  • Documentation enhancements regarding install and extensions (#2505)
  • Home page and search results speed improvements (#2402,#2403)
  • I18n: Added Greek translation and updated other ones (#2506)

Bug fixes:

  • UI fixes (#2507)
  • Fixes for i18n login and logout issues (#2497)
  • Date on add/edit resource breaks if offset is specified (#2383)
  • Fix in organizations read page (#2509)
  • Add synchronous_search plugin to deployment.ini template (#2521)
  • Inconsistent language on license dropdown (#2575)
  • Fix bug in translating lists in multilingual plugin
  • Group autocomplete doesn’t work with multiple words (#2373)
  • Other minor fixes