Support Us

New Broken Link Checker Plugin for CKAN

Sean Hammond - October 21, 2014 in Extensions, Presentations

deadoralive is a new broken-link checker service that works with CKAN and other sites, and ckanext-deadoralive is a CKAN plugin that you can install to integrate your CKAN site with the link checker. The pair have been developed by Open Knowledge as part of our work on the new version of Öppnadata, the Swedish national open data portal.

This quick presentation gives an overview of the link checker’s features and design:

For more details, see these blog posts:

CKAN patch releases 2.0.5, 2.1.3 and 2.2.1 now available

Adrià Mercader - October 15, 2014 in Releases, Uncategorized

We are happy to announce that new patch releases for CKAN 2.0.x, 2.1.x and 2.2.x are now available to download and install.

These patch releases fix important bugs and security issues, so users are strongly encouraged to upgrade to the latest patch release for the CKAN version they are using.

For a list of the fixes included you can check the CHANGELOG:

Patch release upgrades are very straight-forward and do not contain any backwards incompatible changes or involve any change in the requirements, database or Solr schema. As stated in the release policy, the latest patch release is the only one officially supported.

For details on how to upgrade, see the following links depending on your install method:

If you find any issue, you can let the technical team know in the mailing list or the IRC channel.

 

CKAN High-Level Roadmap Consultation – Autumn 2014

Rufus Pollock - October 1, 2014 in Association, Roadmap

We are working to create a Roadmap of key technical and community work for the next 6 months and we want to hear from as many as possible of you in the CKAN community to find out what you think is important.

To aid this process we have created a long-list of Roadmap ideas in a Trello board which you can view, comment, vote on and add to – please take a look now and share your thoughts »

Roadmap Consultation Trello Board – vote, comment and discuss »

More Information

With the CKAN Association Steering Group and Technical Team in place, we can now think more about the CKAN “Roadmap”.

The goal here is to agree a short-list of 4-6 high-level technical and community items that CKAN Association will focus on in the next 6-12m.

The plan is as follows:

  • Build a “long-list” of potential roadmap item and review / prioritize with users and the community over the next few weeks
  • Create a final Roadmap selection (short-list) that will be the focus of our near-term work

Led by Ashley Casovan of the Steering Group, the Steering Group and the Technical Team have already prepped long-list of high-level roadmap items which can be found in trello here:

https://trello.com/b/kvgOZLMs/ckan-association-roadmap-consultation

Please jump in and vote and comment.

FAQ

How does this relate to Github Ideas and Roadmap Tracker

The current ideas and roadmap tracker was used as input for this long-list. The purpose of the trello long-list was:

To have a shorter high-level long-list that focused on key items (the issue tracker now has over 80 items of varying scope and detail) Provide a simple interface in which people (including non-developers) could vote and comment

How will the Roadmap be resourced

The aim of the Roadmap is to produce a list of things that the CKAN Association and general community can specifically focus and execute on. We aim to produce a list that is commensurate with current (and anticipated) resources available to the CKAN Association based on Member commitments.

Where can I find out more about CKAN Roadmaps

See the Roadmap page.

Community Meeting: Friday 26 September 2014

Rufus Pollock - September 25, 2014 in Association, Meetups, News

The next CKAN Association Community meeting is planned for this Friday 26th September at 4pm London / 11am EDT.

One major proposed topic is the Roadmap but please suggest additional topics in the meeting doc or here on list.

To help us manage the meeting and get an idea of numbers if you intend to come please add yourself to the participants list in the meeting doc.

Details

  • When: Friday 26 Sepember 2014 – 4pm London (BST) / 11am US East Coast (EDT) / 5pm European (CET)
  • Where: online (likely via WebEx or similar – link to come)
  • Text chat: http://webchat.freenode.net/?channels=ckan
  • Chair: Jeanne Holm (GSA / Data.Gov)
  • Meeting Doc: Google Doc here – please add your name if you plan to come!

Draft Agenda

Announcing the CKAN Association Steering Group

Rufus Pollock - September 8, 2014 in Association, News

We are delighted to announce that we have finalized the initial membership of the CKAN Association Steering Group. At present it consists of 4 organizations and their representatives as follows:

  • Antonio Acuña (*), Head of Data.gov.uk, UK Cabinet Office
  • Jeanne Holm, Evangelist, Data.gov, U.S. General Services Administration (Data.gov)
  • Pat McDermott and Ashley Casovan, Open Government Secretariat, Treasury Board of Canada (data.gc.ca)
  • Rufus Pollock (+), President (Open Knowledge)

(*) indicates Chair of the Steering Group
(+) indicates Secretary of the Steering Group

Steering Board Members will serve an initial term of 2 years and serve “ex officio” representing their organizations. We will continue to review Steering Group Membership and to consider potential new members as we go forward.

More Information About the Steering Group

The Steering Group is a key part of the CKAN Association. It is made up of key stakeholders who have committed to oversee and steer the CKAN Association going forward. The initial selection of the steering committee was coordinated by Open Knowledge.

Full details of the Steering Group can be found on the Steering Group page.

Senior Python web developer wanted for world leading open source data portal

Sean Hammond - July 20, 2014 in Jobs

Working in the fast-growing area of open data, we build open source tools to drive transparency, accountability and re-use. Our flagship product CKAN runs the official national data portals from the UK to Brazil, US to Australia and many others. We also build data tools and OpenSpending browsers.

We’re looking for someone passionate about the technical challenges of building software that is used as the infrastructure for open data around the world, so come join our growing team to shape the future of the open data ecosystem!

Key Skills

  • Python, JavaScript, HTML, CSS
  • Python web frameworks (we use Pylons and Flask)
  • PostgreSQL and SQLAlchemy
  • Self motivated, self-starter, able to manage your own time

Extra bonus points for:

  • Open source projects/contributions
  • Front end skills, particularly in data-vis
  • You’ve written an app using open data before
  • Experience working in a distributed team

How to apply

Email jobs@okfn.org, with the subject line “Python Developer – CKAN/Services”. Please include:

  1. Your CV
  2. A link to your GitHub (or similar) profile
  3. A cover letter

More about the Job

You will be working as part of a small, dynamic team in a modern, open-source development environment. This role is full-time and we are very happy with remote-working.

We generally work remotely (with strong contingents in London and Berlin), using asynchronous communication (email, IRC, GitHub) but with standups, developer meetings and demos most days (Skype, Google Hangout) and real-world gatherings more than twice a year including at the Open Knowledge Festival. We also try and ensure our developers can attend at least one open source conference a year.

At each level of our software stack we use best-in-class open source software, including Python, Nose, Travis CI and Coveralls, Sphinx and Read the Docs, Flask, Jinja2, Solr, PostgreSQL and SQLAlchemy, JavaScript and jQuery, Bootstrap, Git and Github, and Transifex.

We iterate quickly, and publish working, open-source code early and often.

All of our code is on github – it is open to public scrutiny, and we encourage contributions from third-party developers. This means that we have to write exceptionally clear, readable, well-tested code with excellent documentation.

All code contributions, whether from internal or external developers, are made with GitHub pull requests and we do code reviews in the open on GitHub.

We are engaged with a large and active community of users, developers and translators of our open source software, via our mailing lists, GitHub issues and pull requests, public developer meetings, Stack Overflow and Transifex. We support users in getting started with our software, encourage and mentor new developers, and take on feedback and suggestions for the next releases.

About the Open Knowledge Foundation

The Open Knowledge Foundation (OKF) is an internationally recognized non-profit working to open knowledge and see it used to empower and improve the lives of citizens around the world. We build tools, provide advice and develop communities in the area of open knowledge: data, content and information which can be freely shared and used. We believe that by creating an open knowledge commons we can make a significant contribution to improving governance, research and the economy. The last two years have seen rapid growth in our activities, increasing our annual revenue to £2m and our team to over 35 across four continents. We are a virtual organisation with the whole team working remotely, although we have informal clusters in London, Cambridge and Berlin.

The OKF is an international leader in its field and has extensive experience in building open source tools and communities around open material. The Foundation’s software development work includes some of the most innovative and widely acclaimed projects in the area. For example, its CKAN project is the world’s leading open source data portal platform – used by data.gov, data.gov.uk, the European Commission’s open data portal, and numerous national, regional and local portals from Austria to Brazil. The award winning OpenSpending project enables users to explore over 13 million government spending transactions from around the world. It has an active global network which includes Working Groups and Local Groups in dozens of countries – including groups, ambassadors and partners in 21 of Europe’s 27 Member States.

We’re changing the world by promoting a global shift towards more open ways of working in government, arts, sciences and much more.

The Open Knowledge Data Packager

Sean Hammond - June 9, 2014 in Releases

Today we’re launching the Open Knowledge Data Packager!

Data Packager is a web app for quickly creating and publishing Tabular Data Packages from collections of CSV files on your computer. You can register for a free user account and start creating data packages now, or take a look at a sample data package.

With Data Packager’s simple interface you can create a data package, upload CSV files to it, enter some metadata, and get a web page where users can explore and download your data package. When you login, you’ll be taken to your dashboard, where you’ll see a list of any packages you’ve created so far and Add package button:

Data Packager dashboard screenshot

My Data Packager dashboard

Click the Add package button to create a new data package and you’ll be taken to a form where you can enter the title and other metadata for your package:

Creating a new data package

Creating a new data package

Click on Next: Add CSV files and you’ll be taken to a form where you can upload one or more CSV files to your data package:

Uploading CSV files to a new data package

Uploading CSV files to a new data package

Finally, click on Finish to create your data package. You’ll be taken to your data package’s page:

Browsing your newly created data package

Browsing your newly created data package

You can publish the URL of this page, or share it with anyone who you want to share your data package with.

Why Tabular Data Packages?

Tabular Data Packages (defined by the DataProtocols.org Tabular Data Package spec) are a simple and easy-to-use data publishing and sharing format for the web. A Tabular Data Package is a collection of CSV files with a datapackage.json file. The datapackage.json file contains metadata about the package (title of the package, description, keywords, license, etc.) and schemas for each of the package’s CSV files.

The format is a good compromise between CSV and Excel, providing the simplicity and ease-of-use of CSV with some of the expressivity of full-blown spreadsheets.

The schemas for the CSV files use the JSON Table Schema format, a simple format for tabular data schemas. It includes metadata for each of the CSV file’s columns (column name, type, description, etc.) and optional primary and foreign keys for the file.

Data Packager Features

After you’ve created your data package and uploaded some CSV files to it, Data Packager has a few nice features for you…

Download data packages

The Download Data Package button on your data package’s page will download a ZIP file including all of your package’s CSV files and the datapackage.json file containing the metadata you entered for your package and files, plus schemas for each of your files:

Screenshot of Download Data Package button

Schema browser

Data Packager automatically generates a JSON Table Schema for each CSV file that you upload. The generated schema includes:

  • Column names for each of the file’s columns (taken from the CSV file’s header row, if it has one)

  • The type of the data in each column (string, number, date…), inferred from the values in the columns

  • Some descriptive statistics calculated for numerical columns (minimum and maximum values, mean, standard deviation…)

  • Temporal extents (earliest and latest dates) for date and time columns

By clicking on one of the CSV files on your data package’s page, you can browse the file’s schema using the schema browser. Each file’s page shows a preview of the CSV file contents, and by clicking on the columns in the preview you can inspect the schema for each column:

The schema browser

The schema browser

Schema editor

By clicking the Edit button on one of your CSV file’s pages, you can edit the file’s JSON Table Schema and add your own custom attributes. Data Packager validates all the changes that you make and gives helpful error messages if you try to save an invalid schema.

The schema editor

The schema editor

Primary and foreign keys

If you add primary and foreign keys to a CSV file’s schema, they’ll also be shown on the file’s page.

Primary and foreign keys

Primary and foreign keys

API

All of Data Packager’s features can also be used via its JSON API.

Open Source

Data Packager is 100% open source. You can:

Built with CKAN

Data Packager is built using CKAN, the highly-customisable open source data portal platform. All Data Packager features are implemented by a CKAN extension, ckanext-datapackager.

IVPR / OIC bringing the Weave visualization platform to CKAN

Rufus Pollock - June 5, 2014 in Community, Featured, Visualization

The CKAN Association is always looking to further enhance the visualization options available for the CKAN data management platform. We are therefore delighted to connect with the Institute for Visualization and Perception Research (IVPR) at the University of Massachusetts Lowell and the Open Indicators Consortium who have led the development of Weave (Web-based Analysis and Visualization Environment), a richly featured, open source data visualization platform for the web.

CKAN-WEAVE_demo

WEAVE running using CKAN

Weave is a state-of-the-art system which provides software tools that researchers, educators, analysts, trainers, students, and the general public can use to interactively analyze, visualize and disseminate remote, local or distributed data. Weave is the IVPR’s fifth-generation visualization system and incorporates 20 years of research and embedded patented algorithms. Previous generations of this software were desktop versions. Both the earlier versions and Weave were designed to solve complex problems in a variety of application areas including drug discovery, medicine, economics, and national security. The bulk of Weave’s source code is released under the GNU General Public License (GPLv3), and its core is released under the Mozilla Public License (MPLv2).

Weave_obesity2

To facilitate the connection between CKAN and Weave, IVPR recently developed an adaptor for Weave that facilitates bringing CKAN-indexed CSV or XLS files directly into Weave for creating visualizations. The Connecticut Data Collaborative, a founding OIC member, is currently building a site that will integrate CKAN and Weave.

UK_elections2

For more information about Weave you can:

  • Visit the Weave wiki which provides everything needed to install and explore Weave.
  • See the many OIC websites using Weave here
  • We will keep you posted on new developments as they arise. In the meantime, interested developers should join the Weave-users and Weave-dev google groups to connect with other developers working with Weave.

Introducing the Technical Team and Contributing to the CKAN Code

Rufus Pollock - May 22, 2014 in Association, Community, News

This post follows up the recent post about the CKAN Association to give more information about the new Technical Team (aka official “committers”) and also gives the latest information on how anyone can contribute to CKAN on the technical side of things.

How to Get Involved in Contributing

There are lots of way to contribute to the technical side of CKAN from filing bug reports to coding up new extensions or improving the core code. We’ve recently created a dedicated guide for folks interested in contributed to the core CKAN codebase, along with a new guide on how to extend CKAN with “extensions”.

Furthermore, contributing isn’t just about the “core” CKAN codebase – with CKAN’s rich API there are lots of things to build that never need to go near the core python codebase and allow you to use any language you want – PHP, Javascript, Ruby, Java etc.

For example, everything from a phone app, a visualization dashboard and integrations with third-party services can be done over the API without touching a line of core code. (If you’re interested in these sorts of contributions the “ideas tracker” on github is the best place to find out what people are doing – or share your own ideas).

The Technical Team

Overseeing, coordinating and directing contributions, especially for the core is the responsibility of the Technical Team. The Technical team is made up of experienced CKAN developers who have contributed back to the main CKAN repositories. The Technical Team meet regularly (currently twice a week) to review pull requests and discuss technical matters.

Technical Team membership is open to anyone and membership is granted based on a proven record of competence and contribution. The Technical Team current lead is Ian Ward who has been working on CKAN for data.gc.ca.

More about the Technical Team can be found on the Technical Team home page.

CKAN Community Hangout May 2014

Rufus Pollock - May 19, 2014 in Association, Community, Events, Meetups

The next CKAN Community hangout is on Wednesday 28th May and will be chaired by Antonio Acuña of data.gov.uk.

  • When: Wed 28th May at 4pm London (BST) / 11am East Cost US (EDT) / 5pm European (CET)
  • Where: Online (Google Hangout)
  • Agenda and Signup: online shared meeting doc

Topics to cover include:

  • Community show and tell – people can share what they’re up to
  • Discussion of high-level roadmap
  • CKAN Association updates and discussion

There is an open Google Doc for the meeting with an agenda and more information

We recommend putting yourself down if you’d like to attend so we can plan for numbers.

The meeting will be by Google Hangout and we hope to run it as a hangout on air so it will stream (and get saved) on YouTube.