Support Us

CKAN High-Level Roadmap Consultation – Autumn 2014

Rufus Pollock - October 1, 2014 in Association, Roadmap

We are working to create a Roadmap of key technical and community work for the next 6 months and we want to hear from as many as possible of you in the CKAN community to find out what you think is important.

To aid this process we have created a long-list of Roadmap ideas in a Trello board which you can view, comment, vote on and add to – please take a look now and share your thoughts »

Roadmap Consultation Trello Board – vote, comment and discuss »

More Information

With the CKAN Association Steering Group and Technical Team in place, we can now think more about the CKAN “Roadmap”.

The goal here is to agree a short-list of 4-6 high-level technical and community items that CKAN Association will focus on in the next 6-12m.

The plan is as follows:

  • Build a “long-list” of potential roadmap item and review / prioritize with users and the community over the next few weeks
  • Create a final Roadmap selection (short-list) that will be the focus of our near-term work

Led by Ashley Casovan of the Steering Group, the Steering Group and the Technical Team have already prepped long-list of high-level roadmap items which can be found in trello here:

https://trello.com/b/kvgOZLMs/ckan-association-roadmap-consultation

Please jump in and vote and comment.

FAQ

How does this relate to Github Ideas and Roadmap Tracker

The current ideas and roadmap tracker was used as input for this long-list. The purpose of the trello long-list was:

To have a shorter high-level long-list that focused on key items (the issue tracker now has over 80 items of varying scope and detail) Provide a simple interface in which people (including non-developers) could vote and comment

How will the Roadmap be resourced

The aim of the Roadmap is to produce a list of things that the CKAN Association and general community can specifically focus and execute on. We aim to produce a list that is commensurate with current (and anticipated) resources available to the CKAN Association based on Member commitments.

Where can I find out more about CKAN Roadmaps

See the Roadmap page.

Community Meeting: Friday 26 September 2014

Rufus Pollock - September 25, 2014 in Association, Meetups, News

The next CKAN Association Community meeting is planned for this Friday 26th September at 4pm London / 11am EDT.

One major proposed topic is the Roadmap but please suggest additional topics in the meeting doc or here on list.

To help us manage the meeting and get an idea of numbers if you intend to come please add yourself to the participants list in the meeting doc.

Details

  • When: Friday 26 Sepember 2014 – 4pm London (BST) / 11am US East Coast (EDT) / 5pm European (CET)
  • Where: online (likely via WebEx or similar – link to come)
  • Text chat: http://webchat.freenode.net/?channels=ckan
  • Chair: Jeanne Holm (GSA / Data.Gov)
  • Meeting Doc: Google Doc here – please add your name if you plan to come!

Draft Agenda

Announcing the CKAN Association Steering Group

Rufus Pollock - September 8, 2014 in Association, News

We are delighted to announce that we have finalized the initial membership of the CKAN Association Steering Group. At present it consists of 4 organizations and their representatives as follows:

  • Antonio Acuña (*), Head of Data.gov.uk, UK Cabinet Office
  • Jeanne Holm, Evangelist, Data.gov, U.S. General Services Administration (Data.gov)
  • Pat McDermott and Ashley Casovan, Open Government Secretariat, Treasury Board of Canada (data.gc.ca)
  • Rufus Pollock (+), President (Open Knowledge)

(*) indicates Chair of the Steering Group
(+) indicates Secretary of the Steering Group

Steering Board Members will serve an initial term of 2 years and serve “ex officio” representing their organizations. We will continue to review Steering Group Membership and to consider potential new members as we go forward.

More Information About the Steering Group

The Steering Group is a key part of the CKAN Association. It is made up of key stakeholders who have committed to oversee and steer the CKAN Association going forward. The initial selection of the steering committee was coordinated by Open Knowledge.

Full details of the Steering Group can be found on the Steering Group page.

Senior Python web developer wanted for world leading open source data portal

Sean Hammond - July 20, 2014 in Jobs

Working in the fast-growing area of open data, we build open source tools to drive transparency, accountability and re-use. Our flagship product CKAN runs the official national data portals from the UK to Brazil, US to Australia and many others. We also build data tools and OpenSpending browsers.

We’re looking for someone passionate about the technical challenges of building software that is used as the infrastructure for open data around the world, so come join our growing team to shape the future of the open data ecosystem!

Key Skills

  • Python, JavaScript, HTML, CSS
  • Python web frameworks (we use Pylons and Flask)
  • PostgreSQL and SQLAlchemy
  • Self motivated, self-starter, able to manage your own time

Extra bonus points for:

  • Open source projects/contributions
  • Front end skills, particularly in data-vis
  • You’ve written an app using open data before
  • Experience working in a distributed team

How to apply

Email jobs@okfn.org, with the subject line “Python Developer – CKAN/Services”. Please include:

  1. Your CV
  2. A link to your GitHub (or similar) profile
  3. A cover letter

More about the Job

You will be working as part of a small, dynamic team in a modern, open-source development environment. This role is full-time and we are very happy with remote-working.

We generally work remotely (with strong contingents in London and Berlin), using asynchronous communication (email, IRC, GitHub) but with standups, developer meetings and demos most days (Skype, Google Hangout) and real-world gatherings more than twice a year including at the Open Knowledge Festival. We also try and ensure our developers can attend at least one open source conference a year.

At each level of our software stack we use best-in-class open source software, including Python, Nose, Travis CI and Coveralls, Sphinx and Read the Docs, Flask, Jinja2, Solr, PostgreSQL and SQLAlchemy, JavaScript and jQuery, Bootstrap, Git and Github, and Transifex.

We iterate quickly, and publish working, open-source code early and often.

All of our code is on github – it is open to public scrutiny, and we encourage contributions from third-party developers. This means that we have to write exceptionally clear, readable, well-tested code with excellent documentation.

All code contributions, whether from internal or external developers, are made with GitHub pull requests and we do code reviews in the open on GitHub.

We are engaged with a large and active community of users, developers and translators of our open source software, via our mailing lists, GitHub issues and pull requests, public developer meetings, Stack Overflow and Transifex. We support users in getting started with our software, encourage and mentor new developers, and take on feedback and suggestions for the next releases.

About the Open Knowledge Foundation

The Open Knowledge Foundation (OKF) is an internationally recognized non-profit working to open knowledge and see it used to empower and improve the lives of citizens around the world. We build tools, provide advice and develop communities in the area of open knowledge: data, content and information which can be freely shared and used. We believe that by creating an open knowledge commons we can make a significant contribution to improving governance, research and the economy. The last two years have seen rapid growth in our activities, increasing our annual revenue to £2m and our team to over 35 across four continents. We are a virtual organisation with the whole team working remotely, although we have informal clusters in London, Cambridge and Berlin.

The OKF is an international leader in its field and has extensive experience in building open source tools and communities around open material. The Foundation’s software development work includes some of the most innovative and widely acclaimed projects in the area. For example, its CKAN project is the world’s leading open source data portal platform – used by data.gov, data.gov.uk, the European Commission’s open data portal, and numerous national, regional and local portals from Austria to Brazil. The award winning OpenSpending project enables users to explore over 13 million government spending transactions from around the world. It has an active global network which includes Working Groups and Local Groups in dozens of countries – including groups, ambassadors and partners in 21 of Europe’s 27 Member States.

We’re changing the world by promoting a global shift towards more open ways of working in government, arts, sciences and much more.

The Open Knowledge Data Packager

Sean Hammond - June 9, 2014 in Releases

Today we’re launching the Open Knowledge Data Packager!

Data Packager is a web app for quickly creating and publishing Tabular Data Packages from collections of CSV files on your computer. You can register for a free user account and start creating data packages now, or take a look at a sample data package.

With Data Packager’s simple interface you can create a data package, upload CSV files to it, enter some metadata, and get a web page where users can explore and download your data package. When you login, you’ll be taken to your dashboard, where you’ll see a list of any packages you’ve created so far and Add package button:

Data Packager dashboard screenshot

My Data Packager dashboard

Click the Add package button to create a new data package and you’ll be taken to a form where you can enter the title and other metadata for your package:

Creating a new data package

Creating a new data package

Click on Next: Add CSV files and you’ll be taken to a form where you can upload one or more CSV files to your data package:

Uploading CSV files to a new data package

Uploading CSV files to a new data package

Finally, click on Finish to create your data package. You’ll be taken to your data package’s page:

Browsing your newly created data package

Browsing your newly created data package

You can publish the URL of this page, or share it with anyone who you want to share your data package with.

Why Tabular Data Packages?

Tabular Data Packages (defined by the DataProtocols.org Tabular Data Package spec) are a simple and easy-to-use data publishing and sharing format for the web. A Tabular Data Package is a collection of CSV files with a datapackage.json file. The datapackage.json file contains metadata about the package (title of the package, description, keywords, license, etc.) and schemas for each of the package’s CSV files.

The format is a good compromise between CSV and Excel, providing the simplicity and ease-of-use of CSV with some of the expressivity of full-blown spreadsheets.

The schemas for the CSV files use the JSON Table Schema format, a simple format for tabular data schemas. It includes metadata for each of the CSV file’s columns (column name, type, description, etc.) and optional primary and foreign keys for the file.

Data Packager Features

After you’ve created your data package and uploaded some CSV files to it, Data Packager has a few nice features for you…

Download data packages

The Download Data Package button on your data package’s page will download a ZIP file including all of your package’s CSV files and the datapackage.json file containing the metadata you entered for your package and files, plus schemas for each of your files:

Screenshot of Download Data Package button

Schema browser

Data Packager automatically generates a JSON Table Schema for each CSV file that you upload. The generated schema includes:

  • Column names for each of the file’s columns (taken from the CSV file’s header row, if it has one)

  • The type of the data in each column (string, number, date…), inferred from the values in the columns

  • Some descriptive statistics calculated for numerical columns (minimum and maximum values, mean, standard deviation…)

  • Temporal extents (earliest and latest dates) for date and time columns

By clicking on one of the CSV files on your data package’s page, you can browse the file’s schema using the schema browser. Each file’s page shows a preview of the CSV file contents, and by clicking on the columns in the preview you can inspect the schema for each column:

The schema browser

The schema browser

Schema editor

By clicking the Edit button on one of your CSV file’s pages, you can edit the file’s JSON Table Schema and add your own custom attributes. Data Packager validates all the changes that you make and gives helpful error messages if you try to save an invalid schema.

The schema editor

The schema editor

Primary and foreign keys

If you add primary and foreign keys to a CSV file’s schema, they’ll also be shown on the file’s page.

Primary and foreign keys

Primary and foreign keys

API

All of Data Packager’s features can also be used via its JSON API.

Open Source

Data Packager is 100% open source. You can:

Built with CKAN

Data Packager is built using CKAN, the highly-customisable open source data portal platform. All Data Packager features are implemented by a CKAN extension, ckanext-datapackager.

IVPR / OIC bringing the Weave visualization platform to CKAN

Rufus Pollock - June 5, 2014 in Community, Featured, Visualization

The CKAN Association is always looking to further enhance the visualization options available for the CKAN data management platform. We are therefore delighted to connect with the Institute for Visualization and Perception Research (IVPR) at the University of Massachusetts Lowell and the Open Indicators Consortium who have led the development of Weave (Web-based Analysis and Visualization Environment), a richly featured, open source data visualization platform for the web.

CKAN-WEAVE_demo

WEAVE running using CKAN

Weave is a state-of-the-art system which provides software tools that researchers, educators, analysts, trainers, students, and the general public can use to interactively analyze, visualize and disseminate remote, local or distributed data. Weave is the IVPR’s fifth-generation visualization system and incorporates 20 years of research and embedded patented algorithms. Previous generations of this software were desktop versions. Both the earlier versions and Weave were designed to solve complex problems in a variety of application areas including drug discovery, medicine, economics, and national security. The bulk of Weave’s source code is released under the GNU General Public License (GPLv3), and its core is released under the Mozilla Public License (MPLv2).

Weave_obesity2

To facilitate the connection between CKAN and Weave, IVPR recently developed an adaptor for Weave that facilitates bringing CKAN-indexed CSV or XLS files directly into Weave for creating visualizations. The Connecticut Data Collaborative, a founding OIC member, is currently building a site that will integrate CKAN and Weave.

UK_elections2

For more information about Weave you can:

  • Visit the Weave wiki which provides everything needed to install and explore Weave.
  • See the many OIC websites using Weave here
  • We will keep you posted on new developments as they arise. In the meantime, interested developers should join the Weave-users and Weave-dev google groups to connect with other developers working with Weave.

Introducing the Technical Team and Contributing to the CKAN Code

Rufus Pollock - May 22, 2014 in Association, Community, News

This post follows up the recent post about the CKAN Association to give more information about the new Technical Team (aka official “committers”) and also gives the latest information on how anyone can contribute to CKAN on the technical side of things.

How to Get Involved in Contributing

There are lots of way to contribute to the technical side of CKAN from filing bug reports to coding up new extensions or improving the core code. We’ve recently created a dedicated guide for folks interested in contributed to the core CKAN codebase, along with a new guide on how to extend CKAN with “extensions”.

Furthermore, contributing isn’t just about the “core” CKAN codebase – with CKAN’s rich API there are lots of things to build that never need to go near the core python codebase and allow you to use any language you want – PHP, Javascript, Ruby, Java etc.

For example, everything from a phone app, a visualization dashboard and integrations with third-party services can be done over the API without touching a line of core code. (If you’re interested in these sorts of contributions the “ideas tracker” on github is the best place to find out what people are doing – or share your own ideas).

The Technical Team

Overseeing, coordinating and directing contributions, especially for the core is the responsibility of the Technical Team. The Technical team is made up of experienced CKAN developers who have contributed back to the main CKAN repositories. The Technical Team meet regularly (currently twice a week) to review pull requests and discuss technical matters.

Technical Team membership is open to anyone and membership is granted based on a proven record of competence and contribution. The Technical Team current lead is Ian Ward who has been working on CKAN for data.gc.ca.

More about the Technical Team can be found on the Technical Team home page.

CKAN Community Hangout May 2014

Rufus Pollock - May 19, 2014 in Association, Community, Events, Meetups

The next CKAN Community hangout is on Wednesday 28th May and will be chaired by Antonio Acuña of data.gov.uk.

  • When: Wed 28th May at 4pm London (BST) / 11am East Cost US (EDT) / 5pm European (CET)
  • Where: Online (Google Hangout)
  • Agenda and Signup: online shared meeting doc

Topics to cover include:

  • Community show and tell – people can share what they’re up to
  • Discussion of high-level roadmap
  • CKAN Association updates and discussion

There is an open Google Doc for the meeting with an agenda and more information

We recommend putting yourself down if you’d like to attend so we can plan for numbers.

The meeting will be by Google Hangout and we hope to run it as a hangout on air so it will stream (and get saved) on YouTube.

CKAN Census 2014

Adrià Mercader - May 12, 2014 in Community, Deployments, Featured

Census!

 

Update: We have now also published a Deployment Survey to learn more about how CKAN is deployed and installed:

http://ckan.org/deployment-survey/


CKAN is used by hundreds of organizations across the world to publish their Open Data on-line. More and more CKAN sites are going live and CKAN is being used in many new ways, integrating with other tools and being extended with new features.

We want to get a clearer picture of the current CKAN ecosystem to better understand how CKAN is being used and help scope the project roadmap. And if at the same time we can bring the ckan.org instances page up to date all the better!

So if you are developing or managing a CKAN site or know someone who does, can you spend 5 minutes filling out this quick survey?

http://ckan.org/census

Any help in spreading out the link will be much appreciated as well. The results will make a big difference, and of course, they will be made available to the community later on.

This is the first in a series of surveys that will focus on different aspects of maintaining and using CKAN. In the following weeks we’ll send another one around focused on Deployment and Installation.

Let’s map CKAN usage across the world!

 

Photo by suziesparkle

 

Introducing the CKAN Association

Rufus Pollock - March 18, 2014 in Association, Community, Featured, News

We are pleased to announce the CKAN “Association”. The Association will manage and oversee the CKAN project going forward, supporting the growth of CKAN, its community and stakeholders. The Association reflects more than a year of discussion and consultation with key stakeholders and the wider community.

Key aspects of the Association are:

  • A Steering Group and Advisory Group which oversee the project and represent stakeholders.
  • Specific teams to look after particular areas such as a “Technical Team” to oversee technical development and a “Content and Outreach team” to oversee materials (including project website) and drive community engagement
  • Membership to allow stakeholders to contribute to the longer-term sustainability of the project – more below

The Association has its formal institutional home at the Open Knowledge Foundation but is autonomous and has its own independent governance, in the form the Steering Group which is drawn from major CKAN stakeholders. The Open Knowledge Foundation, who are the original creators of CKAN, continue to contribute to CKAN at all levels but the Association allows others – from government users to suppliers of CKAN services – to have a formal role in the development of CKAN project going forward.

Membership

The CKAN Association will have members. Membership is a way for individuals, companies and organisations to support the CKAN Project and be recognised for doing so. By becoming a member you are helping to ensure the long-term sustainability of CKAN.

Member organizations are expected to contribute resources – either through contributing money or providing in-kind resources such as staff time. Members receive recognition for their contribution through display on the website, participation in events etc.

You can find more information about membership here »

Frequently Asked Questions

Why Create the Association?

Over the last few years CKAN has seen rapid growth in terms of technology, deployments, and the community. It is now the basis of dozens of major sites around the world, including national data portals in the UK, US, Canada, Brazil, Australia, Germany, Austria and Norway. There also has been substantial growth in the developer and vendor community deploying, customising and working with CKAN.

We believe that, as with many open-source projects when they achieve a certain size, the time has come to bring some more structure to the community of CKAN developers and users. By doing so we aim to provide a solid foundation for the future growth of the project, and to empower more explicitly its growing array of stakeholders.

Will the CKAN Association be a Separate Legal Entity?

No, at least not initially. The association will retains its legal home at the Open Knowledge Foundation operating as a self-governed and autonomous project. If a strong need for a separate legal entity arises this is something that the Steering Committee will consider in due course.

How will the CKAN Association relate to the Open Knowledge Foundation’s technical consulting around CKAN?

These 2 activities will be strictly separated. The CKAN Services team at the Open Knowledge Foundation will no doubt participate in the CKAN Association as stakeholders similar to other organizations and groups but will have no special rights or privileges.

What “Assets” will the Association have responsibility for?

The Association will have responsibility for items such as:

  • The primary CKAN codebase
  • The CKAN project roadmap including overseeing and steering technical development of CKAN
  • Oversee and drive user and community engagement
  • The ckan.org website and any related media assets
  • Managing any project finances and resources (e.g. from membership fees)

Will the CKAN Association have dedicated staff?

We imagine that the CKAN Association may appoint dedicated staff on an as needed basis and where there are resources to do so (also note that Members may contribute in kind resources in the form of staff time). However, at least initially, the CKAN Association will not have dedicated staff but will have in-kind support time provided by the Open Knowledge Foundation and other key stakeholders.