Datopian has launched a new CKAN feature that allows users to subscribe to datasets. This is an opt-in feature that sends users an email notification when a dataset to which they are subscribed is changed or updated. Let’s take a look at the feature in more detail.
Datopian are the co-creators, co-stewards and one of the main developers of CKAN. They work with government, NGO and enterprise clients worldwide to build CKAN-based, data-driven solutions, from open data portals to internal data management systems. Through monitoring client use cases for data across the private, public and third sectors, Datopian ensure that CKAN is responding to genuine challenges faced by real organizations. As with many of their feature releases, the suscribe to datasets option was built to address client needs.
Photo by Dele Oke on Unsplash
In order to provide clients with a robust messaging system, Datopian needed to build a feature outside of the main application process.
Before Datopian developed a subscribe to datasets feature, data portal users had no good way of finding out about changes to datasets. Approaches to notifying users of changes include using RSS feeds or CKAN’s built-in email integration. However, these approaches were not applicable for their client’s context because:
Some datasets and resources can change rapidly, and many different types of stakeholders can subscribe to change notifications. This means that anywhere from 50,000 to 200,000 notifications may be broadcast in a given month.
Datopian's client wants to extend the notification feature to support additional notification channels as well as email. A next iteration will add SMS notifications, giving users the choice to receive notifications by SMS, email, or both.
Another advantage of the feature is that the granularity is high. Users can currently receive the following information via email notifications:
The name of the datasets in which a change has taken place.
Whether the change was applied to a whole dataset, or a single resource.
Whether there were changes to the metadata.
Here’s an example notification:
Screenshot section of an example email notification
Fig 1.1. Diagram demonstrates that data curators edit the metadata and data of a dataset or resource to which a user is subscribed.Fig 1.2. Diagram shows, at a high level, the technical design of the data subscription service, including how it interacts with CKAN.
Configure notification frequency - system administrators can determine the frequency with which users receive email notifications. This is particularly helpful for users subscribed to very large datasets that are updated multiple times per minute/hour.
Disable notifications for certain datasets - system administrators may opt to disable notifications for certain datasets for a number of reasons. In particular, companies using CKAN data portals may choose to disable notifications for datasets that are updated frequently, should the cost of mass emailing become too high.
Subscribe to new datasets - CKAN users can receive emails notifying them when new datasets are added to the portal. This is particularly helpful for users monitoring all portal activity.
