Migrating CKAN 2.8 to CKAN 2.9

This is a short guideline on how to migrate from CKAN 2.8 to CKAN 2.9. For those of you who are learning about CKAN now, it is a tool for creating open data websites. It helps you manage and publish collections of data. It is used by national and local governments, research institutions, and other organizations that collect a lot of data. Now, let`s take a look at the process of migration.

How to migrate from CKAN 2.8.9 to CKAN 2.9.4

As is with any open-source project, the first place to check is the community for already migrated extensions from the upstream repositories. If an extension was already migrated, it can be pulled from upstream and just merged with custom HTMLs and the translations for the extension. For the extensions that were not migrated, there is this general blueprint that can be followed, tackling the migration one extension at a time:

• Migrate Python 2 to Python 3
• Migrate the web framework Pylons to Flask
• Integration of the key CKAN dependencies
• Upgrade of the UI in order to be compatible with the changes done through the migration of the web framework Pylons to Flask
• Migrate the current tests from nosetests to pytests

Migrating Python 2 to Python 3 is the simplest task, just simply change python 2 specific syntax to python 3. After this, there are some Python 3 compatibility issues for example bytes vs str handling, unicode, use of basestring, relative imports etc, but these errors can be handled while working on the next steps.

The next three steps are interlinked. With these steps comes code modernization. First, check whether any python libraries that the extension depended on are Python 2 only. If that is the case, you should find suitable replacements or updates. A useful tool for this is the caniusepython3 library, which can help in identifying dependencies that can’t run on Python 3. After this, comes the migration of all the controllers to flask views/blueprints. This means changing all BaseController controllers handled by the IRoutes interface to blueprints handled by IBlueprint interface. Simply put, the code went from looking like this:

import ckan.plugins as p

class MyPlugin(p.SingletonPlugin):


    p.implements(p.IRoutes, inherit=True)

    def before_map(self, _map):
        # New route to custom action
        m.connect(
            '/foo',
            controller='ckanext.my_ext.plugin:MyController',
            action='custom_action')

        # Overriding a core route
        m.connect(
            '/group',
            controller='ckanext.my_ext.plugin:MyController',
            action='custom_group_index')
        return m

class MyController(p.toolkit.BaseController):

    def custom_action(self):
        # ...

    def custom_group_index(self):
        # …

to looking like this:

import ckan.plugins as p


	def custom_action():
	    # ... 

	def custom_group_index():
	    # ...


	class MyPlugin(p.SingletonPlugin):
	
	    p.implements(p.IBlueprint)
	
	    def get_blueprint(self):
	        blueprint = Blueprint('foo', self.__module__)
	        rules = [ 
	            ('/foo', 'custom_action', custom_action),
	            ('/group', 'group_index', custom_group_index), ]
	        for rule in rules:
	            blueprint.add_url_rule(*rule)
	
	        return blueprint

If a plugin registered CLI commands, it should be migrated from paste command to Click CLI commands. These migrated commands are integrated into the existing ecosystem via the new IClick interface.

class ExtPlugin(p.SingletonPlugin)
         p.implements(p.IClick)
         # IClick
         def get_commands(self):
             """Call me via: `ckan hello`"""
             import click
             @click.command()
             def hello():
                 click.echo('Hello, World!')
             return [hello]

For migrating the frontend resources, the resources registered in resource.config or the files in the fantastic location such as ckanext/ext/fanstatic/script.js and ckanext/ext/fanstatic/style.css had to be loaded into any HTML through webassets.yaml. An example web assets the file looks like this:

ext_script: # name of the asset
	  filters: rjsmin # preprocessor for files. Optional
	  output: ckanext-ext/extension.js  # actual path, where the generated file will be stored
	  contents: # list of files that are included in the asset
	    - script.js
	ext_style:  # name of asset
	  output: ckanext-ext/extension.css  # actual path, where generated file will be stored
	  contents: # list of files that are included into asset
	    - style.css

In the case of fantastic it is possible to load CSS and JS files into one asset. For web assets, those files need to be split into two separate assets. So, after migration to webassets.yaml, the templates from loading resources are updated to assets.

{% block scripts %}
  {{ super() }}
  {% resource 'ckanext-ext/script.js' %}
{% endblock scripts %}
  {% block styles %}
  {{ super() }}
  {% resource 'ckanext-ext/style.css' %}
{% endblock styles %}
 
{% block scripts %}
  {{ super() }}
  {% asset 'ckanext-ext/ext_script' %}
{% endblock scripts %}
  {% block styles %}
  {{ super() }}
  {% asset 'ckanext-ext/ext_style' %}
{% endblock styles %}

After this step is completed, there is a functioning extension and with the final step of re-writing the tests, you can be sure that there were no new bugs introduced in the process. The tests were re-written from nosetests to pytests which meant removing imports from nose.tools to import pytest. The assert statements also had to be changed:

assert_equal(x, y)
eq_(x, y)
assert_in(x, y) 

assert_raises(exc, func, *args) 

with assert_raises(exc) as cm:
    func(*args)     
assert 'error msg' in cm.exception


assert x == y
assert x == y
assert x in y
	
with pytest.raises(exc):
    func(*args)
	
with pytest.raises(RuntimeError) as excinfo:
    func(*args)
assert 'error msg' in str(excinfo.value)

The old FunctionalTestBase was replaced by various pytest fixtures. These fixtures are decorators that were added directly to the test class or method. After this last step of migrating the tests was complete, the whole migration of the extension was completed.

Final Thoughts

Migrating from CKAN 2.8 to CKAN 2.9 is an interesting journey! If you have any questions about the migration process, you can always reach out to us here.

____

Note: This post was written by Hristijan Vilos, Software Developer at Keitaro. You can see the original post on their blog.