Extension Development and testing

To run CKAN and develop and/or test extensions locally, one needs to set up development environment as the following:

1. Set up virtual environment:

sudo mkdir -p /etc/ckan/default/  # or other directory of choice
sudo chown `whoami` /etc/ckan/default
python3 -m venv /etc/ckan/default
source /etc/ckan/default/bin/activate
pip install setuptools
pip install --upgrade pip

Note, virtual enviromnent should stay activated during the whole installation process.

2. Install CKAN as a package into virtual environment:

pip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.10.5#egg=ckan[requirements]'
# or for development purposes:
pip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.10.5#egg=ckan[requirements,dev]'

Installation of dependencies may fail due to some known or other compatibility issues, below is an example of troubleshooting.

One of the known issues is psycopg2 building. If installation of psycopg2 fails, go to CKAN requirements.txt now cloned and located at <venv directory>/src/ckan/requirements.txt, change psycopg2==2.9.7 => psycopg2-binary==2.9.9, rerun installation of the dependencies and install CKAN itself separately:

pip install -r <venv directory>/src/ckan/requirements.txt
pip install -r <venv directory>/src/ckan/dev-requirements.txt
pip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.10.5#egg=ckan'

Another case of dependencies incompatibility occurring for CKAN v2.9.10 is PyYAML. In requirements.txt it may be set as pyyaml==5.4.1 or pyyaml==6.0.1, if it causes the following error in the runtime TypeError: load() missing 1 required positional argument: ‘Loader', it should be downgraded to pyyaml==5.3.1.

3. Install required extensions An extension can be cloned first to a desired location and then installed to the virtual environment from file, e.g.:

pip install -e file:///<path to local extension repo>/ckan-fairdatapoint#egg=ckanext-fairdatapoint

For example on mac

pip install -e file:///Users/<username>/Github/gdi-userportal-ckanext-fairdatapoint#egg=ckanext-fairdatapoint  

Or directly from Github location

pip3 install -e git+https://github.com/ckan/ckanext-dcat.git@v2.1.0#egg=ckanext-dcat 

Then install dependencies:

pip install -r <path to local extension repo>/requirements.txt
pip install -r <path to local extension repo>/dev-requirements.txt

For example ckanext-harvest

pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-harvest/master/requirements.txt 
pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-harvest/master/dev-requirements.txt  

4. Set up a postgres database, specify database connection string in ckan.ini and test-core.ini.

Testing strategy for a plugin would depend on plugin functionality. CKAN provides a number of helper functions allowing to generate dummy data in the database or clean the database afterwards. Please review official documentation for detailed info.

To set up testing of a plugin make sure the following settings are configured correctly:

1. Make sure pytest-ckan is installed to the virtual environment. Normally this dependency should be listed in dev-requirements.txt of a plugin and therefore got installed. If you do not see it in pip list output run pip install pytest-ckan.
2. Each plugin has an autogenerated test.ini file. Make sure it points to test-core.ini file of CKAN installation, e.g.:

[app:main]
use = config:/etc/ckan/default/src/ckan/test-core.ini

3. Modify test-core.ini so configuration is correct. Most likely you need to modify sqlalchemy.url parameter so it contains correct connection information: sqlalchemy.url = postgresql://<user>:<password>@localhost/<db_name> If testing of your plugin requires writing to the database it is recommended to set up a separate test instance of postgres database.

Tests can be run via simple pytest --ckan-ini=test.ini command or an extended one with desired options. The following example is to run tests with coverage:

use = config:<path to virtual environment>/default/src/ckan/test-core.ini
<path to virtual environment>/default/bin/pytest --ckan-ini=test.ini --disable-warnings ./ckanext/fairdatapoint --cov ./ckanext/fairdatapoint -vv

Per file/UT within PyCharm. Mark sure the following user environment variable is set

CKAN_INI=test.ini