flynn.gg
Christopher Flynn
Machine Learning
Systems Architect,
PhD Mathematician
Projects
Open Source
Blog
Résumé
GitHub
Blog
Slack messaging from Celery workers
2017-11-27 
In my last blog post, I discussed how to add a Slack messaging layer to Celery by using signals and patching a few of the Task class methods. Since then I put together a proper python package which integrates nicely with any already existing Celery application.
The implementation is similar to the previous blog post, but the package provides several customization options to the end user. For the full set of documentation, check out the project on readthedocs.
While building this package, I thought it would be a good idea to get more familiar with best practices for open source software. I’ll briefly go through some of the tools I learned along the way.
pipenv
Pipenv is a tool that combines the pip installation package with the virtualenv tool suite into one simple environment and package dependency manager. It also made it much easier to manage the package requirements alongside the dev requirements.
pytest
On past projects I’ve been using the built-in Python unit testing framework, unittest. It’s served its purpose but a lot of the community has been moving to a newer framework called pytest. The transition was very simple and the pytest library just made writing tests (especially combinatoric ones) way easier. Since celery-slack provides many customization options, writing tests for each one of them would be absurd. Pytest’s fixtures allows for easy writing of parametrized tests, and writing simple assert statements is much simpler than remembering all the self.assert... statements that are part of the unittest's TestCase class. I was able to set up thousands of tests in only several dozens of lines of code.
Additionally, great extensions like pytest-mock, pytest-cov, and pytest-xdist have been extremely useful in mocking objects for integration tests, building coverage reports, and executing tests in parallel, respectively.
tox
Tox is a python package for running a suite of tests across multiple virtual environments. Since celery-slack has a dependency on celery, I used tox to execute my tests across 3 major/minor versions of Celery, and across 5 different major/minor Python versions. This is to ensure that the package works on Python 2 and 3, but also on older versions of Celery that may still be part of some legacy systems.
To manage the local Python versions, I used a package called pyenv.
travis-ci
I have a bit of experience in continuous integration/deployment using Jenkins. I’m also aware of a few competitors: namely travis-ci and circleci. Most of the open source projects I’ve been using lately have been set up to test on travis-ci.org, a free to use continuous integration tool for open source projects. The process was painless, and the configuration is similar to using tox.
codecov
Codecov is also a free-for-open-source service that tracks code coverage from tests. Simple to setup and integrated easily with the travis-ci service.
readthedocs
Last but not least is readthedocs, a documentation website that integrates with GitHub for continuous deployment of documentation using Sphinx. This service uses GitHub webhooks to rebuild documentation whenever commits are made. The service uses a requirements.txt file to setup the build environment for Python, and also integrates with Google Analytics tracking.
You can find the readthedocs documentation for celery-slack here.
