Hello & welcome to Part 5! This is the section about testing, why testing is important, how to do it, and how to set up your repository to run your tests automatically when you push new code.

Why Shall We Test Our Code?

why shall we test our code

Tests make your life easier and give you confidence in your code. Software is complicated, and tests allow you to be sure that you’ve not broken your project when you change something.

Writing tests slows you down in the short term, but speeds things up in the long term. Further, as someone who is writing scientific code, tests make sure that your work can be easily reproduced.

Philosophically, each test should test one thing. This makes it easier to ensure that we can see what has broken when a test breaks. We’ll see an example of this next.

Fix a Test

Let’s start by working through fixing a failing test, and committing the results to our repository.

Start by making a new branch, with git checkout -b fix-failing-test.

Then open up toymir/tests/test_toymir.py, uncomment the two functions that start with test_hz_to_midi, and then run pytest -v.

🔥💥🔥💥🔥 The tests failed, oh no! 🔥💥🔥💥🔥

It turns out that those tests are testing a function that we’ve commented out.

You can find the function in toymir/freq.py - go ahead and uncomment the code that’s there, and then the tests will pass!

Great! You should also write a new test to verify that this function works correctly for when the input or some part of it is less than or equal to 0. The correct functionality should be that it throws an exception. You can look at the hz_to_period function for reference.

Once pytest -v shows that all tests are passing, commit your changes, merge on github, and then pull the results back to your local.

Checkpoint

Run:

pytest -v

You should see:

platform darwin -- Python 3.6.5, pytest-3.7.2, py-1.5.4, pluggy-0.7.1
rootdir: /Users/thorkell/Code/bmcfee/ismir2018-oss-tutorial, inifile:
collected 5 items

tests/test_toymir.py .....          	                        [100%]

Get Travis Working

We set up our Travis account in Part 2! Travis is a service that runs your tests when you push new code to GitHub. This is useful because it means that you can’t forget to run your tests – and it means that other people’s Pull Requests will have tests run against them.

The first thing we need to do is to activate your repository, so that Travis will run your tests. Visit https://travis-ci.org/profile/<your-github-user-name>. You should see a list of repositories. Press the toggle for this repo, and things will be activated!

Travis is controlled by a file called .travis.yml. This file tells the Travis servers what to do with your code - in our case, just run the tests.

We’ve written the file for you. Let’s take a quick look at it. The install block handles the gruntwork of installing everything, and the script block is what Travis runs. We currently just run pytest.

Let’s modify this to run Flake8 as well. Flake8 checks your code and your code style, to make sure that you’re writing code that is both more correct, and easy to read.

Make a branch wih git checkout -b add-flake8 – then, all you need to do is uncomment that flake8 line above the call to pytest, then commit and push your code. (Don’t worry about all the arguments to flake8 for now!)

Then, open up Travis, (https://travis-ci.org/<your-github-username>/ismir2018-oss-tutorial) and you can see your tests run!

Checkpoint

Visit your page on Travis. You should see your tests running on Travis!

Wait, the tests failed?

Ah, how come our tests failed? We know that our tests are working, so maybe it is this new Flake8 thing! Run flake8 toymir/ locally, and see what happens.

Turns out that Flake8 will not allow unused imports! And our import seaborn in freq.py is never used.

This is very easy to fix – on the same branch, remove that line, and the run our tests and flake8 toymir/ again. Everything should pass! Once it does, commit and push the change, and watch things pass on Travis. Once the tests pass on Travis, merge things to master.

Checkpoint

You’ve seen your tests run on Travis, and have merged the passing code into master!

Gotchas:

  • Rarely, your tests will pass locally but fail on Travis. This is super frustrating, and is usually caused by there being a difference between your local environment and the Travis environment.

Success!

We’ve added two tests, and gotten comfortable running pytest. Furthermore, we’ve enabled Travis, so that our tests always run when we push code to GitHub. In Part 6, we’ll set up our documentation so that it builds automatically, and write some new docs.