How does code coverage report generation with Instanbul work?

Recently I started working on a project that uses Istanbul to measure and report on code coverage of the unit tests. I thought the tool was genius but I asked myself: How does Istanbul actually track code coverage? It seemed like magic. Today I found the time to take a high level  look at the source code of the project and I was able to shed some light on how it actually works.

Please note that this blog post is not to give you instructions about how to use Istanbul. I think there are plenty out there. Please consider this blog post as a work in progress, the information in here might be up to 50 percent wrong (my own estimate). So far my inspection of the source code has been quite high level and here I document how I THINK it works. I will try to update this blog post once I spend more time looking at some of the details of the code. Also I hope that some people will find this (googling “How do Javascript code coverage reports with Istanbul work” didn’t find any relevant results so far) and let me know if there is something wrong with my understanding.

Let’s start with a diagram of some of the core modules within Istanbul:

Diagram of Istanbul modules.

Now this diagram is certainly an outrageous oversimplification, but it will serve us as a base for further discussion. Let’s take a closer look at those modules:

  • instrumenter: enriches the code that is to be tested with extra statements that log the coverage. Such as if you had a function and you wanted to test how many times the function is called you would insert a counter that gets incremented in the first line of the function.
  • walker: ‘walks’ through the code in the form of a tree.
  • esprima parser: an external plugin that is used to parse the code which is to be tested. Esprima spits out a syntax tree that can be walked by the walker.
  • code: well this is the code that is to be covered.
  • coverage object: at some point the code is executed and through the instrumentation the execution of the code is logged. The output of this process is the coverage object. It is simply a json file with information about how many times the statements, functions and branches in the code have been executed.
  • collector: there might be several coverage object generated for different parts of modules of the code. The collector takes these coverage objects and merges them into one coverage object that contains all the information.
  • reporter: takes the final coverage object and turns it into a human readable and condensed¬†coverage report.

In my opinion the instrumenter is at the heart of the whole plugin. Understanding the concept of instrumentation was the entry point for me to understand how code coverage can be measured. This contains the ‘magic’ that I could not get my head around before.

Important to say is that the coverage reporting is virtually independent of the actual unit tests. The only important thing is that the code execution is monitored while the unit tests are executed. This means that the unit tests need to execute the instrumented code in order for the coverage to be recorded. Before diving into the Istanbul source code I thought that the unit tests needed to deeply integrate with the coverage plugin in order for everything to work. Luckily this is not the case which makes the whole thing less error-prone, because it’s less interdependent.

Tadaaa. This was my short overview over how code coverage with Istanbul works. As mentioned before this is a work in progress and I hope to update it as soon as I have more time to dive into the internals of Istanbul more. In the meantime feel free to let me know if you find anything wrong with my explanation or if you know more about a certain area and can help me understand.