I am a fan of code coverage as long as you aim for 100% coverage "of code that should be tested". I don't think you will get some of the benefits of code coverage if you just turn it on for all code and say "let's aim for 80%". A good code coverage tool should allow you to specify what to include or ignore by namespace, class name and even attributes (such as the "GeneratedCode" attribute).
OpenCover is a nice code coverage tool as it meets these needs and it's Open Source. Later in this post I will go over how to set it up just using batch files to keep it nice and simple.
I use code coverage to set myself goals so I know what I am aiming to test in a certain time period. For example, I could say "by the end of the week I want to have all my MVC Controllers tested" - so I add a filter to include the controllers in the code coverage report. So I use it as a way of "defining success" while writing tests.
Keep in mind that having 100% coverage on some code does not mean you have tested all scenarios! All it does is prove you have tested one scenario down that path. So I use it as an indicator of whether something has been tested, not a measure to say the code works in every scenario. But that is a lot better than nothing and accidentally missing some code!
There can be cases where auto-generated code ends up in namespaces you want to include, but you don't want to have the generated code included. An example of this is T4MVC which generates code for your MVC project to create strongly typed references where you usually need to use magic strings. One way I found around this which is included in the batch file above is to use the "excludebyattribute" switch. You will notice above I have put -excludebyattribute:"System.CodeDom.Compiler.GeneratedCodeAttribute" which works if the generated code has that attribute (as T4MVC does).
For a more complicated example, see the code coverage I have created for the DotNetAppStarterKit sample MVC app: https://github.com/nootn/DotNetAppStarterKit/blob/master/DotNetAppStarterKit.SampleMvc.UnitTests/_RunCodeCoverageInOutput.bat.
This is how I got it working (using TeamCity version 7 at the time of writing) with the Code Coverage report showing as an "Artifact" so I can just click the link in TeamCity to view it:
Hopefully this helps you get started on your Code Coverage journey!
OpenCover is a nice code coverage tool as it meets these needs and it's Open Source. Later in this post I will go over how to set it up just using batch files to keep it nice and simple.
I use code coverage to set myself goals so I know what I am aiming to test in a certain time period. For example, I could say "by the end of the week I want to have all my MVC Controllers tested" - so I add a filter to include the controllers in the code coverage report. So I use it as a way of "defining success" while writing tests.
Keep in mind that having 100% coverage on some code does not mean you have tested all scenarios! All it does is prove you have tested one scenario down that path. So I use it as an indicator of whether something has been tested, not a measure to say the code works in every scenario. But that is a lot better than nothing and accidentally missing some code!
Example implementation - Local Development Machine
Sample Application
To test the steps I am documenting here, I have created a very simple sample application. If you want to play with this before adding it to your project or just see how it works, you can download the sample application here: https://github.com/nootn/OpenCoverExample.
Basic Setup with NUnit
For these instructions, I assume you have a Test project with some NUnit tests already succeeding.
- Install the following NuGet packages (it does not matter which project you install them in as they go to the solution level packages):
- NUnit.Runners
- OpenCover
- ReportGenerator
*Take note of the version numbers of each of these packages that were installed as you need them later* - Create a batch file in the root folder of your Test project called "_RunCodeCoverageInOutput.bat"
- In the contents, add the following (replacing 'ProjToTest'; with your project's name, 'TestProj' with the name of your test project, and changing the version numbers to match that of the packages you downloaded:
"..\..\..\packages\OpenCover.4.5.2316\OpenCover.Console.exe" -target:"..\..\..\packages\NUnit.Runners.2.6.3\tools\nunit-console.exe" -targetargs:"/nologo TestProj.dll /noshadow" -filter:"+[ProjToTest]ProjToTest*" -excludebyattribute:"System.CodeDom.Compiler.GeneratedCodeAttribute" -register:user -output:"_CodeCoverageResult.xml"
"..\..\..\packages\ReportGenerator.1.9.1.0\ReportGenerator.exe" "-reports:_CodeCoverageResult.xml" "-targetdir:_CodeCoverageReport" - Go to the properties of the batch file in Visual Studio and change the value of "Copy To Output Directory" to "Copy if newer"
- Build your solution, go to the output directory (E.g. bin\debug\) and run the batch file "_RunCodeCoverageInOutput.bat"
- If it doesn't work - check the console output to diagnose the error - feel free to comment on this post and I will try to help, or refer to the documentation.
- If it does work, you should have a subdirectory created called "_CodeCoverageReport" and in there, open up "index.htm" which shows your nicely generated code coverage report!
You can drill into each of the items on the left to see which lines of code have/have not been run
Include or Exclude more
To include or exclude more namespaces, you just alter the "filter" value. The example filter I have above is "+[ProjToTest]ProjToTest*" which means "assembly = ProjToTest, namespace = ProjToTest". If for example you had a particular namespace within that you wanted to exclude, you could add a "-" to the filter after the "+", E.g: "+[ProjToTest]ProjToTest* -[ProjToTest]ProjToTest.SomeNamespace*". This illustrates you can be very granular by only including certain namespaces, or just include all then exclude specific ones - choose which ever option saves you time and makes sense.There can be cases where auto-generated code ends up in namespaces you want to include, but you don't want to have the generated code included. An example of this is T4MVC which generates code for your MVC project to create strongly typed references where you usually need to use magic strings. One way I found around this which is included in the batch file above is to use the "excludebyattribute" switch. You will notice above I have put -excludebyattribute:"System.CodeDom.Compiler.GeneratedCodeAttribute" which works if the generated code has that attribute (as T4MVC does).
For a more complicated example, see the code coverage I have created for the DotNetAppStarterKit sample MVC app: https://github.com/nootn/DotNetAppStarterKit/blob/master/DotNetAppStarterKit.SampleMvc.UnitTests/_RunCodeCoverageInOutput.bat.
Running in TeamCity with No TeamCity Changes
I couldn't find a TeamCity add-on for OpenCover (there was one for PartCover built in) and the only instructions I could find to get it working involved installing things on the server. So that was another good reason to get this working as a batch file (so it can run in TeamCity self-contained within it's own source code).This is how I got it working (using TeamCity version 7 at the time of writing) with the Code Coverage report showing as an "Artifact" so I can just click the link in TeamCity to view it:
- Create another batch file similar to "_RunCodeCoverageInOutput.bat" above called "_RunCodeCoverageInTeamCity.bat", but remove the "..\..\..\" in front of any executables, because TeamCity will run it from the root level, not have to go up 3 levels (you might have to play with this depending how TeamCity is configured for you)
- Again make the batch file "Copy if newer" as above
- In TeamCity, make sure your Artifact Path includes "_CodeCoverageReport":
- Make a new build step after your solution is built to run the code coverage batch file. It can just be a "Command Line" runner type and it just needs to run the batch file you created:
Hopefully this helps you get started on your Code Coverage journey!
Comments
Post a Comment