= Writing automatic tests for NCM components = '''This is work in progress!! ''' [[TOC]] == Introduction == Most components lack an automatic test suite. The reason is that many operations they do require root access, and it's very difficult to perform any tests locally. Also, previously it was not possible to load an arbitrary profile and feed the `Configuration` object to the component being tested. In any case, the consequences have been fatal: testing is expensive, as it requires: 1. Writing the component. 1. Preparing a real profile with the component under test. 1. SSH-ing into the test machine. 1. Install the RPM, and do whatever tricks are needed to avoid SPMA from removing our test version. 1. Running the component. 1. Iterate. so we don't test nearly enough. And we don't dare to refactor code, because we cannot be completely sure that we are not breaking some existing use case. And regressions appear. This document proposes a strategy for automatically testing any "decent" Quattor code. Decent Quattor code is code that follows the [Development/Code/CodingStyle Coding style guide]. If you are responsible of a module that is not decent, go and fix it first. == Prerequisites == For running any tests, you will need: * A check out of `perl-CAF`, `perl-LC`, `ncm-ncd` and `ccm` in your `PERL5LIB` environment variable. * What I do for `CAF`, `CCM` and `ncm-ncd` is to have a Git clone and run `mvn compile` on each of them. A local installation will work as well. * If you are writing a component (and most likely you are), a check out of the latest version of CCM, and add it to `PERL5LIB`. * The Maven-based build tools, at version 1.19 or later (they will be updated automatically for you, don't worry). * If you have any mockup profiles, the `panc` executable must be in your `PATH`. == Test layout == * Tests are scripts with `.t` extension, stored under `src/test/perl`. * Any resources needed for your tests, such as mockup profiles, should be stored in `src/test/resources`. * There must be a smoke test, called `00-load.t` in that directory. Its only mission is to load your module. If you are using a recent enough version of the Maven build tools ('''not yet released'''), it is already there. == Writing a test == === Test header === Obviously, you will be using your component: In addition to that, the canonical way of running tests in Perl is with `Test::Simple` or `Test::More`. You must use one of these (or any standard module that uses one of these). I recommend you to have a look at `Test::Tutorial`, `Test::Simple` and `Test::More` man pages. You want also your test to inspect any commands run and files used by your component. And you need to be sure that your test doesn't break your work station! To ease this, we provide another testing module: `Test::Quattor`. It will disable any dangerous operations for you, will store any `CAF::Process` and `CAF::File*` objects for later verification. Finally, you want an instance of your component. So, your test script will start like this: {{{ #!perl use strict; use warnings; use NCM::Component::your_component; use Test::More; use Test::Quattor; my $comp = NCM::Component::your_component->new('your_component'); }}} === Testing smaller functions === Don't ever start testing your component with the `Configure` method. Instead, test first the behaviour of some if its lower functions. Just create the arguments they will receive from their callers, and call them yourself: {{{ #!perl my $tree = { "users" => { a => 0, b => 1 } }; my $result = $comp->method_that_takes_users($tree); # Run any tests over the returned $result. }}} === Testing on files === Any `CAF::FileWriter` or `CAF::FileEditor` objects created by your component are now stored internally by `Test::Quattor`. We can request it to give us each "modified" file with `get_file`, which is exported by default. {{{ #!perl # Ensure the file /etc/foo has been opened my $fh = get_file("/etc/foo"); # We have our file object here: ok(defined($fh), "The file was opened"); is("$fh", "Some contents", "The file has received the expected contents"); is(*$fh->{options}->{mode}, 0700, "The file has the expected permissions"); }}} Now, let's imagine that our component was '''editing''' an existing file. In this case, we'll simulate some initial contents for it via the `set_file_contents` function: {{{ #!perl set_file_contents("/etc/passwd", "root:x:0:0:root:/root/:/bin/bash\n"); # Call the function that will manipulate /etc/passwd here. $fh = get_file("/etc/passwd"); }}} === Tests on commands === All commands should be run with `CAF::Process`. And we are accessing them with `get_command`: {{{ #!perl $ok = $component->function_that_calls_ls($args); ok($ok, "The function was successful"); my $cmd = get_command("/bin/ls -lh"); ok(defined($cmd), "ls was invoked"); }}} Now, this `$cmd` hash contains two elements: * `object` is the `CAF::Process` object that encapsulates the command. * `method` is the name of the method that got executed: `run`, `output`, `execute`... For instance, we want to be sure that this command was `run`, with its output discarded: {{{ #!perl is($cmd->{method}, "run", "The correct method was called"); }}} === Testing with real profiles (the `Configure` method) === Sometimes it's more convenient to have a mockup profile, and get that configuration. Here comes my advice: 1. Have a set of test profiles. They should be tiny and have no dependencies on any "standard" templates. Just assign a few values to fill in your component structure. For instance: {{{ #!sh object template test_profile; prefix "/software/components/foo"; "a" = 1; "b" = 2; }}} 1. These profiles must be compiled and a cache created for it. `Test::Quattor` comes in hand here: If you pass any arguments to it, it will compile them and prepare the correct cache: {{{ #!perl use Test::Quattor qw(test_profile another_test_profile); }}} 1. And then, you'll want it to serve you the configuration for some of these, right? {{{ #!perl my $config = get_config_for_profile("test_profile"); }}} 1. And now, you can just run `Configure` on it: {{{ #!perl $comp->Configure($config); }}} 1. Or, if your tests can benefit from it, extract portions of it with `getTree()`: {{{ #!perl my $t = $config->getElement("/software/components/foo")->getTree(); $comp->a_lesser_method($t->{a_subtree}); }}} And test whatever you need to test. Simple, right? Always remember that the `Configure` method should be terribly simple, with almost no logic. The bulk of your tests will focus in the methods of your component that do the real work. == Running the tests == If you set up the `PERL5LIB` environment variable correctly, now all that's missing to run any scripts in `src/test/perl` with `.t' extension. The build tools will do that for us: {{{ #!sh $ mvn test }}} The `test` phase is run before `package`, so you'll have your tests run whenever you create the RPMs for your component: {{{ #!sh $ mvn package }}} If you want to skip these tests (for instance, you cannot modify your `PERL5LIB`), just disable the `module-test` profile: {{{ #!sh $ mvn '-P!module-test' package }}} === Coverage reports === If we want to generate coverage reports, we only need to load `Devel::Cover`. We can do it with the `HARNESS_PERL_SWITCHES` environment variable: {{{ #!sh $ export HARNESS_PERL_SWITCHES='-MDevel::Cover=+ignore,/(test|LC|CCM|CAF|Test|dependency)/' $ mvn test }}} There is now a `cover_db` directory. If we run {{{ #!sh $ cover }}} We'll get a very detailed HTML report. Review the full documentation of `Devel::Cover` for more details. Note that generating the coverage reports slows down the execution of your tests. But it's unvaluable help to find subtle bugs. == Pending issues == How will we ship the `Test::Quattor` module? Is the name appropriate? == Conclusion == I hope this will reduce the effort required to test Quattor code. New components should have a detailed test suite. And older components should receive one. With time, we should see less regressions and less fear to change the code in maintainable ways.