Writing automatic tests for NCM components

This is work in progress!!

TOC

Introduction

We lack an automatic test suite. The reason is that many operations we do require root access, and it's very difficult to perform any tests locally. Also, previously it was not possible to load an arbitrary profile and feed the Configuration object to the component being tested.

In any case, the consequences have been fatal: testing is expensive, as it requires:

1. Writing the component.
2. Preparing a real profile with the component under test.
3. SSH-ing into the test machine.
4. Install the RPM, and do whatever tricks are needed to avoid SPMA from removing our test version.
5. Running the component.
6. Iterate.

so we don't test nearly enough. And we don't dare to refactor code, because we cannot be completely sure that we are not breaking some existing use case. And regressions appear.

This document proposes a strategy for automatically testing any "decent" Quattor code. Decent Quattor code is code that follows the Coding style guide. If you are responsible of a module that is not decent, go and fix it first.

Prerequisites

For running any tests, you will need:

• A check out of perl-CAF and perl-LC in your PERL5LIB environment variable.
• If you are writing a component (and most likely you are), a check out of the latest version of CCM, and add it to PERL5LIB.
• The latest Maven-based build tools (not yet released!!! ).
• If you have any mockup profiles, the panc executable must be in your PATH.

Test layout

• Tests are scripts with .t extension, stored under src/test/perl.
• Any resources needed for your tests, such as mockup profiles, should be stored in src/test/resources.
• There must be a smoke test, called 00-load.t in that directory. Its only mission is to load your module. If you are using a recent enough version of the Maven build tools (not yet released), it is already there.

Writing a test

Test header

Obviously, you will be using your component: In addition to that, the canonical way of running tests in Perl is with Test::Simple or Test::More. You must use one of these (or any standard module that uses one of these). I recommend you to have a look at Test::Tutorial, Test::Simple and Test::More man pages.

You want also your test to inspect any commands run and files used by your component. And you need to be sure that your test doesn't break your work station! To ease this, we provide another testing module: Test::Quattor. It will disable any dangerous operations for you, will store any CAF::Process and CAF::File* objects for later verification.

Finally, you want an instance of your component. So, your test script will start like this:

use strict;
use warnings;
use NCM::Component::your_component;
use Test::More;
use Test::Quattor;

my $comp = NCM::Component::your_component->new('your_component');

Testing smaller functions

Don't ever start testing your component with the Configure method. Instead, test first the behaviour of some if its lower functions. Just create the arguments they will receive from their callers, and call them yourself:

my $tree = {
   "users" => { a => 0,
                b => 1 }
};
my $result = $comp->method_that_takes_users($tree);
# Run any tests over the returned $result.

Testing on files

Any CAF::FileWriter or CAF::FileEditor objects created by your component are now stored internally by Test::Quattor. We can request it to give us each "modified" file with get_file, which is exported by default.

# Ensure the file /etc/foo has been opened
my $fh = get_file("/etc/foo"); # We have our file object here:
ok(defined($fh), "The file was opened");
is("$fh", "Some contents", "The file has received the expected contents");
is(*$fh->{options}->{mode}, 0700, "The file has the expected permissions");

Now, let's imagine that our component was editing an existing file. In this case, we'll simulate some initial contents for it via the set_file_contents function:

set_file_contents("/etc/passwd", "root:x:0:0:root:/root/:/bin/bash\n");
# Call the function that will manipulate /etc/passwd here.
$fh = get_file("/etc/passwd");

Tests on commands

All commands should be run with CAF::Process. And we are accessing them with get_command:

$ok = $component->function_that_calls_ls($args);
ok($ok, "The function was successful");
my $cmd = get_command("/bin/ls -lh");
ok(defined($cmd), "ls was invoked");

Now, this $cmd hash contains two elements:

• object is the CAF::Process object that encapsulates the command.
• method is the name of the method that got executed: run, output, execute...

For instance, we want to be sure that this command was run, with its output discarded:

is($cmd->{method}, "run", "The correct method was called");

Testing the Configure method

This method should have almost no logic. But it still deserves some testing. It must receive a valid EDG::CCM::Configuration object from a profile. And handling this is done by Test::Quattor. This is how we do it:

1. We have a set of test profiles. They are minimalistic, and have no dependencies on any "standard" templates. Just assign a few values to fill in your component structure. For instance:

   object template test_profile;
   
   prefix "/software/components/foo";
   "a" = 1;
   "b" = 2;
   

2. This profile must be compiled, and a cache must be created for it. The Test::Quattor comes in hand here: If you pass any arguments to it, it will compile them and prepare the correct cache:

   use Test::Quattor qw(test_profile another_test_profile);
   

3. And then, you'll want it to serve you the configuration for some of these, right?

   my $config = get_config_for_profile("test_profile");
   

4. And now, you can just run Configure on it:

   $comp->Configure($config);
   

And test whatever you need to test. Simple, right?

Running the tests

If you set up the PERL5LIB environment variable correctly, now all that's missing is to ask the build tools to test:

$ mvn test

The test phase is run before package, so you'll have your tests run whenever you create the RPMs for your component:

$ mvn package

If you want to disable these tests (for instance, a wrong PERL5LIB), just disable the module-test profile:

$ mvn '-P!module-test' package

Conclusion

I hope this will reduce the effort required to test Quattor code. New components should have a detailed test suite. And older components should receive one.