Writing automatic tests for NCM components
TracNav
Table of Contents
Introduction
Most components lack an automatic test suite. The reason is that many operations they do require root access, and it's very difficult to perform any tests locally. Also, previously it was not possible to load an arbitrary profile and feed the Configuration
object to the component being tested.
In any case, the consequences have been fatal: testing is expensive, as it requires:
- Writing the component.
- Preparing a real profile with the component under test.
- SSH-ing into the test machine.
- Install the RPM, and do whatever tricks are needed to avoid SPMA from removing our test version.
- Running the component.
- Iterate.
so we don't test nearly enough. And we don't dare to refactor code, because we cannot be completely sure that we are not breaking some existing use case. And regressions appear.
This document proposes a strategy for automatically testing any "decent" Quattor code. Decent Quattor code is code that follows the Coding style guide. If you are responsible of a module that is not decent, go and fix it first.
Prerequisites
For running any tests, you will need:
- A check out of
perl-CAF
,perl-LC
,ncm-ncd
andccm
in yourPERL5LIB
environment variable.- What I do for
CAF
,CCM
andncm-ncd
is to have a Git clone and runmvn compile
on each of them. A local installation will work as well.
- What I do for
- If you are writing a component (and most likely you are), a check out of the latest version of CCM, and add it to
PERL5LIB
. - The Maven-based build tools, at version 1.28 or later (they will be updated automatically for you, don't worry).
- If you have any mockup profiles, the
panc
executable must be in yourPATH
. - Your Perl environment must include
Test::MockModule
,Test::MockObject
andReadonly
. They are available as RPMs or you can download them from CPAN.
Remark: if you use QWG templates to configure your OS, you can add all the packages required by including template config/os/quattor-development
.
Test layout
- Tests are scripts with
.t
extension, stored undersrc/test/perl
. - Any resources needed for your tests, such as mockup profiles, should be stored in
src/test/resources
. src/test/resources
must contain a file calledccm.cfg
with the following contents:debug 0 get_timeout 1 profile https://www.google.com cache_root target/test/cache retrieve_wait 0 retrieve_retries 1 dbformat CDB_File
- There must be a smoke test, called
00-load.t
in that directory. Its only mission is to load your module. If you are using a recent enough version of the Maven build tools (not yet released), it is already there.
Writing a test
Test header
Obviously, you will be using your component: In addition to that, the canonical way of running tests in Perl is with Test::Simple
or Test::More
. You must use one of these (or any standard module that uses one of these). I recommend you to have a look at Test::Tutorial
, Test::Simple
and Test::More
man pages.
You want also your test to inspect any commands run and files used by your component. And you need to be sure that your test doesn't break your work station! To ease this, we provide another testing module: Test::Quattor
. It will disable any dangerous operations for you, will store any CAF::Process
and CAF::File*
objects for later verification.
Finally, you want an instance of your component. So, your test script will start like this:
use strict; use warnings; use NCM::Component::your_component; use Test::More; use Test::Quattor; my $comp = NCM::Component::your_component->new('your_component');
Testing smaller functions
Don't ever start testing your component with the Configure
method. Instead, test first the behaviour of some if its lower functions. Just create the arguments they will receive from their callers, and call them yourself:
my $tree = { "users" => { a => 0, b => 1 } }; my $result = $comp->method_that_takes_users($tree); # Run any tests over the returned $result.
Testing on files
Any CAF::FileWriter
or CAF::FileEditor
objects created by your component are now stored internally by Test::Quattor
. We can request it to give us each "modified" file with get_file
, which is exported by default.
# Ensure the file /etc/foo has been opened my $fh = get_file("/etc/foo"); # We have our file object here: ok(defined($fh), "The file was opened"); is("$fh", "Some contents", "The file has received the expected contents"); is(*$fh->{options}->{mode}, 0700, "The file has the expected permissions");
Now, let's imagine that our component was editing an existing file. In this case, we'll simulate some initial contents for it via the set_file_contents
function:
set_file_contents("/etc/passwd", "root:x:0:0:root:/root/:/bin/bash\n"); # Call the function that will manipulate /etc/passwd here. $fh = get_file("/etc/passwd");
Tests on commands
All commands should be run with CAF::Process
. And we are accessing them with get_command
:
$ok = $component->function_that_calls_ls($args); ok($ok, "The function was successful"); my $cmd = get_command("/bin/ls -lh"); ok(defined($cmd), "ls was invoked");
Now, this $cmd
hash contains two elements:
object
is theCAF::Process
object that encapsulates the command.method
is the name of the method that got executed:run
,output
,execute
...
For instance, we want to be sure that this command was run
, with its output discarded:
is($cmd->{method}, "run", "The correct method was called");
Finally, we can alter the command status and output for this command with the set_command_status
and set_command_output
functions:
# Set the standard output of the ls -l command. When calling output() on it, it will return foo\nbar set_desired_output("ls -l", "foo\nbar"); # Set the standard error of the ls -l command. set_desired_error("ls -l", "this is an error"); # Sets $? to 1 set_command_status("ls -l", "1");
Testing with real profiles (the Configure
method)
Sometimes it's more convenient to have a mockup profile, and get that configuration. Here comes my advice:
- Have a set of test profiles. They should be tiny and have no dependencies on any "standard" templates. Just assign a few values to fill in your component structure. For instance:
object template test_profile; prefix "/software/components/foo"; "a" = 1; "b" = 2;
- These profiles must be compiled and a cache created for it.
Test::Quattor
comes in hand here: If you pass any arguments to it, it will compile them and prepare the correct cache:use Test::Quattor qw(test_profile another_test_profile);
- And then, you'll want it to serve you the configuration for some of these, right?
my $config = get_config_for_profile("test_profile");
- And now, you can just run
Configure
on it:$comp->Configure($config);
- Or, if your tests can benefit from it, extract portions of it with
getTree()
:my $t = $config->getElement("/software/components/foo")->getTree(); $comp->a_lesser_method($t->{a_subtree});
Your test profiles should live in src/test/resources
.
And test whatever you need to test. Simple, right?
Always remember that the Configure
method should be terribly simple, with almost no logic. The bulk of your tests will focus in the methods of your component that do the real work.
Running the tests
If you set up the PERL5LIB
environment variable correctly, now all that's missing to run any scripts in src/test/perl
with `.t' extension. The build tools will do that for us:
$ mvn test
The test
phase is run before package
, so you'll have your tests run whenever you create the RPMs for your component:
$ mvn package
If you want to skip these tests (for instance, you cannot modify your PERL5LIB
), just disable the module-test
profile:
$ mvn '-P!module-test' package
Coverage reports
If we want to generate coverage reports, we only need to load Devel::Cover
. We can do it with the HARNESS_PERL_SWITCHES
environment variable:
$ export HARNESS_PERL_SWITCHES='-MDevel::Cover=+ignore,/(test|LC|CCM|CAF|Test|dependency)/' $ mvn test
There is now a cover_db
directory. If we run
$ cover
We'll get a very detailed HTML report. Review the full documentation of Devel::Cover
for more details.
Note that generating the coverage reports slows down the execution of your tests. But it's unvaluable help to find subtle bugs.
Customizing the executions
All this test infrastructure calls prove. To pass additional parameters to it, just adjust your ~/.proverc
. This is mine:
# Run tests concurrently: 4 tests -j4 # Run the script I've been modifying lately first --state=new,save
Getting the component output displayed
By default, all the component output produced by $self->info()
, $self->error()
, $self->debug()
, $self->verbose()
*are not displayed* when running the tests. To get it displayed, use one of the following two methodds:
- Pass this option from the command line when executing the test:
mvn -Dprove.args=-v
- Set it as your default configuration when executing the tests, adding the following line to your
~/.proverc
:-v
When doing the one of the actions described above, the component is run in verbose
mode.
Setting debug level in the component
If you'd like to get debug message displayed to help troubleshooting a component, in addition to enabling the display of componennt output, you need to define the environment variable QUATTOR_TEST_LOG_DEBUGLEVEL
to the appropriate debug level (1 to 5).
Conclusion
Unit tests are critical to prevent regressions in new versions and validate new features.
New components MUST HAVE unit tests covering their main functionalities. And when modifying older components, new features MUST also be covered by new unit tests. It is also recommended to add unit tests for existing features in old components.