Written by R3x

How to write ATF tests for NetBSD

I have recently started contributing to the amazing NetBSD foundation. I was thinking of trying out a new OS for a long time. Switching to the NetBSD OS has been a fun change.

My first contribution to the NetBSD foundation was adding regression tests for the Address Sanitizer (ASan) in the Automated Testing Framework(ATF) which NetBSD has. I managed to complete it with the help of my really amazing mentor Kamil. This post is gonna be about the ATF framework that NetBSD has and how to you can add multiple tests with ease.

Intro

In ATF tests we will basically be talking about test programs which are a suite of test cases for a specific application or program.

The ATF suite of Commands

There are a variety of commands that the atf suite offers. These include :

  • atf-check: The versatile command that is a vital part of the checking process. man page
  • atf-run: Command used to run a test program. man page
  • atf-fail: Report failure of a test case.
  • atf-report: used to pretty print the atf-run. man page
  • atf-set: To set atf test conditions.

We will be taking a better look at the syntax and usage later.

Let’s start with the Basics

The ATF testing framework comes preinstalled with a default NetBSD installation. It is used to write tests for various applications and commands in NetBSD.  One can write the Test programs in either the C language or in shell script. In this post I will be dealing with the Bash part.

If you take a look at the source code the test directory is in tests/. On a NetBSD system the compiled tests reside in /usr/tests/.

The command to run the test is atf-run which you can combine with atf-report to provide a pretty neat test result format.

Try running atf-run | atf-report in your netbsd installation. You need to be in the /usr/tests directory since if the file name is not specified then it uses the Atffile to find the tests and run them.

you can also run individual tests with atf-run | atf-report. Keep in mind that you need to be in the same directory as the test-file to do the same.

The basic structure of a Test file

atf_init_test_cases() function: This function should have the test cases that are going to be tested.
Each test needs to be added as atf_add_test_case.

Each test case needs to have a head and a body defined as two separate functions namely _head() and _body().

A basic test structure looks like this :


atf_test_case foo
foo_head() {

}

foo_body() {

}

atf_init_test_cases() {
    atf_add_test_case foo
}

The test function head: The head is mainly used to hold a description of the test case and to list the required programs. The command atf-set is used to set both these fields

Here is an example test case where the name is foo and it requires the ‘echo’ command.


atf_test_case foo
foo_head() {
    atf_set "decsr" "This is a sample test"
    atf_set "require.progs" "echo"
}

The test function body: Here you can add your test and check if the result is up to your satisfaction. You can also add other preliminary checks here if you want to.

A very important and useful function here is the atf-check command which is used to check the result of any function that you need to run.

NOTE: Keep in mind that atf-check has three flags which denote

  •  -s exit code
  •  -o stdout
  •  -e stderr of any process that you run.

We will look into that later.

Below is a sample body for the same foo program that we saw earlier. Assume that foo is trying to check if the cat command is working fine.


foo_body() {
    echo "CHECK" >> new.txt
    atf-check -s exit:0 -e ignore -o match:"CHECK" cat new.txt
}

Now what we just did was to echo CHECK into a file and then used cat to print the output to stdout and check the output.

Let’s now explore deeper into the atf-check command: 

In the previous function, I checked whether the command exited returning 0. I also ignored the stderr buffer and tried to match the stdout buffer for the “CHECK” string in the file.

Some important things to keep in mind regarding the atf–check command.

  • -s or the exit code can be used to match the exit code of a function.
    • Hence if you are writing any C or Cpp tests with this make sure that you include an exit(0); at the end.
    • you can additionally check for different exit values and also check for signals.
    • Add a not before the exit if you want the exit code to be not equal to the given value.
  • -o or the stdout options
    • you can use the inline:”text” to match the text with the string at stdout.
    • match:”regex” can be used to match a regular expression with the stdout.
    • ignore can be used to ignore whatever in the
  • –e or the stderr option is pretty much the same as -o
  • Additionally,  Some test or commands may lead to an error saying the test case exited with a non-okay exit code. this is not permitted. I have not been able to find a good reason for this but if the statements are not necessary you can avoid the check part.

You can refer the manual pages for the same to get a better idea of how they work. I have provided the links below.

Adding a test program

First of all, you need to create an sh file in the tests/ directory in your src/. Add all required test cases for the same. Use the formats given above to help you.

Once you have finished and you feel that you need to test the file without building the whole distribution you can copy the file to your NetBSD distribution /usr/tests/.

Note that you can’t run atf-run on a file without there being an Atffile in that directory

To test the test program you can use atf-run | atf-report. Fix errors, bugs etc.

Once you have done the same you need to modify the Makefile in the directory of your preference in the tests/ directory of the src/.

A typical Makefile looks like this


.include 

TESTSDIR=	${TESTSBASE}/

TESTS_SH=    #
TESTS_SH+=    t_foo
TESTS_SH+=    t_your_next_test
.include

This was an example Makefile for the test t_foo (test files are usually named with t_).

The last change you need to make is to add the name of the file in distrib/sets/lists/tests/. You can use the previous names to assist in adding this part. Please keep in mind that the devs prefer the tests being in alphabetical order.

That’s all… You have added your own test 🙂

Check if everything is proper by building a distribution with ./build.sh distribution

Additional checks: Adding an architecture check

Now your test file is ready. But you want the test to run only in a couple of architectures. Adding an architecture check in all the test_cases? Boring and time-consuming. Why not just add one in the init_test_cases function. How? Look below.


SUPPORT='n'
test_target() {
	if uname -m | grep -q "amd64"; then
		SUPPORT='y'
	fi

	if uname -m | grep -q "i386"; then
		SUPPORT='y'
	fi
}

atf_test_case target_not_supported
target_not_supported_head()
{
	atf_set "descr" "Test forced skip"
}

target_not_supported_body()
{
	atf_skip "Target is not supported"
}

atf_init_test_cases()
{
	test_target
	test $SUPPORT = 'n' && {
		atf_add_test_case target_not_supported
		return 0
	}
        #Your tests here
}

 

I guess that covers all the important things that we had to cover. 🙂

Some Good Links

Here are the links to a couple of tests that I wrote:

 

3 thoughts on “How to write ATF tests for NetBSD”

  1. Pingback: In Other BSDs for 2018/04/14 – DragonFly BSD Digest
  2. Pingback: In Other BSDs for 2018/04/14 – FreshBSD.com
  3. Pingback: Router On A Stick | BSD Now 249 | Jupiter Broadcasting

Leave a comment