Tutorial¶
This manual starts with a hands-on tutorial, that demonstrates various aspect of gnatbdd.
The purpose of this tool is to act as a high-level testing framework, where tests are described in such a way that they can be understood by the various parties involved in the development of an application.
The calculator application¶
In theory, BDD (see Behavior Driven Development) development starts with writing the tests, and then modify the code until the test passes. However, in most cases you will be starting with an existing application, so this tutorial will follow that typical case.
We will therefore assume we have implemented a very simple stack-based
RPN calculator (where operands are pushed on the stack, and then the operation
is applied to the top elements on the stack). Let’s keep it very simple for now,
and assume we have the following src/calculator.ads spec file:
package Calculator is
procedure Reset;
-- Empty the stack
procedure Enter (Value : String);
-- Push a new operand on the stack (if Value is an integer) or
-- compute an operation (only "+" is supported for now)
-- The stack is limited to 5 entries.
function Peek return Integer;
-- Return the value at the top of the stack
end Calculator;
It is not necessary to look at the body to test this application, so we will omit it in this documentation. It is available in the gnatbdd source package should you wish to reproduce and expand this example.
We also need to compile this application. For this, we will use a GNAT
project file, which also allows us to edit the application in GPS and use
gprbuild to build. The project file calc.gpr should contains:
project Calc is
for Source_Dirs use ("src");
for Object_Dir use "obj";
end Calc;
First test: simple addition¶
Let’s now write our first test, checking that we can do a simple addition.
Tests are written in text files with the .feature extension (by
default). They will generally be found in a subdirectory, so let’s create
the file features/first.feature, with the following contents:
Feature: simple operations
A series of basic tests for simple operations
Scenario: simple addition
When I enter "1"
And I enter "2"
And I enter "+"
Then I should read 3
Tests are called Scenarios, and they are grouped into Features. Each Feature is supposed to deal with one specific aspect of the application, but it might take several tests to fully test that feature. In the example above, we have added a simple textual description of the feature, which is similar to a high-level requirement for the feature.
We have then written, in English, a number of steps to execute for this scenario.
Running the test¶
We need two steps to run the test.
The first step is to generate an executable, the driver, which includes (part of) your application’s source code and part of the library provided with gnatbdd. We’ll generate this executable with:
> gnatbdd -Pcalc.gpr
The result of running this test are two new files in the object directory of your
project (in our case these are obj/driver.adb and obj/driver.gpr).
Let’s then compile this driver:
> gprbuild -Pobj/driver.gpr
This second step has generated the executable obj/driver, which is in
charge of parsing each of our features files and execute the corresponding
scenarios.
The two steps above need to be performed only when your application’s code changes, or when the glue code (see below) changes. But you then use this same driver to run (or re-run) any number of features files.
Let’s do that now:
> ./obj/driver --color=no
The output will by default use colors, which we cannot easily reproduce in this manual. So we added the –color=no switch to explicitly disable colors, and so that the output you get is exactly the one we show below. Feel free to experiment without this switch to get colors (if your terminal supports them):
U
Feature: simple operations
A series of basic tests for simple operations
Scenario: Simple addition # first.feature#1:4
When I enter "1" # [UNDEFINED] first.feature:5
And I enter "2" # [UNDEFINED] first.feature:6
And I enter "+" # [UNDEFINED] first.feature:7
Then I should read 3 # [UNDEFINED] first.feature:8
1 features
1 scenarios (1 undefined)
4 steps (4 undefined)
0m0.000s
The first line indicates that the driver has executed one scenario, where it found undefined steps (i.e. it saw some text, but did not know how to execute the corresponding code – not so surprising, we have done nothing so far). The rest of the output shows more details for the scenarios that failed (by default, the driver does not show the scenarios that passed).
Writing the step definitions¶
We now need to write a bit more code to match the English text that describes the steps with the actual code to execute.
This is done through regular expressions associated with subprograms. These subprograms will be searched for in either the sources of your application (by using the project you passed in parameter to gnatbdd) or in any number of additional directories. For now, we will do the latter and create the step definitions outside of our standard project.
We thus create the following features/step_definitions/mysteps1.ads:
package MySteps1 is
-- @when ^I enter "(.*)"$
procedure When_I_Enter (Value : String);
-- @then ^I should read (\d+)$
procedure Then_I_Should_Read (Result : Integer);
end MySteps1;
The most important thing to notice is that this is standard Ada, which will be compiled through gprbuild as usual when we build the driver.
The second thing to notice are the two special comments which contain regular
expressions. They start with one of the keywords @when, @given, @then
(and a few others), which can be used interchangeably. These keywords are
followed by the actual regular expression. The full set of regular expressions
from GNAT.Regpat is supported, although in most cases you will mostly
be using a lot of plain text including one or more parenthesis group. These
parenthesis groups will be automatically passed as parameters to the procedure
whose spec is just after this comment.
For instance, when gnatbdd sees a step that starts with “I enter” and then some quoted text, it will call the subprogram When_I_Enter and pass it the text as parameter.
The second example is slightly more interesting. The regular expression expects to find a number after “I should read”. This number is read as text, but since our subprogram expects an Integer, gnatbdd will automatically convert it to an integer before calling Then_I_Should_Read.
Let’s write the body for these step definitions:
with BDD.Asserts; use BDD.Asserts;
with Calculator; use Calculator;
package body MySteps1 is
procedure When_I_Enter (Value : String) is
begin
Enter (Value);
end When_I_Enter;
procedure Then_I_Should_Read (Result : Integer) is
begin
Assert (Result, Peek);
end Then_I_Should_Read;
end My_Steps1;
This is very simple code, which needs to use both our application’s code (Calculator), and a part of the gnatbdd library which makes its easy to compare integers, strings and various other types. When the assertion fails, an error will be raised by the Assert procedure, and caught by the driver to report an error to the user.
Remember we have put these step definitions in a directory that is not part
of our calc.gpr project. We will need to pass this directory to
gnatbdd, which can either be done on the command line, or in general be done
directly from the project file so that we do not have to repeat them every
time we run gnatbdd. Let’s modify calc.gpr as such:
project Calc is
for Source_Dirs use ("src");
for Object_Dir use "obj";
package GnatBDD is
for Switches use ("--steps=features/step_definitions");
end GnatBDD;
end Calc;
At this point, let’s re-run gnatbdd and recompile the driver:
> gnatbdd -Pcalc
> gprbuild -Pobj/driver.gpr
> ./obj/driver --color=no
And when we rerun our tests, we get the much more satisfying:
.
1 features
1 scenarios (1 passed)
4 steps (4 passed)
0m0.000s
Adding multiple tests¶
Let’s imagine we want to add more tests that will perform essentially the same steps: enter the two operands on the stack, then enter the operation, and check the result. We could simply copy-paste the scenario as we have written it, but that would result in a very long features files.
Instead, we will use another feature, namely Scenario Outlines. Basically,
we will write a generic version of the test, and then a number of examples on
which we want to apply this generic. Here is the code we will now add to our
features/first.feature file:
Scenario Outline: testing operators
When I enter "<first>"
And I enter "<second>"
And I enter "<operation>"
Then I should read <result>
Examples:
| first | second | operation | result |
| 10 | 20 | + | 30 |
| 20 | 10 | - | 10 |
| 10 | 20 | + | 40 |
Each of the names between brackets will be substituted with the value in the corresponding column of the table. Each row will result in one execution of the outline. Since there are no new steps defined, and the application code has not changed, we do not even need to rebuild the driver. We can directly run the test, and get the following output:
.F
Feature: simple operations
A series of basic tests for simple operations
Scenario Outline: testing operators# first.feature#2:10
When I enter "10" # [OK] first.feature:11
And I enter "20" # [OK] first.feature:12
And I enter "+" # [OK] first.feature:13
Then I should read 30 # [OK] first.feature:14
When I enter "20" # [OK] first.feature:11
And I enter "10" # [OK] first.feature:12
And I enter "-" # [FAILED] first.feature:13
Exception name: PROGRAM_ERROR
Message: Unknown operation: -
Call stack traceback locations:
0x10a80ccb3 ...
at BDD.Asserts_Generic.From_Exception::bdd-asserts_generic.adb:139
Then I should read 10 # [SKIPPED] first.feature:14
When I enter "10" # [OK] first.feature:11
And I enter "20" # [FAILED] first.feature:12
Exception name: CONSTRAINT_ERROR
Message: Calculator stack overflow
Call stack traceback locations:
0x10a80c919 ...
at BDD.Asserts_Generic.From_Exception::bdd-asserts_generic.adb:139
And I enter "+" # [SKIPPED] first.feature:13
Then I should read 40 # [SKIPPED] first.feature:14
1 features
2 scenarios (1 passed, 1 failed)
16 steps (11 passed, 2 failed, 3 skipped)
0m0.002s
The first line shows that 2 scenarios were executed, the first one passed, the second one failed.
Hum, not quite clean, why do we have failures ?
- The first example line passed, since 10+20 is indeed equal to 30.
- The second line failed with an unexpected exception. That exception was
properly caught by gnatbdd, which displays some details. If we look
back at our
src/calculator.adsspec, we will see that in fact the only supported operation is “+”, so “-” raises an exception. At this point, the stack of the calculator contains the result of the first scenario, the result of the first example, and the two operands we intended to use for “-”. - So when the third example runs, it pushes 10 on the stack (which now has five elements), and tries to push 20, but the stack overflows and we get another internal exception.
What we forgot to do here is to reset the calculator between each scenario
and each example. For this, we will introduce a new step definition for
“Given an empty calculator”. We could add it to the first scenario, as
well as to the scenario outline. But then we’ll have to remember to write
it for each scenario we might add to this features file. Instead, we will
define a Background block, which basically is a set of steps run before
each scenario and each example. Let’s edit features/first.feature
and add the following before the first scenario:
Background:
Given an empty calculator
We could generate, build and run our driver again, but we know this step is undefined, so all scenarios will be marked as undefined.
It happens that we already have a subprogram that would be appropriate to implement this step. This is Calculator.Reset. It is just missing the regular expression, so we will edit the file and add it:
package Calculator is
-- @given ^an empty calculator$
procedure Reset;
...
end Calculator;
This time, the step is defined in the application sources, but as we mentioned before these are already automatically parsed by gnatbdd, so we can now generate, build and run:
> gnatbdd -Pcalc
> gprbuild -Pobj/driver.gpr
> ./obj/driver --color=no
and now we get:
..F
Feature: simple operations
A series of basic tests for simple operations
Scenario Outline: testing operators# first.feature#2:13
When I enter "10" # [OK] first.feature:14
And I enter "20" # [OK] first.feature:15
And I enter "+" # [OK] first.feature:16
Then I should read 30 # [OK] first.feature:17
When I enter "20" # [OK] first.feature:14
And I enter "10" # [OK] first.feature:15
And I enter "-" # [FAILED] first.feature:16
Exception name: PROGRAM_ERROR
Message: Unknown operation: -
Call stack traceback locations:
0x10e254cb3 ...
at BDD.Asserts_Generic.From_Exception::bdd-asserts_generic.adb:139
Then I should read 10 # [SKIPPED] first.feature:17
When I enter "10" # [OK] first.feature:14
And I enter "20" # [OK] first.feature:15
And I enter "+" # [OK] first.feature:16
Then I should read 40 # [FAILED] first.feature:17
40 /= 30
at MySteps1.Then_I_Should_Read::mysteps1.adb:14
1 features
2 scenarios (1 passed, 1 failed)
17 steps (14 passed, 2 failed, 1 skipped)
0m0.002s
The last exception is gone, and we now get an error (yes, 10+20 is not 40). Notice how the call to Assert lists both the expected and actual values, which is often enough to understand the error without entering the debugger.
We can fix the features file to expect 30 instead of 40.
At the same time, we implement support for “-” in our calculator, since this is now needed to pass our tests. This is the traditional BDD approach: write the test first, then write the code.
Using tables¶
We add an issue previously with the contents of the stack. Let’s add a new test that will check this content.
The idea is to enter a number of values on the stack (we already have a step definition for this), and then compare the stack with an expected output. To describe this expected output, we will make use of another aspect of the features language, the tables. We have already seen an instance of such a table when we defined a scenario outline before, although that was in a slightly different context.
Let’s add the test to features/first.feature:
Scenario: Checking stack contents
When I enter "10"
And I enter "20"
And I enter "30"
And I enter "40"
Then the stack should contain
| value |
| 10 |
| 20 |
| 31 |
| 40 |
And, as usual, we need to provide the step definition. Let’s add it
to features/step_definitions/mysteps.ads, although it could of course
be in any other source file:
with BDD.Tables; use BDD.Tables;
package MySteps1 is
-- @then ^the stack should contain$
procedure Then_The_Stack_Should_Contain (Expected : BDD.Tables.Table);
end MySteps1;
There are a few differences with our previous step definitions. Since the table is given on a different line, it should not be part of the regular expression. But we still want to indicate in the parameters of the subprogram that a table is expected. That should be the last parameter, and its type should be exactly BDD.Tables.Table (this is a textual comparison, no cross-reference from the compiler). With this, gnatbdd will automatically recognize that a table is part of the step and will be passed to the subprogram.
Let’s look at the implementation:
package body MySteps1 is
procedure Then_The_Stack_Should_Contain (Expected : Table) is
Actual : Table := Create;
Stack : constant Integer_Array := Peek_Stack;
begin
for S in Stack'Range loop
Actual.Put (Column => 1, Row => S, Value => Stack (S)'Img);
end loop;
Assert (Expected => Expected, Actual => Actual);
end Then_The_Stack_Should_Contain;
end MySteps1;
Here we build another table from the actual contents of the stack, and do a diff between the two tables. We’ll see when we run the test that gnatbdd is able to nicely display diffs in table, using colors when possible, so that it is easy to see at a glance where any error is.
In the best BDD tradition, we have now implemented a test before we even had
the code for it in our calculator, so now we need to improve our calculator
until the test passes. Here goes the change in src/calculator.ads,
with a corresponding body (not shown here):
package Calculator is
...
type Integer_Array is array (Natural range <>) of Integer;
function Peek_Stack return Integer_Array;
end Calculator;
With these various additions, we can now run our test as usual:
> gnatbdd -Pcalc
> gprbuild -Pobj/driver.gpr
> ./obj/driver --color=no
...F
Feature: A first feature
This is not very useful, we are just testing that we can execute basic
calculator tests.
Scenario: Checking stack contents # first.feature#3:25
When I enter "10" # [OK] first.feature:26
And I enter "20" # [OK] first.feature:27
And I enter "30" # [OK] first.feature:28
And I enter "40" # [OK] first.feature:29
Then the stack should contain # [FAILED] first.feature:30
| 10 |
| 20 |
| 30 /= 31 |
| 40 |
1 features
3 scenarios (2 passed, 1 failed)
22 steps (21 passed, 1 failed)
0m0.003s
The test is obviously wrong, so let’s fix the test, rerun, and all is clean !
This concludes this small tutorial. The rest of this document will go into further details for each of the steps we have seen here, as well as quite a few additional capabilities in gnatbdd.