Steps definitions¶

GNATbdd¶

GNATbdd is the first tool that you will need to run as part of your testing framework.

Its role is to aggregate, into one or more executables various pieces of code:

Part of your application’s code: This is the code you intend to test, and all its closure and dependencies.
Step definitions: These are (hopefully short) subprograms that create the link between the English sentences you wrote in your features files, and the actual code to execute.
GNATbdd’s own library: GNATbdd comes with an extensive library which includes looking for features files to run, parsing them, actually running them (or a subset of them depending on command line switches), and an assert library to help you write your own tests.
Stubs: It is often useful, when running test, to stub a part of your application. This might be done to simplify the test setup (no need for a remove server for instance for a distributed application), to speed up the test by abstracting a slower part of the application, or to help focus the tests. Since you might want to stub different parts of the application depending on the features you are testing, GNATbdd might need to generate multiple executables that will be run for the tests.

All of these will be examined in more details in this section.

The general workflow is thus the following:

GNATbdd library      Ada step definitions
        \                  /
         \                /
          \              /
             GNATbdd    ------------   Application code
                |
                | generate glue code
                | <compile and link>
                |
                \           feature files
                 \             /
                  \           /
                   \         /
                  test driver

Given its dependencies, GNATbdd (and the compiling of the driver) only need to be performed when your application’s code changes, or the step definitions change. This step is not needed when you only change the features files themselves, or even if you add new features files.

Compiling steps¶

Now that you have described in the features file what the test should be doing and what the expected result should be.

This is performed via the various steps defined in the scenario. We now need to associate those steps (English sentences) into actual code.

GNATbdd has one mandatory parameter, -P project.gpr, which points to the project file you are using to build your application.

GNATbdd will parse that project to find all its source files, and look for step definitions in all of the spec files.

Additionally, you can add one or more directories that are not part of your project, but that should also be parsed (recursively). For instance, by specifying --steps=features/step_definitions one or more type, that directory and all its subdirectories will be searched for Ada spec files that might contain step definitions.

When you launch GNATbdd, it will search for all the Ada files in those directories and generate one Ada file, named by default driver.adb. That file is generated in the object directory of the project you passed in argument. That directory should therefore be writable (although it will be created automatically if it does not exist yet).

If you wish to use another name for the driver (and therefore for the generated executable), you can use the --driver=NAME switch on the command line.

Building the driver¶

Once the driver has been generated by GNATbdd, you now need to compile it. GNATbdd has in fact also generated a project file, named by default driver.gpr (that name is also set through the --driver switch).

So all you have to do is hopefully:

> gprbuild -P obj/driver.gpr

where obj/ is the object directory of the application’s project.

The generated project depends on three other projects:

your application’s project: This is referenced through an absolute path, so should always be found.
gnatbdd.gpr: This is a project installed along with GNATbdd. It should be in one of the directories looked for by the compiler (which is automatic if you installed GNATbdd in the same directory as the compiler), or in one of the directories part of the GPR_PROJECT_PATH environment variable.
gnatcoll.gpr: This is the project file for the GNAT Components Collection, which should be available the same way that gnatbdd.gpr is.

Add files for step definitions¶

The steps themselves can basically perform any action you want, and they define whether you are doing black box or white box testing (see below).

The Ada packages should contain code similar to the following:

package My_Steps is

   --  @given ^a user named (.*)$
   procedure Set_User (Name : String);

   --  @then ^I should get (\d+) as a result$
   procedure Check_Result (Expected : Integer);
end My_Steps;

The example above shows two steps defined with special comments. The comment must occur just before the subprogram to which it applies.

Note

Should support custom aspects ? With comments, how do we handle cases where the regexp is too long to fix on a line, except for using pragma Style_Checks(Off).

The comments should start with one of '@given‘, '@then‘ or '@when‘. There is no semantic difference, they only act as a way to help introduce the regexp.

It is recommended that regular expressions always be surrounded with ‘^’ and ‘$’, to indicate they should match the whole step definition, and not just part of it.

Parameter types¶

The regular expressions are matched with the step as found in the *.feature file. The parenthesis groups found in the regexp will be passed as parameters to the procedure. By default, all parameters are passed as strings. If you use another scalar type for the parameter, GNATbdd will use a Type’Value (...) before passing the parameter, and raise an error if the type is incorrect.

GNATbdd provides specific handling for a few parameter types. Note that the type must be written exactly as in the table below (case-insensitive), since GNATbdd does not contain a semantic analyzer to resolve names.

Type	Conversion from regexp match
String	The parenthesis group as matched by the regexp
Ada.Calendar.Time	Converts a date with GNATCOLL.Utils.Time_Value. This supports a number of date and time formats, for instance ‘2015-06-15T12:00:00Z’ or ‘2015-06-15’ or ‘15 jun 2015’ or ‘jun 15, 2015’.
others	Use others’Value to convert from string

Using tables in step definitions¶

Some steps include extra information, like a table or a multi-line string. This information is not part of the regular expression, although the subprogram should have one or more parameters for it. For instance:

with BDD.Tables;   use BDD.Tables;
package My_Steps is

    --  @then ^I should see the following results:$
    procedure Check_Results (Expected : BDD.Tables.Table);

end My_Steps;

Here, GNATbdd will notice that the subprogram has one more parameter than there are parenthesis groups in the regular expression. It then checks for this extra parameter whether the type is BDD.Tables.Table. If this is the case, that parameter will be passed the table that the user wrote as part of the step.

The comparison of the type is purely textual, there is no semantic analysis. So it might be specified exactly as BDD.Tables.Table, even if you are using use clauses.

Assert library¶

The intent is that the steps should raise an exception when the step fails. GNATbdd provides the package BDD.Asserts to help perform the tests and raise the exception when they fail. This package will also make sure a proper error message is logged, showing the expecting and actual outputs.

For instance, the implementation for one of the steps above could be:

with BDD.Asserts;   use BDD.Asserts;

package body My_Steps is
   procedure Check_Result (Expected : Integer) is
      Actual : constant Integer := Get_Current_Result;
   begin
      Assert (Expected, Actual, "Incorrect result");
   end Check_Result;
end My_Steps;

When this test fails, GNATbdd will display extra information, as in:

Then I should get 5 as result     # [FAILED]
   Incorrect result: 5 /= 4 at my_steps.adb:7

Many more variants of Assert exist, which are able to compare a lot of the usual Ada types, as well as more advanced types like lists of strings, or the tables that are used in the feature files to provide data to steps.

Automatic type conversion¶

By default, all the parenthesis group in your regular expressions are associated with String parameters in the subprogram that implements the step.

However, GNATbdd also accepts other types for parameters, and will automatically convert the string to them. The types are matched with string comparison, so they must be defined exactly as how they appear in the following table (casing not withstanding), even if you are using use clauses in your packages.

Type	Description
String	The default parameter type
Integer Natural	Typically associated with (d+) in the regexp
Ada.Calendar.Time	Date (including optional time and timezone) “2014-01-01 13:00:00+01:00” “Thu, 19 Dec 2014 13:59:12” “19/12/2014” “12/19/2014”
BDD.Tables.Table	See Using tables in step definitions
other types	GNATbdd will generate a type’Image call

Predefined Regular Expressions¶

To simplify the writting of your steps, GNATbdd provides a number of predefined regular expressions that can be used in your own regular expressions. These expressions have a name, that can be used in your regexps by using a leading percent sign, as in:

--  @then "^I should get %natural results";
procedure My_Step (Expected : Integer);

The predefined regexps are automatically included in a parenthesis group, so you should not add parenthesis yourself.

Here is the full list of predefined regular expressions:

name	examples	Ada type
integer	-1; 0; 234	String or Integer
float	-1.0E+10; 002E-10	String or Float
natural	2; 56	String or Natural
date	Feb 04, 2014; 2014-02-04	String or Ada.Calendar.Time

You can use the percent symbol twice (“%%”) when you need to ignore a string matching one of the predefined regular expressions. For instance, “%%integer” would match the string “%integer” in the feature file. When you only when to insert a percent sign before any other string, you do not need to duplicate it though (so “%test” can be written exactly as is).

Predefined steps¶

GNATbdd itself includes some predefined steps, which you can immediately use in your .feature file.

Here is the full list of predefined steps:

When I run ‘.*’

This step can be used to spawn an executable (possibly with its arguments) on the local machine.
Then (stdout|stderr) should be .*

Compare the contents of standard output or standard error with an expected output. In general, the last argument would be specified as a multi-string argument in your .feature file.

Note

When an application is using AWS, we could have predefined steps to connect to a web server and compare its output (json, html,...)

Missing step definitions¶

When the *.feature files contain steps that have no corresponding definition, they are are highlighted in a special color, and GNATbdd will display possible definitions for the corresponding subprograms, which you can copy and paste into your Ada file directly. This helps getting started.

Writing steps in python¶

Steps can also be defined in python, by creating python files with contents similar to:

@step_regexp("^I should get :num as a result")
def check_result(expected):
    current = get_current_result()
    if expected != current:
        raise AssertionError(
           "Invalid result %d != %d" % (expected, current))

As usual, any python file found in the features/step_definitions directory or the one set through --steps will be analyzed, and those that use the @step_regexp decorator.

Set up¶

Often, the code for the step definitions will need some initialization (connecting to a database, opening a web browser,...). It is possible to declare any number of such initialization subprograms.

Some of them will be run once per execution of the driver, others will be executed one per feature file.

These subprograms are also declared in the step definitions files, as in:

package My_Steps is

     @setup_driver
     procedure Init_Driver;

     @setup_feature
     procedure Init_Feature;

end My_Steps;

The subprograms never take any parameter. Since they might be called multiple times, they also need to clean up any initialization they might have done before.

Most of the time, it is better for these procedures to be written directly as specific steps in the tests, so @setup_feature should in general be replaced with a Background section in the feature file.

Asynchronous tests¶

Note

Note sure how to implement this yet.

In some cases, it is necessary to stop executing steps to give some time for the application to complete its handling, and then come back to the execution of the test. In particular, this is often necessary when testing graphical user interfaces and other event-based applications.

White box vs Black box testing¶

Testing can be done in various ways, and this section tries to provide a few leads on how to organize and perform your tests.

Black box testing¶

In this mode, the application is spawned with specific arguments, and all interaction with it (input or output) is done only as a user would. It is not possible to examine the value of specific variables, unless they have a direct impact on what can be seen from the outside.

The main advantage is that the application is tested exactly as the user would use it. This mode is compatible with most applications, like command-line application, graphical user interfaces, web servers or embedded applications.

When testing embedded applications, the test driver will run on the host, and the application will be spawned on the target. Communication between the two is the responsibility of the step definition, and could take the form of examining the standard output or communicating via sockets for instance.

No real restriction apply to the way the step definition is written, since it is running on the host, not in the more limited environment of the target.

White box testing¶

In this mode, the step definitions can access all the public parts of your application’s code (or at least the public part of it). As a result, it is possible to inspect in details the actual start of your application, and perhaps catch errors earlier in the code.

One of the inconvenients in this mode is that the steps themselves end up dragging in a lot of the application’s code, which makes the link time for the driver longer.

More importantly, this mode might not be compatible with embedded development, since the driver runs on the host.

Note

Can we run the steps directly on the target in this case, while limiting what features of the code we use like controlled types, memory allocation,...

White box testing can itself be done in one of two ways: either be linking the application’s code within the GNATbdd driver (because the code for the steps would with the application’s own packages), or by spawning an executable and communicating with it via various means (stdin/stdout, sockets, pipes,...)

Table Of Contents

Previous topic

Next topic

This Page

Steps definitions¶

GNATbdd¶

Compiling steps¶

Building the driver¶

Add files for step definitions¶

Parameter types¶

Using tables in step definitions¶

Assert library¶

Automatic type conversion¶

Predefined Regular Expressions¶

Predefined steps¶

Missing step definitions¶

Writing steps in python¶

Set up¶

Asynchronous tests¶

White box vs Black box testing¶

Black box testing¶

White box testing¶