System Testing Team

The System or Integration Testing aims to ensure that code submitted for release in Geant4 coworks with previously tested code and runs in a robust manner. It is assumed that the code has previously been well tested at the "unit" level with a reasonably recent "reference" version. See STT Aims. See also the Tag & Release Policy document.

The System Test Team was established at the Geant4 Niigata Workshop in July 1998. The Working Group is currently composed by:

Gunter Folger, CERN (coordinator)

Gabriele Cosmo, CERN (release management & Q/A)

Alberto Ribon, CERN (physics validation test suite)

Pablo Cirrone, INFN LNS

Francesco Romano, INFN LNS

STT contact: geant4-stt@cern.ch.

STT Aims

To improve the quality and robustness of GEANT4 code at the "System Level", that is to say, to ensure it compiles and runs without crashing on all supported platforms in tests wich cover a wide range of physics, geometries, particles, processes over many events. Although physics validation is not our aim, we will obviously be aware of the physics quality and seek cooperation with the category teams.

STT Assumptions and Requirements

During normal development, tags will be tested and simply accepted or rejected - we call this incremental acceptance. Two weeks before a public release we go into the release phase - see below.

It is assumed that code tagged and announced for System Testing has already been subject to Unit Testing by the category working group.
It is encouraged a steady improvement of Unit tests, e.g., to cover more platforms.
When a category team want to include new code in the next release they should make a tag and inform the System Test Team.
It is encouraged an early announcement of tags, so that tests can be spread over time, last minute tags are discouraged.
Each tag will be subject to tests and accepted or rejected on a short timescale - 2 working days.

If accepted, it will become part of the "reference" version, and category coordinators will be asked to make a formal announcement of the tag to the System Test Team or on the Bonsai database.
If rejected, a new category tag will be requested which fixes the problem(s).

incremental acceptance phase.

Accepted tags will be tagged with a global reference tag from time to time for the convenience of developers.
Two weeks before a release, no new category tags will be accepted. The existing accepted tags will form the basis of the release. A CVS branch may be created, if required, on which essential bug fixes can be made.
This is the release phase.
Problems are fed back to category coordinators; 24 hour acknowledgement is to be expected and a reasonable urgency to provide fixes.
STT has the right to exclude code from a release if it causes problems.

Communicating "Incidents" to Category Coordinators

For each Test Incident, i.e., for each problem or bug, the STT member should make a Test Incident Report as follows:

Mail the category coordinator of the category in which the problem first appears (see geant4/tests/responsibles for a list of coordinators and associates - include the associates if it seems reasonable), copy to the STT mailing list and describe the problem. This procedure is also automated by toggling to "Rejected" in the Bonsai database the tag which may be responsible for the problem.
During the incremental acceptance phase or release phase, ask the category coordinator to fix the problem and submit a new category tag. If the new tag fixes the problem, inform the category coordinator by toggling the tag to "Accepted" in the Bonsai database.
Problems not fixed will be entered in the Problem Tracking System and eventually also mentioned in the release-notes. They will be listed in a summary page, which will be updated at every reference-tag of development; the page will be also announced to category coordinators each time it is updated.

Test Suite

A set of tests in geant4/tests/ excercises the code. Also many examples are used for regular integration testing.

Test Procedures

There are detailed test procedures for members of the System Test Team.

Tags Database

Category tags are notified and recorded in a database, tested then accepted or rejected according to the above procedures.

Release Policy

Hopefully, the above policies will mean that we go into the release phase with no outstanding problems. However, some new problems might emerge during the porting of the code on all supported platforms/compilers. There might also be some agreed "deferred" problems. If these cannot be fixed, and are not killers, they should be listed in the Release Notes.

As also specified in our Tag and Release Policy, if required to create a CVS branch, Testers - including Category Testers - will have to checkout the branch into a separate working directory so as to avoid confusion with the main trunk (development) version. The branch will then be used for testing and any bugs found will have to be fixed there and, if appropriate, also in the development version. This will be done primarily by the Category Testers in response to TIRs.

A CVS branch behaves like an independent version of the code. This means that developers are free to continue developing. The down side is that System and Category Testers have to check out an independent version of the code in a separate working area so that essential bugs (ESSENTIAL bugs only) can be fixed on the branch (and also be fixed in the HEAD if appropriate). But we have discussed this and we think it is an acceptable price to pay for giving us the freedom to patch releases and continue development.

Work plans

[ 2007 ] [ 2008 ] [ 2009 ] [ 2010 ]