These guidelines provide extended rules and information for Nexedi staff on how to write and run unit tests.
Table of Contents
Using Test Suite for Automated Testing¶
Test Suite Definition¶
If you would like to run automated tests on your project, you will have to define what we call a "test suite" document on Nexedi ERP5. Then Nexedi ERP5 will automatically look for you for some free resources to run your tests. Results can be seen in the module "Test Result Module".
Configure Test suite¶
ERP5 Repository is structured like this :
/tests/__init__.py
/product/*
/bt5/*
tests/__init__.py defines a test suite. A test suite is used to describe the list of test that will be executed, and it can be used to handle project specific way of running tests (like using --save). You can see right now in tests/__init__.py of ERP5 repository, there is class "ERP5" to launch all ERP5 unit test, and "PERF" to launch only performance tests.
In order to launch a new test suite, please go on Nexedi ERP5 in the module "Test Suite Module". You can then copy/paste an existing test suite, do some changes on the copy, and then validate.
Here the setting of ERP5-MASTER test suite which is used to test all generic erp5 tests:
https://www.tiolive.com/nexedi/test_suite_module/4/view
On the document level, you can define several parameters:
- Title: title of this test suite. It must be unique, two test suites must have different titles. This title is used to retrieve the result in Test Result Module
- Test Suite: name of the test suite class defined in tests/__init__.py, so for ERP5 it can be "ERP5", "PERF", etc.
- Additional bt5 path list: a list of path for additional bt5 that are not part of the software release. This is useful if the only difference of your project with generic software is some bt5.
- Project Title: link to the project associated to your test suite.
- Distributor: please select ERP5-Projects unless you know what you are doing. It is used to select different clouds of testnodes.
- Priority: How many cores you need. Please do not take more than you need. 15 cores for small project that can runs within 10 minutes on 3 cores is non-sense.
Then you can add Test Suite Repository object for every repository relevant for this test. Most of the time a single repository is needed, however for some specific projects several repositories might be needed. Below possible parameters for a Test Suite Repository.
- URL: url of the repository, it might contain password for private repositories (for private repository, you could use nxdtestbot:opah9Fo9Vi username and password)
- branch: which branch of this repository should be used
- buildout section id: what is the section name of the software to override (the section in the software getting this repository)
- profile-path: path, in first given repository, to a buildout configuration file describing the environment to test.
The automated test suite will checkout every repository in vcs_repository_list, and then as soon as there is a commit in one of them, the list of revision of all theses repositories will be given to Nexedi ERP5. Then Nexedi ERP5 can check if there is other automated test suites running on other computers and will either create a new tests result or ask the current node to synchronize.
Once the testnode knows that it must run tests, it first run slapos commands in order to update/create software and to create new instance (so we have software and instance inside erp5testnode partition). Once the software is built and the instance is created, the command runTestSuite is launched.
Email notification¶
In order to define where email notification will be sent, please setup the field "Stakeholder" on project. If you are using a mailing list, you can create an organisation representing the mailing list (which is a group of person).
We could see later if we can use RSS or ATOM feeds.
Selenium UI tests¶
Running UI tests happens with help of firefox which is run inside a virtual X screen using Xvfb. As these test simulate user interaction with system it is required to make sure that always node-quantity=1.
Writing Test guideline¶
Having tests for the work we do is very important. Generally speaking, writing tests allows to save plenty of time. It allows checking some parts of the code quickly. It allows to automate regression tests. Thanks to them, doing an important refactoring of the code is possible. An without them it's impossible to reach good quality of services.
General Tests Guideline¶
A good test is a test covering well existing code, using small amount of lines of code an which is fast to run.
Whenever possible, it's important to organize the code into distinct parts (documents, classes , methods) that could be tested separately. Like this every part could be easily modified and tested without checking the whole code every time.
It is not necessary to test huge amount of data. Let's say you would like to test the method setTitle on a document, checking a very few values is enough (like simple ascii value and utf8 values with a string in Japanese), you do not need to test various complete dictionaries.
Live Tests Guideline¶
Please make sure to use live tests all the time. Old tests must be converted to be compatible with live tests.
What is a live tests ? A live test is a test that we can launch directly through the ERP5 interface. A live tests create data inside your instance, therefore it must be written in such a way to not destroy everything.
It is really much better when the code is well orgazined and when it is possible to test subpart of the code directly instead of doing only global tests. When an unit test takes only 1 second, it is very fast to improve the related code. When a test takes 50 minutes, it is a nightmare to check any change and this is very inefficient.
What a test MUST NOT DO:
It's not necessary to destroy all data to make good tests. So here some tips in how to deal with existing data:
- You might use Sequence to retrieve object you just created.
- If you do reporting, then you can pass a parameter to ignore previously created data, like passing creation_date=<date of the beginning of your test.
- Instead of doing:
module.newContent()
self.assertEquals(1, len(module))
, you can do:
initial_count = len(module)
module.newContent()
self.assertEquals(inital_count+1, len(module))
- You might create new modules only for test (like test_person_module)
- You can prefix data
It's really not an issue to cumulate data from the run of various tests. It's even very helpful in many cases. When you do new developments, you directly have data to works with. And you already have examples of data of previous run, it can help solving issues more quickly.
Live tests and production environment¶
Live tests are really good candidates to check if a production server is running correctly. Some advices above can help, but there is some risk. Like it can be possible to show to users data entered by tests. Some messages could be wrongly sent to external software (like bank money transfer !). Some data entered by users could be altered.
So some rules and good usage practice must be written to do live test on production environment in a safe way.
Here we will publish more informations once we have more experience with it.
Testing infrastructure¶
Architecture¶¶
Architecture¶¶
While working on a project, everyone doing features should write tests that should be launched automatically. We make the distinction between different types of tests:
- Unit Tests : tests to check anything from low level code to user features
- Functional UI tests : tests of user interfaces, usually by controlling web browser. This type of test is now merged with Unit Tests and we
- Performance tests : tests written with the idea to check the evolution of execution speed over time. We usually strongly avoid any decrease of performance
- Scalability tests : tests written to check if a software could scale correctly. It check if we could process more work while we increase hardware capacity. It allows to detect regression in scalability
Tests are controled by a Testing Master : an ERP5 instance used to control clouds of testing machines. In Nexedi, the Testing Master is Nexedi ERP5
Tests are defined on the Testing Master by developpers on a Test Suite in test suite module. A Test suite contains all informations needed to know which test should be launched (where to find code, how to retrieve the list of tests to execute, what is the software used).
Each test suite needs to be attached to a Distributor. A distributor is a document on the Testing Master allowing to define a cloud of testing machine. We have distinct usage of distributors :
- ERP5 Project Distributor : distributor for the main pool of machines executing tests. Many testing machines are associated to it. It is possible that several machine works in the same time on the same test suite to execute many tests in parallel.
- ERP5 Performance Distributor : distributor associated for now to a single machine. Idea is to launch all tests sequencially to avoid any noise
- Some Project Distributor : for some reasons (like specific hardware), some project needs their own distributor to have a dedicated pool of machines
- Scalability Distributor : it is associated with a few testing machines that would execute only scalability tests. Usually, all machines part associated to the distributor would be in the same network to avoid latency between them.
The software used to control tests on a machine is called testnode. Depending on machines, we can create one or more test node per machine. Each testnode is associated to a distributor.
Unit test are run by launching one or several nodes, each launching one or more parallel tests. All nodes are synchronized through Testing Master which is in charge of distributing the work and to make sure every node is testing same revision.
Each node is installed thanks to Slapos tools through Slapos Master (Vifib). Code is handled by git, and nodes are in charge of checking if there is new commit, and if there is new commit they can ask Nexedi ERP5 if there is some work to do.
Functional Tests¶
See (outdated) op-network-devel-functional-testing
Nexedi ERP5 Task API¶
Nexedi ERP5 provide an API to start, stop unit test. This API also allows to report failures, results (tests passed or failed), status and allows to know if there still ongoing work. The API is available here.
Any tool test running reporting results on Nexedi ERP5 must use this API. Soon a library will exist to allow using it easily. This API allow to handle from simple cases (like one unit test at a time on a single server) to complex cases (an unit test running on several servers, with different number of cores on each server). For now a working example using this library is available in runTestSuite and ERP5TypeTestSuite.
Test Nodes¶
Test nodes are created on vifib.net using nexed_development_service account. Their configuration is quite trivial, please see existing examples if you wish to create a new test node. A typical configuration is:
<?xml version='1.0' encoding='utf-8'?>
<instance>
<parameter id="test-node-title">COMP533-3Nodes-ERP5PROJECT1</parameter>
<parameter id="node-quantity">3</parameter>
<parameter id="test-suite-master-url">https://webbuildbot:xxx@www.tiolive.com/nexedi/portal_task_distribution/erp5_project</parameter>
</instance>
Parameters:
- test-node-title: title of the test node, must be unique and should give idea of the computer used.
- node-quantity: how many parallel test to run
- test-suite-master-url: url of the distributor, make sure to select the right one depending on wich "cloud" you want your test node to belongs to (erp5_project, erp5_performance, erp5_scalability ?)
For a 4 core server (8 with hyperthreading), it is good to create only 2 test nodes with a node-quantity of 3. Like this at most 6 threads are used. Also having only 2 test nodes is important, if we use 6 testnodes of 1 core, then we need to install much more software than if we have only 2 nodes. So to save disk space, it is better to not have too much test nodes.
Nexedi ERP5 Test Suite Distribution¶
Test Nodes¶ doesn't know in advance on which project they are going to work on. Therefore every test node is defined with the url of a distributor. The test node will call "startTestSuite" on the distributor and it will get all needed parameters to work on one or many projects.
The first time a test node calls startTestSuite, Nexedi ERP5 is going to look if this test node already exists. If not, then it will be created under test node module.
From time to time, an alarm (task_distributor_alarm_optimize) looks at all defined test suites and available test nodes and distribute the work to do. This alarm avoid moving test suite from a test node to another uselessly. In the same time, this alarm is checking for all test node that seems dead (10 hours without sending message) and invalidate them. Like this test suite allocated to a dead test node will be moved to another test node automatically.
Nexedi ERP5 synchronization¶
Let's say we have 2 testing nodes daemon (A and B) running on two different computers. Each daemon is doing checkout or update of repositories. Since A and B can run with several minutes or even more of interval, Nexedi ERP5 needs to make sure that both A and B are testing same revision. Therefore testnode A (assuming no test is already started) will do:
- Checkout/update repository 1, get revision x, checkout/update repository 2, get revision y.
- Ask Nexedi ERP5 to start unit test for a particular test_suite title with following pair of repository and revisions:
[('repository 1',x), ('repository 2',y)].
- And then Nexedi ERP5 check if there is already running test, if not create new one and then returns pair to tests. So here we assume there was not test, so it returns
[('repository 1',x), ('repository 2',y)].
- Then runTestSuite is launched for
[('repository 1',x), ('repository 2',y)]
And then testnode B will do (running a little bit after testnode A):
- checkout/update repository 1, get revision X, checkout/update repository 2, get revision Y.
- Ask Nexedi ERP5 to start unit test for a particular test_suite title with following pair of repository and revisions:
[('repository 1',X), ('repository 2',Y)].
- And then Nexedi ERP5 check if there is already running test, see there is one, so returns previous commit x, and y;
[('repository 1',x), ('repository 2',y)].
- Then runTestSuite reset git repositories and launch test for
[('repository 1',x), ('repository 2',y)].
Like this we are sure that all computers running the same test suite will be synchronized.
Hardcode erp5testrunner directory¶
Sometimes developer may need to create a testnode, and log in. To see what happened in that testnode when running a test suite. In order to do that. Developer need to create a webrunner and install erp5testnode software release. Before compile the erp5testnode, you need go to software/erp5testnode/instance-default.cfg . Change the working-directory to a custom directory path, shorter is better, e.g: /srv/slapgrid/slappartX/t. Don't forget to change the md5sum in the software.cfg. See the example in here: [HARDCODED] Hardcode erp5testrunner directory
Then when the testnode download the repositories which defined in a test suite. It will store the code in this working-directory. Then developer should follow the Test Nodes section in above, add that three parameters in Webrunner -> Services -> Parameter.
Important notes:
- Again, this step should be done just right after the webrunner setup was done, before install the software release.
- Like described in above, the testnode download the repositories which defined in a test suite. But if you invalidated that test suite. The repositories which get downloaded in the will be
working-directory deleted. More specifically, suppose your working directory is /srv/slapgrid/slappartX/t, and your reference of test suite is aaa, then your repositories will be cloned in /srv/slapgrid/slappartX/t/aaa. If you invalidate that test suite and validate it again. The reference will be changed, like bbb. Now aaa will be delete, everything you left in that directory has gone.
- If you keep the testsuite stay in the validated state. Change the code in the repository which cloned by testnode under working directory is not recommended either. Because testnode has a mechanism to "rollback" to the commit and branch which defined in the test suite. If you did some modification to the repositories which under the working directory. It could be revert silently.
Related Articles¶