This documents how different parts of ERP5 should be tested, as well as what tools are available. For information on how to use tools, please take a look at HowTo.

Table of Contents

• Testing Frameworks
  • ERP5TypeTestCase
    • SecurityTestCase
  • ERP5 Test Tool
  • erp5mechanize
  • Simple Scripting
  • Test Cases
• How to Select a Testing Framework
  • Unit Testing
  • Functional Testing
  • User Interface Testing
  • Performance Testing
  • System Testing
  • Manual Testing
• When You Should Write Tests
• Related Articles

Testing Frameworks¶

ERP5TypeTestCase¶

ERP5TypeTestCase is based on ZopeTestCase, and it provides an unit testing framework mainly. ERP5TypeTestCase generates a portal site on the fly, and executes one or more test suites on it, then discards the temporary site.

The advantage of using ERP5TypeTestCase is that you can obtain fine-grained control of behaviours, because you write Python code in unrestricted environment. The disadvantage is that the framework is often too low-level, if you want to test a feature closer to the user.

SecurityTestCase¶

SecurityTestCase is a class extending ERP5TypeTestCase, and adding tools for permissions and security checks.

Assertions such as:

assertUserCanAccessDocument
assertUserCanAddDocument
assertUserCanChangeLocalRoles

and many others are defined there to ease your work.

ERP5 Test Tool¶

ERP5 Test Tool is a tool available under a portal site, named portal_tests. This is based on Zelenium, which, in turn, is based on Selenium. You can make as many as test suites you wish to have under portal_tests, and you can run tests manually or semi-automatically by passing ?auto=true to portal_tests on a web browser.

The advantage in ERP5 Test Tool is that you can interact with ERP5 in a visible form, and it is relatively easy to write tests on Selenium. The disadvantage is that Selenium is basically one-directional, thus you cannot put complex expressions in it. Therefore, it is often necessary to write Python Scripts separately, and call them from tests.

erp5mechanize¶

erp5mechanize is based on mechanize, and provides a tool to control a browser-like object from Python.

Although writing tests with erp5mechanize is not as intuitive as with Selenium, erp5mechanize is more flexible and less resource consuming. Thus erp5mechanize is suitable to perform complex scenarios, simulate many users, and profile real performance.

Simple Scripting¶

This is not very specific to ERP5, but it is also possible to simply run a batch script from a command line, for example, by using wget or the urllib module in Python.

Test Cases¶

Test Cases¶ are documents stored in Test Case Module, and managed by the Business Template erp5_consulting. Each Test Case documents how to test a business process with ERP5, in order to ensure that an implementation follows project requirements.

The original purpose of Test Cases was to specify how ERP5 is configured for a user, but this is considered a part of testing frameworks in ERP5.

How to Select a Testing Framework¶

Which testing framework should be used depends on what you want to test. It is critical to select the most appropriate one in a long run.

Unit Testing¶

When you want to test a unit -- a small component of a system -- you should use ERP5TypeTestCase. There is no alternative which gives you the same level of control.

Functional Testing¶

This depends on a meaning of functional testing. In ERP5, functional testing means cross-units testing, sometimes known as integration testing. For this purpose, you should use ERP5TypeTestCase. Although ERP5TypeTestCase mainly targets at unit testing, there is no much difference between unit testing and functional testing when writing tests in ERP5. Thus reusing the same framework makes sense.

User Interface Testing¶

User interface testing is often assumed to be a part of functional testing, but we distinguish them for clarity. When you want to test a feature that interacts with the user directly, such as ListBox, it is an user interface testing.

For most UI testing, you should use ERP5 Test Tool, because this is the most effective way to describe and check an UI. But if you need to describe a complex scenario with which involves multiple actors on a module, you should use erp5mechanize, because Selenium is not that powerful.

Performance Testing¶

Testing performance is a difficult issue, and there is no simple answer for this.

If you test the performance of use cases, you should use erp5mechanize, because the other methods are not flexible enough or too much resource-sensitive.

If you test the performance of a small component of a system, you can use ERP5TypeTestCase for this purpose.

If you test extremely simple cases, such as stressing a system with high loads, simple scripting is the most efficient.

System Testing¶

Umigumi has a tool called umitest which is suitable for system-level testing. This way, you can test if ERP5 works well in a given installation of a whole operating system, but it is likely that you need to combine other testing frameworks to perform precise checks, especially ERP5 Test Tool.

Manual Testing¶

The last resort is manual testing, namely, testing a feature manually by human. Unfortunately, not all tests can be completely automated or too hard to be automated, thus using manual tests is inevitable from case to case. For example, if you want to make sure that a page is rendered with conformance to a user's preference in a given browser, it is nearly impossible to determine whether a page is rendered appropriately in a mechanical way.

You should write Test Cases for manual testing. But note that this is the last resort. Manual testing is not automated by computers, so it costs, and it won't be performed regularly. Whenever possible, you should use other methods. Use manual testing, only if you may not avoid it by any means.

When You Should Write Tests¶

In principle, whenever. Whenever that makes sense.

If you find a feature which is not tested, you should write one or more tests for it.

If you find a bug which is not tested, you must write a test before fixing it.

If you write a new feature, you should write one or more tests for it.

Also, do not forget to run existing tests when you modify anything, in order to make sure that you would not break any existing feature. If you introduce incompatibilities, you must update all tests which are affected.

Related Articles¶