History: From Mainframe to Desktop

A Central Bank in charge of the monetary union in height countries decided at the end of the 90s to migrate from a traditional information system based on an accounting software running on central mainframe and a lot of paperwork to a Zero Paper approach based on Common Off-The Shelf Software (COTS) and a modern IP networking. Satellite connectivity and IP routing was deployed all over the 20+ branches of the central bank using technologies from Intelsat and CISCO. Desktop computers provided by IBM were installed to replace terminals. Oracle applications were deployed and configured, partly by third parties and partly by engineers of the Central Bank, to implement accounting and payroll. In just a few year, the Central Bank modernized its information system.

The Central Bank then discovered that open source could provide a lot of flexibility and cost savings. Initial email platform was based on Lotus Notes technology while productivity software was using Microsoft Office. Both were replaced by open source. Email migrated to Thunderbird and Postfix. Microsoft Office was replaced by Open Office. Management of printers and file sharing was migrated to SAMBA. By using open source, it was no longer required to plan investments one year in advance in the IT department budget, which in numerous cases helped providing a solution in short time to the users of the Central Bank, without having to break the management rules of the Central Bank in terms of purchase. Use of open source also helps reducing costs tremendously, actually more than 2000 € per user.

Yet, Common Off-The Shelf Software (COTS) failed in one case: the implementation of core business processes of the bank in relation with currency issuing, monetary circulation and the management of banking transactions. A standard proprietary ERP was first configured and deployed. However, a lot of custom developments were required to implement regulatory requirements of the bank. The security model of the standard proprietary ERP could not match with the rule based approach access control policies in the Central Bank. And, as soon as the number of users increased, scalability problems prevented using the standard proprietary ERP. In addition, at that time, the supplier of the standard proprietary ERP supported only Internet Explorer which prevented the Central Bank moving further on towards open source desktop.

The Central Bank thus searched for an alternative which could support its regulatory requirements, which could scale and which would be compatible with any Web browsers including over a high latency VSAT network.

Challenge: 60 regulatory workflows, 10,000 security groups

Central Bank operations derives from international treaties and regulatory guides which need to be approved either by finance ministers or by executives of the Central Bank. They influence directly the way banking business processes are implemented. It usually takes longer to change such regulatory requirements than it takes to implement an ERP. An ERP for a Central Bank should thus be able to adapt its processes and workflows to regulatory requirements.

The Central Bank has 64 core business processes. We can name a few here. monetary emission and destruction, check payment, account transfers, traveler check sales and purchases. It consists of 25 branches, each of which has at least 290 sites such as the main vault, counters, cash sorting buckets. Staff of the bank is organized by functions and directorates. There are 35 functions in the bank such as: accountant, analyst, cashier. There are 8 directorates, each of which has divisions and offices. The total number of directorates, divisions and offices is 4943.

Only certain staff with certain function, working in certain branch or site and in a certain directorate or division are allowed to perform certain actions as part of one of the 64 workflows. This makes a total of 64 x 4943 x 35 = 11072320 possible security groups to support in relation with the configuration of workflows in the ERP.

The Central Bank also needs to interface to other banks using SWIFT network for example. Each regulatory workflow should thus be configured in such a way that the result of a transaction feeds an appropriate application: accounting, payroll, clearing, international money transfer, etc.

After an international tender, ERP5 open source ERP was selected as the core platform of the Central Bank to implement its regulatory requirements.

Implementation: Document based Implementation to Match Existing Procedures

ERP5 is a document oriented ERP based on a unified model of business: the Unified Business Model (UBM). Thanks to UBM, ERP5 does not need more than 10 tables to store core ERP properties. Most traditional ERPs would need thousands of tables to achieve the same result. Also, ERP5 documentary approach can adapt to existing documents and to existing processes of an organization. ERP5 documentary approch thus does not require changing habits or internal processes. This was essential in the case of the Central Bank because of the regulatory nature of its core processes.

The implementation approach which was used at the Central Bank is called: "ERP5 Implementation Process". This approach is fairly simple: describe existing business processes through test cases, collect all paper documents which exist in the bank and synchronize data from and to ERP5. However, before doing those three steps, some abstract analysis is required in order to make sure that the Unified Business Model applies to the Central Bank. Although we have never found in 10 years a single business process which does not fit into the Unified Business Model, it is safer to spend a few weeks doing some theoretical assessment before investing a couple of man year in implementation effort.

The asbtract analysis of the Central Bank core processes has lead to a domain specific business template called "ERP5 Banking", which consists of a few hundre lines of python code. "ERP5 Banking" encapsulates generic business rules which eventually simplify the implementation of any application in the field of banking and payment. It is now used for example to implement mobile payment systems, tolling systems and virtual currency clearinghouse. "ERP5 Banking" has thus been designed as an application of the Unified Business Model with enough genericty to cover any banking process besides those of a Central Bank.

The project was conducted in 3 phases: prototyping, implementation and finalization.

During the prototyping phase, the project was then divided in 3 teams. A first team collected all daily management samples for all documents of the central bank: forms, receipts, reports, etc. This team consisted mostly of engineers of the Central Bank. A second team implemented a prototype for a single process. This second team included a mix of Nexedi engineers and engineers of the Central Bank. The third team implemented a synhronization system for bank acount holders. This third team consisted of Nexedi engineers.

The collection process of samples for all daily management documents helped defining the project scope: all transaction forms, receipts, reports, workflows interfaces have been considered. Through the prototype implementation, a template module was created with unit test coverage, accelerated input form, reports, interface to an external system and documentation. This prototype demonstrated the practical suitability of the "ERP5 Banking" abstraction". Synchronization of account data through SyncML protocol demonstrated the possibility to interface Oracle-based applications and ERP5 either to migrate legacy data or to update data in legacy applications.

The implementation as such could then start. Each of the 60 workflows was modelled after existing paper documents used for daily management of the Central Bank. Each paper document - such as a money transfer form - usually shows different states of validation materializd by handwritten signatures which can later trace responsibilities for decision making. Each state of validation acts as workflow state which is then configured in ERP5 Web based workflow editor. Each paper document also shows different properties such as date, account number, amount, currency, etc. Each property was mapped to existing properties and base classes of ERP5 Unified Business Model. Once a workflow and form was configured, security and permissions were defined in order to make sure that only certain profile of staff could perform certain changes of workflow state, alsa known as workflow transitions. The last transition of workflow usually trigerred an insertion of data in the Central Bank event bus which acts as a central vehicle to interface all applications with the accounting application. For each workflow, a set of unit tests and functional tests were developped to prevent any future regressions.

The implementation process took about a year. In addition to daily management documents, standard regulatory reports were implemented with the gaol to reproduce perfectly existing paper reports. Making sure that the new ERP5 based application would work exactly as the current paper based workflows was a clear objective and - as we will see bellow - a key success factor. The first acceptance tests, organized through so-called "laboratory" based on explicit test cases written by staff of the Central Bank showed a functional match of more than 94%. The remaining 6% was due to a few features which were not consistent with daily business processes, which changed during the implementation or for which non regulatory documents were used.

As part of the finalization phase, a few more weeks of development were needed to take into account results of the first user acceptance test. A second user acceptance laboratory, 9 months after the end of the implementation phase, showed a 100% feature coverage. The reader will thus wonder what was finalization about if... nearly all features were there. The finalization phase was about scalability. A first tentative implementation of proprietary ERP had already failed due to scalability. For this new ERP5 implementation, Nexedi took the decision to wait until it would be possible to demonstrate the ability of ERP5 to handle twice as much data with twice as much transactions as the Central Bank expectations over the next 5 years after the start of production.

Scalability tests were thus developped to mimic the exact usage of ERP5, including delays, time to discussion with customers, lunch break, etc. Initial results were terrible. As soon as 2 users were entering transactions at the same time, ERP5 transaction management system was preventing one of the two to complete their transaction. The reason was simple: banking transactions were assigned a sequential incremental reference which is incompatible with truely parallel processing of long transactions. The transaction numbering was thus changed to include a random part after a date prefix. Little by little, by optimizing indices and table structures in MySQL, by using new algorithms for activities in ERP5, by accelerating ERP5 catalog, high performance with 300 concurrent users could be reached. Moreover, the whole ERP5 application could be fully tested with 300 simulated users before going into production.

On february 4th 2008, all existing procedures of the Central Bank were moved at the same time in all 25 branches from paper and spreadsheet to ERP5.

Success Factors: Customer-Supplier Joint Implementation and Novel Security Model

ERP5 ability to implement the 64 regulatory workflows with high flexibility and productivity is obviously a key success factor. ERP5 user interface, which relies on standard HTML forms and a concept of "single HTTP request per transaction" was essential to make the application usable over high latency VSAT networks. But they do not explain alone the successful implementation.

The strong involvement of staff of the Central Bank in the project tremendously helped accessing the know how on regulatory procedures and reports. Without staff of the Central Bank, Nexedi could never have reached alone the same result. Staff of the Central Bank involved in the project are mainly engineers trained in excellent universities. One other key staff in the project is an ex-branch manager with more than 20 years experience and knowledge of all procedures. The project also remained secret during more than 2 years. It was announced only 6 months before being put in production. Initially many executives of the Bank thought the project would fail and thus did not interfere. This helped reducing the bandwith required to manage the project and helped finalizing it in short time. In contrast, many ERP projects in the world fail because too many executives are involved, extend the feature requests and create a bandwidth bottleneck in project management.

Other ERP5 key technologies also played in key role in the project. The use of Zope Object Database (ZODB) to store banking data with a NoSQL approach showed excellent scalibility, probably better than the scalability of any relational database. The use of activities, ERP5 asynchronous parallel processing technology was instrumental in processing a lot of transactions in parallel and grouping them by small batches in near real time.

Another important success factor is the novel model of ERP5 to manage access permissions. Most applications frameworks use a user, group
and role approach. Users are member of a group. A group is granted a role in a context. Only certain roles can perform certain actions or access certain data. This model simple and suitable for simple applications. But as soon as the number of workflows is high, two phenomenon happen: role explosion and group explosion. Application developers tend to create a few roles for each workflow. With 64 workflows, we have about 300 roles. Application developers also create a group for each combination of function, diretcorate and site. In the case of the Central Bank, creating 1500000 groups would be required.

To solve group explosion and role explosion, ERP5 introduces a new approach to permission management: rule based permissions. First, ERP5 acknowledges that most workflows in management do not need more than 5 roles, which are called the "5A" in ERP5 terminology: Author, Assignee, Assignor, Associate and Auditor. The Author role defines who can create a new transaction. The Assignee role defines who is in charge of initiating the transaction. The Assignor role defines who is in charge of validating the transaction. The Associate role defines who may interfere in the validation process. The Auditor role defines who can access the document for consultation only.

ERP5 then provides a rule based system which can be used to specify which functions, directorates and branches are assigned which of the "5A" roles. Rules can look like "all users of the same directorate as the author of the document and with a function of vault manager" or "all users of the headquarters with a function of accoutant". Each rule thus consist of 2 or 3 lines of predicate definition. ERP5 interpretes predicates in an efficient way and attaches to each transaction a collection of so-called security codes which are stored in the database with transactions. Permissions to access transactions are thus intimately related to the Zope Object Database persistent machinery. This reduces the risk of information leaks which often occure whenever security and permissions are externalized to another system. Security codes are also indexed efficiently in MySQL relational database. Querying a list of transactions visible by a given user of the system can thus benefit from fast indices. This is again a strong advantage of ERP5 architecture compated to approaches based on external permission management systems or filtering.

Solving Difficulties: Open Source for Reactive Support

When the project started on february 1st, everthing went fine. All users could use ERP5. It was scalable as expected. However, a few transactions were not properly indexed in ERP5 catalog. Zope Object Database and MySQL showed transactional behaviours which had never been detected before and which could lead to blocked workflows. This behaviours had never been found during 1 year, even though simulation of Bank processes was with a security ratio of 2.

Apparently, reality was a bit different from simulation. Certain delays between transactions could not be similated. After some research, Nexedi developers identified the source of transactional mis behaviour in a complex combination of Zope core database, ERP5 activity system and MySQL. Within 24 hours, by changing one line in the source code of Zope and ERP5, the problem could be solved.

Imagine now the same situation with a proprietary database, a proprietary application server and a proprietary ERP, each of which with support contracts. This kind of situation is very difficult to analyze without access to the source code of the three components at the same time, and without a team with direct access to the knowledge of core components. This kind of team is close to impossible to find in the world of proprietary software.

Only open source allows to build such teams. Nexedi has learnt through scalability testing of ERP5 how to recompile MySQL - now MariaDB - and optimize performance. Nexedi team has learnt the internals of Zope by conducting scalability tests during one year on large data set. With a proprietary database and application server, this would not have been possible.

Future Evolutions: SlapOS Cloud for Global Banking, Distributed NoSQL

After 4 year of safe opearion, ERP5 has evolved at the Central Bank to include more workflows. Interfacing with the automated real tim clearing system and with the international money transfer is now integrated completely integrated to ERP5. This has saved large amounts of budget to the Central Bank.

Staff of the Central Bank is able to extend ERP5, create reports, interface ERP5 with external systems such as automatic teller machined or new payment systems. ERP5 has also been interface to JEDOX PALO business intelligence software to provide user-configurable online reporting.

Recently, ERP5 was ported to Zope 2.12. The latest upgrade in the bank has used SlapOS Cloud Operating system. The whole build, deploy and run process of ERP5 Banking is now fully automated by SlapOS.

In the future, ERP5 will evolve to adopt a new user interface mostly based on Javascript and JSON, while retaining the "one HTTP request per transaction". A new version of the Zope Object Database, called NEO and which relies on distributed storage, is planned to support very large data sizes and automated disaster recovery. After five years of research, NEO can now provide better performance than ZEO with high reliability. With NEO, ERP5 Banking can sustain 20 years of data. MySQL will also probably be replaced by MariaDB due to the lack of relevant commercial support after MySQL the takeover of Sun Microsystems by Oracle. On the other hand, MariaDB now enjoys rapid development and support for critical applications.

The story behind MariaDB also demonstrates the power of open source over the long term. Even in case of hostile takeover, developers of an open source can still escape and can keep on providing support to users of the open source project. For large organizations such as the Central Bank, ERP5 open sourceness is the best guarantee to maintain its core system for then next 20 years.