The customer was embarking on a project to be the first in the industry to offer a state-run lottery sales and online Internet gaming product. The project required the implementation of a high performance service bus that could handle peak loads of up to 10,000 transactions per second. Because the system supports online internet gambling, regulations require that every transaction be logged, thus requiring a logging system that could keep pace with the service bus and log every transaction as it occurred in real time. The requirement was to create one log entry when the transaction started and one when it ended, therefore the persistence mechanism had to support up to 20,000 inserts per second. In addition to the high volume of operations that needed to be persisted, the logging system had to provide the following additional characteristics.
- High Availability & Automatic Fail-over. Every transaction had to be logged. If a transaction could not be logged, then the transaction could not occur and revenue would be lost. As such, the logging system needed to be highly available, using clustering technology that supports automatic fail-over if a node in the cluster goes down.
- Disaster Recovery. The logging service needed to support off-site disaster recovery via log-shipping, replication or mirroring.
- Scalability. The selected storage/database technology backing the logging service needed to scale as the transaction volume on the system scaled. Traditionally, database systems scale either vertically (bigger, faster machines) or horizontally (additional machines). Initial requirements needed to support 20,000 log entries inserted per second, and needed to allow for growth well beyond that.
- Durability. Every transaction logged must be committed and be permanent. This means that writes need to be committed to disk and/or replicated to other nodes before they can be considered durable. Lazy-writing or flushing of data to disk cannot be used, because a system failure would lose data in memory that has not yet been written to disk.
The objective was to select a persistence technology to meet the above requirements as well as implement a logging service that encapsulated the persistence technology and exposed the logging operations to the service bus.