Utilising Orkhestra and supporting Code Magus Tools, CML is able to offer an efficient, repeatable and consistent environment where Orkhestra drives the testing of the non-functional portions of applications. This non-functional testing is used to determine server capacity requirements, response times and throughput under various loads and the system’s (or elements of the system’s) behaviour under long running; extreme loads; and fail over and recovery scenarios reporting on recovery time and impact on (successful) customer throughput. Tests include determination of the profile under extreme conditions to check and / or ensure best behaviour under these conditions. For example, whether throughput collapses past saturation or remains level.
As well as supporting the hardware/environment testing, we also provide a Workload analysis from a business point of view in order to support business requirements. During an NFT a model of the system under test is built up of various entities such as a community of users, devices and peer businesses to reproduce load to align to the workload analysis. CML installs resource monitoring probes on machines (wintel, solaris, linux, AIX, zSeries) then collects resource data during the test. The test is then varied in order to obtain strong correlations (if they are available) by doing a statistical analysis of the tools measurements against the probes measurements. The tools measurements typically include response times and throughputs against successful, failed or fault responses.
The CML Tools facilitate complete end-to-end testing: including all systems and all security schemes that would be required in a production environment from extracting entities according to an application Entity Relationship (ER) model from applicable systems where entities satisfy test criteria; Through shuffling the entities; to using the entities in a non-immediate reuse manner; to final analysis. The end to end nature of testing allows tests to avoid cache hot-spots, network bottlenecks and Data Base locking and hence avoiding any skewing of the test results.
CML also implements Non-functional testing across applications that hand-off to each other and which require more than one channel to operate concurrently; For example authentication that provides a certificate to inject into a WAP application,
It has, at times, been necessary to invoke applications on different channels in the same NFT test.
Examples of this are:
- invoking a web service to reset a user’s password thereby receiving a temporary password then,
- invoking a proprietary application to reset that user’s PIN and finally invoking the Web-based retail application to change the temporary password into a permanent one. This is all done in the same state machine.
- another example of this was in the middle of executing a web-based application, a different application was invoked to perform a financial transaction, once again all in the same state machine
- a more complex example is the hand-off from one state machine testing a Mobile Application platform to another state machine executing a WAP application. During the hand-off, the URL and user’s security token are sent from one state machine to the other and a banking transaction is executed; Thus mimicking the behaviour of a mobile banking platform.
- obtaining a one-time password on one channel and then using that as the credentials on another.
For end to end NFT the CML Tools facilitate maintenance of large amounts of application data in a usable state. All security contexts of a large number of users may be easily maintained. These include users, certificates, profiles, roles, etc., which in turn includes maintaining userIDs, passwords and PINs.
The CML Tools also help to maintain schemes for the management and maintenance of driving test data and instruments, such as mag stripe cards, chip/ICC cards, for Debit, Credit, Fleet and private label applications.
CML makes use of external tools to complete analysis of any NFT run. R  is a language and environment for statistical computing and graphics, gplot  is a portable command-line driven graphing utility. These tools are used to generate statistical results often rendered in an applicable graphical view.
Non-functional testing of large-scale Web-based applications
CML also supports complex data extraction for end-to-end testing with:
- Automated provisioning of identities and generation and linking of certificates used for encryption.
- Provision of web-service simulators to simulate external service provider applications
- Scripting of Web applications produced by frameworks such as .net and ajax and automatic generation of state machines
Non-functional testing of large-scale networked applications using proprietary or custom network protocols
Additional functionality to the Orkhestra tool suite allows for automated scripting state machine generation and testing of applications which use proprietary or custom protocols. An example of this is the Entersekt EMA server binary protocol.
The ability to monitor production server systems or systems under test in real time and retrospectively is key to understanding the throughput, responsiveness and overall health of a system in order to affect changes for improving that system. CML probes (querying operating system and server application indicators) and CML server applications generate metrics and forward them on to the CML Metric Recorder and Proxy. The Metric Recorder and Proxy can record the metrics to a log for later analysis or replay and also forward the metrics in real time to a real time consumer such as CML Serfboard where they can be rendered on a dash board. Thresholds can be set so that support staff are notified as soon as the indicators pass the threshold value. These notifications are either visual on the dash board or back office channels such as text or email.
Metric recording utilizes Serfboard, Probes – Windows, SNMP, AIX and Solaris, gplot and R.
Orkhestra and the CML Tools support many Non-functional banking specific applications tests including:
- Mobile applications through Wireless application Protocol (WAP), USSD, XML, HTTPS and proprietary cell network to application hosting servers. Application types dealt with include Mobile retail banking, Internet retail banking, mobile business banking.
- ATM, POS, Integrators (retail acquirers) – NIPS BICISO Specification Postilion Switch environment Version: v1.43, ATM types tester, proprietary protocol and message layouts, TermApp – Postilion TermApp.ISO External Interface Specification Version 5.0, ACI Postilion PostBridge – Postilion PostBridge Interface Specification Version 8, ACI Base24/BIC ISO – ACI BIC ISO Standards Manual BASE24® and NIPS BICISO Specification Postilion Switch Environment Version: v1.43
- Retail applications include functions for: Mobile top-ups, pre-paids, transfers, payments, beneficiary maintenance, standing and debit orders
- Internet business banking
- Back office applications with traditional screen interfaces (such as IBM 3270, Data Stream, or VT100/VT220 type terminals).
- Web-server hosted applications, web-services for business to business, application to application
Vehicle/asset finance applications
- Systems that interact with local/country transaction switches
- Card associations such as MasterCard International, VISA International and AMEX/American Express and Diners
- Automated cheque processing and cash management applications
- CRM/Customer Relationship and Customer Information Management applications
- Biometric based verification and identification applications and Biometric management applications
- Payment systems
- Credit and loan applications
- Globus Teminos T24 banking applications including, but not limited to Teller, ATM, POS
- Wall Street Treasury applications