Final Report of Suspended Trading Session on May 12

6.23.2009

Tokyo Commodity Exchange Inc. (TOCOM or “the Exchange”) reported today that it has determined what ultimately caused the connectivity failure between the Member systems and the Exchange system which led to the suspended trading session on May 12, along with the preventive measures and disciplinary actions to those who are responsible for trading system operation.

TOCOM has found it very regrettable that the suspended trading session on May 12 between 11:35 AM and 3:00 PM caused inconveniences to market participants and other parities concerned. The Exchange has taken preventive measures on the following day, and has made sincere and continuous efforts to identify causes of the problem.

1. Cause of the problem

It has been identified that the combination of the following three factors caused the connectivity failure.

(1)In configuration of the concerned router, it was set to log messages as well as to display messages to the console.

(2)The network service provider repeatedly executed cutoff and reconnection in a very short time during their construction work on the line connected to the concerned router on May 12 from 2:00 AM. This fact has become clear by the log messages of the router.

(3)The routers were set to configure the heartbeat between them more frequently than usual based on recommendation from NADAQ OMX (package software vendor). Such a configuration has been adopted according to NASDAQ OMX’s recommendation because TOCOM’s trading system was built on the network, which can be switched over the line without affecting market participants’ trading system. Therefore the heartbeat interval has been set to configure monitoring more frequently than it is in the ordinary cases.         

   

In other words, this connectivity failure occurred because network line construction (above #2) and the router’s configuration (above #1 and 3) have jointly resulted in causing the high CPU loads of the secondary router and primary router. The former became high from 2:00 AM and the same status continued after the day session has commenced, and consequently, the latter gradually fell in the same situation.

With regard to such router configuration, NTT Data, support and maintenance provider for the Exchange’s new trading system, consider the configuration of logging and heartbeat are effective as maintenance requirements for the swift detection of connectivity failure and post analysis. They also recognize the method of monitoring routers to be legitimate because CPU utilization has always been normal throughout the six months between the acceptance test in last November and the launch in this May. However, NTT Data had planned to execute statistical monitoring and to improve router’s configuration, if necessary, after the system went live.   

In case there was an extraordinary event in relation with our data center, such as this construction work on the network line, it would have been necessary to verify that there was no negative effect after the construction and before the trading session with the utmost care and attention. It is so, even if the monitoring method is free of defect under normal operational environment.

2. Preventive Measures

(1)Preventive Measures Already in Place

TOCOM has already deactivated the router’s logging in order to prevent the same incident from reoccurring. Additionally, the following preventive measures have been put in place.

  • Established a monitoring system for router’s CPU loads
    TOCOM established a system to monitor CPU load of the router at the Exchange’s data center since the trading session resumed on May 12. On May 15, the Exchange has installed monitoring software for the router CPU loads, which performs real-time monitoring and issues an alert when the CPU load reaches beyond threshold level
  • Established a procedure for restoration (without suspending the trading  session) in case of the reoccurrence of the same incident
    TOCOM will recover the network connectivity promptly by rebooting the troubled router in case of (?) detecting the same incident at the data center.
 
 
 

(2)Additional Preventive Measures

NTT Data has proposed to TOCOM that log messages of the router be sent not to the console, but to a separate server, under the policy of sustaining such maintenance requirements that allow prompt analysis in case of network connectivity incidents because the connection failure on May 12 was caused not by defect of the router itself, but by severe condition of its use. TOCOM will decide whether or not to implement the proposed measure after receiving the final proposal from NTT Data.

 Additionally, TOCOM has requested NTT Data to report a study of similar incidents and a proposal of the improved operational system before the end of July. The Exchange will execute necessary measures based on this report.

3.  Internal disciplinary actions

In May 27, TOCOM has accepted President and CEO Masaaki Nangaku’s proposal to voluntarily return 10% of his one monthly compensation to take responsibility for the Exchange’s failure to fulfill its duty to provide a stable market and the suspended trading session on May 12.

Following the determination of the ultimate cause of the connectivity failure, TOCOM acknowledged insufficiency of management for safe and stable market operation and its board of directors gave reprimand to Nangaku. He gave reprimand to Kazunari Hayakawa, Executive Managing Officer, and Yuichi Fukui, Executive Officer, both of whom are responsible for the trading system operation.

The last, but not the least, TOCOM once again express deepest apologies for causing inconveniences to market participants and concerned parties with the connectivity failure and consequent suspended trading on May 12.

TOP