Failure in Railway Signal Box Altona Germany Real

Failure in Railway Signal Box, Altona Germany Real Time System Failure A Case Study Vijai Raghunathan Instructor : Dr. Lumpp Course : EE 585 Fault Tolerant System, Fall 2006 1 University of Kentucky

Introduction l l 2 In 12 March 1995, German railway replaced switch tower with a computerized system. Computer System – by Siemens Railway Station Involved – Hamburg Altona – extremely busy station with 30000 passengers every day. University of Kentucky

Altona Station 3 University of Kentucky

The Old System l l 4 250 rail shunts A few hundreds of signals 7 major switch stands 50 experienced switchmen University of Kentucky

Electromechanical Systems 5 University of Kentucky

The New System l l l l 6 18 switch stands All the above controlled by Intel 486 based real time systems One central operating and displaying system (BAR 16) coordinating all stands. BAR 16 – 16 bit interface BAR 16 – redundant hardware with 2 processors, ram and no disk Incompatible with the old one……. cannot run in parallel with old one. Needs 40 switchmen lesser than old system. University of Kentucky

The Computer System 7 University of Kentucky

Failure of New System l l 8 BAR 16 failed immediately after start up. Cause not found for hours. So Altona station temporarily shut down. Passengers – forced to take other railway routes 25 kilometers away. University of Kentucky

Cause l l l 9 Programming Error Possible Stack Overflow Condition Routine handling stack overflow went into dead loop. Allocation of stack required a few more bytes over 3500 bytes. But RAM was only 3500 bytes. University of Kentucky

What happened ? Actual Hardware Present 3500 bytes 10 RAM Program’s stack size Stack Size (algorithm) Error zone * Size>3500 bytes University of Kentucky

Bug Fixed l l l 11 The bug appeared only twice in 4 days. Finally on Wednesday (system failed on Sunday), bug was fixed. A new RAM with 4000 bytes was built to allocate the stack routine. 4000 bytes (500 more to give a safe upper limit) Rail traffic back at 1: 59 pm on Wednesday. University of Kentucky

Conclusion l l l 12 Hidden Faults – tough to find! Siemens Manager – said the team felt the stack routine would not be used due to the presence of a good RTOS. Assumption that “stack routine” will never be used was a mistake. Expect your software to run in the worst situations while designing RT Systems. Altona Incident - Another Lesson Learnt by engineers! University of Kentucky

References l l l 13 http: //catless. ncl. ac. uk/Risks/16. 93. html#subj 1. 1 http: //delivery. acm. org/10. 1145/780000/773574/p 7 neumann. pdf? key 1=773574&key 2=7387399511&coll=GUIDE& dl=GUIDE&CFID=1337449&CFTOKEN=44661065 http: //www 5. informatik. tu-muenchen. de/~huckle/bugse. html University of Kentucky