Testing modern signalling systems

Testing new and modified signalling systems has always been vitally important to ensure that they work safely at the time of commissioning. Some signalling testing processes have changed little since safety logic started to be implemented by software and processors, rather than by relays. Signalling systems are also becoming more complicated, interconnected and software controlled. So how can signalling engineers ensure that modern software processor-based signalling is tested correctly? This was the subject of a presidential paper, presented to the IRSE in March by Robin Lee of Park Signalling. Robin had been assisted with the preparation of the paper by the IRSE International Technical Committee.

Robin began by explaining that signalling communication-based control systems, such as ETCS Level 2, will reduce the number of trackside assets such as signals. Functionality is shifting away from the trackside onto trains. On-board systems, often from different suppliers, interface with trackside equipment for movement authorities, and such systems are now connected by IP communications and radio.

Another change, at least for European railways, is interoperability. More trains running across borders and the separation of the railway infrastructure managers, operators, and vehicle owners, together with phased procurement, increase the number of on-board systems that need to interface with each other. All this adds to the number of equipment types that need testing.

The social and political environment is also changing. Railways are recovering fast from Covid-19 and becoming busy again, and it’s not easy to close railways to renew, test, and commission new systems. Cost savings also need to be made to signalling without degrading safety. There is also a need to increase personal safety by minimising lineside working.

Testing today
Today, the testing of signalling is mainly based on independent inspection and functional testing at a number of levels – from single sub-systems to full final configuration testing. A significant amount of this occurs off-site prior to the integration of the whole system, with designs and application data reviewed and individual sub-systems tested independently. Items are then installed and connected, and the system inspected and tested as a whole before it is validated by ‘principles testing’. Testing and inspection evidence is essential for safety case compliance and for the system to be signed into use.

Unfortunately, certain hazards can never be addressed by off-site testing, such as proving that a point machine has been installed and wired up correctly in the right location. Care must be taken to properly test any connections made on-site to ensure the off-site testing is still valid for the final system. Pre-formed connectors can help, but blockades will always be required for on-site system testing.

Robin explained that, for over 30 years, electronic interlocking data has been tested off-site by simulating the trackside environment. Routes can be set and results seen on a real or mimic panel, and test harnesses can be used to simulate trackside equipment states. Other interfaces, such as the operator interface and maintainers terminal can also be tested without disrupting the railway.

Test coverage
It was explained that not all signalling systems can be fully tested and not all combinations of inputs and internal states can be covered, and it is only possible to fully test simple logic where a limited number of inputs, each with only a limited number of states, exist. For example, a relay only has two states (energised and de-energised) dependant on one input, which can be tested. But there is a third state where the contacts are open circuit whilst the relay transitions and it could be subject to unexpected voltages.

Assumptions are similarly made with computer-based interlocking hardware and software. Complex software quickly causes the number of combinations of inputs and internal states to multiply, making it difficult to test all of the combinations. Just 30 train detection inputs with two states could result in over a billion test cases. Therefore, assumptions have to be made that certain inputs and internal states do not affect functions where they are not intended to be related.

Cab-based technologies such as ETCS make this problem worse. Trains report their position within 0.1 metres or 1 metre, which the Radio Block Centre (RBC) may be expected to respond to with a large number of possible movement authorities and other data. These cannot all be tested. ETCS can also apply temporary speed restrictions, further increasing the combinations of messages that could be sent to trains. When compared to reading a single signal aspect, it is also harder to manually evaluate if the movement authority information is correct.

When alterations are made to an interlocking, it is considered acceptable to only perform testing on the related functions, on the basis that the functionality of everything else will not be affected. However, when testing very complex systems, the risk of testing only related functions will also increase.

Robin and the ITC recommend that a justification should be produced in a regression test plan. This should include which elements will be re-tested and which will not, with a rationale for why the untested parts remain safe.

Testing aims to reveal defects, human errors, and random faults during all stages of a signalling implementation. This applies at different levels, from singular components (say a relay) up to the full signalling system and should be traceable to the system requirements and the Reliability, Availability, Maintainability, Safety and Security (RAMSS) analysis. Having explained the problems, Robin discussed some potential solutions.

Formal methods
Mathematical methods can be used to remove the need for ‘testing’, instead replacing it with ‘proof’ that certain requirements have been met. This is particularly suited for proving simple rules such as “points shall not move if the dead-locking train detection section is occupied”, to full suites of signalling principles.

However, this still cannot prove the whole system is safe, as use of the proof in the real world relies on assumptions and the specification itself being correct. For example, formal methods cannot prove that point detection contacts are attached to the intended point or that a train detection boundary is not unexpectantly foul of another line. ‘Formal methods’ is a tool and not the complete answer.

Installing a new system in parallel or ‘on top of’ the existing signalling system has been used for some projects, and Robin recommended reading about a Hong Kong project described in January 2020’s IRSE News. This can allow the new signalling system to be tested and allow confidence in the reliability of the new system to be gained gradually during possessions outside of normal operating hours. This method is easier if the new system is cab-based with new axle counter-based train detection systems and avoids new signals having to be covered/ uncovered at the beginning/end of each testing stage.

New train detection systems can be permanently active if they do not interfere with the old system. Points and level crossings, however, are likely to require manual or automatic switch over between the new and old systems, which is not easy and takes time, resource and is expensive.

The benefits have to be balanced against the expense of testers and other staff on site over many weeks, as well as the cost of setting up and giving back multiple possessions, rather than a single possession. New hazards can also be introduced with combinations of system states, interfaces, and human confusion whilst the systems co-exist. New designs of equipment could perhaps be designed to allow for easier switching between old and new control systems, Robin suggested.

Alternatively, in metro systems there are examples of shadow-running Communication-Based Train Control (CBTC) systems, while the existing system continues to control movements and maintain safety until the new CBTC system have been proven.

Standardised interfaces, boundaries
Standardisation of interfaces between the interlocking and other boundaries, such as other interlockings, RBC, and fringes, can reduce testing said Robin, with different features of the interfaces standardised. These include the protocol, physical interface, functionality and/or application rules of the interface.

The good news for the future is that some of this should be addressed in EULYNX (a European 14 Infrastructure Manager initiative to standardise interfaces and elements of the signalling system) and ETCS specifications, which should reduce bespoke solutions and increase the re-usability of testing.

EULYNX specifies the data interface for interlocking-to-interlocking communications and standardisation should allow testing functionality to be re-used, and it should also be possible to easily upgrade and test one side of the interface without affecting the other side.

Remote ‘on-site’ and train testing
Traditional trackside testing involves testers walking through the installation. However, the availability of trainborne video/data collection systems and less trackside assets with cab-based signalling systems now allows more testing to be carried out remotely and/or from the train.

If data is recorded from a test train during a possession, it does not necessarily have to be analysed there and then. It could be done anytime and anywhere. Video/train data could confirm the correct location of trackside items such as balises, and the correlation of point positions could be confirmed by analysing video recordings, possibly automatically suggested Robin. Likewise, Robin asked, could axle-counter system configuration data and head installations be verified by service/test train movements and position reports by forward facing videos?

Fuzz testing
Fuzz testing is an automated software testing method that injects invalid, malformed, or unexpected inputs into a system to reveal software defects and vulnerabilities and can be applied at software component and application levels.

It has become a popular method of testing various software systems, including regular web services, apps, and commercial desktop software. Fuzz testing focuses on identifying reliability/stability issues rather than demonstrating correct functionality. Its main advantage is that commercial off the shelf software can perform such testing without the user needing to write any bespoke tests.

Robin asked if fuzz testing could be used to test signalling application data. Inputs (signaller actions) could be commanded randomly, simulated trains could react in random but legal ways, and defined rules could be monitored (for example points do not move under a train or two-trains cannot approach head-to-head, preventing either from continuing). Could machine learning improve such testing, by learning from human testers?

Digital twins
With testing railway signalling, a digital twin could be a virtual model. Various tools such as simulation, software in the loop, full interface protocols/ latency simulation, and real hardware in the loop, could increase the realism of a digital twin test setup. This could be achieved by just using the schematic layout and asset information (such as control tables) available. Links to Building Information Modelling (BIM) could also help.

This could simulate a number of signalling systems covering a particular area of railway where the simulated systems replicate a significant proportion of their functionality, but do not consist of the ‘real’ software. A comprehensive digital twin for testing would consist of all connected systems, utilising the same software as is in use in the real world, with the ability to run on the same hardware (a physical twin).

Digital twins are normally maintained throughout the whole life of the real twin, and if the digital twin is kept up to date it is much more likely to help integrate and re-test the full system as it is upgraded over the life of the signalling system.

The EUYLNX initiative is aimed at opening up the signalling market and integrating different products from different suppliers. Therefore, the Infrastructure Manager (IM) is best placed to own the digital twin and this will limit suppliers being able to increase ‘supplier lock-in’. Contractual arrangements should ensure suppliers are required to provide suitable digital or physical twin components for any systems they supply.

There are situations where suppliers have provided twins of their products for their customer’s use or shared usage by the supplier and customer. Suppliers must also be required to support these by keeping the twin representative of the real system and to provide technical support. The supplier should be encouraged to make use of the wider digital twin to perform its own integration testing in addition to the IM’s own activities. As signalling assets can last for decades it is important that long term contractual arrangements are established at the beginning of a scheme.

Standardisation makes it easier to establish and operate a test environment based on a digital twin. Standardised IP based systems allow the test harness to inspect, check for expected and unexpected messages, and inject additional messages into the interface without requiring supplier specific knowledge. This also may allow more tests to be re-used, or at least easily automatically generated for signalling applications from different suppliers.

To manage complexity, the test infrastructure may require standardisation, for example the interfaces between the systems under test and the test software, and to eliminate the manual steps required to set up, run the tests, and analyse the results.

Unfortunately, not everything can be tested using a digital twin. For example, it cannot easily check that the equipment/configuration has been installed in the right location. It cannot test radio coverage, nor that the digital map of the layout (and its gradients and other data) is correct. However, digital twins may be able to support the majority of the testing of a signalling system throughout its life and digital twin technology is likely to improve over time.

Case studies
Robin and the ITC have looked at case studies of schemes around the world where some of these solutions have already been applied. These include Singapore’s North-South and East West Lines (NSEWL) which has recently been re-signalled from a track-coded automatic train control system to modern moving-block CBTC system.

The supplier provided the trackside and onboard signalling system, including equipping new trains and retrofitting existing rolling stock and provided a dedicated CBTC simulation facility. The Land Transport Authority (LTA), the owner of the metro system, now requires simulation labs, or digital twins, to be provided for all new lines. The NSEWL has had such a testing facility since 2018, consisting of real signalling hardware, including the trackside and onboard equipment.

It is used by the supplier, LTA, and the Operator SMRT to test operational scenarios, for stress testing, system validation, and to investigate software patches. It has also been used for training, timetable testing and testing of passenger information systems.
Bane NOR, the Norwegian IM, is renewing its whole network with ETCS Level 2 based signalling. A single interlocking/RBC/trackside supplier, and a single Traffic Management System (TMS) supplier are to be used to minimise complexity and ensure standardised equipment is used. This will even result in a single point machine and axle counter type for the whole country.

Campus Nyland is Bane NOR’s test and training lab which is used by suppliers and their own testers. They have found the lab to be very valuable. It allows testing of all parts of the system: interlocking, RBC, traffic management, communications network, GSM-R, object controllers, and real point machines and points. The lab was even adapted to allow remote operation during Covid-19.

Conclusion
Robin outlined the ways in which testing has changed and could continue to evolve, and encouragingly the case studies discussed are already applying some of the solutions explained by Robin and give hope for the future. Automatic repeatable tests have been found to be achievable and highly valuable in testing new systems. Infrastructure managers will need to assess the benefits and cost of digital twins and to contractually require suppliers to support and contribute to a digital twin throughout the life of the asset.

There are likely to be other testing methods emerging and it’s an area where signalling can learn from other industries with far larger R&D resources than rail. It will be interesting to see what future techniques are developed and used for testing safety critical software signalling systems.