The automation of the largest and fastest machine in the world

When the largest particle accelerator ever built becomes operational at CERN, the nuclear research centre in Geneva, physicists hope to gain new insight into matter and what holds it together. The accelerator and its safety will be controlled and monitored by 130 control systems featuring ‘hardened’ automation technology from Siemens.

The underground circular track of the LHC, 27 km in length

Over 10,000 people from around 60 countries have participated in the realisation of perhaps the most important project in basic research, the particle accelerator (Large Hadron Collider, or LHC) at the European Organisation for Nuclear Research, CERN. The LHC is built in a circular tunnel, 27 km in circumference and 50 to 150 metres underground, extending from Lake Geneva to the French Jura.
When the accelerator becomes operational, two proton beams will be fired in opposite directions and brought to collision in multiple collision detector chambers. Scientists estimate there will be around 600 million collisions per second, providing a correspondingly enormous amount of data to help answer essential questions in physics.
Some of the demanding technical requirements include:
 9,600 magnets to guide the proton beams, including more than 1,200 superconducting dipole magnets each 14.2 metres in length;
 To produce the superconductive state, the magnets are first cooled to -193°C using gaseous helium. The temperature of the helium gas is decreased progressively by using over 10,000 tons of liquid nitrogen and secondly to -271°C with almost 60 tons of liquid helium. The refrigerators have a capacity of 18 kW each;
 Of the four main detectors in the LHC, the ATLAS with a length of 46 metres, a diameter of 25 metres and a weight of 7,000 tons is the greatest ‘apparatus’ of its kind. It is used as a multi-detector, among other things to detect the mysterious Higgs boson and dark matter particles;
 A modular service vehicle (TIM) has been designed for automated measurement and inspection work. It can travel through the entire accelerator tunnel on a monorail—and be stopped if necessary using Profisafe via Industrial Wireless LAN;
 Hundreds of Simatic S7-300 and S7-400 controllers, including 36 Powering Interlock Controllers (PIC) with centralised on and off functions for the power supply, ensure high availability and reliability of all critical systems;
 The supervisory system PVSS (Process Visualisation and Control System) from ETM, a subsidiary of Siemens AG, provides visualisation and monitoring of most of the control systems in the LHC, whose annual data traffic of about 15 petabytes (15 million GB) would fill more than 1.7 million double-sided DVDs.
One reason for CERN’s extensive use of controllers and processors from Siemens is the desire to employ proven commercial technology for the automation of the systems. Another reason is that the controllers had to demonstrate their reliability in a series of rigorous tests, in a way that has hardly ever been performed and continues to be performed for any other application.
Protection against external access
‘Redundant installations such as the Simatic S7-400H fault-tolerant type of controllers may offer a high degree of operational safety. But who can guarantee that no one will take over the controller, crash it and compromise its security?’ asks Dr. Stefan Lüders from the computer security team of the IT department at CERN. ‘Most controllers, field devices and even actuators are now directly connected to Ethernet.’
The team led by Dr. Lüders therefore developed a special test bench for dedicated examination of the vulnerability of controllers, SCADA (Supervisory Control and Data Acquisition) systems and other Ethernet-connected devices in the market to cyber-attacks. This not only relates to protection against hackers with more or less criminal intent, but also against viruses and worms that can be introduced through a variety of channels—including USB sticks and CF cards. In contrast to the usual patches that can be installed in an office environment, controllers cannot be easily updated daily with the latest antivirus protection, even if it is available.
As part of the validation of controllers used at CERN, at the test bench on Control System Security at CERN (TOCSSiC), 31 devices from seven manufacturers were systematically tested for penetration resistance with the vulnerability scanners Nessus and Netwox. Taking all different firmware versions into account, this led to 53 tests in total. In addition to interference through overload (Denial of Service, DoS), the tests also included provoked attacks on vulnerabilities in operating systems by infiltration of malicious software and ‘malicious’ manipulation of TCP/IP-based protocols. About one third of the tested devices failed these tests and has shown severe security problems.
Approximately one third of the devices came from the Simatic S7 product series, some with an integrated Ethernet interface, some with separate communication processors, such as the CP 343-1 Lean for the S7-300 series.

Simatic in endurance test: Comprehensive penetration testing was performed with network scanners to test and optimise the security of controllers against attacks from the network

The poor test results led to a ‘very productive interaction with Siemens’ and ultimately made ‘Simatic controllers significantly more secure over the years; now they meet the stringent requirements at CERN,’ summarises Dr. Lüders.
Robust even under proton bombardment
The field devices of the Simatic ET 200M distributed I/O were tested for their resilience in the immediate vicinity of the particle accelerator and bombarded with protons for this purpose. This durability test especially targets the I/O cards of the ET 200M DP modules. Siemens engineers maintained the mean time between failures (MTBF) required by CERN under proton bombardment by exchanging the optocoupler on the cards.
Another example is the robust control of the 1,400 helium supply valves for cooling the magnets in the accelerator. The enclosure of the Sipart PS2 electro pneumatic positioner located on the valves only contains passive electronics, which is resistant to radiation. The active electronics, however, is installed in cabinet drawers in parallel service tunnel or alcoves. Each drawer can contain three modules; each module is connected to the field by a cable of up to one kilometre and groups five positioners.
The operational safety of the entire LHC is guaranteed by a system of Powering Interlock Controllers (PIC), consisting of a total of 36 Simatic S7-300 controllers with CPU 319-3 PN/DP—the fastest in this series. The PIC ensures that all safety conditions are met prior to the powering and during the operation of the magnets. These conditions range from the proper operating temperature of the magnets and errors in the cooler to power converters and readiness of the emergency off circuits and the uninterruptible power supplies (UPS).
When critical events or failures occur, the proton beam can be quickly switched off within a few milliseconds. The reliability was demonstrated during the initial startup of the LHC in September 2008.
Redundant detector safety
A Detector Safety System (DSS) is responsible for the immediate status monitoring of the detectors and the protection of essential detector equipment. It consists of a controller-based front-end for safety-critical tasks and a SCADA back-end for configuration and monitoring. Two fault-tolerant Simatic S7-400 controllers with CPU 414-4H executing the same process code and constantly synchronising operate in the redundant front-end, independent of the back-end. If a problem occurs, the ‘good’ controller automatically assumes exclusive operation until the other has been updated. The sensors and actuators for the alarm matrix of the DSS are also configured with redundant connections through up to 32 ET 200M distributed I/O modules to Profibus, the power supply and Ethernet communications to the back-end using Simatic CP 443-1. The controllers are certified for applications in accordance with safety category SIL 2.
The operator at the back-end computer, running PVSS, determines the information to be collected and analysed and the pre-defined safety measures to be taken by the front-end controllers. The code executed at the controller end is identical on all the DSS installation. This code is entirely data-driven, and the data are taken from the PVSS configuration database. This enables the DSS to be easily adapted to the form, implementation and evaluation of the experiments at any time (and even online).
‘We originally based the data communication between the front and back-end on the OPC protocol, using a Siemens OPC Server’, says Giulio Morpurgo, responsible for the development of the DSS.
Once ETM made available an enhanced version of the S7 protocol following a CERN requirement, which directly interfaces PVSS with Siemens PLCs while still working in a way functionally identical to OPC, the project adopted such solution, for sake of simplicity and to avoid an unnecessary additional layer. Each tag in the PVSS application is assigned an I/O address of the PLC memory.
‘We only had to adapt these addresses to the new syntax to migrate from OPC to the new interface. The S7 driver fully satisfies our requirements, and it also supports the redundancy we tried to implement at all levels in the DSS, by implementing the automatic switch between the two CPUs, should one communication line manifest a problem,’ says Mr. Morpurgo.
The world's longest refrigerator
One of the biggest challenges in the automation of the LHC, however, was undoubtedly safe cooling—or cryogenics—for the superconducting magnets guiding and accelerating the two beams. In the ‘longest refrigerator in the world,’ sixteen Simatic S7-400 controllers (two per sector) with CPU 416-2 control each about 250 closed loops and 500 alarms and interlocks within a cycle of less than 500 ms. The 15,000 radiation-tolerant sensors and actuators in the direct vicinity of the magnets are accessed via Profibus or WorldFIP fieldbuses, which comprise a few kilometres of optical fibre. Each pair of controllers is flanked by eight front-end industrial PCs interfacing the WorldFIP bus lines.

Total SCADA: The universal process control and visualisation system, PVSS, in the Central Control Centre at CERN ensures trouble-free control and monitoring of the particle accelerator’s entire technical infrastructure in a homogeneous user interface

The above-mentioned 180 positioners per sector with split electronics for controlling the helium valves are connected via Profibus (1.5 Mbps). There are also a total of 52 remote switch boxes for 5,000 cryogenic instruments at the eight injection points of the LHC. At point 4, the two opposing proton beams are accelerated ultimately to 7 TeV (Tera electron volt) by four times four superconducting RF resonators.
In each sector, one of the two S7-400 controllers is assigned to a 2,460 metre arc section in the LHC. The other controls the cryogenics in each of the 270 metre straight sections located near the injection areas.
‘The reliability of Simatic in this system is very important,’ stressed Dr. Paulo Gomes, senior engineer for the cryogenic instrumentation. ‘CERN comprehensively tested PLCs of several manufacturers and finally specified two brands. Of these two, Simatic met our rigorous demands for continuous fail-safe operation.’ Yet the production of helium in the LHC refrigerators has been designed using the other CERN standard PLC brand.
‘Our control engineers ensured a smooth integration between the two PLC brands allowing fluent data exchange between them and, moreover, an optimal and flexible control system architecture,’ said Dr. Enrique Blanco, from the Industrial Controls and Electronics group and one of the automation engineers leading the cryogenics control system project.
The connection to the Cryo-Scada, based on PVSS, is provided by CP 443-1 Advanced communication processors that ensure a highly flat communication structure requiring little configuration effort even in such complex systems, and avoid the need for additional gateways. These processors achieved the best results in the above-mentioned tests for robustness against network attacks. Data bandwidth has also been optimised by using a CERN in-house development event-driven communication mechanism where the PLC only sends data to the supervisory data server whenever the data have changed over a certain threshold.
In addition to Cryo-Scada as a server for operator access to all relevant data and commands for supervising the cryogenic process in the CERN Control Centre (CCC), the cryogenics instrumentation engineers also use the Cryogenic Instrumentation Expert Tool (CIET) running PVSS for the monitoring, configuration and parameter assignment of every WorldFIP channel. Both PVSS supervisory applications are running on the Linux SLC4 operating system, which turns out to be a dramatic increase of the overall performance.
This entire giant project with a rather large number of sensors and actuators has been optimally designed and developed following an approach based on Unicos (Unified Industrial Control System).
‘The Unicos framework has been developed at CERN and allows an automatic code generation to fully configure a control system, furthermore, independent of the PLC platform. This minimises the commissioning time and allows concentrating the effort in custom development maximising the productivity of the control engineers,’ highlighted Dr. Blanco.
Visualisation of over one million monitoring channels per detector
The process control and visualisation software, PVSS, was standardised CERN-wide and advised for all SCADA functions in 2002. According to Dr. Blanco, ‘At CERN there was a large market survey and a working group to select and support a limited number of products. From a total of more than a hundred competitor solutions, in the end it was PVSS that best suited our demands for openness, scalability and flexibility as a supervision system for the LHC.’
Here is something to illustrate the gigantic proportions involved. Each of the four large detectors contains over one million I/O monitoring channels. In addition, there is a variety of sub-detectors that can also operate as stand-alone systems if needed. This requires a highly distributed architecture of hundreds of computers.
Whereas all the data from the PVSS software is visualised on a totally homogeneous user interface, it looks completely different below this. Controllers work with I/O standard modules in many fields. More often, however, special input and output modules are employed, usually controlled by dedicated software, PC-based and VMEbus platforms with different operating systems.

The active electronics of the electro-pneumatic positioner Sipart PS2, for the helium supply to the cryogenic magnets

PVSS not only currently supports Windows and Linux, but also Solaris 10. In addition, the detector control systems at CERN matches an approach with hierarchical arrangement of finite machines (Finite State Machines, FSM) which is somewhat untypical in industrial SCADA systems.
‘An FSM toolkit was developed at CERN and applied in all the LHC detectors,’ explains Dr. Blanco and emphasises, ‘PVSS openness allowed the FSM toolkit smooth integration proving to be extremely flexible compared to other commercial SCADA systems.’
Train Inspection Monorail
TIM is a relatively small but no less exemplary high-tech project, its name is derived from the French ‘Train Inspection Monorail’. It is a modular, driverless vehicle in the form of a train that travels through the entire accelerator tunnel suspended from an overhead monorail to perform fully automated inspections and measurements. The modules are assembled to build-up a train according to the demands and service task at hand. TIM was developed for use when entering the tunnel would be too dangerous for personnel, for example, during tests, commissioning, or cryogenic cool-down of the magnets.
During the actual operation of the LHC, TIM remains in a protected park position so that the proton resistance of the electronics used in the modules is not a criterion; operational safety is, however. ‘In case of doubt, we must ensure that the vehicle will always stop immediately if it encounters people still in the tunnel, or an unexpected obstacle,’ says Keith Kershaw, head of the handling technology team.
TIM is equipped with a laser scanner for this purpose, and is controlled by a fail-safe Simatic S7-300 with CPU 315F-2 PN/DP, which triggers the emergency stop through the Profisafe protocol. ‘This not only reduces the required wiring in the very tight space of the modules, but also the work needed to wire the respective vehicle configurations,’ continues Mr. Kershaw. Because IWLAN (Industrial Wireless LAN) components from the Simatic NET product range are used for the communication between the modules. A Scalance W744-1 Pro is used as a client at the Profinet interface of the controller to connect up to eight nodes. An Access Point Scalance W-788 is installed in each of the modules. The communication with the monitoring computer is performed via GSM/Edge with a teleservice modem as a link on the vehicle.
TIM modules, according to Mr. Kershaw, are being developed for tele-operation applications including remote radiation level surveys around the LHC ring. Another application is the precise alignments checking of the LHC collimators that are used to stop high energy particles that deviate from the desired beam orbits. Remote handling of the highly-radioactive collimators is also being studied.
Pioneering co-operation in CERN openlab
Siemens has been a partner in the CERN openlab since 2008 in order to further strengthen the successful cooperation with CERN and also to be able to offer customers the innovations tested at CERN. Apart from several companies in the IT industry, Siemens is the sole industrial sponsor and development partner in the CERN openlab. Its Automation and Control Competence Centre is designed to further improve the hardware and software of automation products for future use in the LHC, including data security and openness in software engineering for such complex system landscapes. The co-operation between CERN and Siemens in the CERN openlab covers two main areas:
Continuous improvement of the security for programmable logic controllers (PLC) to protect against viruses and hacker attacks. The new modular Simatic S7-1200 controllers will be tested for this in the near future.
Development of software solutions for the deployment of SCADA and PLC software in large configurations. The aim here is the future capability of performing centralised, efficient updates for engineering software and application data in such a complex environment. In addition, there is the development of solutions for handling huge amounts of data for visualisation in the SCADA field.
Nicolas Mader Siemens AG Industry Automation Switzerland, and Dipl.-Ing. (Univ.) Karsten Schneider Siemens AG Industry Automation, Nürnberg

Source: http://www.controlengeurope.com/article.aspx?ArticleID=31000