Company Post

Building a Digital Twin at Gran Telescopio Canarias

Situated on the island of La Palma in the Canary Islands, Gran Telescopio Canarias, or GRANTECAN is the largest optical telescope of the world, where scientists observe the sky to further our knowledge of the universe. Astrophysicist and GRANTECAN Director Romano Corradi (also a TouchDesigner user) has been steering the digital transformation of GRANTECAN.
Building a functional virtual replica or “digital twin” of the telescope enabled real data to be fed into, processed and analyzed by the digital twin to inform the facility’s actual operation, maintenance, and development while providing another layer of safety to the operational staff. 
For this undertaking Romano elected to work with TouchDesigner “knowing its versatility to treat nearly any kind of data, wide interoperability, and its capability to provide real-time rendered 3D models”, and y=f(x) was brought in to lend their inimitable skills and creativity to bring the digital twin to life. Our conversations with Romano and y=f(x)'s Roy and Tim Gerritsen, Rogier van Ostaijen and Darien Brito sheds light on this groundbreaking initiative and process.

Romano Corradi: Gran Telescopio Canarias (GTC, or GRANTECAN), with its 10.4m collecting mirror composed 36 hexagonal glass ceramic segments, is the largest optical and infrared telescope in operation. Located in one of the best sites for astronomical research, the Observatorio del Roque de los Muchachos on the summit of the island of La Palma (Canary Islands, Spain), GRANTECAN provides front-end astronomical observations to the scientific community in nearly all fields of astrophysics. This includes the search for the first galaxies and stars that formed in the Universe, the investigation of the nature of dark matter and dark energy, the study of blackholes, the search for planets around other stars, or the formation of chemical elements and molecules which determine the history of cosmos and ultimately the development of life.

Large telescopes such as Gran Telescopio Canarias, hereafter GRANTECAN, are complex machines, with massive optics and mechanics tuned to nanometre precision, thousands of moving axes, extreme cryogenic systems, varied electronics and sensors, all managed by a sophisticated control software. One can therefore see GRANTECAN as a 24/7/365 factory with complex machinery, without forgetting that its (intangible) products are not physical assets, but knowledge about the Universe.

As a state-of-the-art factory, at GRANTECAN we are in the process to transform our operational model according to the paradigm of the so-called Industry 4.0.

The basic ingredient of Industry 4.0 is digital transformation aimed to optimise and accelerate business and production processes. Powerful digital enablers of industry 4.0 is the panoply of connected sensors that are grouped under the umbrella of Internet Of Things (IoT, note that 40 billion connected IoT devices are forecasted in the world by the end of 2023!), as well as robots, cobots, drones, machine’s sensors, and the corresponding huge flow of data which is to be analysed using Artificial Intelligence, Big Data technologies, etc, to feed diagnostics and prescriptions back to the operational processes.

The digital transformation at GRANTECAN is described in the graphic below, where all different agents considered so far are listed. As it is a scalable process that allows the inclusion of heterogenous information, in the future other operational and scientific aspects will be considered.

Derivative: What was the impetus for bringing TouchDesigner to the project?

Romano Corradi: At the tip-of-the-iceberg of this technological effort (the Enterprise layer in the above graphic), we have been seeking for an innovative, attractive user interface. TouchDesigner, that we used in the past for outreach activities, was our choice, knowing its versatility to treat nearly any kind of data, wide interoperability, and in particular its capability to provide real-time renderized 3D models.

The nature of TouchDesigner, principally devised to create artistic products, allows a much more versatile, dynamical, visually attractive, and user-friendly approach as compared to the standard “engineering” panels that are generally used to visualize factory’s data.

Derivative: What was the objective when you were considering bringing TouchDesigner into the project? We understood it was to radially improve the visualization of the state of the entire telescope facility.  For the purpose of (1) improving productivity of the engineers, and (2) giving the public a better understanding of how the telescope facility functions on a day-to-day basis.

Romano Corradi: Our goal was to build an effective way to display the real-time status of the telescope and the installation, informing people in charge of the installation of status changes, faults and emergencies, security issues, the environmental conditions that may allow or not the astronomical observation, and the most relevant operational data.

An additional important goal is the empowerment of our workers, facilitating their tasks, removing repetitive activities, providing high-level digital tools, in other words providing opportunities to develop their talent and expertise.

This fits the next-level of the present industrial revolution, in which this transformation aims to be human-centric and with the explicit goal to preserve our planet (Industry 5.0).

Derivative: Can you explain the concept of a "digital twin" and how you are applying it at GRANTECAN?

Romano Corradi: Using the glossary of industry 4.0 (but we did not know it when we started!), we have been building a digital twin of GRANTECAN. There is still no complete consensus on the exact definition of a digital twin, as it is a relatively new concept, but the main idea is that a digital twin is a virtual copy of a physical entity (a building, a factory, a machine, a person), which is fed with real data that can be analyzed in the virtual space, providing informative feedback to the real entity that facilitates its operation, maintenance, and development. The “technology” of digital twins is very rapidly growing, as part of the digital transformation of industries. It can therefore be considered another essential digital enabler of the transformation to Industry 4.0.  Can TouchDesigner enter the whole game of digital twins (see for example the Digital Twin Consortium) as a competitive solution? I do not know the answer in general, but for GRANTECAN we are certainly achieving a successful solution, with the added value in terms of creativity and aesthetics that provides a platform originally devised for different ecosystems (art, entertainment, branding,..) such as TouchDesigner.

In a first phase, a 3D model of the whole GRANTECAN installation (telescope and annexed buildings) has been ingested into TouchDesigner, and system’s parameters can now be visualized in nearly-real-time on the top of the 3D model using customizable views, controlled using a multi-touch control monitor.

The output is presented in a large display in the telescope’s control room. Other panels were built, a dashboard with general information about specific subsystems, and an interactive display of the various optical and infrared surveillance cameras distributed throughout the installation.

Digital Twin mirroring the telescope’s activity. Credit: ©2015 Gianluca Lombardi/GTC & GRANTECAN

Derivative: What you've achieved is quite remarkable, can you tell us a bit about your development partner and process?

Romano Corradi: This development was done by y=f(x) with the assistance of GRANTECAN engineers. At the beginning it was a bit challenging, as two different worlds (art and science engineering) were meeting and had to talk the same language and adapt “on-the-fly” the scope of the development, the timing and the working methodology. Instrumental to this scope was a visit of Roy, Tim, Darien, and Rogier from y=f(x) to our installation at the Observatorio del Roque de los Muchachos at the top of the island of La Palma (Canary Islands, Spain).

After a quick adaptation, it was really a pleasure to work with y=f(x), at GRANTECAN we not only appreciate their enormous competence and skills, but also their availability to adapt to our idiosyncrasies (e.g. continuous adaptation of the planification to fit operational incidences, delays in the supplies of the necessary hardware, ..), and above all a cordiality that makes this project fully enjoyable.

The project continues, but even at its present stage feedback is enthusiastic from nearly any people who have seen it at work: our technicians, engineers, and astronomers, the general public, collaborating technological companies, other telescopes, and even the world leading companies in the development of digital twins, when we showed them some samples at the IoT Solutions World Congress in Barcelona on February 2023.

We are now in the definition of the next phases of the collaboration with y=f(x). What has been done so far discloses so many possible developments (some of them extending to the outreach of GRANTECAN to the general public) that we are making the effort to understand in which ones TouchDesigner provides unique, most valuable solutions.

Derivative: For the y=(f)x team, what was your motivation and interest in taking on this project?

y=f(x): We helped Jakob Povel realize an installation for the visitors area of the GRANTECAN a few years ago, and later met with Romano Corradi at the Montreal TouchDesigner Summit. Both of these events were extremely enjoyable and inspiring, so it wasn't really a question of whether or not we would want to take on this project when the opportunity arose.

We think the study of astrophysics is a fascinating field, and as people interested in the intersection of art and science, we could not have asked for a more exciting opportunity to work with such an exceptionally well-engineered machine, on a project with such passionate, smart and amazing people.

The integration of all the data points that GRANTECAN has to offer presented a technical challenge we were keen to take since solving it implied the creation of a system that could be scalable and forced us to design a robust strategy that could be later adapted and expanded.

Part of the difficulty in scientific communication is that the language, whether oral, written or visual, tends to be very technical and perhaps a bit too dry for the general public. We designed our visual communication so that it was aesthetically pleasing as well as functional.

Our on-site visit was a privilege to hear first-hand from the engineering team, and learn about the inner mechanisms of the machines, the data parsing and the many different disciplines that converge in the research facility. The planning and effort that goes into making such a complicated device as a telescope of this kind is astonishing. And what better than to have had guidance from the big boss Romano himself during the whole trip?

Derivative: You’ve put a lot of effort into gathering and centralizing data into the FIWARE data center and IoT cloud system. Can you give us some specific examples of the data gathered?

Romano Corradi: As a first step of the project, development was focussed on conveying into FIWARE environmental information (meteorological, air quality, visual inspection of sky conditions), security aspects (presence of people in critical areas), the monitoring of services such as the photovoltaic energy production and its use by electric cars, the water consumption, as well as basic information for the night operation in terms of the telescope position, temperature of its structure, path of the light through the different mirrors and to the science instruments, and the instruments status.

Derivative: Can you give some examples of how that data is massaged in FIWARE to make it more useful to the engineers.

Romano Corradi: So far, FIWARE collects real-time information, which is then visualized either using the Grafana software to produce specific “engineering” views, or by TouchDesigner which provides a more general, comprehensive, and immediate view easy to understand by any type of user .

Derivative: TouchDesigner then has a simple protocol/API to obtain massaged or raw sensor data from FIWARE.  Is the data you are receiving all expressed in JSON?

y=f(x): The data we obtain from FIWARE comes in via the JSON standard. We transform the information into data objects in our TouchDesigner system and store it using dependable dictionaries that facilitate referencing across the board.

We also designed a monitoring mechanism that logs information and reports when there are critical events in the system that require the attention of the operational staff.

The capturing and rendering of the surveillance cameras had to be tailor-made. The data we obtain comes, for the most part, in Real Time Streaming Protocol (RTSP), but a portion of it also comes from web-hosted images. That implies that the system needs periodic refreshments and a dual-capturing mechanism. The challenge was creating a solution that could work for more than 20 cameras at maximum resolution while remaining performant.

Furthermore, various cameras have an additional infrared feed, which comes in a different resolution than the optical feed. It was necessary to account for it when presenting a unified and seamlessly integrated overview.

The touch control panel allows a versatile look into the digital twin and its varied information.

Derivative: How did you get the model of the facility (where did it come from), what file format, and what did you do to it when you got it to make it more useful?

y=f(x): Grantecan supplied us with an intricately detailed version of the telescope,  thoughtfully crafted in the Blender format. We did not have to reduce the polygon count and could use the model as is, which was a pleasant surprise. We exported the different elements of the model in separate FBX files, so our asset directory was well-organized and easy to understand.

Derivative: I suppose a major challenge is to show the various bits of data in the clearest way possible to the viewer. What are your thoughts on the approach you took?  It seems like you chose a monocolor coloring except for the data that you want to enunciate, for example.

y=f(x): Striking the right balance between looks and functionality became a crucial challenge. This system serves multiple purposes—it keeps an eye on things for the operations team and also acts as a way to explain the process to the public.

With lots of data involved, things could easily get too messy and overwhelming. So, we decided to create a 3D environment that shows everything in real time. The centrepiece is a detailed model of the telescope.

We set up different ways to show information and change the view using pre-defined setups. By using different layers and colours, we could make some parts more noticeable than others, depending on what was important.

As an aesthetic starting point, we used sci-fi movies as a reference to blend everything in a futuristic look and feel.[We like this!]

Derivative: You said that currently, you are focused on using snapshots of data at the current time. But to understand it better you need to look at how the data changes over time. What are your plans in that regard?

Romano Corradi: The next step of the development of the project will be to process data to determine the evolution of the operational processes, to automatise and optimise them, predict and act against failures, and avoid human errors. This will be done by applying different kinds of analysis methods, including machine learning, as an intermediate layer before visualization is done with either Grafana or TouchDesigner.

Derivative: Give us one example where that would be useful (that you may implement first).

Romano Corradi: A critical parameter for a telescope such as GRANTECAN is to maximise the image quality (sharpness of the observations of stars and galaxies). This depends on the status of the telescope’s mirrors and their mutual alignment (only the primary mirror has 326 degree of freedom which has to be adjusted in real time), the continuous cleaning process of mirrors, the mechanical performance of the telescope (which needs to move with the highest precision to follow the apparent rotation of the night sky), but also the ventilation of the dome to minimise air turbulence, which is controlled by a complex natural and forced ventilation in the dome. Putting together real-time information on all these aspects, analysing them, and providing an immediate and clear visualisation for the use of the night operators is one of goals of the project.

Derivative: You said already the engineers have been able to get almost instantaneous alerts whereas before that data may have been buried in a sea of numbers. Can you give an example of that?

Romano Corradi: The telescope structure is good example, as it contains a large number of temperature sensors, which are now immediately visualized in the interactive 3D model, which allow to understand where the alert occurs and act accordingly. This was not possible before, as data were stored in a dynamic table where the precise location of the sensor was not obvious.

Derivative: You talked a bit about the expanding the data gathered to include energy use and production, and vegetation information. Have you taken some steps to integrate that, and look at some of the data?

Romano Corradi: Global electricity consumption, photovoltaic production, and use of the electricity by our e-cars are already monitored by the system. The next step would be to differentiate in detail the energy consumption by each telescope’s subsystem, to understand the most efficient energy saving options.

Derivative: Then what questions will you be asking, or what will you look to be optimizing with that extra environmental data?

Romano Corradi: It is crucial to predict in advance the expected “optical” quality of the night in order to plan accordingly the scientific programmes to be executed. This includes the evolution of temperature, humidity, wind intensity and direction, air turbulence and dust content, and moonlight. As weather can change very quickly in the mountain, we need to be able to react quickly by closing the dome of the telescope to prevent any damage by wind, or even simple condensation of water onto its optical surfaces.

Derivative: And finally, what are some next steps and future goals for the work you are planning at GRANTECAN?

Romano Corradi: y=f(x) and GRANTECAN are now working on enriching the information that is dealt by the digital twin, including a comprehensive view of the status of the three main mirrors of the telescope. Another goal is to develop a new,  immersive and interactive audiovisual experience for the more than ten thousand people visiting the telescope every year. The idea is to create a live connection between people and the telescope, using the digital twin, cameras, sensors, as well as an autonomous robot that will guide people throughout the installation explaining with the voice of GRANTECAN how it works and its scientific results. It is here that the project will also meet the general public, creating a link between engineering, science, and people through the power of an audiovisual interactive experience that only a real-time, versatile and fully interoperable platform like TouchDesigner can allow.


Follow GRANTECAN: Web | Twitter | Intstagram
Follow y=f(x): Web | Intstagram

You may also read our 2018 feature with Romano Corradi about Using TouchDesigner For Science Communication at the Gran Telescopio Canarias and stay tuned for more developments in the digital transformation at GRANTECAN.