Elasticsearch at RTE: Blackout Prevention through Weather Prediction

About RTE

At the core of the power system, RTE (Réseau de transport d’électricité) keeps the balance between power consumption and generation. Twenty-four hours a day and seven days a week, we play a key role in directing the flow of electricity and maximizing power system efficiency for our customers and the community. We convey electricity throughout mainland France, from power generation facilities to industrial consumers who are connected to the transmission grid, and to the distribution grid which provide the link between RTE and end users. We operate France’s high and extra-high voltage transmission system, the biggest in Europe.

Our Daily Challenge

The electrical resistance of a power line causes it to produce more heat as the current it carries increases. If this heat is not sufficiently dissipated, the metal conductor in the line may soften to the extent that it sags under its own weight between supporting structures. If the line sags too low, a flash over to nearby objects (such as trees) may occur, causing a transient increase in current. Automatic protective relays detect the excessively high current and quickly disconnect the line, with the load previously carried by the line transferred to other lines. If the other lines do not have enough spare capacity to accommodate the extra current, their overload protection will react as well, causing a cascading failure. Eventually, this can lead to a widespread power outage (blackout), like the one that occurred in Northeastern and Midwestern United States and the Canadian province of Ontario on Thursday, August 14, 2003.

This incident had major adverse effects on the proper functioning of the regional economy, administration, public services, and more generally, on people’s daily lives. Power plants went offline to prevent damage in the case of an overload, forcing homes and businesses to limit power usage. Some areas lost water pressure because pumps lacked power, causing potential contamination of the water supply. Railroad service, airports, gas stations, and oil refineries had to interrupt service due to lack of electricity. Cellular communication devices were disrupted and cable television systems were disabled. Large numbers of factories were closed in the affected area, and others outside the area were forced to close or slow work because of supply problems and the need to conserve energy while the grid was being stabilized.

Unleashing the Power of Numerical Weather Prediction Data

Basically, the problem we are trying to solve consists of dynamically determining the sag margin without violating clearance requirements. A way of solving this problem is Dynamic Line Rating (DLR). The DLR prediction model aims to answer this simple question:
What is a transmission line’s maximum instantaneous current carrying capacity after accounting for the effects of weather (temperature, wind, and solar radiation) on thermal damage and line sag?

Clearance requirements for power lines

To answer that question, we used data provided by Météo France, the French national meteorological service. This data is formatted into GRIB2 files that can be sourced from Météo France’s open data platform. GRIB is a file format for the storage and transport of gridded meteorological data, such as output from the Numerical Weather Prediction model. It is designed to be self-describing, compact, and portable across computer architectures. The GRIB standard was designed and is maintained by the World Meteorological Organization.

How did the Elastic Stack Help us Respond to the Challenge?

The goal of the POC was to provide the easiest and most powerful access to this weather data to end users. We found out that Elasticsearch’s powerful ingest capabilities, geo indexing, and query features could help us achieve our goal efficiently and at scale both in terms of throughput and storage size.

Let’s go deeper in how we built the data processing stream using the Elastic Stack.

Architecture: Elasticsearch and Logstash Combined with Kafka

Architecture: Elasticsearch and Logstash Combined with Kafka

The data processing pipeline consists of four stages:

Data pre-processing: First, we extracted the needed data from the GRIB2 file to a flat CSV file. For this, we have developed custom code to wrap a CLI utility called wgrib2, provided by the Climate Prediction Center of the US National Weather Service.

Data buffering: Next, we buffered each piece of data by pushing it into a Kafka topic.

Data indexing: Then we read messages from the Kafka topic using the Logstash input plugin to extract the desired fields and build the JSON document according to our Elasticsearch mapping specification. We have also indexed the geographical location of the power lines using the geoshape datatype.

Elasticsearch weather data mapping:

 "_all": { "enabled": false }, "_source": { "enabled": true, "excludes": ["rid", "debut", "fin", "location"] }, "properties": { "requestid": { "type": "keyword" }, "debut": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" }, "fin": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" }, "code": { "type": "keyword" }, "valeur": { "coerce": false, "type": "scaled_float", "scaling_factor": 1000 }, "location": { "type": "geo_shape", "tree": "quadtree", "precision": "1100m" } }

Data querying: Finally, the data could easily be accessed using an Elasticsearch query combining the weather data and the geographical location of the power lines.

Key Benefits of Using the Elastic Stack and Kafka

  1. The whole pipeline is:
    • Simple to schedule
    • Stateless and therefore scalable
    • Replayable
    • Modularized into simple autonomous tasks
  2. GRIB2 records are translated to universal JSON Elasticsearch documents, enabling requests from a much wider range of clients.
    • Most client side technologies are able to query the JSON-based RESTful API
    • Requesters are not bound to GRIB2 enabled software packages
  3. Kafka is a broker with topic partitioning and message retention, which is great:
    • Decoupling the consumption of data from production
    • The ability to be stopped and and continue ingesting data from the same point when restarted
    • The ability to replay the ingest phase of a pipeline repeatedly into multiple consumers, with no change to the source configuration
    • The option to deliver messages to different consumers depending on readers grouping configurations and origin

Key facts of the performance test

Improved Disc Use Performance with Elasticsearch: Tuning correctly the weather data document mapping with the correct index options helped us reduce the index storage size and the document unitary size by 25%.

Graph: Improved Disc Use Performance with Elasticsearch

Increased Memory Stability with Elasticsearch: We have found no memory leaks, even during heavy indexing process. Stopping indexing has made memory turn back to a standard constant value.

Linear Scaling with Logstash: With correctly configured topic partitions on Kafka, we have tested the Logstash scaling option and we found that having two Logstash pipelines running instead of one made the ingestion process run twice as fast. With only one Logstash pipeline, we reached an ingesting rate of 275,000 documents per minute, adding a second Logstash pipeline doubled the throughput.

Geo Querying Performance with Elasticsearch: Filtering millions of documents by intersecting two complex geoshapes in two different indexes took only two seconds per request, which is a very satisfying response time.

Outlook: Providing a Robust yet Easy to Use Platform for Weather Data Access

The POC has demonstrated the feasibility of ingesting millions of records, combining them with geographical locations in a query, and sending back a ready to use data set for the end user, with great indexing and querying performance.

Our next challenge is to build a scalable (billions of records over multiple weather sources), easy to use, robust platform for grid experts to access weather data. The goal is to provide a powerful tool to discover, experiment with, and build new prediction models or improve the accuracy of existing ones. Eventually, these models will go into production to more accurately predict the effects of the weather on our transmission system assets and help us optimize their availability for the benefit of our customers and the community.

Using X-Pack Machine Learning and Alerting

The Elastic Stack combined with the X-Pack alerting and machine learning features can help us identify when the weather predictions values associated with other metrics make the line to deviate from its normal clearance range and automatically alert on risks of thermal damage and line sag.


Akli RAHMOUN is a project manager, technical software architect and developer at RTE.
He enjoys spending time finding solutions for accelerating decision making process using technology.

Tony BLANCHARD is an architect and trainer at ZENIKA. He has been working on the Elastic stack since 2012 to answer architecture, technical and prototyping questions.