Data is often referred to as the new oil, the world’s most valuable resource, but while industries like finance and healthcare are already exploiting the benefits, the energy sector has lagged behind.

Today’s energy system has its roots in the 1900s and mostly analogue with some digital “add-ons”, but the demands of the future, particularly around the need to decarbonise and decentralise energy generation and distribution has galvanised support for a modern digitalised system able to handle increasingly complex scenarios.

Advanced data-gathering and analytic techniques offer utilities the opportunity to better forecast supply and demand, troubleshoot issues on the network, manage energy constraints and improve compliance with regulatory requests. They can help reduce energy losses, uncover customer usage patterns and improve customer understanding of carbon impacts and energy costs.

Now the UK government, regulators, and technology innovators are pushing a range of initiatives designed to unlock the power of data in the energy system. The Energy Data Taskforce (EDT), a body established by government, Ofgem, and Innovate UK, has published a data strategy for the sector based on two key principles: filling in data gaps by requiring new and better quality data; and maximising its value by embedding the presumption that all data is open.

The strategy aligns closely with a wider blueprint, devised by the National Infrastructure Commission (NIC) and now being delivered by the Centre for Digital Built Britain (CDBB) to create a digital framework for infrastructure data, including new data standards and a national digital twin. Meanwhile, data pioneers are showing how emerging technologies like machine learning and predictive analytics can uncover new insights and fill in data blanks.

But this ambitious digital overhaul faces hurdles, legacy infrastructure, particularly low-voltage networks, has a dearth of available data, which raises the prospect of major investments in upgrades to sensors. Data protection obligations under GDPR and concerns around data security and commercially sensitive data may restrict what customer and systems data can be shared. And the UK’s pressing skills gap could stymie the ongoing need for data engineers and data-literate technicians.

“The biggest challenge is regulation related to getting access to the data we need,” says Laura Sandys, chair of the EDT. “We will hopefully see legislation come through over the next couple of months with a greater emphasis on data and data retrieval. There will be a greater focus on data needed to manage the system. Ofgem has made statements that it will look at the whole issue of regulation and the need to make data and data requirements business as usual, rather than something unusual, which is incredibly important.”

Power moves

The UK energy market is undergoing a historic and rapid transformation as emerging technologies and the impacts of climate change converge. Homeowners are making greater use of low carbon technologies (LCTs) such as solar panels and electric vehicles, the second phase of the domestic smart meter rollout is well under way (there are 14.9 million smart meters installed so far, a quarter of which operate in “smart” mode) and businesses are digitalising to allow for increased flexibility and optimised processes using smart control systems.

As generation is distributed and demand fluctuates at greater rates distribution network operators (DNOs) will need to take a more active role in managing local electricity generation and use, which challenges the traditional model.

Data is recognised as a key enabler to many of these processes. It will also underpin goals set out in the EU’s Clean Energy Package, a set of new directives designed to integrate energy markets and underpin the transition to a low carbon, more flexible energy system.

Better access to quality energy data has major implications. Granular data about energy networks and their generation capacity can improve optimisation and help deliver energy more consistently and reliably. It can facilitate the cross-vector provision of energy by improving understanding of the ability to shift demand from one energy source to another, for example gas to electricity using hybrid heat pumps.

More comprehensive data on network assets can make it easier to forecast investment needs. Dr Richard Dobson, practice manager – data systems, at the Energy Systems Catapult who helped develop the EDT report, told Flex: “Having more data on networks can help scenario planning of what would happen. If for example we had a 30 per cent uptake in electric vehicle users in a local area, how would the system handle that?”

When systems data is cross-referenced with customer demand data, even greater value can be extracted, he says: “There may be services that consumers want to purchase that are also good for the energy system, such as heat as a service, or managed EV charging, whereby the customer can plug in wherever they want and the system knows the specific car and the associated account to be billed.”

A greater focus on digitisation can improve regulatory oversight, says Andrew Burgess, deputy director of energy system transition at Ofgem: “The more useful data that we can analyse and interpret, the better the information we have to regulate the industry. Data helps us make better decisions in our price controls, understand the money that regulated monopolies need in future and what they have to deliver in return for that money. The better data that is provided to flexibility providers the more we will start to see better value alternatives to traditional approaches to running the network, which will save money for all consumers.”

Plugging gaps

The EDT report warns that data in the energy system is often of poor quality, inaccurate, or missing, while valuable data is often restricted or hard to find. To address these shortcomings, it recommends the setting up of an asset registration strategy to coordinate the registration of energy assets, and a unified digital system map of the energy system to increase the visibility of infrastructure and assets.

The latter recommendation reflects a wider goal set by the NIC to create a digital system map of all UK network infrastructure and ultimately a national digital twin of network infrastructure. The work is being led by CDBB under which the Digital Framework Task Group, which recently published its first output, The Gemini Principles. This paper sets out proposed principles to guide the national twin and the information management framework required to enable it.

Sarah Hayes, project director for regulation at the NIC, who was also on the steering group that helped develop the EDT strategy, told Flex: “The information management framework will bring together industry standards bodies, technology companies and consultants, etc to develop a common language for data about infrastructure across the built environment. It’s roots are in building information modelling (BIM), a system applied to construction projects that puts all data in the same format and the same language. We want to apply this to existing infrastructure, most of which is old and we need to be able to get more out of it.”

A lack of information about what data already exists is another challenge that creates inefficiency under the current energy system. “It sounds very ‘meta’, but it means that people search for data that doesn’t exist, for a very long time in some cases, or they build a business model based on a false notion that they will be able to get something that doesn’t exist,” says Dobson. “Companies also re-collect data that they could have gathered from an existing source at a much lower cost, potentially with a higher degree of accuracy.”

The taskforce therefore recommends that a data catalogue is set up to capture all available energy system datasets across government, Ofgem and industry as standardised metadata, with industry participation in the catalogue mandated via regulatory and policy frameworks.

Missing data is a particular concern on older networks, but is there a way to avoid the major investments required to upgrade them with sensors? Emerging technologies such as data analytics and machine learning could provide a solution.

Electralink, an organisation that runs a data transfer service for the energy sector, has been working with the DNO Western Power Distribution to explore how machine learning can interrogate customer energy consumption patterns to reveal network constraints.

Legal restrictions currently prevent access to granular domestic smart meter data, so the project explored how machine learning can be applied to non-smart consumption data already available in Electralink’s archive.

Dan Hopkinson, director of data and transformation at Electralink tells Flex: “Using machine learning on the data we were able to identify some very strong correlations, and the potential for the existence of a lot of low carbon technologies that DNOs are not currently aware of. The data is a strong indicator if a technology like solar panels, or an EV recharger, has been applied to a property.”

Hopkinson admits that the model is not perfect, but the intention in future research is to incorporate additional data sets, including contextual data, switching behaviour, and demographic data such as property type, to create a more accurate picture, This could allow DNOs to model the low voltage network without the need for protected consumer data.

“This form of virtual network monitoring could improve our understanding and therefore reduce costs by ensuring that real-time monitoring is only put where it is needed,” says Hopkinson.

Benefits multiplied

Data may be the new oil, but if, like the black stuff, too much of it is allowed to accumulate in the hands of private companies there is a danger it will not empower and benefit the rest of society.

Reducing barriers to public and private sector data can attract innovators that create operational efficiencies or develop new business models; it can increase the speed that new markets develop and improve decision-making across the energy system. Data sharing improves transparency, which can give the regulator and third parties better oversight of costs and energy efficiency.

In a drive to exploit these benefits, the EDT recommends that all energy system data should be presumed open, and that data owners who want to restrict access must first justify the reasons why to the regulator.

The NIC’s Hayes comments: “Our report Data for the Public Good made points about the need to share data within industries and across industries and how the value that can be unlocked is greater than the value that can accrue to one company alone. The typical commercial approach is that all information is commercially sensitive, but when you go through data in more detail you often find that lots can be shared.”

The taskforce recommends that companies adopt an “openness triage” process that considers a range of risk factors and develops appropriate mitigation mechanisms to maximise the potential for sharing.

The opportunities around open data have already been demonstrated by Elexon, whose balancing mechanism reporting service (BMRS) provides free data for anyone to use under an open online licence.

The BMRS has over 1,000 daily users on average and provides key information on the wholesale electricity market, including near real-time updates on which sources are being used to generate electricity, demand levels and imbalance prices. It also publishes data required by the EU REMIT regulations, such as outages at power stations.

Elexon provided input to the taskforce on the sort of companies and parties that access BMRS data and what they use it for. Mark Bygraves, chief executive at Elexon, says: “This data is extremely valuable to existing companies and new entrants, because understanding supply and demand patterns helps companies to develop the right products and services for consumers.”

Open data will be crucial to support the sector’s transformation to decentralised energy, he adds: “Many consumers will provide demand-side response whereby they reduce, or increase their electricity consumption at certain times in exchange for payments from energy companies. We are likely to see greater use of electricity storage and electric vehicles that act as ‘virtual storage’ by being available to top up supply when they are not in use … This means that as an industry we need to ensure easy access to clear data on when these services are needed by network companies and suppliers, network companies also need clear data on the capability of electricity storage, so they know how much power it can provide, when, and for how long.”

Zeroing in

As technology continues its seemingly inexorable march, progress is under way to implement the recommendations made by the EDT, since the publication of the strategy in June this year.

Ofgem has said it intends to include the recommendations in several regulatory position papers and has progressed them as part of the next business plan update for RIIO2. The regulator has started making its metadata (data that describes data sets) more available and has updated its metadata practices to ensure they are interoperable with the data catalogue proposed by the taskforce. It is also exploring options to use Innovate UK’s energy revolution funding to stimulate investment related to digital infrastructure recommendations made by the EDT.

But the road ahead could be bumpy given the need to overhaul ingrained procedures and attitudes. Andrew Burgess comments: “There is a cultural change, for the energy industry as well as for Ofgem, around understanding that data on commonly used assets, such as the energy system, is presumed open to all, with the burden of evidence on the data custodian/processor.”

He says the digital architecture solutions proposed by the taskforce will only succeed if they are properly coordinated with other digital services, using a process of iterative delivery that avoids monolithic “old-style IT service delivery practices”.

The EDT has set a primary goal to get its first two recommendations peppered through regulation and legislation over the next few years. The data catalogue is a near-term target, with progress on its development already made by Ofgem, BEIS and Innovate UK.

“The energy sector might be behind the data curve, but if everyone supports our strategy we could find ourselves ahead of the curve into the future,” says EDT’s Sandys. “Soon all areas of infrastructure are going to become interrelated, which is why it is so important to get conflation around approaches and standards, etc. These are exciting times indeed for infrastructure.”

Lessons from Estonia

Estonia is leading the rest of Europe in its use of smart meter data with a platform that allows consumers to control who can access their energy data.

The Estfeed data exchange platform was developed by Elering, the country’s independent gas and electricity transmission system operator, to connect smart meter data from various hubs and sources to energy service providers, including suppliers and app developers, via a legal and secure system.

Powered by X-Road software, it features a transparent data transport layer with a consent management system that allows individual consumers to choose who to share data with, the purposes it is used for, and view who has accessed the data. Individuals can also track their personal energy consumption, and what energy they have produced.

The platform also provides access to other data, such as stock market electricity prices, and weather forecasts, to help energy service providers and other market players optimise or innovate new services.

Estafeed went live in September 2017 and each month consumers give about 500 to 1,000 consents to access their data. According to Elering, the data has allowed companies to innovate more personalised services that lower consumer energy bills. For example, building monitoring service providers have accessed data to optimise consumption and lower energy costs in business buildings. Energy consumption data has been shared with energy package comparison tools and electricity and gas suppliers to create personalised offers and enable consumer switching.

Dr Richard Dobson, practice manager – data systems, at the Energy Systems Catapult, tells Flex: “As a secure data communications platform, X-Road underpins the smart metering system in Estonia and means that all consumers have complete visibility of what’s going on with their data. They have the ability to say ‘I would like organisation X, Y or Z to access my data’ and then remove that access at any time. It is really exciting.”

Estonia is already a pioneer in smart meters, having achieved 100 per cent coverage in homes and businesses back in 2014. Recent developments in EU regulation, including the General Data Protection Reform and the Clean Energy Package have sparked wider interest in Estafeed. Elering now plans to expand the system to connect to 100 million electricity meters across Europe by 2020.

This article first appeared in the latest edition of Utility Week’s Flex supplment, here.

Or download the full pdf of the latest issue of Flex, here