Flex 4: It’s all about the data – Utility Week Flex 4: It's all about the data - Utility Week

Link to the home page

Flex is a free quarterly supplement produced by Utility Week that is available online, in print and as a pdf download.

The Oct-Dec 2019 issue first appeared in the 11th October issue of Utility Week and a pdf can be downloaded here.


The data revolution


The UK’s energy system arrives late to the data revolution, but thanks to new government-backed data strategies, industry initiatives and cutting edge technologies like digital twins and machine learning, it is paving the way for a brighter digital future. Stephen Cousins kicks off our Flex data special.

Data is often referred to as the new oil, the world’s most valuable resource, but while industries like finance and healthcare are already exploiting the benefits, the energy sector has lagged behind.

Today’s energy system has its roots in the 1900s and mostly analogue with some digital “add-ons”, but the demands of the future, particularly around the need to decarbonise and decentralise energy generation and distribution has galvanised support for a modern digitalised system able to handle increasingly complex scenarios.

Advanced data-gathering and analytic techniques offer utilities the opportunity to better forecast supply and demand, troubleshoot issues on the network, manage energy constraints and improve compliance with regulatory requests. They can help reduce energy losses, uncover customer usage patterns and improve customer understanding of carbon impacts and energy costs.

Now the UK government, regulators, and technology innovators are pushing a range of initiatives designed to unlock the power of data in the energy system. The Energy Data Taskforce (EDT), a body established by government, Ofgem, and Innovate UK, has published a data strategy for the sector based on two key principles: filling in data gaps by requiring new and better quality data; and maximising its value by embedding the presumption that all data is open.

The strategy aligns closely with a wider blueprint, devised by the National Infrastructure Commission (NIC) and now being delivered by the Centre for Digital Built Britain (CDBB) to create a digital framework for infrastructure data, including new data standards and a national digital twin. Meanwhile, data pioneers are showing how emerging technologies like machine learning and predictive analytics can uncover new insights and fill in data blanks.

But this ambitious digital overhaul faces hurdles, legacy infrastructure, particularly low-voltage networks, has a dearth of available data, which raises the prospect of major investments in upgrades to sensors. Data protection obligations under GDPR and concerns around data security and commercially sensitive data may restrict what customer and systems data can be shared. And the UK’s pressing skills gap could stymie the ongoing need for data engineers and data-literate technicians.

“The biggest challenge is regulation related to getting access to the data we need,” says Laura Sandys, chair of the EDT. “We will hopefully see legislation come through over the next couple of months with a greater emphasis on data and data retrieval. There will be a greater focus on data needed to manage the system. Ofgem has made statements that it will look at the whole issue of regulation and the need to make data and data requirements business as usual, rather than something unusual, which is incredibly important.”cPower moves

The UK energy market is undergoing a historic and rapid transformation as emerging technologies and the impacts of climate change converge. Homeowners are making greater use of low carbon technologies (LCTs) such as solar panels and electric vehicles, the second phase of the domestic smart meter rollout is well under way (there are 14.9 million smart meters installed so far, a quarter of which operate in “smart” mode) and businesses are digitalising to allow for increased flexibility and optimised processes using smart control systems.

As generation is distributed and demand fluctuates at greater rates distribution network operators (DNOs) will need to take a more active role in managing local electricity generation and use, which challenges the traditional model.

Data is recognised as a key enabler to many of these processes. It will also underpin goals set out in the EU’s Clean Energy Package, a set of new directives designed to integrate energy markets and underpin the transition to a low carbon, more flexible energy system.

Better access to quality energy data has major implications. Granular data about energy networks and their generation capacity can improve optimisation and help deliver energy more consistently and reliably. It can facilitate the cross-vector provision of energy by improving understanding of the ability to shift demand from one energy source to another, for example gas to electricity using hybrid heat pumps.

More comprehensive data on network assets can make it easier to forecast investment needs. Dr Richard Dobson, practice manager – data systems, at the Energy Systems Catapult who helped develop the EDT report, told Flex: “Having more data on networks can help scenario planning of what would happen. If for example we had a 30 per cent uptake in electric vehicle users in a local area, how would the system handle that?”

When systems data is cross-referenced with customer demand data, even greater value can be extracted, he says: “There may be services that consumers want to purchase that are also good for the energy system, such as heat as a service, or managed EV charging, whereby the customer can plug in wherever they want and the system knows the specific car and the associated account to be billed.”

A greater focus on digitisation can improve regulatory oversight, says Andrew Burgess, deputy director of energy system transition at Ofgem: “The more useful data that we can analyse and interpret, the better the information we have to regulate the industry. Data helps us make better decisions in our price controls, understand the money that regulated monopolies need in future and what they have to deliver in return for that money. The better data that is provided to flexibility providers the more we will start to see better value alternatives to traditional approaches to running the network, which will save money for all consumers.”

Plugging gaps

The EDT report warns that data in the energy system is often of poor quality, inaccurate, or missing, while valuable data is often restricted or hard to find. To address these shortcomings, it recommends the setting up of an asset registration strategy to coordinate the registration of energy assets, and a unified digital system map of the energy system to increase the visibility of infrastructure and assets.

The latter recommendation reflects a wider goal set by the NIC to create a digital system map of all UK network infrastructure and ultimately a national digital twin of network infrastructure. The work is being led by CDBB under which the Digital Framework Task Group, which recently published its first output, The Gemini Principles. This paper sets out proposed principles to guide the national twin and the information management framework required to enable it.

Sarah Hayes, project director for regulation at the NIC, who was also on the steering group that helped develop the EDT strategy, told Flex: “The information management framework will bring together industry standards bodies, technology companies and consultants, etc to develop a common language for data about infrastructure across the built environment. It’s roots are in building information modelling (BIM), a system applied to construction projects that puts all data in the same format and the same language. We want to apply this to existing infrastructure, most of which is old and we need to be able to get more out of it.”

A lack of information about what data already exists is another challenge that creates inefficiency under the current energy system. “It sounds very ‘meta’, but it means that people search for data that doesn’t exist, for a very long time in some cases, or they build a business model based on a false notion that they will be able to get something that doesn’t exist,” says Dobson. “Companies also re-collect data that they could have gathered from an existing source at a much lower cost, potentially with a higher degree of accuracy.”

The taskforce therefore recommends that a data catalogue is set up to capture all available energy system datasets across government, Ofgem and industry as standardised metadata, with industry participation in the catalogue mandated via regulatory and policy frameworks.

Missing data is a particular concern on older networks, but is there a way to avoid the major investments required to upgrade them with sensors? Emerging technologies such as data analytics and machine learning could provide a solution.

Electralink, an organisation that runs a data transfer service for the energy sector, has been working with the DNO Western Power Distribution to explore how machine learning can interrogate customer energy consumption patterns to reveal network constraints.

Legal restrictions currently prevent access to granular domestic smart meter data, so the project explored how machine learning can be applied to non-smart consumption data already available in Electralink’s archive.

Dan Hopkinson, director of data and transformation at Electralink tells Flex: “Using machine learning on the data we were able to identify some very strong correlations, and the potential for the existence of a lot of low carbon technologies that DNOs are not currently aware of. The data is a strong indicator if a technology like solar panels, or an EV recharger, has been applied to a property.”

Hopkinson admits that the model is not perfect, but the intention in future research is to incorporate additional data sets, including contextual data, switching behaviour, and demographic data such as property type, to create a more accurate picture, This could allow DNOs to model the low voltage network without the need for protected consumer data.

“This form of virtual network monitoring could improve our understanding and therefore reduce costs by ensuring that real-time monitoring is only put where it is needed,” says Hopkinson.

Benefits multiplied

Data may be the new oil, but if, like the black stuff, too much of it is allowed to accumulate in the hands of private companies there is a danger it will not empower and benefit the rest of society.

Reducing barriers to public and private sector data can attract innovators that create operational efficiencies or develop new business models; it can increase the speed that new markets develop and improve decision-making across the energy system. Data sharing improves transparency, which can give the regulator and third parties better oversight of costs and energy efficiency.

In a drive to exploit these benefits, the EDT recommends that all energy system data should be presumed open, and that data owners who want to restrict access must first justify the reasons why to the regulator.

The NIC’s Hayes comments: “Our report Data for the Public Good made points about the need to share data within industries and across industries and how the value that can be unlocked is greater than the value that can accrue to one company alone. The typical commercial approach is that all information is commercially sensitive, but when you go through data in more detail you often find that lots can be shared.”

The taskforce recommends that companies adopt an “openness triage” process that considers a range of risk factors and develops appropriate mitigation mechanisms to maximise the potential for sharing.

The opportunities around open data have already been demonstrated by Elexon, whose balancing mechanism reporting service (BMRS) provides free data for anyone to use under an open online licence.

The BMRS has over 1,000 daily users on average and provides key information on the wholesale electricity market, including near real-time updates on which sources are being used to generate electricity, demand levels and imbalance prices. It also publishes data required by the EU REMIT regulations, such as outages at power stations.

Elexon provided input to the taskforce on the sort of companies and parties that access BMRS data and what they use it for. Mark Bygraves, chief executive at Elexon, says: “This data is extremely valuable to existing companies and new entrants, because understanding supply and demand patterns helps companies to develop the right products and services for consumers.”

Open data will be crucial to support the sector’s transformation to decentralised energy, he adds: “Many consumers will provide demand-side response whereby they reduce, or increase their electricity consumption at certain times in exchange for payments from energy companies. We are likely to see greater use of electricity storage and electric vehicles that act as ‘virtual storage’ by being available to top up supply when they are not in use … This means that as an industry we need to ensure easy access to clear data on when these services are needed by network companies and suppliers, network companies also need clear data on the capability of electricity storage, so they know how much power it can provide, when, and for how long.”

Zeroing in

As technology continues its seemingly inexorable march, progress is under way to implement the recommendations made by the EDT, since the publication of the strategy in June this year.

Ofgem has said it intends to include the recommendations in several regulatory position papers and has progressed them as part of the next business plan update for RIIO2. The regulator has started making its metadata (data that describes data sets) more available and has updated its metadata practices to ensure they are interoperable with the data catalogue proposed by the taskforce. It is also exploring options to use Innovate UK’s energy revolution funding to stimulate investment related to digital infrastructure recommendations made by the EDT.

But the road ahead could be bumpy given the need to overhaul ingrained procedures and attitudes. Andrew Burgess comments: “There is a cultural change, for the energy industry as well as for Ofgem, around understanding that data on commonly used assets, such as the energy system, is presumed open to all, with the burden of evidence on the data custodian/processor.”

He says the digital architecture solutions proposed by the taskforce will only succeed if they are properly coordinated with other digital services, using a process of iterative delivery that avoids monolithic “old-style IT service delivery practices”.

The EDT has set a primary goal to get its first two recommendations [to direct the sector to adopt the principle of digitalisation of the energy system, and to adopt the principle that energy system data should be presumed open] peppered through regulation and legislation over the next few years. The data catalogue is a near-term target, with progress on its development already made by Ofgem, BEIS and Innovate UK.

“The energy sector might be behind the data curve, but if everyone supports our strategy we could find ourselves ahead of the curve into the future,” says EDT’s Sandys. “Soon all areas of infrastructure are going to become interrelated, which is why it is so important to get conflation around approaches and standards, etc. These are exciting times indeed for infrastructure.”

Stephen Cousins is a freelance journalist

Lessons from Estonia

Estonia is leading the rest of Europe in its use of smart meter data with a platform that allows consumers to control who can access their energy data.

The Estfeed data exchange platform was developed by Elering, the country’s independent gas and electricity transmission system operator, to connect smart meter data from various hubs and sources to energy service providers, including suppliers and app developers, via a legal and secure system.

Powered by X-Road software, it features a transparent data transport layer with a consent management system that allows individual consumers to choose who to share data with, the purposes it is used for, and view who has accessed the data. Individuals can also track their personal energy consumption, and what energy they have produced.

The platform also provides access to other data, such as stock market electricity prices, and weather forecasts, to help energy service providers and other market players optimise or innovate new services.

Estafeed went live in September 2017 and each month consumers give about 500 to 1,000 consents to access their data. According to Elering, the data has allowed companies to innovate more personalised services that lower consumer energy bills. For example, building monitoring service providers have accessed data to optimise consumption and lower energy costs in business buildings. Energy consumption data has been shared with energy package comparison tools and electricity and gas suppliers to create personalised offers and enable consumer switching.

Dr Richard Dobson, practice manager – data systems, at the Energy Systems Catapult, tells Flex: “As a secure data communications platform, X-Road underpins the smart metering system in Estonia and means that all consumers have complete visibility of what’s going on with their data. They have the ability to say ‘I would like organisation X, Y or Z to access my data’ and then remove that access at any time. It is really exciting.”

Estonia is already a pioneer in smart meters, having achieved 100 per cent coverage in homes and businesses back in 2014. Recent developments in EU regulation, including the General Data Protection Reform and the Clean Energy Package have sparked wider interest in Estafeed. Elering now plans to expand the system to connect to 100 million electricity meters across Europe by 2020.

Data dozen

Dr Richard Dobson
Practice manager – data systems, at the Energy Systems Catapult

Dr Richard Dobson, practice manager – data systems, at the Energy Systems Catapult, lists the top types of data the energy sector needs for transformation – and how close we are to getting it.

1 Network infrastructure data

Network data (including network maps and logical network models) is a fundamental enabler for efficiency and innovation. Understanding where the network is, how it is connected and the limitations of the system is key to optimising the system operation, develop new business models (DSO transition, flexibility markets, local energy markets, peer to peer markets, etc.) and driving technology innovation.


Data exists for the high voltage/pressures but not at lower voltage/pressures and the data is not generally openly available

2 Consumer needs and preferences

Understanding consumer needs and preferences is absolutely critical to enable the sector to develop low carbon solutions which consumers want to buy.


There is a need for more research but robust data collection processes exist, such as the ESC Living Lab (open and paid)

3 Granular demand data

Granular consumer and industry demand data is vital to many critical industry processes and a range of new business models.


Many ‘non-smart’ meters still exist, the Smart Meter Implementation Programme is progressing but access is difficult and many innovative organisations are choosing to recollect data rather than integrate

4 Geospatial data

Data which describes the layout and characteristics of the physical world including the location, size and shape of buildings and road network are vitally important for planning, building and operating energy systems.


ata is of high quality and is widely available (open and paid)

5 Distributed energy resource data

Distributed Energy Resources (DERs) are increasingly common and there are a number of registration portals but there is no consistent dataset which provides an accurate view of all assets and their capabilities. Greater accuracy and accessibility of DER data would enable growth of flexibility / local energy markets, optimisation of investment and improved system operation.


Some sources exists but the data is incomplete, inconsistent and not readily accessible – the removal of the FIT register has made this situation worse. The EDTF recommendation 4 addresses this point

6 Network capacity and constraint data

Network capacity and constraint information is the result of a blend of network monitoring and modelling. This information is critical to streamline the connections process, identify opportunities for innovation and drive investment into the right locations within system.


Some high-level information available but it is inconsistent and not frequently updated

7 Planning data

Local Authority planning data is important to enable the energy system to grow and adapt to the changing needs of the nation. Having visibility of developments further in advance could enable more innovative and efficient energy system solutions.


Generally available but with inconsistent formats

8 Asset monitoring data

Asset monitoring data includes technical asset monitoring (network, generation, storage, etc.), smart meter metrics and the associated models. The blend of this monitoring data enables actors to operate the system effectively, identify areas for optimisation / innovation and utilise proactive network maintenance models.


Data is collected for some assets but it is not ubiquitous and the data is not widely shared

9 Market data

Data relating to energy markets and the financial operation of the energy system. This information is critical as it lets innovators identify and target areas where costs are disproportionately high, enables carbon pricing to have maximum impact and enables price signals to drive beneficial behaviours.


Existing market data is often available but via paid services, emerging markets are less transparent.

10 Cross infrastructure data

Sharing data across infrastructure sectors (water, communications, transport, waste, etc.) could help to minimise public disruption from groundworks, reduce costs for consumers and identify innovation opportunities.


There are good projects such as the Geospatial Commission Underground Asset Register but data which is currently openly available is limited

11 Weather data

Granular weather data is fundamental for modelling output of renewable generation, asset condition, asset efficiency, etc.


Data accurate and widely available (open and paid)

12 Demographic data

This is critical for modelling, business model innovation and low carbon transition modelling.


Data is of a good quality and widely available
(open and paid)




Drax takes outage testing to the cloud

The biomass and coal-fired power station in North Yorkshire is using eviFile’s workflow tool to create an end-to-end digital audit trail of the some 4,000 individual tests that must be carried out on each generator unit when they go through a maintenance outage.


The digital auditing solution replaces an existing paper-based system that had proved difficult to track and monitor progress.


Michelle Desmond, outage co-ordinator and project sponsor at Drax Group told Flex: “Streamlining the way we work will lead to cost savings – it’s difficult to quantify those savings at this stage, but we believe it could be up to £1m a year. The new system is saving teams time and making our internal processes more efficient.”


Outage testing is designed to ensure that generators are safe, fully operational and performing at maximum output and efficiency. Units are shut down for up to 12 weeks every four years while a team of over 30 engineers and almost 1,000 contractors completes tests across different systems. The complex process must be planned at least a year in advance.


EviFile’s digital workflow solution can manage a high volume of concurrent tests and ensures that each test is validated and assured by the relevant authorities within Drax, at technical support provider, Atkins, and ultimately insurers that provide coverage for any issues that may occur on a unit.


Luke Allen, managing director of eviFile told Flex: “Through a process of Digital Progressive assurance, project partners can access high-quality, real-time digital information showing exactly how project delivery, repairs and maintenance and inspection processes are performing.

This enables businesses to reduce the potential of cost overruns by providing better visibility on the health of a project.”


A customisable control panel ensures that every test is sent to the right contractor at the right time and all relevant sign offs are obtained before progressing to the next phase.


The Drax project raised some unique challenges. Tests had to take place inside a boiler, which required a more robust solution that on other projects and using offline tablets that were synched to the eviFile cloud when a connection was available.


The latter proved a challenge with device specification on the Drax network, said Allen: “A base station outside of the outage location was setup and each test recorded as a draft and then completed by a tech clerk immediately after the draft was finalised. It meant a small delay in results hitting the system …In future all outage tests will be completed through eviFile on Drax-verified iPads and the data synced in real time.”


According to Desmond there was an initial upfront period when Drax had to “iron out the kinks and get used to using a digital platform”, but now it is halfway through the second outage using eviFile, the process is smoother with minimal support required.


The secure, digital cloud-based solution has proved a more efficient and secure than the previous process of recording using pen and paper with sign off involving paper reports being passed between contractors and departments, said Desmond: “We have made significant efficiency savings in terms of productivity and visibility of progress made during the outage and dramatically improved the quality of the data we now receive and have access to.”


Smart meters

Why smart meters remain an untapped resource

Smart meters form the foundation of the utility sector’s digital transformation. But what has the industry learnt about how to use the vast quantities of data at their disposal? Rachel Willcox finds out.

The government has described replacement of all gas and electricity meters with smart meters as an essential national energy infrastructure upgrade for Great Britain that will help make our energy system cheaper, more efficient and reliable and be a key enabler of the transition to a low carbon economy.

Already there are 14.9 million smart and advanced meters operating in homes and businesses across Great Britain, although the transition could hardly be described as seamless. It has been thwarted by missed deadlines, interoperability issues, and installation rates suffering a loss of momentum as the government commitment to ensuring that every home and small business in the country is offered one. It’s relief all round that the Department of Business, Energy and Industrial Strategy (BEIS) is now consulting on extending the deadline from 2020 to 2024.

Analysis from BEIS suggests that smart meters will slash £300 million off consumers’ bills in 2020, rising to more than £1.2 billion a year by 2030 – or £47 per household – though those figures have been disputed. For energy providers, applying analytics to the vast quantities of useful data utilities collect from customers can uncover new customer usage patterns, better forecast demand, and improve compliance with regulatory requests, as well as prevent fraud and reduce loss.

The majority of meter installations to date have been first generation smart meters (SMETS1), a standard defined by government to ensure minimum common functionality ahead of the national smart metering communications infrastructure being in place that will pave the way for interoperability between all energy suppliers. The long-awaited introduction of SMETS2 – the second generation of smart meters – has given a new impetus to the use of smart data.

Utility companies will benefit because, by better understanding how and why their customers use energy, they can offer products and services specifically tailored to their customers’ needs. This includes dynamic tariffs, localised peer-to-peer energy trading and faster switching
times, explains Bjoern Reinke, director, Smart Metering and Systems of Intelligence at Drax, which has worked on smart metering projects with customers including Haven Power and Opus Energy.

As we move towards a decentralised energy system, where the production and consumption patterns become hard to predict, the access to data is a key enabler for the transition towards a sustainable future, says Christian Chudoba, founder and chief executive of digital energy platform Lumenaza.

Against a backdrop of compelling benefits, many energy providers remain tight-lipped on how smart meter data is being used to their competitive advantage. A spokeswoman from SSE told Utility Week the commercially sensitive nature of the subject meant it was unable to share details of how it was using data from the 1.45 million SMETS1 and 250,000 SMETS2 meters currently installed among its customer base. Other utilities approached by Utility Week were similarly reluctant to be interviewed.

Although energy providers are in an excellent position to make the most of customer data, in practice most customers aren’t noticing any benefits, says Mustafa Atik, an energy and utilities expert at customer experience specialist Quadient. Only a quarter of UK consumers believe they have been given a better service or advice thanks to their utility companies’ application of their data, while 61 per cent have seen no evidence of it being used at all, according to Quadient’s research.

With data use such a critical part of the modern customer experience, utility providers risk wasting their efforts if it brings no customer-facing benefits and creates the impression that customers ultimately don’t matter, Atik says.

Ted Hopcroft, an energy expert at PA Consulting, says availability of the data and knowing what to do with it once you have it remains a stumbling block, not to mention the intricacy of integrating SMETS1 and SMETS2 data. “Companies will need this composite integrated view to reap the benefits of smart data and for distribution network operators in particular, some critical mass of data will be required to deliver the potential benefits.”

Meanwhile, security concerns prevail and the impact on consumer trust is not to be sniffed at. “Some consumers question if the smart meter is spying on their household, while others are concerned about hackers gaining unauthorised access to their home networks,” explains Atik. There’s also the worry that software incompatibilities could hinder switching or that insight into usage could even prompt suppliers to bump up energy tariffs at peak times such as Christmas.

Katie Russell, head of data & analytics at Ovo Energy, says customer communication is key. We reassure them that we use encryption, controlled access and other security, including ways agreed with the government, to make sure the information transmitted from their smart meters is secure. At the same time, strict new regulations and codes of practice, governed by Ofgem, keep smart meter data private and safe, and all customer personal data is safeguarded by the Data Protection Act.

Being conscious too that customers may not want their data used for anything other than billing, Reinke warns: “It’s really important that the energy suppliers only inform customers about products and services that are relevant and likely to be of interest to them. Many consumers are already worried about the perceived invasion of privacy in relation to technology such as Alexa. The balance needs to be right. Using the data to give too many notifications to customers could result in disinterest as could those notifications which don’t add any value.”

Making smart meters more useful

Under GDPR, a homeowner’s consent is required for their supplier to take and use their half-hourly readings (see p16). If this permission is rescinded, the data collected must be deleted and returned to the customer. Explaining why data is collected and how it is used should help to alleviate this hurdle, says Shona Toms, senior associate at law firm Osborne Clarke.

As data sharing becomes more commonplace in the energy market, the main challenge is not data protection or security but competition and data ownership. “There is a tension between a company wanting to hold on to data for its own benefit and uploading the data to an open platform for the benefit of the industry at large,” Toms adds.

Smart meters are ultimately useless if providers don’t use the data they provide in the right way. Organisations must have a clear view of what they want to achieve with the data, with a clear line of sight from their KPIs to the data provision, Hopcroft says: They must avoid the trap of drowning in data or creating a function that generates endless graphs and dashboards showing things that are interesting to know but don’t drive business value.

The potential of smart meter data remains largely untapped. Applying machine learning so that data can be analysed at a much more granular level represents a huge opportunity. “Anomaly detection can provide a great service to customers because it can also identify inefficient equipment and that which is likely to fail.”

The ability to dynamically update tariffs, switch loads and turn on other sources of energy, such as battery storage, are also largely untapped. In February 2019 Octopus launched time-of-use tariffs for consumption, that effectively paid customers to use electricity when there is too much power on the UK system, and in April became the first supplier offering a Smart Export Guarantee tariff.

Marissa Papas, managing consultant at Capgemini, believes using smart meter data in isolation is a missed opportunity.  By combining the data points gathered through smart metering with other data sources, for example demographics and behavioural, utility companies can turn the data into something useful and relevant

“The true winners in the market will be the ones who use data to address business concerns and customer needs and are agile and proactive in their response,” Papas adds. “Utilities of the future will need to have a proactive and engaged relationship with their customers. Building and maintaining trust with customers is crucial in this highly competitive market.”

The key to successful use of smart meter data is to focus on the outcome and value for the end customer and for the energy system,” warns Chudoba. “After all, the smart meter data is very important, but it is an enabler; a piece in the puzzle of the energy transition.”

Rachel Willcox is a freelance journalist

OVO Energy

Over 45% of Ovo’s customer base is now smart metered, having installed over one million individual smart meters to date. Katie Russell is head of data & analytics at the company

What issues does use of smart meter
data present?

When we use customers data to build products, we need to ensure that we are transparent in how we make any personalised calculations and recommendations and that customers have enough information to understand how they are being billed as well as understanding the additional information on how they could save money. Often this means simplifying complicated concepts and only showing the most relevant information.

What does the future hold in store?

We will we show our customers how combining smart meters with Internet of Things innovations in the home will magnify their energy efficiency and offer them benefits. These benefits will be amplified for customers who grant us permission to securely collect the IoT data itself.

What advice would you pass on to others based on your experience to date?

Give customers control of their data and preferences. Listen to their concerns and reassure them about your security practices and how their data is safeguarded. From a safety perspective, invest in training. All our engineers are trained externally by accredited training providers. In product development, A/B test and track customer feedback to ensure the products are as informative and effective as possible.



Transparency, consent, ownership, privacy…when it comes to data utilities firms must protect consumer rights.

By Alex Underwood and Jeremy Godley

The introduction of smart technologies promises to revolutionise the sector by helping to improve energy efficiency, facilitate the introduction of new products and provide long-term opportunities for cost benefits. However, there is growing concern around how companies use consumers’ data to drive these changes. Moreover, following a spate of recent scandals, increasing numbers of consumers are beginning to raise questions around the ethics of big data and are querying what steps companies, regulators and legislators are taking to ensure their data is adequately protected.

Data ethics

Data ethics is primarily concerned with a number of key principles, including, data transparency, consent, ownership, privacy, and data availability. Previously, the energy sector rarely encountered issues concerning the principles of data ethics because traditional utility business models collected only rudimentary data relating to the levels of consumption on customer premises.

However, the introduction of smart meters, among other technologies, has meant energy utilities are accumulating more consumer data than ever before. As we report in the previous feature (see p15) there is potential for energy utilities to acquire unprecedented amounts of data about their customers. As utilities continue to develop novel practices to collect, analyse and monetise consumer data, there is the inherent risk that the digitisation of the energy industry will result in more instances of energy utilities encroaching on consumers’ data rights.

In the UK, the General Data Protection Regulation 2016/679 (GDPR) and the Data Protection Act 2018 (DPA) govern the processing of data. This covers all and any data which can be used to directly or indirectly identify natural persons. Detailed energy consumption data from smart meters is likely to be considered “personal data” for the purposes of the data protection legislation, meaning energy utilities that handle customer consumption data will be expected to comply with the GDPR.

One of the key principles of the GDPR is that personal data must be processed in a fair and transparent manner. In practice, this means energy utilities must provide a privacy note to individuals setting out how and why personal data is being processed. Energy utilities will need to make sure they keep their privacy policies updated as and when new purposes arise. This is not always straightforward because big data analytics often result in processing data for new and novel purposes.

In addition to the requirement to process data in a fair and transparent manner, energy utilities must also have a lawful basis for processing personal data. Out of the six available lawful basis, consent or legitimate interest are likely to be the most appropriate. Consent must be freely given, specific, informed and unambiguous to be valid under the GDPR. Consumers must have the right to withdraw consent at any time. Alternatively, energy utilities may seek to rely on legitimate interest. This is appropriate where energy utilities are processing data for a legitimate reason which the data subject could reasonably expect and is doing so in a manner that is likely to have minimal privacy implications.

Importantly, the GDPR makes clear that the ultimate owner of data is the individual. Individuals have the right to consent, to withdraw consent, to restrict processing, to access copies of personal data held, to correct inaccuracies and a right to be forgotten. Since the GDPR came into effect, the Information Commissioners Office (ICO) has shown that it is willing to enforce these consumer rights by issuing a number of large fines to companies in breach of the legislation.

One of the most challenging principles to protect is privacy. This is due to the intrinsic nature of big data; the more information a company has about a consumer, the more likely that company is to develop products that benefit the consumer. We have seen through a number of high-profile media stories how companies within the tech industry are able to obtain personal details about our lives and personalities. Until now, the energy industry was largely immune from this.

However, the large increase in the collection of data by utilities has left the consumer more exposed to these types of infringement of privacy. Based on consumption data alone a company is able to ascertain someone’s identity, their employment status and be able to glean information about their lifestyle and habits. In addition, and similarly to a number of other data-orientated industries, as we permit companies greater access to our personal lives we open ourselves up to increased security risks.

For example, burglars may be able to use the data from consumer smart systems to target empty households. Despite this, energy consumers appear willing to forgo aspects of their personal privacy in exchange for the benefits associated with data sharing – recent research undertaken on behalf of Smart Energy GB showed that only 5 per cent of the people surveyed raised privacy within the energy industry as a concern.

In addition to the requirements under the data privacy legislation described above, energy companies are also required to comply with the Data Access and Privacy Framework. The framework was designed to safeguard consumers’ interests, while enabling stakeholders proportionate access to data to facilitate innovation. It has been implemented through a range of regulatory requirements, including the smart energy code (SEC), energy supply licence conditions and the energy distribution standard licence conditions. The requirements under the framework can be broadly split into the following categories.

Energy supplier access to domestic consumption data

Energy suppliers are permitted to access monthly and daily energy consumption data from domestic customers in order to fulfil their regulated duties, for example, to provide the consumer with accurate bills or statements. Energy suppliers can only access energy consumption data that is more than daily, such as half-hourly data, providing they have the consumer’s consent to do so. As part of this process, the energy supplier must explain to the consumer how their data will be used and that they have a right to withdraw consent.

The government has claimed that the rules are designed to ensure that consumers are able to make informed choices about sharing their detailed consumption data. The onus is on the energy supplier to clearly explain why they wish to access consumer data and to incentivise the consumer into sharing their data by offering products and services in return.

Energy network operator access to domestic consumption data

Energy network operators are under more stringent requirements than suppliers. Under the framework, they are only able to obtain consumption data relating to periods of less than one month if they have obtained the consumer’s consent and have implemented Ofgem-approved procedures to ensure the data can no longer be associated with a customer at a particular premises. The exceptions to this are more limited than for energy suppliers and include where they have reasonable grounds to suspect theft.

Third party access to consumption data

The ability for consumers to share their data with unlicensed third parties is widely regarded as the biggest potential for innovation, permitting third parties to develop new products to help consumers manage their energy use. Under the GDPR, individuals have a right to data portability, which allows individuals to obtain and reuse their own personal data for different purposes across different services. This helps individuals take advantage of applications and services that help find a better deal or to develop a better understanding of their energy consumption habits. To further support this, the government established the smart energy code, which enables third parties to access energy consumption data directly from smart meters subject to a number of privacy safeguards, including acquiring the consumer’s consent.

Alex Underwood and Jeremy Godley are lawyers at Osborne Clarke


Vulnerable customers

Shared learning

In advance of a Priority Service Register data-sharing programme being rolled out nationwide by April 2020, what have United Utilities and Electricity North West learnt from piloting the scheme, asks Robyn Wilson.

It’s been two years since the water and energy industries started to publicly explore the benefits of sharing data on vulnerable customers as a way to improve services.

Set against the recent backdrop of big data, regulators Ofgem, Ofwat and the UK Regulators’ Network published a report in October 2017 calling for greater cross-sector collaboration and the sharing of non-financial vulnerability data through the Priority Service Register (PSR). The PSR is a free service provided by suppliers, which vulnerable customers can sign up to receive various services such as advance notice of planned power cuts. Up until this point, each utility held its own register.

The report prompted a series of pilots to get under way between major companies in energy, water and third-party industries to see how this tie-up would work in practice, with an update report published by the regulators the following year. This highlighted a number of PSR data-sharing pilots, including United Utilities and Electricity North West, who explain about what they’ve learnt from the pilot.

How did the pilot work in practice?

Running over a 14-week period between February and April 2018, the companies conducted a two-way data-sharing pilot in which they gained explicit consent from their mutual customers to share their PSR data.

Explaining the mechanics of the pilot, ENW’s customer director, Stephanie Trubshaw says: “We started off making sure we did everything simply, so we didn’t worry about the existing data we had – we just dealt with new customers. So, when we or United Utilities were registering a new customer, we would offer to register them on both [PSRs].”

This data and the customer’s consent would then be recorded as well as being shared with their suppliers, subject to the customer’s consent. During this process they confirmed the preferred method and time of contact with customers.

“We [ENW and UU] had regular weekly meetings to understand the challenges we were facing, such as how people recorded their date or their name, to make sure we could actually use the data in a specific format,” Trubshaw adds.

This led the teams to ensure that full names were taken from customers, rather than abbreviated versions, as well as the customer’s whole address. Though a seemingly small step, this was critical, since it meant the data was taken down by both companies in the same format and could therefore be used and shared effectively.

“By getting the correct matching data, we could just load it into the databases to prevent manual handling or double entry, which makes a difference when you get into higher volumes.”

What was the customer response?

“The first few months, the uptake was very mixed,” says Trubshaw. “Customers were fine with what we were offering but they weren’t as sure of the benefits of the water [PSR] and this was because our agents probably weren’t skilled enough.”

As a result, both companies trained their agents in the trial to understand the benefits of cross-referencing the datasets. This would allow them to fully explain these benefits to customers. These would include a company who had identified a customer vulnerability being able to share it through the PSR, resulting in a coordinated and more efficient response that could ultimately benefit both customer and company.

Advantages for the customer really come to the fore when considering extreme weather events. As United Utilities customer and people director Louise Beardmore notes: “When it comes to floods, we are responding as multiple agencies at the same time. If you think of the flooding we’ve had recently in the North West, it means that United Utilities and Electricity North West are responding together.

“We know which customers need our help and support, so when it comes to restoring the electricity supply or water supply, we know where those customers are.”

Trubshaw says that after carrying out the training, they saw a notable increase in registrations. Customer uptake increased to about 85 per cent, from 42 per cent when they started in the first week. By the end of March 2018, she said about 4,500 customers had agreed to having their data used by both parties.

What are the lessons learned?

Perhaps one of the most important lessons learned to date is a company’s ability to have “the human touch” in order to maintain customer confidence during this process.

As United Utilities’ Beardmore explains: “One of the biggest areas of learning is that you need to make sure your staff are properly trained to be able to communicate the benefits of data sharing and the benefits they will see from being on someone else’s PSR.

“You’ve got to be very confident about how the data is going to be used and for what purpose because this is about trust for customers, so how their data is exchanged and stored is very important.”

An example of this, she says, is telling the customer that you are using the data to tailor the support provided by both companies such as making sure a blind customer is registered as in need of a braille bill at the same time as being clear the data will not be used for marketing or debt collection.

To this Trubshaw adds that it is also really important that your staff feel comfortable with the services both companies are offering so they can comfortably and convincingly promote it. She advises companies to have specialists to head the programme and to train specific teams of between 10-12 agents to start with, to ensure processes are tested before being rolled out more widely.

“Realistically it’s better to start off with a specific group to ensure you’re doing it on a scale that you can learn from. Ensuring that you have good project management, with follow up calls the next day to cross-reference and check what happened is also really key to ensuring that you are delivering on what people are receiving at the other end.”

There is a consensus that the scale of rolling out the nationwide project, coordination between the many organisations that are sharing data will undoubtedly be a challenge. The good news is that the technology for data sharing is not proving to be an issue; described by one source within the industry as “straightforward”.

But companies need to be wary of balancing both the need to have the “human touch” with the necessity of being rigid and accurate when gathering volume data at this scale. Staff training is therefore essential to the success of a nationwide rollout.

Robyn Wilson is a freelance journalist



Expert views

Securing the Brave New World: can utilities trust their consumers’ data to AI?

sponsored post

Dr David Scholefield, CISSP OPST, Chief Information Security Officer at Flexys Solutions

Machine Learning and AI may be ‘flavour of the month’ with the media talking excitedly about a new dawn in intelligent computers. Behind the hype, there is a quiet revolution happening that will change all of our lives in ways that we are only just beginning to understand. However, with the new opportunities come genuine risks and security challenges that need to be carefully managed.

What is machine learning?

Machine learning (ML) software works by training itself to find patterns in large quantities of data about a given subject area. Once trained the software can recognise those patterns when shown new data.

This may not sound much, but take the example of diagnosing cancer from MRI scans and it suddenly looks more interesting. An ML system can be shown thousands of MRI scans of patients with a specific type of cancer and thousands of scans from patients without cancer, and as a result the program can work out how to make the correct diagnosis from any future MRI scan. A machine learning program may discover patterns that even the best specialists haven’t seen.

In a commercial setting, including the utility sector, AI also has enormous potential. One example in debt collection is the identification of vulnerable customers who may need additional support, as well as identifying those who are likely to self-manage their outstanding debts. This intelligent segmentation means that resources can be concentrated on those customers who need it most, driving down cost and improving customer satisfaction.

But what are the risks?

Most security concerns around machine learning centre on privacy and autonomous decision making. Firstly, in order for AI systems to become expert in specific human behaviours, they must analyse a great deal of relevant personal information. There is a concern that this information may not be adequately protected from disclosure when the AI interacts with other systems.

Secondly, there is a concern that the system might make incorrect decisions that impact on people’s lives in a negative way without the ‘common sense’ or experienced oversight of a human agent.

Although these concerns are legitimate, there are effective systems and controls that allow AI to be used in a manner that protects privacy and protects against unmoderated AI decision making.


When AI learns about behaviour patterns from data concerning a large group of individuals, there is no need to include information that identifies any specific individual. For example, the name of a person isn’t relevant to the learning process: the AI is trying to learn about people in general, not any one specific person.

Responsible providers of ML applications will strictly anonymise any learning data so that the AI never sees any information that identifies any individual. In addition, once the AI has completed its learning, strong data separation controls ensure that the data it has used is securely stored away from the AI system itself and can never be seen or accessed by any third party. There is therefore no way to ‘trick’ the AI into revealing anything specific about the people the data was sourced from.

Autonomous decision making

Another area of potential concern is around autonomous decision making, where AI might make incorrect decisions without any human oversight or intervention. With the new GDPR restrictions, this kind of machine learning faces even more controls and restrictions.

AI should only ever be used to support or augment existing human decision making. At no stage should AI systems make definitive or irreversible decisions that might adversely affect any person’s future. AI should empower its users to make more informed and effective decisions but not attempt to replace existing knowledge and experience.

The future is bright

With controls around privacy and autonomy, AI can support existing business functions by learning about customers and using that learning to improve future interactions with them. There’s a revolution coming, and businesses that adopt secure AI now will certainly have an advantage over those who are left behind!

To find out more about the areas covered in this article or for more details of our solutions and services contact us on 0117 428 5741 or email enquiries@flexys.com

See also, podcasts at:

Episode 1: https://soundcloud.com/user-204957968/flexyspodcastmlai

Episode 2: https://soundcloud.com/user-204957968/its-maths-not-magic-machine-learning-is-like-giving-handguns-to-chimpanzees-episode-2

Making Digital Simple For Everyone

sponsored post

By Mark Simpson, Sales and Marketing Director, Mando

Over the past few years inclusivity has been a rising challenge for the utilities sector, one which is increasingly becoming a focus with regulators, with Ofwat and Ofgem calling for better service for all customers.

Whilst companies are shifting more and more services online and finding new ways for customers to self-serve, serving vulnerable customers with additional needs has typically been an afterthought of digital transformation, and customer experience can vary greatly.

Digital customer experience agency Mando has just released a new report Making Digital Simple For Everyone: Improving experiences for vulnerable customers. This is the culmination of interviews with charities and support groups, and roundtable events with utilities leaders.

The report explores common challenges that prevent utilities from delivering a positive experience to priority groups, hearing and sharing perspectives on how digital technology can make lives easier.

Data needs to be shared and accurate

Customer data is a common challenge across the utilities sector – ensuring its accuracy, keeping it up-to-date and incentivising customers to share this with providers. The onus has always been on the customer to register themselves as vulnerable and the size of the problem comes when you look at vastly differing numbers on priority services registers: 345,000 for water, 6 million for electricity and 4.8 million for gas. A successful pioneering data-sharing project was piloted between Electricity North West and United Utilities last year, and a sector-wide initiative is due to go live in 2020, sharing the data of customers eligible for priority services.

Personalisation can highlight support services

Personalisation is one area which has become synonymous with good customer experience. When harnessed in the right way, this can be used to better serve vulnerable customers. Digital can allow you to determine whether someone is pre-vulnerable based on their journey through a website, such as long dwell times, or confused or illogical movement – and personalised content can be tailored to their perceived needs, like making calls-to-action to feature the various support schemes offered. Care is to be taken though, personalisation can sometimes be seen as intrusive by a vulnerable person.

Everything everywhere – but integration is key

Customers are increasingly switching to digital channels and expect to be communicated to within their channel of choice. This experience, whilst improving in many areas, sometimes falls down when customers miss a payment and they drop straight into a telephone service. For a customer with additional needs, this can be difficult, particularly with multiple vulnerabilities like a hearing or speech impairment, who may struggle on the phone. The key is to consider the end-to-end journey for all customer groups, ensuring that they can begin on one channel and continue on another, whilst seamlessly integrating new and emerging channels like voice assistants and chatbots.

Prioritising accessibility

Accessibility is rarely at the forefront of digital transformation and there remains a fine balance between meeting business objectives and serving the needs of all customer groups. However, with 11 million people in the UK living with a mental or physical disability, it is something that cannot be ignored. By rethinking how we design simpler experiences that can scale to more people in new ways, they become more accessible to everyone.

AI enhances opportunities for all

AI can enhance the intelligence of both new and existing digital applications, making an impact on how utilities companies can best serve vulnerable customer groups. This is relevant, not just for the customers themselves but also for internal teams. For example, an AI-powered bot can help internal teams to surface FAQs more quickly whilst assisting a customer in the call centre, allowing the agent to more closely focus on the customer themselves. These technologies are far-reaching, including speech to text bots, language translation, and predictive analytics.

With the right approach, digital can enable companies to better serve everyone; creating efficiencies to allow frontline staff more time to spend with those in complex situations or offering up new channels that allow people to overcome their challenges.

To download and read the full report, please visit www.mando.agency/flex

Crying over spills? Use data to save water

sponsored post

By Rik Gunderson, UK Utility Director, Software AG

Water shortages impact every aspect of life; it is fundamental to not only human life, but also to the transportation and manufacturing processes that define our modern economies.

Health, commerce, the economy, education all rely on a clean and available water supply. Leaks, pollution, drought are the enemies of water. Over 2 billion people live without safe water at home.

The World Bank estimates that more than 126 million cubic meters of water lost to leaks and theft cost nearly $40 billion per year in waste and foregone revenues.

Water is life, in more ways than one. And the regulators are watching. In the UK, an environment watchdog said the performance of some utilities was “simply unacceptable,” and that England would run short of water within 25 years unless action is taken.

So, what can be done? It all boils down to data.

From leak detection, to incident management and maintenance, data collected from Internet of Things devices can be the catalyst in a utility’s effort to save water.

A highly efficient water utility that can report honestly and accurately on improvements in leakage, pollution control and environmental issues – with a customer centric approach – will demonstrate a clear leadership position.

This requires complex integration between detection and fix, and Software AG is ideally positioned to help, using our unique capabilities. Working with our partners at water utilities, we have formulated a platform for growth and are launching a water management solution.

Using this platform, utilities can:

Detect and report leaks: Using a variety of leading-edge, industry standard Internet of Things monitors and sensors, then analysing the data in real time, you will be alerted as to leakage severity and environmental impact (using geofencing and geospatial referencing).

Manage resources: Integration capabilities trigger activities within your own resource management/ project team planning/route management systems. They direct the right teams to triage in the field to achieve the most effective tasks in the most optimal time.

Handling incidents: A structured working practice definition onsite, via rugged mobile devices, ensures an incident in the field is governed from the moment the resources arrive through every decision process. This will ensure the best-practice approach is taken to the successful outcome of any episode.

Inform and report: Clear accountability, auditable data, straightforward and open reporting will be built into the next round of water resource management plans. This will ensure that utilities get a “fair shake” in the press when accidents do happen, as details of incident detection improvements, environmental considerations, leakage reduction, predictive “AI assisted” analysis of future state, can be delivered to customers for complete awareness of your activities.

Realise ongoing maintenance improvements: Our partners want to be able to analyse current network deployments, points of failure, maintenance reports and descriptive analysis in order to accurately target ongoing work efforts at the most effective parts of the complex network. By utilising our machine learning engine and artificial intelligence analytics tools, our solution ensures that the most cost-effective efforts are being made in a time-sensitive frame. This means utilities can make the very best decisions on the direction and prioritisation of highly skilled and expensive projects.

Here at Software AG we take water conservation and management seriously, and we are working toward a solution to help water utilities better manage their supply. This is digital automated water management and we call it the Cumulocity IoT Solution Accelerator for Water Management. The solution, created in partnership with telco Telstra, provides utilities with a quick way to gain insights from meter data to minimize costs involved with supplying water to customers.

We are excited to be helping create a solution that will address one of the greatest challenges of the next century – the management of our natural resources; first and foremost – water.

To start a conversation around how you can better harness the assets, data, people and processes in your company, please contact:
Rik Gunderson, UK Utility, email: Directorrik.gunderson@softwareag.com


Request a call back