How the Met Office is preparing for an explosive growth in data

SUMMARY:

CIO Charles Ewen says linkages with technology providers will help the Met Office generate big value for the public purse.

Huge increases in the amount of data produced, stored and processed means public sector organisations must think now about how they will work with external providers to create benefits for the citizen and other partner organisations.

That is the view of Charles Ewen, CIO at the Met Office, who is helping the UK’s weather service manage an exponential growth in information. Ewen joined the Met Office eight years ago and moved into the CIO position five years ago. He has spent the past few years pushing a digital transformation initiative at the organisation, which has included the procurement of three Cray XC40 supercomputers.

These systems, which are three of the world’s 50 largest supercomputers, are capable of processing more than 14,000 trillion arithmetic operations per second. The supercomputing project is expected to enable £2bn of socio-economic benefits across the UK through enhanced resilience to severe weather and related hazards. Potential benefits include better forecasting at airports, more sophisticated modelling related to flooding, improved information for energy markets, and new climate impacts research.

Transforming through data

The Met Office, therefore, is at the leading-edge when it comes to using big data. The organisation draws on its array of human and supercomputing resources to create 3,000 tailored forecasts and briefings a day for a range of customers, including government bodies, armed forces, private sector enterprises and the public. Ewen recognises the Met Office is blazing a trail when it comes to data use:

Nearly all businesses aren’t past the inflection point for exponential data growth yet. The only organisations that are at that point are agencies like the Met Office and Cern, and then enterprises like Amazon and Google in the private sector space. We’re talking about very big numbers – and, when you start doubling very big numbers, then every time that you double it, if forces further transformation.

Ewen refers to the exponential growth in data, which leads researcher IDC to predict the amount of data around the globe will grow to 163 zettabytes, each of which is a trillion gigabytes, by 2025. That total is ten times the 16.1ZB of data generated in 2016. While access to this data creates a host of opportunities, Ewen warns IT leaders in all sectors that the inexorable rise in information creates significant challenges:

In five to ten years, the amount of data starts to become so big, you can’t move anywhere – you can’t even write it to a file system because it takes too long. Then, value must be extracted in a new way. We’re already doing research and development in the science domain, which involves largely physicists and mathematicians in orchestrated programmes. But with industry, it’s really centred on server-less technologies, so things like Google Cloud Functions and AWS Lambda, where you can move problems to data as opposed to data to problems.

Ewen says this shift in approach represents a new paradigm in data use. Public sector organisations, like their private enterprise counterparts, tend to generate value by moving data from one place to another. Ewen expects a new methodology to emerge, where a range of technologies and processes help IT leaders and their organisations to process information in real time. He mentions a range of advanced technologies that will help IT decision makers move problems to data, but suggests the exact nature of this approach is still up for debate:

Technologies like blockchain and secure containers are valid and will play a crucial role in this journey. The basic thing to stress is many people’s rules on IT will start to fall over. Take data lakes, for example. They’re great, but good luck trying to hold your own information when you start trying to work with an exabyte of data. You at least need to be ready to be put that information somewhere where you can pull it together. But the good news is there be will a method in the future – and this is tricky, because we don’t know what it is yet – that will help with these problems and help data to flow.

Embracing the future

Ewen says progress from outside organisations is already being made. He points to Greengrass, which is the client-side Internet of Things computational engine for Amazon Web Services. These kinds of next-generation developments will help the Met Office to continue to help its partner organisations make the most of the huge amount of weather data the government agency produces:

So, there are technologies emerging that will help us process the huge amount of information that’s going to come online during the next five to ten years, but they won’t come from us. This transformation relies on the world of Amazon, Google and IBM. It’s a great area to be involved in and it’s right at the cutting edge. We do work very closely with these providers because we’ve got a hard use case. And because we have a hard use case, we’re beginning to flush out some of those scaling problems about what happens when data gets huge.

It is also worth noting that other public sector organisations might find it harder to manage the transition to exponential data growth. Many government agencies have worked hard to open data to the public. While the initiative has produced positive results, Ewen is concerned that too many executives believe the opening of data is a panacea from which business benefits will simply unfurl. He says IT decision makers in all areas should recognise effective data management is a significant challenge, particularly as the amount of information continues to rise inexorably:

People get confused sometimes about open data equalling some kind of reusable playing field. That ceases to be the case at scale. Just making the data available isn’t useful if it’s too difficult to access the data. So, the debate around making something open versus useful, and who pays for that transition for how it moves from open data to becoming useful data, involves a lot of work. You’re challenging quite a lot of policies and processes, because most of the world isn’t yet used to an open way of working.

My take

There’s a lot of hype around the explosion in information, but big data doesn’t get much bigger than the insight-led initiatives of the Met Office. The leading-edge work being undertaken by the weather agency is a great example of a UK public sector organisation leading its field. Its strong links with partner organisations and technology providers will help CIO Charles Ewen to keep pushing boundaries during the next decade.

Image credit - Via pixabay