Our Drilling & Well teams have placed fiber optic cables alongside each Johan Sverdrup wellbore, which we can use as microphones. These microphones transmit the data equivalent of 10,000 Netflix movies each second onto a processing plant on shore.

Here, a data engineering team has managed to figure out how to process the data into a more manageable size while maintaining the main information content of certain aspects of the data.

These data are then sent to Fo.tone in real-time. The traditional way of doing this is to store the data on hard drives offshore, then flying them to shore by helicopter, and afterwards processing the data - which usually meant a 2–3-week delay.  

Now, this entire operation takes less than two seconds from the sound being recorded to it showing in Fo.tone. 

“The next step, which we already started, is analyzing the data further to see what we can learn and improve from it. Then we can continue to utilize machine learning to automate well operations and monitoring,” Kjetil explains.

To find out, they didn’t just look for one example of an obvious solution and run with it - they started testing and trying different services. 

“We are working in 2-week agile sprints, trying new things and improving on it constantly - which we still do,” John Harald says. 

They soon landed on Azure Data Explorer as one of the key components for handling the large amounts of data, released by Microsoft only 6 months before they started.

“It’s a lot of fun to be able to experiment and learn like we have. Especially when you’re as interested in working with Azure, data engineering and big data as we are. We’ve been able to test things at a very early stage, which has been incredibly interesting,” John Harald says.

“The business area has had the same approach as well. No one really knew what the data would contain, where it could lead or what opportunities it had. It was really a big research project on both sides,” he adds.

Splitting data

When data flows from offshore and onto land, it’s a matter of terabytes daily and there’s a lot of processing to be done before it arrives in Fo.tone. As soon as data hits land it’s processed & decimated and then sent into Azure to an Event Hub. From there, data is sent to Azure Data Explorer where it’s stored in order to do continuous queries. 

“To be able to visualize the data to the end user with as low latency as possible, we decided to stream the data directly to the front end using Azure Service Bus. This means we can keep latency down to just two seconds from data being recorded offshore until it's shown in Fo.tone,” John Harald explains.   

The way the Fo.tone team used Azure Event Hub and Service Bus in combination was new to everyone, even to its creators - which even led to the team giving presentations to Microsoft and presenting their use case.

The team made it happen through the old-school tradition of brainstorming, but also by following certain design principles. One of them was Separation of Concerns (SoC). This is most apparent in the heat map component, which visualizes the acoustic data that’s live streamed into the front-end. 

“This component was built from the ground up utilizing WebGL, as we were unable to locate an existing solution that fulfilled our requirements and could be extended in the directions we wanted to go,” Frits says.

For the MVP, getting basic frequency-band data navigation and visualization was the top priority.

 These types of data came with their own sets of constraint, allowing them to make certain assumptions that simplified implementation and meant they could achieve the initial milestones without taking on too much at a time. While the algorithms were basic and lots of potential for improvement, the foundation was set.  

"After passing the MVP stage, our former assumptions were relaxed, and optimizations and improvements were added. We also broke down modules and their responsibilities were made more explicit and limited, which makes the code easier to understand as you don't need to consider the whole picture," Frits explains.

Nothing is set in stone

 Several of the modules for the heatmap became so generic that they were reused for other time-range-oriented features: observations (the data tagging tool), timeseries data (e.g. production data), the raw audio spectrogram display. This sped up implementation of those parts.

 When making the transition to temperature data, which featured substantially relaxed timing constraints compared to acoustic data, they wrote a new data representation module. Then, they only needed to make minor updates to the visualization module to accommodate this.  

“The loose coupling of the code brings it close to a plug-and-play architecture, where you just connect the appropriate parts you need for the type of data that is supposed to be displayed,” Frits says.