New Zealand Does Not Exist
Let’s start our adventure into the world of data management by jumping straight down the rabbit hole. Or more precisely, a borehole in New Zealand. Fortuitously, I happen to have some data about boreholes in New Zealand sitting on a disk drive on my desk.
Leave aside for a moment the fact that I have this data pack sitting in my desk drawer, and you probably don’t. Try and relate anyway.
As I write, the drive is not plugged in, I can’t see what’s on it and I don’t know if it works. However, even without connecting to a computer I can see that the disk label itself contains some valuable data about the data.
The label tells me that this disk contains information from New Zealand, about that country’s petroleum and mineral resources, that it’s from 2017, it’s issued by the government, and that it’s probably called the ‘new zealand petroleum exploration data pack’. (If you do want your own copy you can order it here).
I should really keep a record of these “data about the data pack” somewhere and make sure that it travels with every copy of the contents of this disk so that future generations of data consumers that don’t have access to the original media can understand where it came from.
Also, as this won’t be the only data pack that I have to work with, or the only media in my desk drawer, I should probably start some sort of catalog to keep track of all my incoming datasets like this. However, I’ll do that later, as I am short of time right now and I have to focus on the bigger problems. (for the benefit of the non-nerds in the audience, know that the last sentence is dripping with intentional sarcasm).
I plug the disk into the computer, and it auto-mounts to my mac. I see a bunch of files at the root and (being the trusting soul that I am) click on the file called ‘start.html’. Chrome opens and we are treated to the following start page.
Without leaving the comfort of my suburban West-London home, I am transported to the beautiful shores of middle-earthNew Zealand. The rather cryptic images and language indicate that the data we are looking at is slightly more interesting than your standard list of customers, products, sales, and invoices.
In fact, this data package is a vast resource of scientific information gathered since 1866 by companies prospecting for oil and gas in New Zealand. It is maintained by the New Zealand government as part of what is known as a National Data Repository.
Regardless of your opinion on the role of oil and gas in the energy transition, this is a hugely important collection of data about our planet. The data in this pack has been acquired at huge expense. Drilling wells and shooting seismic surveys costs oil companies billlions of dollars every year and is essential to understanding the location, volume and quality of natural resources in an area of interest.
In a shift in policy in the 1990s, forward looking countries in Europe (notably Norway) realised that part of an international oil company’s licence to operate in their country must include an obligation to report this critical scientific data to the government. Oil companies come and go, but the data, which is our main connection to deep-subsurface assets, remains with the nation. It is for this reason that we are now able to freely access billions of dollars worth of scientific data from the comfort of our desktops.
For the connoisseur of data this collection is an opportunity to take a romp through a complex and richly interesting set of data that will test all of our data management skills.
For data practitioner in the energy industry it is simply one of hundreds of sources of truth about the earth’s subsurface. For now, let’s enjoy this single source, but later we will consider what it means to continuously integrate data from multiple sources, each containing at least this level of complexity, often with much lower levels of consistency and quality and much higher volumes.
This is a baby dataset, but what a beautiful baby it is...