I started this blog to talk about the evolving world of data analytics and data management. The goal is to share opinions, tutorials, and best practices (or at times worst practices) in both data technology and managing development teams. The theme will be new technologies and new approaches that are changing how data analytics and data engineering are done within modern organizations. I have been working in the data warehousing field for a while now and the approach across the industry has changed drastically since I started – or at least it should have changed drastically. Data has grown, modern systems have been built to deal with today’s problems, and there is a new generation of employees that want access to the data and will learn how to use it.
My goal has always been to make important data available for driving insight and decision making. I learned from many experts in the industry the importance of making data clean and easy to use so that the managers and analysts for the business do not incorrectly interpret the data. This is important to consider because you do not want people to make poor decisions because of confusion in the metrics. This concern leads to carefully collecting important data and only exposing it to others in the organization once it is defined, cleansed, and documented — because then they cannot draw incorrect conclusions. The goal is spot on, and the approach I theoretically agree with. The problem: it takes too long to make the data perfect and you must predict how it will be interpreted by the people who see the data or there will still be confusion (unless they miraculously read the definitions and documentation you provide to them).
The solution: free the data from the vault you kept it locked away in and make all of it available to those that can get value out of it! I have many thoughts on how to do this responsibly, but the underlying message is make data easily accessible. If you keep the data locked away you will miss out on the value many other organizations are getting from their data.