Data Engineering Practices

Big Data Kickstart

Big Data is everywhere, here is how to manage it…

Managing big data is critical for many organizations. Analytics can improve products and inform critical business decisions. Using data can provide distinct advantages, and it’s likely that an organization’s competitors are already leveraging their data. But if you have not started down the path yet, it can be a challenge to get your big data initiative off the ground.

Often the world of big data sounds like a magical place where smart data geeks just take data and come back with the answers to all of life’s questions. As you may suspect, it doesn’t actually work this way. There is hard technical work and even harder shifts in how organizational leaders think about the role of data. But the hard work is worth it if it makes the company better and customers happier, which we know is possible based on all the success stories that have been shared publicly. If you are wondering how to approach managing and leveraging data to improve your organization, the steps outlined below provide a way to approach data and analytics with a focus on building a foundation. This foundation will help an organization deal with big data (think many terabytes or petabytes). These steps are meant to drive a long term data focus so you may approach some of the steps in parallel and you will definitely iterate on the steps in the long run.

1. Collect

The first step in getting value from data is to collect it, big or small. Rather than spending days, weeks, or months discussing which data to collect I urge you to start with the mindset that all data related to your organization should be collected. I prefer collecting the data into one data ecosystem to minimize the challenge of finding the data when it’s time to do analysis, but even if you choose a decentralized approach to collection and storage it is important to set standards across departments. You should provide guidance across your organization on how data should be captured and history stored for all processes. You should aim to capture both local data as well as third party services you might use. You would expect to capture all the business-side data from CRMs (such as Salesforce), your company website, accounting systems, HR systems, online marketing tools, and so on. There is also a wealth of data to collect on the product or service you provide (web application). This could include data captured by the system, error and application logs, web analytics (such as Adobe or Google analytics), and customer feedback through surveys. Analysts in your organization will inevitably choose to tie together data from each source system at some point so storing it in one big data ecosystem (such as Hadoop) can decrease the effort needed to join the data together.

2. Make Available

The next focus is making sure those who want to get value from the data can access it. The data should be useful with tools they are able to use. For some teams this will be a spreadsheet and for others it will be more sophisticated reporting software. You will want to focus more on self-service when there is a lot of data to choose from. For big data environments you will want to eliminate roadblocks. A common roadblock is relying on a “one-off” data request process that relies on an overloaded engineering team to create or modify views specific to each analytics request. Ideally all of the data collected will automatically be available for analysts to pull into the tool of their choice without having to request it. Realistically, however, you will likely want to pick some high value data sets to start. This will allow you to think about how to make all the data available in the future without getting caught up with fully automating steps that might need to change once you have a little more experience.

Once you make your first data set available you should market it to the internal teams and offer support for them to get started. This part of the process is a great place to gather feedback on the value, utility and ease of use for others. With this feedback and the experience gained by collecting and making it available in your big data environment, you should be able to decide how to improve the process to be scalable for many more data sets.

3. Train

Once your first data set is available, you should begin putting significant energy into training. You will need to educate analysts on retrieving data and your organization’s decision makers on how they can get the information they need from the data. You will need to cater to varying levels of technical proficiency ranging from those with formal technology training to those in business units who became analysts along the way, so several levels of training will be most effective. One of the goals of training is that department leaders should know what capabilities exist, in particular who they can talk to regarding their department’s data goals as well as any requests for analytics. You will need to provide the support for those individuals handling departmental analytics to be successful, especially in the early stages of your big data journey. A lot of the training time will be focused on making sure the data analysts/scientists can access the data and transform it in a way that fits their project, but you must not neglect making sure the department leaders know how involved they should be in driving the projects and understanding what was learned with each one.

4. Trust

To unlock the potential of big data, there is a vital need to trust others in your organization with the hard work of making sense of it. You can’t rely entirely on a central data team for all of your reporting and analytics. The reason you need to trust others in the organization is simple: there is too much data with too much potential to improve your organization to keep it locked up where only a small group of privileged data experts can access it. An exception to this is that you will still keep financial and personally identifiable information (PII) locked up tight. One example of trust is when an analyst has an idea to take product usage data and feed it into a model to determine what type of emails should be sent to improve conversions (more future purchases, longer retention, etc). Senior data scientists may have concerns about the analyst attempting this project on their own while no data experts available to partner on the project. When you have trust as a component of your data strategy, you should have an open communication channel for the work done by the analyst to happen independent from a central data team and be reviewed if needed. You have a mindset that the analyst can be trusted to be honest with their department leaders about their limitations and the department leaders will be careful to weigh the risks before rolling out customer impacting changes based on any analysis that hasn’t been vetted by the central data team. In practice, department leaders are unlikely to change strategy based on analysis that contradicts their experience and instinct, so the use of data is a step forward and carries a low risk of steering the organization in the wrong direction without proper due diligence.

5. Improve

There is a strong need in big data initiatives to focus on improvement as a consistent part of the work. Despite what you may have heard, big data systems are not cheap and the amount of data that can be collected is growing every year. Effective data initiatives are less about setting up a perfect process to organize people and more about building systems to automate the steps to collect data and make it available. For most organizations, taking an iterative approach to these steps is the best chance for success since results will be evident in a reasonable timeframe. You should focus on a single use case initially with a plan to refactor the process as you expand to more use cases. It is critical that you make time for the effort to refactor after the first project is deployed to production. This will allow your engineers to spend less time trying to develop perfect automated solutions for the ‘Collect’ and ‘Make Available’ steps and provide a chance for feedback and adjustments in the ‘Train’ and ‘Trust’ steps. Data engineers will find solutions to the collect and make available steps, but these alone will not provide the impact you desire without strong organizational effort to train and trust. All of the prior steps are opportunities to learn and most organizations will need to make adjustments to be successful regardless of how much time they spend on the planning and design.


ETL tool vs custom code

I used to help sell an ETL tool that had a graphical drag and drop interface. I really did like the tool because with a little training you could quickly build a basic ETL job. I still like these types of tools if pulling data from a database that has a static or slow changing data model. However, at my current company we do not use an ETL tool because I suggested we are better off without one. While it is possible we will use an ETL tool one day for certain tasks, we currently prefer Python and SQL to move and process our data. The primary reasons we went down this path is for increased flexibility, portability, and maintainability.

One of my top regrets leading a Data Warehousing team that used an ETL tool is that we felt limited by what the tool was capable of doing. Elements of ETL that were not as important when the team started were not easily supported by the tool. The best example of this is reading from a RESTful API. Another was working with JSON data as a source. With these examples we could easily find a tool that can do this for us, but what else will we encounter in the future? At my current company we are consuming RabbitMQ messages and using Kafka for data streaming, and we would not have known to plan for a tool that works well for these use cases. Since we are using Python (and Spark and Scala) we have no limits on what is possible for us to build. There are a lot of libraries that are already built which we can leverage, and we can modify our libraries as new ideas come up rather than being stuck with what a tool provides out of the box. We choose to focus on building a data flow engine in many cases over having one script per table or source. This amount of control over the code that moves data allows us to build up an engine that supports many configurations while keeping the base code backward compatible for data sets already flowing through the system. In many cases we trade off having a longer ramp up period to get our first build working in order to have more flexibility and control down the road, but it cuts down the amount of frustrating rework when systems change.

Another benefit of coding your own ETL is that you can change databases, servers, and data formats without applying changes all over your code base. We have already taken our library for reading SQL Server data and written a similar version that works for Postgres. With how our ETL jobs are set up we just switched out the library import on the relevant scripts and didn’t have to dig in to the logic that was running. I think this leads to better maintainability as well since if you find something is taking up a lot of your time you just build it into the overall system. I remember having alerts at 2 am because the metadata of a table had changed and our ETL tool couldn’t load the data without us refreshing the metadata in the job. With most of our python code we can handle new columns added to the source data and either add that column to the destination table or just ignore it until we decide to modify the destination. This really has decreased time spent getting mad at the system administrators who disrupted our morning by adding a new custom field, though there is still plenty of work to do to ease the pain of changed datatypes and renamed columns.

I am sure there are plenty of different tools out there that do every thing you could want to do (at least according to their sales team), but I love the flexibility, control, and maintainability of writing our own applications to move data. It was worked out well for us as we have transitioned to building out a data platform rather than focusing on just tools to load an analytic data warehouse (but that is a topic for another time).