Overview

What is an Automated Data Pipeline? 

An automated data pipeline (ADP) is an end-to-end solution for automating the ingestion, transformation, storage, and presentation of data on a scalable platform. This BoosterPack demonstrates how a series of open-source tools can be integrated to create an ADP. In the Sample Solution, two types of data are used to showcase this capability:  

1. stock market data, and  
2. news data.  

The main tools used in this BoosterPack are Apache Airflow, Apache Kafka, and MySQL. The entire solution is deployed on a one-node Kubernetes cluster, which is created by leveraging the same technique published in the “Automate Cloud Orchestration with Kubernetes” DAIR BoosterPack. 

What value does it add to my business? 

This solution: 

• automates the processes of ingesting, processing, storing, and presenting data. 
• has zero licensing cost as all the tools used in this BoosterPack are open source. 
• is cloud agnostic and can be deployed in various cloud platforms. 

• implements a microservices-based architecture where the entire application is a collection of loosely coupled services. The individual services are independently deployable, highly maintainable, and testable. 
• can be adapted for various business cases. 

The advantage of this BoosterPack is that it enables you to select and integrate a range of open-source tools to provide a reliable, scalable, extendable end-to-end solution for real-time (or near real-time) data management 

Why choose an Automated Data Pipeline over the alternatives? 

Traditionally, organizations must hire architects (data or enterprise) to design the architecture and then have a development team develop the required services. Today, an equivalent ADP framework would take several months to build and cost thousands of dollars to start development from the ground up. 

This BoosterPack Sample Solution follows a generalized approach which suits most common data management projects. The Sample Solution helps organizations get started quickly by using this solution as a foundation and then customizing it according to their specific business use case. 

The tools used in this solution can be downloaded for free and most have been used in previous BoosterPack Sample Solutions. This BoosterPack provides the ability to select and integrate all these tools in a way that produces a reliable, scalable, and extendable end-to-end solution for real-time (or near real-time) data management.   

Best Practices

HTTPS Encryption

It is strongly recommended you use a valid HTTPS certificate to ensure communications between client and server are encrypted. Refer to the Security section in the Sample Solution on the use of TLS certificates. Let’s Encrypt can be used if the organization does not have an existing SSL/TLS certificate and/or domain registered in DNS, but this is not the preferred production solution.

Backup Strategy

Argus was designed for deployment on K3s, a lightweight Kubernetes implementation installed on a single VM. K3s simplify the process so that you don’t need to thoroughly understand and implement underlying components required to operate Kubernetes. This allows the solution to be easily backed up by using snapshots of the VM and recovery through VM restores. Not everyone will adopt this backup strategy, but it does allow teams to quickly test out Argus and recover to a known copy quickly without resorting to expensive backup solutions. Organizations who want to use Argus in a full Kubernetes environment can access the YAML files from the source code repository and customize the source to meet their needs (for example, if they wish to use Kubernetes packages like FluxCD for more DevOps oriented workflows).

Admin Passwords

Strong default admin passwords have been supplied by default for this Sample Solution. Do not keep passwords in the source code repository; instead, use the ‘secrets’ capability supplied by the CI/CD pipeline tools or cloud providers to securely hide them.

Tips & Traps

The code in this Sample Solution can be customized according to your specific business use case:  

• The ingestion service can be customized to ingest data from any other live data feed such as IoT devices, data from a WebSocket, etc. 
• Processing service can be customized to include code for data cleaning, transformations to a different schema, joining with reference datasets, imputation, augment with calculated columns, etc. 
• Analytical services can be used to build simple to complex machine learning or deep learning models. The models can be re-trained on the data ingested via cold path and later deployed on the cloud. The processing service can then invoke the most recent saved model in real-time to predict data that arrive in a hot path, thus getting real-time prediction results. 

Resources

Tutorials 

The table below provides some useful tutorials to help you get “hands on” experience with the tools and technologies used in this package.

Documentation

The following list provides links to some useful references for learning more about the tools and technologies used in this Sample Solution.

Support

Intelius Analytics has published the source code for this BoosterPack as an open source project on Github. If you find a bug, issue, or have improvement ideas, you can open an in issue in this Git repository. 

In addition, you can post your questions in the DAIR Slack channel or via email to [email protected]. 

Automated Data Pipeline (ADP) Sample Solution

For organizations that need a low-cost framework to automate their data management processes, the ADP Sample Solution is a great option. It demonstrates how open-source tools can be integrated, creating a framework based on micro-services architecture on a scalable platform that can be customized based on your organization’s business needs.  

This can be accomplished without the need to embark on the time-consuming and costly journey of developing a highly customized framework from scratch, employing skilled data architects and engineers, or relying on costly SaaS solutions from vendors and public cloud providers.  

Click here for instructions on how to implement the ADP Sample Solution.