Contents
Introduction
Workflow Management is the process of automating a set of tasks that produces a specific outcome. Which indeed helps in optimizing and improving the workflow or process for better output by reducing the errors and eliminates the manual repetition. Every organization and department has its own use cases for example HR might need to automate employee onboarding workflow, so each of the workflows is different here we will see how vizB – a data analytics tool, makes use of the workflow management system.
Workflow in vizB
Workflow is nothing but a small, well-structured set of tasks that is repeated constantly to get a single output which is converted to a workflow management system and then automated.
Types of workflow
- Sequential workflow
- Parallel workflow
Sequential workflow
A workflow when each step depends or waits on the completion of the previous step is considered to be a sequential workflow.
Parallel workflow
When multiple tasks are performed simultaneously at the same time is known as a parallel workflow.
Key components of a workflow
- Inputs
- Transformation
- Outputs
- Visualizations
Inputs
In vizB inputs are nothing but getting information like orders, customers, products from various e-commerce stores hosted by eCommerce platforms like Shopify, Magento, and Bigcommerce. Through API’s we will be able to collect the input data required from these platforms.
Transformation
After getting the input data. The data is cleaned, analyzed, and processed to get the desired features from the input data and transformed into a common data format with common feature names to make it generic inside vizB. This transformation process involves data cleaning like removing duplicates, dropping empty values, etc. data manipulation by using specific algorithms and mathematical functions useful information extracted from the data and model fitting is nothing but training the machine learning models with the acquired data gives predictive analytics like sales forecasting, demand forecasting, etc. which helps in various business decisions.
Outputs
Once the data transformation is completed the data is uploaded to a database as a table. vizB uses GCP’s BigQuery database for storing the data which gives 10x better performance for reading and uploading large amounts of data when compared to other RDBMS.
Visualizations
Finally after uploading the data to the database. With customized API’s written in python backend, the data is sent to the frontend which uses react and d3 JS to load the graphs and tables in the website dashboards.
Workflow Management Tool used in vizB
vizB uses prefect – an open-source workflow management tool that helps in not only automating the workflow but also gives a real-time cloud dashboard for monitoring the workflow helps to know at what stage the flow run is, whether the flow is successful or not. Identifying errors and troubleshooting, time taken for the flow to complete, etc. prefect offers sequential, parallel, and combination of both the flow types. vizB uses a sequential workflow method for better computation performance. Prefect is available to python via pip install which is more pythonic and simple to implement.
Apache Airflow is another popular workflow management tool. How prefect differs from it and what are all its advantages can be identified through the below link.
Docs – https://docs.prefect.io/core/getting_started/why-not-airflow.html#overview
Conclusion
I believe this blog would have given in-depth knowledge on workflow management systems and how vizB utilizes it for automating its data pipeline which indeed results in better performance by reducing the tedious work routines.
