What is data engineering
Data Engineering
Ever Wondered Why this 2 letter word is a buzzword in the industry and on the Internet?
So, I’ll try to explain it in my own words by first taking the first word The mighty “Data”.
and later “Engineering”.
So, What is Data then?
Data can be a collection of numbers, facts, text, files, and almost everything in the digital world (Arguably!).
There is this nice Image which shows Types of data -
Now, It’s time for the second word What is Engineering? Engineering words remind me of Engineers. Engineers are people who do stuff like practical stuff. So, I made this definition that Engineering is an application or Practical implementation of ideas, maths, and theories.
Now, that we have covered both words let’s jump right back to “Data Engineering”.
It refers to implementing tools, techniques, architectures, and ideas that enable the smooth flow of data, its storage, usage everything. Just like civil Engineers use bridges, roads, Fly-overs, etc. for the smooth flow of traffic. Data Engineer builds system or data pipelines that do the job.
In the past companies referred data engineering to ETL pipelines that handle the data in an organization. In the modern world, data engineering has become a type of software engineering focused on data, especially infrastructure, warehouse, mining, reporting, forecasting, and AI systems, and with the scale of data generating in today’s world. Most of the approaches use cloud infrastructure.
References -
Data Engineering wikipedia (2023) Wikipedia. Available at: https://en.wikipedia.org/wiki/Data_engineering (Accessed: 23 September 2023).
Data image (2023) Wikipedia. Available at: https://en.wikipedia.org/wiki/Data#/media/File:Data_types_-_en.svg (Accessed: 23 September 2023).
Data wikipedia (2023) Wikipedia. Available at: https://en.wikipedia.org/wiki/Data (Accessed: 23 September 2023).
Engineering (2023) Wikipedia. Available at: https://en.wikipedia.org/wiki/Engineering (Accessed: 23 September 2023).