When it comes to the medical profession, doctors get all the glory. In the world of IoT, it’s data scientists who get most of the attention and acclaim. They extract critical intelligence from big data so businesses can make informed decisions on the spot. But they don’t do their work in a vacuum. Data scientists can’t dazzle their industries without data engineers. These unheralded champions, equivalent to nurses, ensure that big data keeps flowing. As anyone who works in the medical profession will tell you, it’s the nurses who keep the hospital running.
What exactly do data engineers do? They work behind the scenes to design and maintain the networks and software that keep the big data pipeline operating. Like a hospital’s nursing staff, data engineers set the stage and keep it running. The roles of data scientists and data engineers can be confusing because they have some overlap. Data engineer and data scientist are not different titles for the same job, however. The two jobs require different skills and experience. Some data scientists can do data engineering. Some data engineers can do data analysis and data visualization.
The roles do have distinctions, however. For instance, large applications call for the skills of data engineers. Research is a primary focus of the data scientist. Like nurses, data engineers are a special breed. The best have certain personality traits that help them excel: focus, mechanical aptitude, patience and persistence. Good data engineers get down in the trenches. They want to understand how and why data pipelines work—or don’t work. Data engineers need patience and persistence to set things right.
To do modeling, data scientists need data engineers to gather, store and process data so they can analyze it for insights. Responsible for data management, data engineers handle procedures, guidelines and standards. They develop data-management technologies and software-engineering tools. They design custom software and discover ways to recover from disasters. They improve data reliability, efficiency and quality. User-defined functions and analytics are part of a data engineer’s job, too.
In contrast, data scientists take a big-picture view of things and have a less nuts-and-bolts relationship with data. They handle analytic projects that arise from the needs of the business. Data scientists also take on data-mining architectures, modeling standards, reporting and data methodologies. They manage data-mining-system performance and efficiency, too.
Because they build and maintain the data pipelines that send information to data scientists, the work of data engineers is very valuable. They can run basic learning models if they understand algorithms. But data scientists tackle business problems that take sophisticated machine-learning algorithms. The best data scientists adapt machine-learning models to meet the changing requirements of the business or agency.
Tools for Tough Big Data Challenges
The challenges of database integration and unstructured big data are handled by the data engineers. They must clean up that unstructured data before they pass it to anyone in the organization who needs it. Like nurses who prep patients for surgery, data engineers prepare the foundation for data scientists to work easily with data. They should know data warehousing, database design, data collection and transfer, and coding.
The part of the data pipeline on which data engineers are focusing on determines which tools they will use. Data engineers at the rear of the pipeline build APIs for data consumption, integrate data sets from external sources and analyze how the data is used to support business growth.
Although these professionals have many languages to choose from, Python is a good option. Data engineers use it to write code related to data ingestion. Python can talk to any data store, such as NoSQL and RDBMS. Data engineers may have to use big data technologies such as Hadoop and Spark to suggest improvements on the basis of how data is used.
Data engineers have many tools at their disposal, including the following:
- Spark
- NoSQL databases (e.g., Cassandra and MongoDB)
- Hadoop and related tools such as HBase, Hive and Pig
- Pentaho
- VMware
- JavaScript
Data Scientists and Data Engineers: Growth on the Horizon
Data engineers can expect to make a good living. In the United States, their average salary is $95,526. The low end of the pay scale is $65,000 and the high end is $121,000. U.S. demand for these jobs should grow 15 percent by 2024—that’s faster than the average for all U.S. occupations. Some of the biggest names in business, as well as the U.S. government, are ramping their requirements for both positions.
A study last year from the Economist Intelligence Unit surveyed 422 executives in the U.S. and Europe. The survey asked them about the digital skills most in demand among industries such as financial services, health care, manufacturing and retail. Forty-three percent of the executives said that in three years, analytics and big data skills will be the most important digital capabilities at their companies.
As life and business become increasingly data driven, demand for both data engineers and data scientists will continue to rise. Now is the time for data professionals to acquire or build on their skills so they will be well positioned for career advancement and job security.
About the Author
Learning@Cisco product manager Neeraj Chadha has more than 20 years of experience in the networking industry. Over that time, he has functioned as a software developer and network engineer, as well as in various aspects of product management. Currently, he guides the overall product strategy and evolution of Cisco courseware and certifications for wireless, collaboration and big data and analytics. Neeraj’s primary areas of focus include technology trends, digital transformation, continuing education and product strategy.
