This role focuses on designing, implementing, and maintaining scalable enterprise ETL processes and robust data pipelines for a global client base. The Data Engineer will leverage big data frameworks such as Apache Spark and Hadoop, along with SQL and Python, to optimize data processing and ensure high data quality.

The role involves collaborating with cross-functional teams to automate routine tasks and deliver accurate, high-value data solutions across various industries.

Required Education

Bachelor’s degree in a quantitative discipline such as Engineering, Mathematics, Finance, Business, or a related field.
Equivalent practical experience may also be considered.

Required Qualifications / Skills / Experience

Experience as a Data Engineer or in a similar role, with a strong understanding of data engineering concepts and methodologies.
Strong knowledge of writing and optimizing SQL queries to retrieve, manipulate, and analyze data efficiently.
Hands-on experience with big data technologies, including:
Apache Spark:
PySpark
Spark SQL
Spark Streaming
Hadoop ecosystem:
HDFS / Ozone
Hive
YARN
Understanding of data modeling concepts and database design to support scalable data solutions.
Familiarity with Python.
Ability to analyze and troubleshoot data issues and provide solutions with minimal supervision.
Basic knowledge of testing and validating data to ensure accuracy and consistency in data pipelines.
Excellent verbal and written communication skills, with the ability to explain complex ideas clearly to both technical and non-technical stakeholders.

Preferred Qualifications / Skills / Experience

Experience supporting the design, implementation, and maintenance of enterprise ETL processes for global data platforms.
Ability to develop scalable and efficient code for data processing while ensuring timely availability and accessibility.
Experience leveraging big data processing frameworks such as Apache Spark and Hadoop to build and optimize data pipelines.
Ability to collaborate with senior engineers to resolve data challenges and maintain high data quality.
Experience supporting data delivery processes alongside Data Engineers and Analysts to provide accurate, high-value data solutions across different clients and industries.
Ability to build strong working relationships with team members and clients while contributing to local and global projects.

Job Duties / Responsibilities

Learn and apply industry best practices, including:
Version control
Code reviews
Data validation
Quality assurance processes
Use SQL and database technologies to optimize data processing and reduce processing time for large datasets.
Design, implement, and maintain data pipelines using:
ETL frameworks
Orchestration tools
Distributed data processing engines
Participate in automation initiatives to streamline routine data tasks and improve operational efficiency.
Ensure compliance with client internal policies and applicable external regulations.

Key Technical Skills

Programming: Python, SQL
Big Data Technologies: Apache Spark, PySpark, Spark SQL, Spark Streaming, Hadoop, HDFS/Ozone, Hive, YARN
Data Engineering: ETL, Data Pipelines, Data Modeling, Database Design, Data Validation
Engineering Practices: Version Control, Code Reviews, Automation, Troubleshooting

Employment Details

Estimated Duration: 3 months
Shift: Standard Work Hours

Pay: $61.00 - $67.00 per hour

Application Question(s):

Do you have a Bachelor’s degree in Engineering, Mathematics, Finance, Business, Computer Science, or another quantitative discipline (or equivalent practical experience)?
Do you have professional experience working as a Data Engineer or in a similar data-focused role?
How would you rate your SQL skills, particularly in writing and optimizing queries for large datasets?
Do you have hands-on experience developing or maintaining ETL/ELT processes and data pipelines?
Have you worked with Apache Spark technologies (such as PySpark, Spark SQL, or Spark Streaming)?
Are you proficient in Python for data processing, automation, or data engineering tasks?
Do you have experience working with Hadoop ecosystem technologies (such as HDFS/Ozone, Hive, or YARN)?
Do you have experience with data modeling, database design, and building scalable data solutions?
Have you performed data validation, testing, or troubleshooting to ensure data accuracy and consistency in pipelines?
Are you comfortable collaborating with technical and non-technical stakeholders to communicate data solutions and resolve issues?

Work Location: In person

Jobility

New Jobs

Data Engineer II