Overview
We are seeking a highly skilled Data Engineer to join our dynamic team. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines and architectures that support advanced analytics and business intelligence initiatives. This role offers an exciting opportunity to work with cutting-edge technologies such as AWS, Azure Data Lake, Hadoop, Spark, and Big Data tools, contributing to the organization’s data-driven decision-making processes. The successful applicant will possess strong technical expertise, excellent problem-solving skills, and a passion for data analysis and architecture.
Responsibilities
- Design, develop, and optimize ETL processes using tools such as Informatica, Talend, and custom scripting with Shell Scripting and Bash (Unix shell).
- Build and maintain large-scale data pipelines utilizing Apache Hive, Spark, and cloud platforms like AWS and Azure Data Lake.
- Develop robust data models and schemas for data warehouses using SQL, Microsoft SQL Server, and Oracle databases.
- Collaborate with cross-functional teams to integrate various data sources, including Linked Data, RESTful APIs, and other external datasets.
- Implement data quality checks, validation procedures, and ensure data security standards are met.
- Support data analysis efforts by creating efficient database designs, performing complex queries, and enabling seamless access to structured and unstructured data.
- Participate in Agile development cycles, contributing to continuous improvement of data architecture solutions.
- Assist in model training, analysis, and visualization tasks using tools like Looker for insightful reporting.
Experience
- Proven experience working with cloud-based platforms such as AWS (including services like S3, Glue) or Azure Data Lake.
- Strong proficiency in programming languages including Python, Java, and scripting languages like VBA.
- Extensive knowledge of big data technologies such as Hadoop, Spark, and distributed storage systems.
- Hands-on experience with database design, management of large datasets, and data warehousing solutions.
- Familiarity with analytics tools, including experience with modeling training, analysis skills, and visualization platforms like Looker or similar BI tools.
- Experience working within an Agile environment with a focus on collaborative development processes. This position offers a challenging environment for innovative problem-solvers eager to leverage their technical expertise in a fast-paced setting focused on transforming raw data into actionable insights.
Pay: $114,700.76 - $138,134.25 per year
Work Location: Remote
