DCI Job Requirement for:
Data Pipeline Reliability Engineer
Location: NYC, NY
Job Description:
Serve as a Data Pipeline Reliability Engineer (DPRE) on a cross-functional team
Be the first to respond to data health failures on key pipelines.
Read code to identify fixes and make changes to improve the performance and stability of data pipelines
Help to ensure our customers’ missions are supported with updated and accurate data
Build, optimize, and maintain data pipelines to improve their efficiency and resilience.
Serve as a first responder
- Triaging, troubleshooting, and coordinating the resolution of technical issues
- Diagnose, resolve, and prevent issues encountered in the field
- Implement/maintain automated monitoring to detect data quality issues
Read and write code changes and modify the monitoring set-up where necessary
Know and understanding how to navigate the pipelines and documentation
Follow SOPs to contact other teams and data providers when
Data is incorrect
Not received on time
Communicate outages with the end users of a pipeline
Contribute to and monitoring tooling improvements (where feasible)
Qualifications:
Strong engineering background
Proficiency with programming languages such as Python, Java, and/or SQL
Basic parallel data processing experience
Basics understanding of optimizing Spark jobs
Experience performing root cause analysis and documentation of findings
Understanding/experience with data concepts such as:
- Data warehousing
- Data Lakes
- Data governance
- Data Liniage
Understanding of networking concepts (DNS, VPNs, Load Balancing)
Experience with the following tools highly preferred:
- Observability tools (Ex. Grafana)
- Data Pipeline tools (Ex. Airflow)
- Cloud tools (Ex: AWS, Azure, Google Cloud)
- IaC tools (Ex. Terraform)
Required: Active Secret Security Clearance
Job Type: Full-time
$160,000-$200,000
Job Type: Full-time
Pay: $160,000.00 - $200,000.00 per year
Benefits:
- 401(k)
- 401(k) matching
- Dental insurance
- Flexible schedule
- Flexible spending account
- Health insurance
- Health savings account
- Life insurance
- Paid time off
- Parental leave
- Professional development assistance
- Relocation assistance
- Retirement plan
- Tuition reimbursement
- Vision insurance
Experience:
- data engineering: 5 years (Preferred)
- python coding/programming : 5 years (Preferred)
- Foundry: 2 years (Preferred)
- building and managing data pipelines : 2 years (Preferred)
Security clearance:
Work Location: In person
