Role Summary:
We are seeking a Senior Databricks Data Engineer (QA Testing) with strong hands on experience building and optimizing data pipelines on Databricks. This role focuses on implementation, performance, and reliability while working within a well defined architectural framework.
Key Responsibilities:
- Develop and maintain data pipelines on Databricks using medallion architecture principles
- Build, optimize, and maintain Databricks notebooks for ingestion, transformation, and analytics
- Implement data flows across boundary systems ensuring reliability, scalability, and data quality
- Apply performance tuning techniques including query optimization, cluster sizing, and Delta Lake optimization
- Work with Unity Catalog to manage schemas, permissions, and data access controls
- Collaborate closely with architects to implement approved data designs and standards
- Troubleshoot pipeline failures, data issues, and performance bottlenecks
- Ensure code quality, documentation, and operational readiness of data solutions
Required Qualifications:
- Strong hands on experience with Databricks and Delta Lake
- Strong hands-on experience with AWS services such as S3, EC2, Lambda, Glue, and Redshift.
- Working experience with IBM DataStage for ETL development, data integration, and data transformation.
- Proven ability to design metadata-driven test harnesses that scale — parameterized test configs that can run against 500 jobs today and 80K tomorrow without rearchitecting
- Experience with data validation tooling (Great Expectations, Deequ, or custom PySpark validation libraries) for row count, schema, transformation logic, and output comparison checks
- Performance and throughput benchmarking — rows/sec, latency comparisons between legacy (DataStage) and target (Databricks) pipelines
- Experience with Iceberg Tables
- Proven experience implementing medallion architecture in real world use cases
- Experience building notebooks using Python, SQL, or Scala
- Solid understanding of data integration across multiple systems
- Strong performance tuning and debugging skills in Databricks environments
- Experience working in enterprise scale data platforms
Scale & Automation Mindset
- Experience designing test coverage strategies for large-scale migrations (thousands of jobs), including prioritization by complexity tier (Low/Medium/High) and risk scoring
- Ability to build reusable test templates that align with ingestion patterns — Full Load, Incremental/CDC, File Ingest, API — so each converted job inherits its test suite from the framework
- Comfort with GraphFrames or lineage-based dependency analysis to sequence integration tests based on upstream/downstream job relationships
Nice to Have:
- Databricks certification
- Experience working with governed data platforms using Unity Catalog
Pay: $60.00 - $70.00 per hour
Experience:
- Databricks: 5 years (Required)
- DataStage: 5 years (Preferred)
- Metadata: 5 years (Preferred)
- AWS: 5 years (Preferred)
- Data Validation: 5 years (Preferred)
- Deeq: 1 year (Preferred)
- Great Expectations: 1 year (Preferred)
- Pyspark: 5 years (Required)
- Iceberg: 3 years (Preferred)
Work Location: Remote
