Description:
JOB DESCRIPTION
JOB TITLE: Senior Data Engineer
DEPARTMENT: 500 – IT
REPORTS TO: Chief of Information Officer
JOB SUMMARY:
The Senior Data Engineer – Databricks is responsible for building and evolving PharmaLink’s enterprise data, analytics, and AI platform. This hands-on role spans the modern Databricks ecosystem, including data engineering, lakehouse architecture, analytics and dashboards, AI, Databricks Apps, and Lakebase. The role requires strong engineering fundamentals, independent problem-solving, technical ownership, and the ability to design, build, and operate reliable, scalable production solutions.
A hands-on builder approach is essential, with a focus on developing solutions rather than simply administering platforms. The ideal candidate can work across data engineering, platform architecture, analytics, AI, and data applications; independently investigate complex problems; and take solutions from initial design through production.
KEY RESPONSIBILITIES:
-
Design, build, and operate enterprise data solutions on Databricks.
-
Develop scalable ETL/ELT pipelines using Python, PySpark, Spark SQL, and Delta Lake.
-
Design curated data models and reusable data products for analytics, applications, and AI.
-
Implement Unity Catalog, security, lineage, access controls, and data-governance capabilities.
-
Build and optimize Databricks SQL Warehouses and AI/BI Dashboards.
-
Develop data and AI applications using Databricks Apps.
-
Design and implement appropriate solutions using Databricks Lakebase.
-
Enable AI/ML and Generative AI use cases using Databricks capabilities.
-
Build automated data-quality, reconciliation, anomaly-detection, and monitoring processes.
-
Integrate data from databases, SaaS platforms, APIs, files, and other enterprise systems.
-
Optimize Spark workloads, SQL queries, pipelines, performance, and platform cost.
-
Establish engineering standards for Git, CI/CD, testing, monitoring, and production deployment.
-
Troubleshoot complex production and data issues and perform root-cause analysis.
-
Partner with Data Governance, BI, application, and business teams to deliver trusted data products.
Requirements:
QUALIFICATIONS:
-
8+ years of experience in data engineering, data platforms, or related engineering roles.
-
4+ years of hands-on Databricks experience in production environments.
-
Advanced SQL, Python, PySpark, and Apache Spark skills.
-
Strong experience with Delta Lake and Unity Catalog.
-
Strong ETL/ELT, data modeling, and enterprise data architecture experience.
-
Experience with Databricks SQL and AI/BI Dashboards.
-
Familiarity with Databricks capabilities across AI/ML, Generative AI, Databricks Apps, and Lakebase.
-
Strong relational database and data-engineering fundamentals.
-
Experience with Git, CI/CD, automated testing, monitoring, and production support.
-
Experience working with cloud-based enterprise data platforms.
-
Must maintain a positive attitude in a high-performance team environment.
-
Must pass background check and random drug screenings.
-
Must be adaptable and flexible in dealing with a diverse work force
PREFERRED QUALIFICATIONS:
-
Databricks on Microsoft Azure.
-
Azure Data Factory.
-
Microsoft SQL Server.
-
Fivetran or similar CDC/ELT technologies.
-
Power BI.
-
Databricks Workflows.
-
MLflow and AI/model lifecycle capabilities.
-
API and application development experience.
-
Azure DevOps or GitHub-based CI/CD.
-
Databricks professional certification.
-
Experience in healthcare, life sciences, or another regulated industry.
PHYSICAL REQUIREMENTS:
-
Requires light physical activity performing non-strenuous daily activities of an administrative nature (i.e., sustained phone, headset, and keyboard use) as well as periods of prolonged sitting.
