SpectraMedix - Lead Databricks Engineer - US Healthcare Analytics

Description

About Us :

SpectraMedix is at the forefront of transforming healthcare, empowering health plans and health systems with the tools and insights they need to provide better, more cost-effective care to the communities they serve. Our cutting-edge platform delivers advanced analytics that help organizations excel in the transition to new value-based payment (VBP) models.

By combining advanced technology with a team of seasoned experts, we provide a unique solution tailored to meet the needs of the healthcare industry during this critical phase of value-based care evolution. Our platform is strategically designed to guide health plans, accountable care organizations, and health systems in their journey toward achieving success in value-based payment initiatives. We are committed to creating an environment where innovation thrives, employees are empowered to drive meaningful change, and every team member plays a key role in reshaping the value and quality equation in healthcare.

Diversity & Inclusivity

At SpectraMedix, we believe that diversity drives innovation and inclusivity fosters collaboration. We are committed to creating a workplace where everyone feels valued, respected, and empowered to bring their authentic selves to work. We celebrate diverse perspectives and are dedicated to providing equal opportunities for all, regardless of race, gender, age, disability, or background. Join us in building a culture of belonging and mutual respect.

What Are We Looking For

We are seeking a Lead Data Engineer with strong expertise in Python and PySpark to design, build, and migrate scalable data pipelines on Databricks using Apache Spark. The role focuses on implementing and managing Medallion Architecture (Bronze, Silver, Gold) with Delta Lake to deliver reliable, high-performance data solutions. The successful candidate will optimize Databricks jobs and Spark workloads, enhance ETL/ELT pipelines, support batch and streaming data processing, and provide technical mentorship while leading Databricks-focused proof-of-concept initiatives

Location : Gurgaon (Hybrid)

Roles & Responsibilities

Migration and Implementation : Lead the migration of legacy Java-based data pipelines to Databricks; design and maintain scalable data pipelines using Databricks and Spark.

Data Engineering and Management : Create, maintain, and update data models; build infrastructure for optimal data extraction, transformation, and loading (ETL).

Performance Analysis and Optimization : Analyse and optimize Databricks jobs and queries; monitor and tune Databricks environments for scalability and reliability. Address data-related technical issues; assist teams with data transformation workloads; perform root cause analysis to identify improvement opportunities.

Technical Improvement : Implement best practices for efficient data processing and storage; develop processes that support data transformation, workload management, data structures, and metadata.

Technical Mentorship : Mentor a team of data engineers; serve as the go-to person for Databricks-related queries; conduct Proof of Concepts (POCs) to demonstrate Databricks capabilities

Required Experience

• Proficiency in Python / PySpark
• Proven experience with Databricks and Spark, including building and optimizing data pipelines.
• Strong data modeling experience and advanced SQL knowledge.
• Experience with Delta Lake and performance analysis in Databricks.
• Knowledge of message queuing, stream processing, and scalable big data stores.
• Experience with Talend is preferable.
  
  

Non-Technical / Behavioral Competencies Required

• Must have worked with US based clients in onsite/offshore delivery model
• Strong verbal and written communication, technical articulation, listening, and presentation skills are essential
• Should have proven analytical and problem-solving skills
• Demonstrated expertise in prioritization, time management, and stakeholder management (both internal and external) is necessary
• Should be a quick learner, self-starter, proactive, and an effective team player.
• Must have experience working under tight deadlines within a matrix organizational structure
  
  

(ref:hirist.tech)