Principal Software Engineer - Python with Azure

Job Description

Key Responsibilities ? Technical Leadership & Architecture ? Act as a technical architect, ensuring alignment between the work of multiple development teams in data engineering and data science.? Design scalable, high-performance data processing solutions within? Azure and Databricks , considering factors such as governance, security, and maintainability.? Drive the adoption of best practices in software development, including CI/CD, testing strategies, and cloud-native architecture.? Work closely with Product Owners and Engineering Managers to translate business needs into technical solutions.? Hands-on Development & Technical Excellence ? Lead by example through high-quality coding, code reviews, and proof-of-concept development.? Solve complex engineering problems and contribute to critical design decisions.? Ensure effective use of? Azure services ,? Databricks , and data processing frameworks such as? Apache Spark, Delta Lake, and Airflow .? Develop and optimise? data pipelines, data transformations, and ML workflows ?in a cloud environment.? Line Management & Team Development ? Provide line management to engineers, ensuring their professional growth and development.? Conduct performance reviews, set development goals, and mentor team members to enhance their skills.? Foster a collaborative and high-performing engineering culture, promoting knowledge sharing and continuous improvement.? Support hiring, onboarding, and career development initiatives within the engineering team.? Collaboration & Cross-Team Coordination ? Act as the technical glue between? data engineers, data scientists, and software developers , ensuring smooth integration of different components.? Provide mentorship and guidance to developers, helping them level up their skills and technical understanding.? Work with DevOps teams to improve deployment pipelines, observability, and infrastructure as code.? Engage with stakeholders across the business, translating technical concepts into business-relevant insights.? Governance, Security & Data Best Practices ? Champion? data governance, lineage, and security ?across the platform.? Advocate for and implement scalable? data architecture patterns , such as? Data Mesh, Lakehouse, or event-driven pipelines .? Ensure compliance with industry standards, internal policies, and regulatory requirements.? Qualifications Requirements & Experience ? 10+ years of strong software development experience required Strong software engineering background with experience in designing and building production-grade applications in? Python, Scala, Java, or similar languages .? Proven experience with? Azure-based data platforms , specifically? Databricks, Azure Data Lake, Azure Synapse, and Azure Functions .? Expertise in? Apache Spark ?and? Delta Lake , with experience building large-scale distributed data pipelines.? Experience with? workflow orchestration tools ?like? Apache Airflow .? Cloud experience in? Azure , including? containerisation (Docker, Kubernetes) ?and? infrastructure as code (Terraform, Bicep) .? Strong knowledge of modern software architecture, including? microservices, event-driven systems, and distributed computing .? Experience leading teams in an? agile ?environment, with a strong understanding of? CI/CD pipelines, automated testing, and DevOps practices .? Excellent problem-solving and communication skills, with the ability to engage with both technical and non-technical stakeholders.? Previous experience in? data engineering, data science, or analytics-heavy environments ?is highly desirable.? Proven? line management experience , including mentoring, career development, and performance management of engineering teams.