Artificial Intelligence Engineer

Job Description

Position: Senior AI Engineer Experience: 5+ Years Location: Hyderabad, Telangana, India Employment Type: Full-Time Joining: Immediate hiring Openings: 12 Position Overview We are hiring a Senior AI Engineer ? Data Lake Integration with 5+ years of experience to lead end-to-end AI/ML development on Azure. The role focuses on building a Data Lake Factory from scratch, developing large-scale feature engineering pipelines in Azure Databricks, and integrating deeply with ADLS Gen2 Silver/Gold zones. The engineer will design Feature Stores, optimize distributed data processing, and implement enterprise-grade MLOps using Azure Machine Learning and CI/CD. This position requires strong expertise in Azure, PySpark, Databricks, AKS, and secure, production-ready AI deployment for real-time inference and model monitoring. Key Responsibilities ? Build a scalable, fully governed Data Lake Factory building from scratch ?Data Lake Integration & Feature Engineering (Core Focus) ?Data Lake Access & Consumption: Expertly navigate and access high-volume, multi-structured data from Azure Data Lake Storage Gen2 (ADLS Gen2), particularly within the curated Silver and Gold data zones. ?Feature Store Development: Lead the design, implementation, and maintenance of a Feature Store (e.g., using Databricks Feature Store or a custom solution) to centralize, manage, and version features derived from the Data Lake for both model training and real-time serving. ?Large-Scale Data Preparation: Utilize Azure Databricks and PySpark to perform complex, distributed data transformations and feature engineering, ensuring datasets are optimized for model training performance and scalability. ?Data Quality for AI: Collaborate with Data Engineers to establish data quality checks and validation pipelines specifically tailored to the inputs required for ML models, ensuring consistency between training and inference data to prevent drift. MLOps and Production Deployment MLOps Pipeline Leadership: Design and govern the MLOps workflow using Azure Machine Learning and Azure DevOps/GitHub Actions, covering automated retraining, model testing, versioning, and secure deployment. Model Serving: Deploy high-throughput, low-latency models for real-time inference using highly scalable Azure services like Azure Kubernetes Service (AKS) or Azure Functions. Monitoring and Optimization: Implement robust model monitoring (e.g., tracking data drift, concept drift, and prediction latency) and integrate automated alerts and remediation logic within the Data Lake/MLOps ecosystem. Security: Ensure compliance with data governance and security policies (e.g., RBAC, encryption) when accessing data from ADLS Gen2 for model building and deployment. Required Technical Skills Experience (5+ Years): Demonstrated experience taking ML models from R&D to production-grade, scalable systems. Cloud Data Platform: Deep practical experience with Azure Data Lake Storage Gen2 structure, security (ACLs), and data access patterns. Big Data Processing: Expertise in Python and PySpark for high-performance feature engineering on large datasets, ideally within Azure Databricks. MLOps Tools: Expert-level knowledge of the Azure Machine Learning service for MLOps orchestration. Deployment: Strong experience with Docker and scalable model serving platforms like Azure Kubernetes Service (AKS) or comparable cloud services. Software Engineering: Excellent command of software engineering principles, CI/CD, version control (Git), and code quality standards. Feature Store: Experience designing, implementing, or working with a Feature Store concept is highly desirable.