AI Infrastructure Engineer

Experience: 3–5 Years

About the Role

We are looking for an experienced AI Infrastructure Engineer to build and scale the backend infrastructure powering next-generation AI applications. You will be responsible for designing highly available, low-latency systems that orchestrate multiple AI models, manage real-time conversations, and ensure resilient AI inference at production scale.

This role is ideal for engineers who enjoy solving distributed systems challenges, optimizing AI infrastructure, and building production-grade platforms that serve thousands of concurrent users.

Key Responsibilities

AI Infrastructure & Orchestration

• Design and implement asynchronous, event-driven AI orchestration systems.
• Build and maintain multi-agent AI workflows for complex conversational experiences.
• Own end-to-end latency optimization from user request to AI response.
• Develop resilient AI inference pipelines with retries, circuit breakers, and graceful fallback strategies.
• Implement intelligent request routing and load balancing across multiple AI models and providers.
• Migrate AI conversation services from monolithic architecture to scalable microservices.
• Build WebSocket/SSE-based streaming infrastructure for real-time AI responses.
• Optimize prompt execution, context management, batching, and response caching.
• Improve credit/data retrieval pipelines feeding AI conversations through efficient caching strategies.
• Design observability dashboards for latency, throughput, fallback triggers, and system health.

Required Skills & Experience

Backend & Distributed Systems

• 3–5 years of experience building production backend systems serving 10,000+ concurrent users.
• Strong expertise in asynchronous and event-driven architectures.
• Hands-on experience with:
• Kafka, RabbitMQ, or other message queues
• WebSockets or Server-Sent Events (SSE)
• Event streaming architectures
• Experience designing scalable microservices.

AI Infrastructure

• Production experience integrating multiple LLM providers such as:
• OpenAI
• Anthropic
• Gemini
• Azure OpenAI
• Self-hosted models using vLLM, Triton, or TensorFlow Serving
• Experience implementing:
• AI request routing
• Retry mechanisms
• Provider fallback strategies
• Rate-limit handling
• Understanding of:
• Prompt optimization
• Context window management
• Conversation state management
• Multi-turn AI interactions
• Experience troubleshooting production AI inference issues under heavy load.

Performance & Scalability

• Strong knowledge of caching strategies:
• Redis
• In-memory caching
• CDN-based caching
• Experience optimizing latency and throughput for AI applications.
• Familiarity with concurrency, load balancing, and fault-tolerant architectures.

Good to Have

• Experience with Agentic AI frameworks such as:
• LangGraph
• LangChain
• CrewAI
• Custom orchestration frameworks
• Experience building streaming chat applications similar to ChatGPT.
• Previous experience in fintech or payments.
• Experience decomposing monolithic systems into microservices.
• Familiarity with observability and monitoring tools:
• Grafana
• Prometheus
• Datadog
• Startup experience with ownership of end-to-end systems.

Preferred Technology Stack

• Languages: Python, Go, Java, or Node.js
• Caching: Redis
• Messaging: Kafka, RabbitMQ
• Streaming: WebSockets, Server-Sent Events (SSE)
• AI Platforms: OpenAI, Anthropic, Gemini, Azure OpenAI, vLLM, Triton, TensorFlow Serving
• Observability: Grafana, Prometheus, Datadog
• Architecture: Microservices, Event-Driven Systems

What Success Looks Like

• Deliver highly available AI infrastructure with minimal latency.
• Build resilient AI systems that gracefully handle provider failures.
• Improve response times through intelligent routing, caching, and batching.
• Scale AI conversations reliably for thousands of concurrent users.
• Drive engineering excellence through observability, automation, and performance optimization.

Why Join Us?

• Work on cutting-edge AI infrastructure powering real-world applications.
• Solve complex distributed systems and large-scale AI orchestration challenges.
• Own critical technical decisions impacting customer experience and business growth.
• Opportunity to shape and scale the AI platform as the company grows.
• Clear path toward technical leadership within the AI infrastructure team.