Kafka developer

Job Description

About Company : Our client is a global technology cons,ulting and digital solutions company, resulting from the merger . It focuses on helping businesses leverage digital technologies to reimagine business models, accelerate innovation, and drive growth. The company provides services across various industries, including digital transformation, consulting, and application development. Role: Kafka developer Location:Bangalore I nterview Mode : virtual Work mode : Hybrid Job type : C2H Immediate Joiner's :15 Day's Experience:6-12 JD: Event Streaming Team Roles Introduction This document lists all the roles and responsibilities found within the Event Streaming team. The goal of defining this is to introduce clear structure and expectations regarding competence in the case of future scale up(s) of the team. Roles Architect The role of an architect in the event streaming area means you have significant experience in designing and building entire systems centered around scalable publish/subscribe messaging. An architect must be able to steer application teams in the direction of success and provide sound technical advice, based on verifiable facts and best practices, backed up by case studies, literature, and personal experience. It's your responsibility to go beyond the boundaries of your domain to make sure your domain fits into the overall architecture. This also means that an architect must be able to understand when event-driven architectures or publish/subscribe messaging are not required and when alternatives (REST APIs or message queues) would work better. As an architect, you also need to be aware of the possibility that the application team that you're supporting might not be aware of the importance of certain dimensions that influence system design, such as: Data consistency and the effect message ordering might have on it Scalability - just because a system works well in a PoC means little if there's a clear bottleneck that only becomes noticeable under actual load Reliability - designing reliable systems is difficult and the redundancy required for stability/reliability likely costs money; it's up to the architect to be aware of this and to be able to articulate the importance of reliability, even if it means higher costs Maintainability - maintainability has many different subdimensions that need to be considered, such as: Team competence - the optimal solution might require significant technical expertise that an application team might not have; proposing a simpler, less optimal solution in cases where optimization isn't important might be better long term Technical debt - how likely is it that the system will need to be drastically changed in the near future, as a result of design decisions made now? Cost - cost needs to be controlled and considered at all times; if a proposed solution is likely to have a significant cost, this needs to be explicitly stated in the design phase, ideally together with a rough cost estimate In practice, these dimensions are expressed through known architectural and design patterns and concepts, which an architect is required to understand: Publish/Subscribe Deadletter queues/topics Event-driven architecture Event Sourcing CQRS CDC Transactional Outbox Complex Event Processing Stream processing Schema Evolution/Schema Management Domain-Driven Design CAP theorem Loose-coupling Delivery guarantees High-availability Network topologies Push vs pull GitOps Data governance In addition to a deep understanding of the listed concepts, an architect also needs to have the following skills: Excellent presentation skills Ability to articulate design choices to others Ability to derive technical architectures from non-technical or unclear/undefined requirements Deep understanding of Apache Kafka and the surrounding ecosystem (schema registries, Kafka Connect, Kafka Streams, Kafka client libraries, Spark Structured Streaming) Deep understanding/knowledge of public clouds, especially focusing on Azure, with focus on services such as Azure Functions, Azure Logic Apps, Azure App Service, Azure Kubernetes Service, OpenShift (Kubernetes in general), Azure Databricks, Azure Stream Analytics, Azure Event Hubs, Azure Service Bus, Azure Event Grid, Azure Data Lake Gen2/Azure Blob Storage Good understanding of private networking and network topologies used in enterprises Good understanding of software design patterns and knowledge of popular programming languages (Java, C#, Javascript, Python) with in-depth knowledge of at least one programming language Deep understanding of Infrastructure-as-Code concepts, knowledge of popular scripting tools and languages (Terraform, Helm, Bash), and CI/CD frameworks (GitHub Actions, Azure Pipelines, etc.) Participate and contribute to the development of the technical roadmap of the team Tech Lead The role of a tech lead in the event streaming area means you have significant experience in implementing entire systems centered around scalable, publish/subscribe messaging systems. It's your responsibility to be able to map architectural diagrams to the actual implementation of a working system, taking into account all best practices, while also being aware of specific technical limitations of various implementation details (e.g., cloud services, client library restrictions and issues) and resolving them together with architects. A tech lead needs to have the following skills: Good presentation skills Deep understanding of Apache Kafka and the surrounding ecosystem (schema registries, Kafka Connect, Kafka Streams, Kafka client libraries, Spark Structured Streaming) Independently resolving issues when deploying and setting up any sort of infrastructure (like cloud services) or applications (e.g., a Spring Boot application on App Service, or a Python Azure Function) Deep understanding of Kubernetes and Docker Deep understanding/knowledge of public clouds, especially focusing on Azure, with focus on services such as Azure Functions, Azure Logic Apps, Azure App Service, Azure Kubernetes Service, OpenShift (Kubernetes in general), Azure Databricks, Azure Stream Analytics, Azure Event Hubs, Azure Service Bus, Azure Event Grid, Azure Data Lake Gen2/Azure Blob Storage Deep understanding of private networking and network topologies used in enterprises (e.g., private endpoints, VNets/VPCs, Private Link, DNS, firewalls, Hub Spoke topologies, etc.) Deep understanding of software design patterns and knowledge of popular programming languages (Java, C#, JavaScript, Python) with in-depth knowledge of at least one programming language Deep understanding of Infrastructure-as-Code concepts, and in-depth knowledge of at least one popular scripting tool and language (Terraform, Helm, Bash), and CI/CD framework (GitHub Actions, Azure Pipelines, etc.) Experience with debugging software applications and setting up logging, observing, and monitoring Good understanding of developer portals such as Backstage, API Catalogs with OpenAPI and AsyncAPI specifications, templating (Jinja), API gateways (Azure API Management, IBM API Connect), WebSockets Experience in building event-driven microservices and REST APIs