Senior Data Engineer

Job Description

Role - Senior Data Engineer Experience - 5-7 Years Location - Remote (India) About 1DigitalStack.ai 1DigitalStack.ai combines AI and deep eCommerce data to help global brands grow faster on online marketplaces. Our platforms deliver advanced analytics, actionable intelligence, and media automation ? enabling brands to optimize visibility, efficiency, and sales performance at scale. We partner with India?s top consumer companies ? Unilever, Marico, Coca-Cola, Tata Consumer, Dabur, and Unicharm ? across 125+ marketplaces globally. Backed by leading venture investors and powered by a 220+ member team, we?re in our $5?10M growth journey, scaling rapidly across categories and geographies to redefine how brands win on digital shelves. About Role This is a high-impact, hands-on engineering role owning the core data systems that power our analytics, AI, and automation stack. You?ll work closely with the CTO and Engineering Leads and independently manage large, high-throughput data pipelines that process millions of events. Responsibilities : ? Build and maintain high-throughput, real-time data pipelines using Kafka/Pulsar with Spark, Flink, and distributed compute engines. ? Design fault-tolerant systems with zero-data-loss principles ? checkpointing, replay logic, DLQs, deduplication, and back-pressure handling. ? Implement data observability ? quality checks, SLA alerts, anomaly detection, lineage, and metadata insights. ? Design and manage Iceberg-based lakehouse tables (Polaris/Gravitino catalogs, schema evolution, compaction). ? Build fast OLAP layers using ClickHouse / StarRocks. ? Model data across bronze ? silver ? gold layers for downstream teams. ? Migrate and modernize legacy pipelines into scalable, distributed workflows. ? Orchestrate ETL workloads using Airflow, DBT, Dagster, SQLMesh. ? Optimize SQL transformations and distributed execution across Trino/Spark. ? Ensure strict security and governance across all data layers ? access control, encryption, auditability. ? Collaborate with backend, analytics, and platform teams for seamless data delivery. Requirements Core Technical Skills ? Extremely strong SQL ? window functions, query planning, optimization. ? High comfort working with distributed & parallel workloads. ? Hands-on experience with some-many of these technologies : Apache Spark, Apache Flink, Trino, Apache Kafka, Apache Pulsar, Apache Beam ? Advanced experience in Python (preferred) or Java (strong fundamentals). ? Strong understanding of Parquet, Apache Iceberg, and Iceberg REST catalogs (Polaris / Gravitino). ? Experience with OLAP databases ? ClickHouse / StarRocks. ? Experience with semantic layers ? Cube.js or similar. ? Strong experience building pipelines with Airflow, DBT, Dagster, SQLMesh. Foundational Strengths ? Solid understanding of data structures & algorithms ? sorting, searching, memory models. ? Strong grasp of OLTP vs OLAP, indexing, query execution, and storage formats. ? Ability to debug distributed systems end-to-end (compute, storage, network, orchestration). ? Familiarity with cloud environments, containerization (Docker), and monitoring. ? Experience with large-scale data ? high throughput, billions of rows, large parallel workloads. ? Awareness of cost optimization in compute & storage. Good to Have ? Experience with emerging stream processors ? Dagster, RisingWave, Arroyo. ? Kubernetes, Terraform, or cloud-native big-data stacks. Mindset ? Strong ownership ? takes systems from design ? build ? monitor. ? Self-driven, independent, and comfortable making technical decisions. ? High attention to reliability, data accuracy, and operational excellence. ? Naturally grows into broader technical responsibility as the platform scales. Why 1DS is a great choice ? High-trust, no-politics culture ? we value communication, ownership, and accountability ? Collaborative, ego-free team ? building together is in our DNA ? Learning-first environment ? mentorship, peer reviews, and exposure to real business impact ? Modern stack + autonomy ? your voice shapes how we build ? VC-funded & scaling fast ? 250+ strong, building from India for the world