Parallel Software Engineer

Job Description

Mountain View, CA / Hyderabad, IN / Remote About DataPelago DataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing. We are looking for specialists to join our engineering team and shape the future of accelerated data processing. The Opportunity As a Parallel Software Engineer, you will be a key individual contributor in developing advanced parallel software that unlocks the full potential of diverse hardware accelerators including GPUs and SIMD-capable CPUs. You will enhance functional breadth, performance, scale, and reliability of data processing operators that are integral to our data processing engine. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers. What You'll Do Architect: Influence the architecture of how our data processing engine harnesses the par- allelism in GPUs and SIMD-capable CPUs efficiently in processing diverse large-scale data. Design: Lead design of functional and performance enhancements to operators and func- tions that are accelerated by our engine. Core Development: Individually design, implement, test, optimize, and maintain parallel implementations of operators and functions on diverse acceleration hardware. Innovation and Differentiation: Analyze advances in accelerated computing hardware, pro- gramming models, and related tools and ensure our engine extends technology and product leadership. Collaboration: Partner effectively with the execution engine engineering team in integrating parallel software components with the overall engine. Continuous Improvement: Foster best practices in design and code reviews, testing, CI/CD, and issue resolution to maintain highest product quality, security, efficiency, & productivity. What You'll Bring Bachelor's degree in Computer Science or a related field with 5+ years of relevant experi- ence OR a Master's degree in Computer Science or a related field with 3+ years of relevant experience. 3+ years of deep technical experience in developing production applications that process large scale data using SIMD-extensions of CPUs (e.g., AVX), GPU programming models (e.g., CUDA, ROCm), or equivalent accelerated computing framework. Demonstrated experience working with software libraries, development tools, and profiling tools specific to parallel and accelerated computing. Demonstrated experience in troubleshooting and resolving functional and performance anomalies in both pre- and post-production scenarios. Strong knowledge of computer architecture. Exceptional programming skills in C, C++. Extensive development experience in Linux environments. Strong analytical and problem-solving skills with a passion for performance optimization. Location Considerations: We value face-to-face collaboration, but recognize that talent can be found anywhere. Our engineer- ing team works at our headquarters in Mountain View, CA, at our India office in Hyderabad, and at remote locations. Why Join DataPelago? Technology Leadership: Shape the architecture and development of core parallel data pro- cessing functions of our engine. Cutting-Edge Innovation: Work on challenging problems at the forefront of accelerated computing and data processing. Significant Impact: Your contributions will directly impact the performance and scalability of our mission-critical platform. Growth: Expand your technical expertise and scope of responsibilities working with other talented engineers and with a growing product. Competitive compensation, stock options, comprehensive benefits package, leadership de- velopment opportunities.