Senior Manager - Cloudera Admin

About Role: The role is for a technical shift leads to provide governance and oversight of the BAU operations. He/she will need to possess right skillsets for new data lake solutions planned this year; it is important to bring in right skillsets for successful implementation of the production engineering initiatives for existing and upcoming system rollouts.

In addition to BAU, the lead will be responsible for driving various initiatives such as Shift left initiative for enabling production standards readiness for new Datawarehouse platforms, Automation for new platform, Refactoring the ETL workflows, Drive data quality initiatives

Cloudera experience: 5 - 10 years of relevant experience in Cloudera platform installation.

Performance tuning experience in Apache Spark / Pyspark, Spark SQL

Key Skills/ Roles & Responsibilities:

• Cloudera platform installation
• Performance tuning experience in Apache Spark / Pyspark, Spark SQL
• Cluster configuration, maintenance, troubleshooting and monitoring
• CDP Private Cloud Base Installation
• HDFS Topology and Roles, HDFS Performance and Fault Tolerance, HDFS and Hadoop Security Overview
• Data Ingest - Ingesting Data, Best Practices for Importing Data
• Data Flow - NiFi Architecture, Apache Kafka, Apache Kafka Cluster Architecture, Apache Kafka Command Line Tools
• Data Access and Discovery - Apache Hive, Apache Impala, Apache Impala Tuning, Managing and Configuring Hue, Hue Authentication and Authorization
• Data Compute - Running Applications on YARN, MapReduce Applications, YARN Memory, and CPU Settings, Tez, ACID, Spark
• Managing Resources - Capacity Scheduler, Managing Queues, Impala Query Scheduling
• Security - Data Governance with SDX, Hadoop Security, Hadoop Authentication Using Kerberos, Hadoop Encryption, Apache Ranger, Apache Atlas, Backup and Recovery
• Private Cloud Capabilities
• Plan, design, build various production tooling related solutions.
• Identifies better ways of working and seed and drive continuous improvement initiatives.
• Review the architecture for SPOF ‘s and resiliency.
• Manage Day to Day Security Operations (Daily checklist, monitoring for availability/performance, user access management, troubleshooting & resolving user issues, managing user queries,) & validating reports
• Drive escalated issues or major outage.
• Tracking and escalating issues with Vendors/Service Providers/ OEM/other connected system teams.
• Identify, respond, troubleshoot, remediate issues in timely manner and carry out root cause analysis
• Ensure that all security controls are implemented as per security guidelines/best practices
• Review and action reports - Incident, Capacity & other on weekly/ monthly basis.
• Managing Audit & Compliance. Co-ordinate with different stakeholders to get audit observations closed related to IT security.
• Knowledge and practice of ITIL processes
• Conduct periodic MIS/ Review meetings with vendors/ OEM to track status of operational and implementation activities.
• Design Request for proposal (RFP) for various DWH solution.
• Contribute to activities/ AMC Renewals/ Contract Renewals/ License Renewals of DWH solutions.
• Must be able to manage & guide the team.
• Good communication and presentation skills
• Ability to learn new engineering tools and apply best practices
• Managing the Vendors, assigning the tasks & closure of same
• Evaluate BI operations effectiveness and create and execute a plan to increase maturity.
• Understand and follow NSE policies and practices
• Cloudera experience (Administrator and/or Developer)
• Datawarehouse, ETL, Shell scripting and Reporting
• Spark, Python
• Hadoop, Hive, Impala, Hue
• Strong ITIL