Senior Administrator - Python (IN)
HCLTech
HCLTech
Join our team as a Senior Python & Databricks Developer, focusing on designing and optimizing robust data engineering and analytics solutions within the Databricks environment. This role is key in collaborating with data engineers, data scientists, and stakeholders to construct dependable ETL pipelines and support advanced analytics initiatives.
This position requires extensive experience in building and refining large-scale data processing systems. You will be instrumental in ensuring data integrity, efficiency, and the successful implementation of complex data workflows.
Key responsibilities include designing, developing, and maintaining ETL/ELT pipelines using Python (PySpark) and Databricks. You will build and optimize Spark jobs for extensive data processing, develop reusable Python code, and implement data transformations. Additionally, you'll manage Databricks notebooks, jobs, and workflows, optimize Spark performance, and work with Delta Lake.
Data integration from various sources like RDBMS, cloud storage, and APIs is crucial. You will also collaborate with cloud platforms (Azure/AWS/GCP), integrate with CI/CD tools, and utilize Git for version control. Active participation in Agile ceremonies and client interaction, including demos and troubleshooting, is expected.
We are seeking candidates with a strong foundation in Python, Databricks, and Apache Spark (PySpark). Essential skills include a solid understanding of data warehousing concepts, experience with Delta Lake, strong SQL abilities, and proficiency in handling large datasets and distributed systems.
Experience with at least one major cloud platform (Azure, AWS, or GCP) and familiarity with CI/CD, Git, and Agile tooling are necessary. Excellent problem-solving, analytical, communication, and documentation skills are vital for success in this client-facing role within an Agile/Scrum framework.
HCLTech
IT Consulting