Join our Regulatory & Financial Risk team as a Data Engineer Manager in Chennai. This role is with Deloitte Touche Tohmatsu India LLP, part of the Strategy, Risk & Transaction group. You will contribute to mitigating risks and uncovering opportunities for value creation through cutting-edge risk management services.

We are seeking a seasoned professional to design, develop, and maintain robust, scalable data processing solutions. The ideal candidate will possess deep expertise in PySpark and a comprehensive understanding of the data engineering landscape, including distributed computing and SDLC-based project development.

This position offers an exciting opportunity to work on complex data challenges within a dynamic and supportive team environment.

Key responsibilities include designing, developing, and optimizing large-scale data pipelines using PySpark. You will build and maintain ETL/ELT workflows, working extensively with big data technologies like Hadoop, Spark, and Impala. Collaborating with cross-functional teams, ensuring code quality through rigorous reviews, and optimizing Spark job performance are crucial aspects of this role.

Further duties involve data modeling, transformation, and pipeline orchestration. You will actively participate in the entire SDLC, from requirement analysis to deployment. Troubleshooting data processing issues, contributing to automation and monitoring frameworks, and ensuring high availability of infrastructure are also key expectations.

Essential qualifications include strong hands-on experience with Python and PySpark, alongside advanced SQL skills for querying and performance tuning. A solid understanding of distributed data systems (Hadoop, Spark ecosystem) and experience with Impala, Hive, or SQL Server are required. Familiarity with data structures, algorithms, performance optimization, and ETL pipeline development is vital.

Knowledge of Agile/Scrum methodologies and version control systems like Git is expected. Experience with large datasets in production environments is a must. Exposure to cloud platforms (Azure, AWS, GCP) and workflow orchestration tools like Airflow is highly desirable. Strong analytical, problem-solving, and collaboration skills are essential for success in this fast-paced role.