Pyspark
Infosys
Infosys
Join a collaborative data engineering team focused on powering reliable analytics and smarter business decisions. This role involves building and optimizing scalable data processing pipelines using PySpark and Spark, working closely with engineers, analysts, and stakeholders. You'll transform raw data into trusted, high-quality datasets and contribute to a culture that values clean engineering, performance, and continuous learning. This is a prime opportunity to grow your impact with modern big data technologies in a fast-paced environment.
- Design, develop, and maintain scalable ETL/ELT pipelines using PySpark for batch and incremental processing. - Build and optimize Apache Spark jobs for performance, partitioning, caching, and efficient transformations. - Perform data cleansing, validation, and reconciliation to ensure data accuracy and consistency. - Collaborate with cross-functional teams to translate requirements into robust data processing solutions. - Troubleshoot pipeline failures, analyze logs, and implement fixes for improved reliability. - Write clean, maintainable code with reusable components and clear documentation. - Support deployment and operationalization of Spark workloads, including monitoring. - Contribute to code reviews and uphold engineering best practices.
- B.Tech, M.Tech, MCA, or M.Sc degree. - 2-3 years of experience in data engineering or large-scale data processing. - Strong hands-on experience with PySpark for building data pipelines and transformations. - Working knowledge of Apache Spark concepts (RDD/DataFrame, joins, shuffles, performance). - Ability to effectively debug Spark applications and resolve data/job issues. - Experience optimizing Spark workloads for performance and cost efficiency is preferred. - Familiarity with building end-to-end data pipelines with strong data quality checks. - Experience working in agile teams and improving engineering standards for data pipelines.
Infosys
IT Consulting