Pyspark

Infosys

2–3 yrs Bengaluru Full Time Work from office
Infosys logo
Posted : today
Actively hiring

Job description

Join a collaborative data engineering team focused on powering reliable analytics and smarter business decisions. This role involves building and optimizing scalable data processing pipelines using PySpark and Spark, working closely with engineers, analysts, and stakeholders. You'll transform raw data into trusted, high-quality datasets and contribute to a culture that values clean engineering, performance, and continuous learning. This is a prime opportunity to grow your impact with modern big data technologies in a fast-paced environment.

Responsibilities

- Design, develop, and maintain scalable ETL/ELT pipelines using PySpark for batch and incremental processing. - Build and optimize Apache Spark jobs for performance, partitioning, caching, and efficient transformations. - Perform data cleansing, validation, and reconciliation to ensure data accuracy and consistency. - Collaborate with cross-functional teams to translate requirements into robust data processing solutions. - Troubleshoot pipeline failures, analyze logs, and implement fixes for improved reliability. - Write clean, maintainable code with reusable components and clear documentation. - Support deployment and operationalization of Spark workloads, including monitoring. - Contribute to code reviews and uphold engineering best practices.

Qualifications

- B.Tech, M.Tech, MCA, or M.Sc degree. - 2-3 years of experience in data engineering or large-scale data processing. - Strong hands-on experience with PySpark for building data pipelines and transformations. - Working knowledge of Apache Spark concepts (RDD/DataFrame, joins, shuffles, performance). - Ability to effectively debug Spark applications and resolve data/job issues. - Experience optimizing Spark workloads for performance and cost efficiency is preferred. - Familiarity with building end-to-end data pipelines with strong data quality checks. - Experience working in agile teams and improving engineering standards for data pipelines.

Essential Skills

PySparkApache SparkETLELT

Good to Have

SQLHadoopHiveKafkaAirflow

Highlights

  • Actively hiring

More Details

RolePyspark
DepartmentData Engineering, Data & Analytics
Employment TypeFull Time, Work from office

About the Company

Infosys logo

Infosys

IT Consulting

Pyspark at Infosys | SkillMX | SkillMX