PySpark Developer

Infosys

2–3 yrs Bengaluru Full Time Work from office
Infosys logo
Posted : today
Actively hiring

Job description

Seeking a proficient PySpark Developer to design, build, and manage sophisticated data processing pipelines. This role involves developing efficient ETL workflows for enterprise data platforms, handling large datasets, and collaborating with cross-functional teams to deliver impactful data solutions.

This is an exciting opportunity to contribute to a dynamic data and analytics unit within Infosys Limited.

Responsibilities

Key responsibilities include developing and maintaining robust data pipelines using PySpark. You will engineer ETL/ELT processes for data ingestion, transformation, and loading, while writing optimized PySpark code for efficient data processing.

Work involves handling diverse data sources, optimizing SQL queries, and troubleshooting data quality and performance issues. Collaboration with Data Engineers, Analysts, and Business teams is essential for understanding requirements and participating in code reviews to uphold data engineering best practices. Monitoring and supporting production data pipelines will also be part of your role.

Qualifications

Essential qualifications include strong PySpark experience, solid Python programming skills, and hands-on SQL proficiency. A good understanding of ETL/ELT concepts and data warehousing is required, along with experience in handling large datasets and distributed processing.

Familiarity with Spark SQL, DataFrames, and Spark transformations is necessary. The role also requires experience in a Linux/Unix environment. Exposure to cloud platforms like AWS, Azure, or GCP, knowledge of Databricks, and experience with workflow orchestration tools such as Airflow are considered advantageous.

Essential Skills

PySparkPythonSQLETLData WarehousingBig DataSpark SQLDataFramesSpark TransformationsLinux/Unix

Good to Have

AWSAzureGCPDatabricksAirflowCI/CD

Highlights

  • Actively hiring

More Details

RolePySpark Developer
DepartmentData Engineering
Employment TypeFull Time, Work from office

About the Company

Infosys logo

Infosys

IT Consulting

PySpark Developer at Infosys | SkillMX | SkillMX