PySpark Developer
Infosys
Infosys
Seeking a proficient PySpark Developer to design, build, and manage sophisticated data processing pipelines. This role involves developing efficient ETL workflows for enterprise data platforms, handling large datasets, and collaborating with cross-functional teams to deliver impactful data solutions.
This is an exciting opportunity to contribute to a dynamic data and analytics unit within Infosys Limited.
Key responsibilities include developing and maintaining robust data pipelines using PySpark. You will engineer ETL/ELT processes for data ingestion, transformation, and loading, while writing optimized PySpark code for efficient data processing.
Work involves handling diverse data sources, optimizing SQL queries, and troubleshooting data quality and performance issues. Collaboration with Data Engineers, Analysts, and Business teams is essential for understanding requirements and participating in code reviews to uphold data engineering best practices. Monitoring and supporting production data pipelines will also be part of your role.
Essential qualifications include strong PySpark experience, solid Python programming skills, and hands-on SQL proficiency. A good understanding of ETL/ELT concepts and data warehousing is required, along with experience in handling large datasets and distributed processing.
Familiarity with Spark SQL, DataFrames, and Spark transformations is necessary. The role also requires experience in a Linux/Unix environment. Exposure to cloud platforms like AWS, Azure, or GCP, knowledge of Databricks, and experience with workflow orchestration tools such as Airflow are considered advantageous.
Infosys
IT Consulting