Pyspark
Infosys
Infosys
Join a dynamic, collaborative environment where data drives intelligent decisions and enhances customer experiences. This role involves hands-on work with extensive datasets to create robust, high-performance data processing solutions utilizing PySpark and Spark. You will collaborate with engineers, analysts, and stakeholders to understand business needs, develop scalable pipelines, and consistently improve data quality and performance. If you thrive on solving complex data challenges, optimizing distributed workloads, and owning projects from conception to delivery, this is an excellent opportunity to expand your influence.
We encourage idea sharing, peer learning, and contributing to a culture that values clarity, craftsmanship, and ongoing improvement.
Key responsibilities include designing, developing, and maintaining scalable batch data pipelines using PySpark and Apache Spark for large datasets. This involves performing data ingestion, transformation, and enrichment while ensuring accuracy, completeness, and consistency. Optimization of Spark jobs for performance, including partitioning, caching, joins, and shuffles, is crucial for improving runtime efficiency and resource utilization.
Implementing robust error handling, logging, and monitoring for reliable pipeline execution and faster issue resolution is also a key part of the role. Collaboration with cross-functional teams to gather requirements, define data contracts, and deliver well-documented solutions is expected. You will also conduct code reviews, adhere to engineering best practices, and contribute to reusable components and standards. Troubleshooting production issues, performing root-cause analysis, and driving corrective/preventive actions are essential duties.
A Bachelor’s degree (or equivalent) in Engineering, Computer Science, IT, or a related field is required, along with 3–5 years of experience in data engineering or big data development. Strong hands-on experience with PySpark and Apache Spark for building data processing workflows is essential, complemented by a solid understanding of distributed data processing concepts and performance tuning fundamentals.
Candidates must demonstrate the ability to translate business requirements into technical implementations and deliver within project timelines. Experience building end-to-end Spark applications, including orchestration, dependency management, and production support readiness, is preferred. Strong data transformation skills with a focus on data quality checks, reconciliation, and pipeline reliability are also beneficial.
Infosys
IT Consulting