Hadoop / PySpark

Infosys

9–11 yrs Bengaluru Full Time Hybrid (office + remote)
Infosys logo
Posted : today
Actively hiring

Job description

Step into a high-impact consulting role focused on designing and delivering scalable big data solutions. This position empowers you to transform complex datasets into actionable insights. Collaborate closely with data engineers, architects, analysts, and business stakeholders to shape modern data platforms using Hadoop ecosystems and PySpark-based processing.

Guide teams through best practices, performance tuning, and reliable delivery, blending hands-on problem-solving with technical leadership. This role is perfect for individuals who thrive on mentoring, driving technical decisions, and building resilient, efficient, and production-ready data pipelines. If solving large-scale data challenges and enabling teams to achieve measurable outcomes motivates you, you'll find a collaborative environment that values ownership, clarity, and continuous improvement.

Responsibilities

Lead the end-to-end delivery of big data solutions leveraging Hadoop and PySpark, from initial requirements through to production rollout. Design and implement scalable batch/ETL pipelines for large datasets, prioritizing reliability and performance.

Drive technical architecture discussions, establish standards, and ensure adherence to best practices for distributed data processing. Optimize Spark jobs using strategies like partitioning, caching, shuffle tuning, and efficient file formats.

Collaborate with stakeholders to translate business needs into technical specifications, delivery plans, and milestones. Establish robust data quality checks, validation frameworks, and operational monitoring for production pipelines.

Perform root-cause analysis for pipeline failures and performance bottlenecks, implementing preventive fixes. Mentor engineers through code and design reviews, fostering knowledge sharing to enhance team capabilities.

Ensure comprehensive documentation, runbooks, and handover artifacts are created and maintained for support readiness.

Qualifications

A Bachelor’s or Master’s degree (or equivalent) in Engineering/Technology/Computer Applications/Science (e.g., BE/BTech/MTech/MCA/MSc). You should possess 9–11 years of overall experience in data engineering and large-scale data processing environments.

Demonstrate strong hands-on experience with Hadoop ecosystem components and distributed data processing concepts. Extensive practical experience in building data pipelines using PySpark is essential. Proven ability to lead technical delivery, guide teams effectively, and manage stakeholder expectations within consulting engagements is required.

Essential Skills

PySparkHadoop

Highlights

  • Actively hiring

More Details

RoleHadoop / PySpark
DepartmentData Engineering, Data & Analytics
Employment TypeFull Time, Hybrid (office + remote)

About the Company

Infosys logo

Infosys

IT Consulting

Hadoop / PySpark at Infosys | SkillMX | SkillMX