Hadoop / PySpark

Infosys

2–5 yrs Bengaluru Full Time Work from office
Infosys logo
Posted : today
Actively hiring

Job description

Join a data-driven team focused on transforming large-scale information into actionable insights. This role involves collaborating with engineers, analysts, and stakeholders to build and optimize robust big-data solutions for reporting, analytics, and downstream applications.

You will gain hands-on experience with modern distributed processing, contribute to production-grade pipelines, and learn best practices for performance, quality, and governance. If you thrive on solving complex data challenges in a collaborative environment that values curiosity and teamwork, this opportunity offers significant technical growth and the chance to make a tangible impact across projects and business outcomes.

Responsibilities

Design, develop, and support scalable data processing workflows using Hadoop and PySpark for batch and large-volume data handling.

Build and maintain data pipelines for ingesting, transforming, and validating data from diverse sources into curated datasets.

Optimize Spark jobs for enhanced performance, including partitioning, caching, and shuffle tuning, to improve overall pipeline efficiency and reliability.

Conduct data quality checks, perform reconciliation, and identify root causes for pipeline failures or data anomalies.

Collaborate with cross-functional teams to understand requirements, translate them into technical solutions, and deliver projects on time.

Create comprehensive technical documentation for workflows, data mappings, and operational runbooks.

Participate in code reviews, adhere to engineering best practices, and contribute to the continuous improvement of standards and tooling.

Qualifications

Possess 2–5 years of experience in big data engineering or data processing.

Hold a Bachelor's or Master's degree in Engineering, Computer Science, or a related field (e.g., BTech, BE, MCA, MSc, MTech).

Demonstrate hands-on experience with Hadoop ecosystem concepts, including HDFS and distributed processing fundamentals.

Have practical experience developing data transformations using PySpark.

Exhibit strong problem-solving skills with the ability to debug data and job execution issues in distributed environments.

Preferred qualifications include experience with end-to-end ETL/ELT pipelines, Spark optimization, data modeling, production support practices, and strong communication skills for stakeholder collaboration.

Essential Skills

HadoopPySparkData ProcessingData PipelinesSpark OptimizationETLELTData QualityProblem SolvingDebuggingCode ReviewsTechnical Documentation

Good to Have

HiveHBaseKafkaAirflowScalaData ModelingMonitoringIncident TriagePipeline Resiliency

Highlights

  • Actively hiring

More Details

RoleHadoop / PySpark
DepartmentData Engineering
Employment TypeFull Time, Work from office

About the Company

Infosys logo

Infosys

IT Consulting

Hadoop / PySpark at Infosys | SkillMX | SkillMX