Hadoop / PySpark

Infosys

5–9 yrs Bengaluru Full Time Work from office
Infosys logo
Posted : today
Actively hiring

Job description

Lead the charge in developing cutting-edge data platforms and empowering teams to extract valuable, high-impact insights from massive datasets. This technology leadership role places you at the nexus of engineering excellence and collaborative innovation. You will drive architectural decisions, define delivery benchmarks, and empower your team to build robust, scalable solutions leveraging Hadoop and PySpark.

Collaborate closely with business stakeholders to grasp their needs, transforming them into resilient data pipelines. Ensure top-tier performance, quality, and governance across the entire data ecosystem. If you thrive on tackling complex data challenges, guiding fellow engineers, and championing best practices in Big Data processing, this position offers a significant opportunity to lead impactful work while fostering a culture of ownership, continuous learning, and improvement.

Responsibilities

Spearhead the design and development of scalable Big Data solutions using Hadoop and PySpark for both batch and large-scale processing needs. Architect and implement comprehensive end-to-end data pipelines, focusing on reliability, performance optimization, and efficient resource management within Hadoop environments. Develop and refine Hive data models, queries, and partitioning strategies to effectively support analytics and downstream data consumption. Lead technical planning, accurate estimation, and successful delivery of data engineering projects, ensuring adherence to timelines and stringent quality standards. Establish clear coding standards, conduct thorough code reviews, and enforce best practices to guarantee maintainability, robust testing, and production readiness. Diagnose and resolve production issues, conduct in-depth root-cause analysis, and implement proactive measures to enhance system stability and throughput. Partner with product, analytics, and platform teams to translate business requirements into scalable and effective technical solutions. Mentor team members, provide guidance on technical decisions, and foster skill development in areas such as Hadoop, PySpark, Big Data, and Hive.

Qualifications

A strong educational foundation with a BTECH, MTECH, MCA, or MSC degree, or an equivalent qualification. A minimum of 5 to 9 years of comprehensive experience, demonstrating deep hands-on expertise in utilizing Hadoop and PySpark for large-scale data processing. Proven track record in building and managing Big Data pipelines, coupled with experience in using Hive for efficient querying and data modeling. A solid understanding of distributed processing concepts, performance optimization techniques, and best practices for data reliability. Demonstrated experience in leading technical execution, including conducting code reviews, participating in design discussions, and taking ownership of project delivery.

Essential Skills

HadoopPySparkHiveBig Data

Good to Have

Spark SQLYARNHDFSOozieAirflow

Highlights

  • Actively hiring

More Details

RoleHadoop / PySpark
IndustryInformation Technology & Services
DepartmentData Engineering
Employment TypeFull Time, Work from office

About the Company

Infosys logo

Infosys

Information Technology & Services

Hadoop / PySpark at Infosys | SkillMX | SkillMX