Databricks, Pyspark

Infosys

6–8 yrs Chennai Full Time Work from office
Infosys logo
Posted : today
Actively hiring

Job description

Join a dynamic data engineering team focused on building modern analytics platforms and transforming raw data into actionable insights. This role involves leading the design and implementation of scalable data pipelines using Databricks and PySpark. You will collaborate with analysts, data scientists, and platform teams to accelerate decision-making across the organization. Emphasizing strong engineering practices, clean code, performance optimization, and reliable operations, you'll also guide best practices and mentor team members. This is an excellent opportunity for individuals passionate about solving complex data challenges, optimizing distributed workloads, and constructing resilient, secure, and adaptable systems within a collaborative, growth-oriented environment.

Responsibilities

- Lead the development of end-to-end data pipelines on Databricks using PySpark for batch and incremental processing. - Design scalable data models and curated datasets for analytics and downstream consumption. - Write and optimize advanced SQL for transformations, validations, and performance-critical queries. - Implement robust data quality checks, reconciliation, and monitoring for trusted datasets. - Tune Spark jobs for performance and cost efficiency (partitioning, caching, file formats, cluster sizing). - Establish coding standards, reusable frameworks, and review practices for improved maintainability. - Collaborate with stakeholders to translate requirements into technical designs and delivery plans. - Troubleshoot production issues, perform root-cause analysis, and drive preventive improvements. - Mentor team members and provide technical guidance on design, implementation, and optimization.

Qualifications

Minimum qualifications include a BTECH, MTECH, MCA, or MSC in Computer Science, Information Technology, or a related field. A minimum of 6–8 years of overall experience in data engineering or large-scale data processing is required. Strong hands-on experience with PySpark for distributed data processing and transformation logic is essential, as is proficiency with Databricks for building, running, and managing data workloads. Advanced SQL skills, including complex joins, window functions, and query optimization, are necessary. Experience building reliable pipelines with a strong focus on data quality, performance, and stability is also a key requirement. Preferred qualifications include experience designing lakehouse architectures, proficiency in Spark optimization techniques, ability to build reusable PySpark utilities, experience with orchestration and scheduling, and a proven track record of technical leadership.

Essential Skills

DatabricksPySparkSQLData QualityPerformance TuningData PipelinesData ModelingETLDistributed SystemsCloud Data PlatformsDatabase TechnologiesBig DataData Processing

Good to Have

Lakehouse ArchitecturesSpark OptimizationOrchestrationSchedulingTechnical Leadership

Highlights

  • Actively hiring

More Details

RoleDatabricks, Pyspark
IndustryTechnology
DepartmentData Engineering, Data & Analytics
Employment TypeFull Time, Work from office

About the Company

Infosys logo

Infosys

Technology

Databricks, Pyspark at Infosys | SkillMX | SkillMX