Databricks

Infosys

3–5 yrs Bengaluru Full Time Hybrid (office + remote)
Infosys logo
Posted : today
Actively hiring

Job description

Shape the future of data-driven decisions by joining a collaborative data engineering team. This role is instrumental in building reliable data pipelines on Databricks, transforming raw data into high-quality datasets. You will partner with stakeholders to understand data needs, enhance performance, and ensure seamless data delivery from ingestion to validation. Embrace this opportunity to grow your technical expertise in optimizing distributed processing and improving data platform operations, contributing to impactful outcomes in an environment that values ownership and innovation.

Responsibilities

Develop and maintain scalable ETL pipelines using Databricks and PySpark, processing large datasets efficiently. Implement data transformations, cleansing, and enrichment aligned with business needs. Optimize Spark jobs for performance and cost through tuning partitions, caching, and cluster configurations. Build reusable notebooks/jobs and support workload scheduling and orchestration within Databricks. Perform data validation and quality checks to ensure curated dataset accuracy. Troubleshoot pipeline failures, analyze logs, and resolve issues for stable production operations. Collaborate with cross-functional teams to gather requirements and deliver iterative enhancements. Maintain clear technical documentation for pipelines, transformations, and operational runbooks.

Qualifications

Possess a Bachelor’s degree (or equivalent) in Engineering, Technology, Computer Science, or a related field. Bring 3–5 years of experience in data engineering or similar roles, with hands-on Databricks expertise. Demonstrate strong PySpark development skills for distributed data processing. Proven ability to build and support production ETL pipelines. Exhibit the capability to analyze data issues, debug Spark/ETL jobs, and implement effective fixes. Experience designing end-to-end data workflows on Databricks, including scheduling and monitoring, is preferred. Strong understanding of data modeling and building curated datasets for analytics use cases is advantageous. Familiarity with Delta Lake concepts and best practices for lakehouse implementations is a plus.

Essential Skills

DatabricksPySparkETL

Good to Have

Delta LakeSpark SQLData ModelingWorkflow OrchestrationPerformance Tuning

Highlights

  • Actively hiring

More Details

RoleDatabricks
DepartmentData Engineering, Data & Analytics
Employment TypeFull Time, Hybrid (office + remote)

About the Company

Infosys logo

Infosys

IT Consulting

Databricks at Infosys | SkillMX | SkillMX