Spark-Scala, Databricks

Infosys

3–5 yrs Bengaluru Full Time Hybrid (office + remote)
Infosys logo
Posted : today
Actively hiring

Job description

Join a dynamic engineering team focused on transforming complex data into actionable insights that drive critical business decisions. This role involves hands-on work with Spark-Scala and Databricks to build robust, scalable data pipelines. You will optimize distributed processing and ensure the delivery of reliable datasets for analytics and subsequent applications. Collaboration with data engineers, analysts, and stakeholders is key to understanding needs and developing effective ETL solutions, while continuously enhancing performance and quality. It's an excellent opportunity to grow your expertise in modern cloud data practices and take ownership from development through production support within a supportive and accountable culture.

Responsibilities

- Design, develop, and maintain scalable ETL pipelines using Spark-Scala on Databricks for both batch and incremental processing. - Implement data transformations, joins, aggregations, and validations to ensure data accuracy and consistency. - Construct reusable Spark components, adhering to best practices for modular and maintainable code. - Optimize Spark job performance and cost-efficiency through partitioning, caching, and shuffle optimization. - Monitor job execution, troubleshoot failures, and provide timely production support with thorough root-cause analysis. - Enhance pipeline reliability by implementing robust logging, error handling, and data quality checks. - Collaborate with cross-functional teams to gather requirements and translate them into comprehensive technical solutions. - Engage in code reviews, documentation, and knowledge sharing to elevate team standards. - Support release cycles by validating outputs, ensuring backward compatibility, and maintaining deployment readiness.

Qualifications

- Hold a Bachelor’s or Master’s degree (e.g., BE/BTech/MSc/MCA/MTech or equivalent). - Possess 3–5 years of experience in data engineering or ETL development. - Demonstrate strong hands-on experience with Spark using Scala and working on Databricks. - Exhibit a solid understanding of ETL concepts, data transformations, and effective pipeline troubleshooting. - Ability to write clean, testable code and collaborate effectively with both technical and non-technical stakeholders. - Preferred experience includes building end-to-end pipelines on Databricks, utilizing notebooks, jobs/workflows, and understanding cluster configurations. - Strong SQL skills and experience integrating Spark pipelines with structured data sources are advantageous. - Familiarity with data quality frameworks, reconciliation strategies, and automated validation checks is a plus. - Exposure to CI/CD practices for data engineering, including version control, automated testing, and release management, is beneficial. - Proven ability to optimize distributed workloads and achieve measurable improvements in runtime and stability.

Essential Skills

SparkScalaDatabricks

Good to Have

SQLDelta LakeApache AirflowAzure Data Lake Storage (ADLS)Git

Highlights

  • Actively hiring

More Details

RoleSpark-Scala, Databricks
DepartmentData Engineering, Data & Analytics
Employment TypeFull Time, Hybrid (office + remote)

About the Company

Infosys logo

Infosys

IT Consulting

Spark-Scala, Databricks at Infosys | SkillMX | SkillMX