Databricks, Pyspark
Infosys
Infosys
Join a dynamic data engineering team focused on building modern analytics platforms and transforming raw data into actionable insights. This role involves leading the design and implementation of scalable data pipelines using Databricks and PySpark. You will collaborate with analysts, data scientists, and platform teams to accelerate decision-making across the organization. Emphasizing strong engineering practices, clean code, performance optimization, and reliable operations, you'll also guide best practices and mentor team members. This is an excellent opportunity for individuals passionate about solving complex data challenges, optimizing distributed workloads, and constructing resilient, secure, and adaptable systems within a collaborative, growth-oriented environment.
- Lead the development of end-to-end data pipelines on Databricks using PySpark for batch and incremental processing. - Design scalable data models and curated datasets for analytics and downstream consumption. - Write and optimize advanced SQL for transformations, validations, and performance-critical queries. - Implement robust data quality checks, reconciliation, and monitoring for trusted datasets. - Tune Spark jobs for performance and cost efficiency (partitioning, caching, file formats, cluster sizing). - Establish coding standards, reusable frameworks, and review practices for improved maintainability. - Collaborate with stakeholders to translate requirements into technical designs and delivery plans. - Troubleshoot production issues, perform root-cause analysis, and drive preventive improvements. - Mentor team members and provide technical guidance on design, implementation, and optimization.
Minimum qualifications include a BTECH, MTECH, MCA, or MSC in Computer Science, Information Technology, or a related field. A minimum of 6–8 years of overall experience in data engineering or large-scale data processing is required. Strong hands-on experience with PySpark for distributed data processing and transformation logic is essential, as is proficiency with Databricks for building, running, and managing data workloads. Advanced SQL skills, including complex joins, window functions, and query optimization, are necessary. Experience building reliable pipelines with a strong focus on data quality, performance, and stability is also a key requirement. Preferred qualifications include experience designing lakehouse architectures, proficiency in Spark optimization techniques, ability to build reusable PySpark utilities, experience with orchestration and scheduling, and a proven track record of technical leadership.
Infosys
Technology