Databricks, PySpark, Python - Data Engineer

Infosys

7–9 yrs Thiruvananthapuram Full Time Work from office
Infosys logo
Posted : today
Actively hiring

Job description

Join our dynamic Data & Analytics Unit as a Senior Consultant, specializing in cutting-edge data engineering solutions. This role offers an exciting opportunity to lead initiatives in designing and building robust data platforms that drive enterprise analytics. If you're passionate about leveraging Databricks, PySpark, and Python to solve complex data challenges, we encourage you to apply.

This position is ideal for an experienced professional looking to make a significant impact within a leading technology organization. You will be instrumental in shaping the future of data-driven decision-making.

Responsibilities

Lead the end-to-end design, development, and deployment of advanced data engineering solutions using Databricks and PySpark. Crafting scalable ETL/ELT pipelines and sophisticated data transformation frameworks will be central to your role. You will also be responsible for writing highly efficient Python and SQL code for data processing and analytics, ensuring seamless integration with enterprise analytics platforms.

Collaborate closely with various stakeholders to meticulously understand business needs and translate them into effective technical strategies. Optimize Spark jobs for peak performance, scalability, and cost-efficiency. Furthermore, you will uphold stringent data quality, governance, and security standards throughout all projects. Mentoring junior team members, conducting code reviews, and providing essential technical guidance are also key aspects of this leadership position.

Qualifications

A minimum of 7-9 years of comprehensive IT experience is required, with a strong emphasis on hands-on expertise in Databricks and PySpark. Exceptional proficiency in Python programming and SQL is essential for this role. Demonstrable experience in building enterprise-scale data pipelines and data lake solutions is a must.

A solid grasp of distributed computing concepts and Spark architecture is crucial. Proven experience in performance tuning and optimization techniques will be highly valued. Excellent stakeholder management and team coordination skills are paramount for successful collaboration and project delivery. A Bachelor of Engineering, B.Tech, BCA, or BSc, along with an MBA, MCA, ME, MSc, or MTech, are among the preferred educational backgrounds.

Essential Skills

DatabricksPySparkPythonSQLData PipelinesData TransformationETLELTSpark ArchitecturePerformance TuningStakeholder ManagementTeam Coordination

Highlights

  • Actively hiring

More Details

RoleDatabricks, PySpark, Python - Data Engineer
DepartmentData Engineering
Employment TypeFull Time, Work from office

About the Company

Infosys logo

Infosys

IT Consulting

Databricks, PySpark, Python - Data Engineer at Infosys | SkillMX | SkillMX