Databricks, PySpark, Python - Data Engineer
Infosys
Infosys
Join our dynamic Data & Analytics Unit as a Senior Consultant, specializing in cutting-edge data engineering solutions. This role offers an exciting opportunity to lead initiatives in designing and building robust data platforms that drive enterprise analytics. If you're passionate about leveraging Databricks, PySpark, and Python to solve complex data challenges, we encourage you to apply.
This position is ideal for an experienced professional looking to make a significant impact within a leading technology organization. You will be instrumental in shaping the future of data-driven decision-making.
Lead the end-to-end design, development, and deployment of advanced data engineering solutions using Databricks and PySpark. Crafting scalable ETL/ELT pipelines and sophisticated data transformation frameworks will be central to your role. You will also be responsible for writing highly efficient Python and SQL code for data processing and analytics, ensuring seamless integration with enterprise analytics platforms.
Collaborate closely with various stakeholders to meticulously understand business needs and translate them into effective technical strategies. Optimize Spark jobs for peak performance, scalability, and cost-efficiency. Furthermore, you will uphold stringent data quality, governance, and security standards throughout all projects. Mentoring junior team members, conducting code reviews, and providing essential technical guidance are also key aspects of this leadership position.
A minimum of 7-9 years of comprehensive IT experience is required, with a strong emphasis on hands-on expertise in Databricks and PySpark. Exceptional proficiency in Python programming and SQL is essential for this role. Demonstrable experience in building enterprise-scale data pipelines and data lake solutions is a must.
A solid grasp of distributed computing concepts and Spark architecture is crucial. Proven experience in performance tuning and optimization techniques will be highly valued. Excellent stakeholder management and team coordination skills are paramount for successful collaboration and project delivery. A Bachelor of Engineering, B.Tech, BCA, or BSc, along with an MBA, MCA, ME, MSc, or MTech, are among the preferred educational backgrounds.
Infosys
IT Consulting