Join our team as an experienced Databricks Developer, focusing on crafting and maintaining robust, scalable data pipelines and analytics solutions. You will leverage your deep understanding of Apache Spark, Python, SQL, and cloud data engineering technologies to drive enterprise data integration and analytics initiatives forward.

This role is integral to designing, developing, and optimizing data solutions within a collaborative environment. We seek a proactive individual adept at handling complex data challenges and contributing to best-in-class data engineering practices.

Your primary duties will involve architecting and implementing ETL/ELT pipelines within Databricks, alongside developing high-performance data processing frameworks using Apache Spark (PySpark). You will build and refine both batch and streaming data pipelines, seamlessly integrating data from diverse sources including databases, APIs, and cloud storage.

Key tasks include performing data transformation, cleansing, and validation, as well as creating and optimizing Delta Lake tables for efficient data storage. You will also be responsible for developing reusable Databricks notebooks, workflows, and jobs, while actively monitoring and troubleshooting pipeline performance and failures. Collaboration with stakeholders to understand data requirements and adherence to coding standards, version control, and CI/CD practices are also essential.

A strong foundation in Databricks, Apache Spark/PySpark, Python, and SQL is mandatory for this position. Demonstrated experience in ETL/ELT development and working with Delta Lake is crucial, complemented by a solid understanding of data warehousing concepts. Familiarity with Git and version control is also required.

Preferred qualifications include experience with major cloud platforms such as Microsoft Azure (Azure Databricks, ADLS Gen2, Azure Data Factory), AWS (S3, Glue, EMR), or Google Cloud (BigQuery, Cloud Storage). Experience with streaming technologies like Spark Structured Streaming and orchestration tools such as Airflow is beneficial. An understanding of data governance and security principles is also a plus.