We are seeking a skilled Technical Leader to drive the design and development of robust data solutions. This role focuses on building scalable batch and near-real-time ETL/ELT pipelines utilizing cutting-edge technologies.

This is an opportunity to work with advanced cloud infrastructure and implement best practices in CI/CD. You will be instrumental in orchestrating complex data workflows and ensuring seamless deployments.

Lead the design and development of scalable batch and near-real-time ETL/ELT pipelines using Snowflake (AWS) and Apache Spark (PySpark, Spark SQL, Structured Streaming). Build and manage structured streaming pipelines leveraging Apache Kafka and Spark Structured Streaming. Design and implement dimensional data models, including Fact/Dimension tables and SCD Type 2. Orchestrate data pipelines efficiently using Databricks Workflows or Apache Airflow. Integrate CI/CD pipelines with Jenkins and Git (Bitbucket/GitHub) for automated deployments across development, UAT, and production environments.

Proficiency in core technologies such as Apache Spark (Core, Structured Streaming), PySpark, Databricks, and advanced SQL. Experience with DevOps and CI/CD tools including Jenkins, Git, GitHub, and Bitbucket. Strong programming skills in Python and SQL. Familiarity with cloud infrastructure, particularly AWS (S3, EMR, EC2, IAM, CloudWatch) and Databricks Runtime & Cluster Management. Experience with streaming and integration technologies like Apache Kafka and Snowflake integration is preferred. A background in financial services, payouts, or enterprise data platforms is advantageous. Hands-on experience with Delta Lake optimization and incremental processing strategies is a plus. A solid understanding of distributed computing principles is essential.