Join our dynamic team focused on mitigating risk and uncovering opportunities within the financial and regulatory landscape. We are seeking a skilled PySpark Developer with a passion for designing and implementing robust data processing solutions. This role offers the chance to work with cutting-edge big data technologies and contribute to critical financial risk management initiatives.

This position is ideal for a proactive individual eager to dive into end-to-end data pipeline development. You'll be instrumental in building and optimizing scalable systems that handle large-scale data efficiently, ensuring the integrity and performance of our data infrastructure.

Key responsibilities include designing, developing, and optimizing large-scale data pipelines using PySpark, alongside building and maintaining ETL/ELT workflows. You will leverage big data technologies like Hadoop, Spark, and Impala, collaborating closely with data scientists and product teams. Performance optimization for Spark jobs, data modeling, and participation in the full Software Development Life Cycle (SDLC) are also integral to this role. You'll troubleshoot data processing issues, contribute to automation efforts, and ensure code quality through best practices and reviews.

Essential qualifications include strong hands-on experience with Python and PySpark, coupled with robust SQL skills for querying and performance tuning. Familiarity with distributed data systems such as Hadoop and the Spark ecosystem is crucial. Experience with relational databases like SQL Server or Impala, a solid understanding of data structures, algorithms, and performance optimization are also required.

Knowledge of ETL pipeline development, data warehousing concepts, and SDLC methodologies (Agile/Scrum) is preferred. Exposure to cloud platforms (Azure, AWS, GCP) and workflow orchestration tools like Airflow or Oozie is beneficial. Strong analytical abilities, problem-solving skills, and excellent communication are vital for success in this fast-paced environment.