Join our team as a Data Engineer and play a vital role in shaping our data platform. You will collaborate with Integration Architects and Data Architects to ensure seamless integration between systems and data models. This position offers an opportunity to contribute to the blueprint and design of our data platform, driving innovation in data generation, collection, and processing.

This role focuses on building robust data solutions. You will be responsible for designing, developing, and maintaining systems that handle data generation, collection, and processing. The core of this role involves creating efficient data pipelines, upholding data quality standards, and implementing ETL processes for effective data migration and deployment across various systems.

Build, maintain, and optimize scalable data pipelines for large-scale data processing as a key member of the data engineering team. Develop and implement ETL/ELT processes using PySpark and Spark to move and transform data from diverse sources. Assist in designing and deploying solutions across major cloud platforms like AWS, Azure, or GCP. Support the maintenance of Big Data processing frameworks and data lakes for structured and unstructured data.

Collaborate with data scientists, analysts, and other engineers to ensure data accuracy and availability. Implement secure and efficient data ingestion strategies. Work on real-time streaming and batch data processing for high-volume workloads. Develop reusable code for ETL operations and contribute to performance tuning of Spark jobs. Assist in maintaining governance and data security practices on cloud platforms.

A minimum of 3 years of experience is required for this role. You must possess a strong proficiency in Databricks Unified Data Analytics Platform. Experience with major cloud platforms such as AWS, Azure, or GCP for data engineering is essential. Strong coding skills in Python, coupled with proficiency in PySpark or Spark frameworks, are crucial for building scalable data pipelines.

Understanding of Big Data architectures, data storage, and processing concepts is necessary. Familiarity with cloud-native data storage solutions (e.g., S3, Blob Storage, BigQuery, Redshift) and data orchestration tools like Apache Airflow is beneficial. Knowledge of data formats such as Parquet, Avro, or JSON, along with strong SQL and database technology skills, is expected. Excellent troubleshooting, debugging, and performance optimization abilities are required.