Senior Test Specialist (IN)
HCLTech
HCLTech
Drive the creation and deployment of comprehensive data pipelines on cloud environments, with a preference for GCP. This role involves constructing efficient data ingestion, transformation, and processing workflows utilizing distributed technologies such as Spark, Flink, or Storm.
Develop advanced ELT/ETL solutions and migration strategies, including the modernization of legacy Datastage systems. Collaborate with diverse database technologies, encompassing relational, NoSQL, MPP, and columnar stores like BigQuery, Redshift, and MongoDB.
Implement real-time streaming and messaging pipelines using Kafka, Pulsar, or Pub/Sub. Design optimized and scalable data models tailored for various consumption needs, incorporating strategies like partitioning, sharding, bucketing, and aggregation. Focus on enhancing performance across storage, compute, and query layers.
Design and implement end-to-end data pipelines on cloud platforms, prioritizing GCP. Build scalable data ingestion, transformation, and processing workflows using distributed technologies (e.g., Spark, Flink, Storm). Develop robust ELT/ETL processing and migration pipelines, including modernization of legacy Datastage. Work with a variety of database technologies: relational, NoSQL, MPP, and columnar stores (e.g., BigQuery, Redshift, MongoDB). Implement streaming and messaging-based pipelines using Kafka, Pulsar, or Pub/Sub. Create optimized, scalable data models for diverse consumption patterns, applying partitioning, sharding, bucketing, and aggregation. Apply performance tuning and optimization across storage, compute, and query layers.
Proven experience in designing and delivering end-to-end data pipelines on cloud platforms, particularly GCP. Proficiency in building scalable data ingestion, transformation, and processing workflows using distributed technologies like Spark, Flink, or Storm. Experience developing robust ELT/ETL processing and migration pipelines, with an understanding of legacy system modernization. Familiarity with a broad range of database technologies, including relational, NoSQL, MPP, and columnar stores (e.g., BigQuery, Redshift, Azure SQLDW, HBase, MongoDB). Skilled in implementing streaming and messaging-based pipelines using Kafka, Pulsar, or Pub/Sub. Ability to build optimized, scalable data models and apply performance tuning across various layers.
HCLTech
IT Consulting