Technical Lead (IN)
HCLTech
HCLTech
Seeking an experienced Data Engineer to lead the migration from legacy ETL/iPaaS tools like SnapLogic and Datastage to a modern ELT architecture leveraging Fivetran. This role involves designing and implementing automated data ingestion and staging processes, ensuring robust data validation, and building custom solutions for unsupported applications.
You will be instrumental in architecting the target ELT state within Fivetran, guaranteeing seamless data flow and integrity throughout the transition from existing systems. A deep understanding of Fivetran's capabilities and connector ecosystem is essential for success.
Design and implement the target ELT state using Fivetran for automated data ingestion and staging. Develop and manage custom Fivetran extensions using Python, REST APIs, and Cloud Functions for legacy applications. Conduct comprehensive audits of existing ETL/iPaaS configurations to map data sources, targets, and transformations. Establish strict data validation protocols for parallel testing and to ensure 100% data parity. Oversee the historical data backfill process, addressing challenges like API rate limits and large volume sync errors.
Demonstrated expertise in data engineering with a strong track record of migrating from legacy ETL/iPaaS tools (SnapLogic, Datastage) to modern ELT stacks. Proficiency in Fivetran, including a solid understanding of its architecture and various source connectors. Experience in building custom extensions using Python, REST APIs, and Cloud Functions. Ability to design and implement robust data validation and testing frameworks to prevent data loss. Skills in troubleshooting issues related to API rate limits, large volume sync errors, and network bottlenecks.
HCLTech
IT Consulting