Join Iris Software, recognized as one of India's Top 25 Best Workplaces in the IT industry, and contribute to your most impactful career work. We are a rapidly expanding IT services company committed to being a trusted technology partner and empowering professionals to achieve their full potential.

With a global presence across India, the U.S.A., and Canada, Iris Software enables enterprise clients in financial services, healthcare, transportation & logistics, and professional services to drive technology-enabled transformations. Our expertise spans complex, mission-critical applications utilizing cutting-edge technologies in Application & Product Engineering, Data & Analytics, Cloud, DevOps, Data & MLOps, Quality Engineering, and Business Automation.

As a Senior Data Engineer, you will be instrumental in designing scalable data engineering solutions using PySpark and advanced distributed data processing frameworks. You will define robust data ingestion, transformation, and processing architectures aligned with business objectives. Responsibilities include designing and optimizing Snowflake or Delta Lake on Databricks solutions, leading the implementation of high-performance batch and streaming data pipelines, and architecting event-driven data architectures with technologies like Apache Kafka or Amazon Kinesis.

Further duties involve defining data streaming standards, integration frameworks, and scalable processing patterns, along with architecting workflow orchestration solutions using Apache Airflow or Databricks Workflows. You will establish comprehensive monitoring, scheduling, and operational controls to ensure reliable pipeline execution. Driving data quality, validation, reconciliation, and governance practices is crucial, as is designing solutions that adhere to modern Lakehouse architecture principles, data observability practices, and platform engineering standards.

Additionally, you will drive the development of business-focused data products by enhancing data quality, discoverability, usability, documentation, and trusted data consumption. Promoting the responsible use of AI-assisted engineering capabilities to boost development productivity, testing, documentation, and overall engineering quality is key. You will review data pipeline designs and implementations to ensure adherence to engineering, scalability, and performance standards, and troubleshoot complex data processing, workflow, and streaming platform issues through detailed root cause analysis. Mentoring team members on relevant technologies and best practices, and collaborating with various teams and stakeholders for end-to-end data platform delivery are also core responsibilities.

This role requires 7-8 years of experience in data engineering, with mandatory expertise in Amazon Kinesis, Apache Spark, Data Quality & Validation, PySpark, SQL, Apache Airflow, and Delta Lake on Databricks. Familiarity with Python and basic DevOps & CI/CD principles is beneficial.

Key behavioral competencies include demonstrating strong ownership, effective collaboration with diverse teams and stakeholders, and a commitment to quality-focused engineering through proactive validation and continuous improvement. Strong analytical thinking is required to tackle complex data engineering and platform challenges. Adaptability in managing evolving technologies, data ecosystems, and business requirements is essential, alongside effective communication regarding delivery status, risks, dependencies, and improvement opportunities.

Attention to detail across data architecture, pipeline design, testing, and implementation activities is paramount. You should encourage continuous improvement in data engineering practices and platform operations, support knowledge sharing and mentoring, and balance scalability, performance, reliability, and business priorities. Promoting innovation by adopting modern data engineering practices, platform engineering principles, and AI-assisted development approaches to enhance engineering productivity and solution quality is also expected.