Software Development Engineer - Agentic Data Infrastructure, Partner Data And Analytics

Amazon

3+ yrs Chennai Full Time Work from office
Amazon logo
Posted : today
Actively hiring

Job description

Join the Data Services team within PDNA (Partner Data And Analytics) to build and manage the Agentic Data Platform. This platform is crucial for Amazon's Selling Partner analytics, serving over 900,000 Selling Partners and 30,000 Amazonians. We handle more than 4,000 standardized metrics and real-time streaming pipelines, validated at peak Black Friday scale. Our Curated Data Layer (CDL) is Amazon's strategic infrastructure for Selling Partner GenAI applications, including Seller Assistant and Supply Chain Expert. We are pioneering a future where intelligent agents autonomously manage data pipelines and deliver insights at scale.

Your role will involve constructing the essential infrastructure layer that bridges raw data and intelligent agents. Consider this the operating system for agentic data, encompassing MCP servers, query engines, schema contracts, streaming backbones, and validation services. This position is focused on systems engineering for agentic AI, building the foundational infrastructure that empowers these advanced agents.

Responsibilities

Design and construct MCP servers and data access APIs. Engineer robust real-time streaming backbones using technologies like Spark Structured Streaming, Kafka/MSK, Kinesis, and Lambda. Develop agent orchestration infrastructure, including tool registries, execution planners, state stores, and handoff protocols for seamless agent coordination. Build trust and validation infrastructure to ensure the reliability of agent-generated outputs. Extend the lakehouse for agentic consumption by creating metadata contracts, schema registries, and catalog APIs on top of our Iceberg-based infrastructure. Take ownership of your systems end-to-end: design, build, test, deploy, operate, and document. Ensure your infrastructure is observable, self-healing, and resilient. Participate in on-call rotations, drive root-cause analysis, and permanently resolve operational issues.

Qualifications

A minimum of 3 years of experience with big data technologies such as AWS, Hadoop, Spark, Pig, Hive, Lucene/SOLR, or Storm/Samza. Possession of a Bachelor's degree in Computer Science, Computer Engineering, or equivalent practical experience. Demonstrated proficiency in at least one systems programming language, including Java, Python, Kotlin, TypeScript, Scala, or Rust. Proven experience in designing and operating production services, including API design, understanding of distributed system patterns (consistency, partitioning, fault tolerance), and service-oriented architectures. Strong foundational knowledge in data structures, algorithms, concurrency, and system design principles. Experience with data infrastructure components like lakehouse architectures (Iceberg), query engines (Athena, Trino), or metadata/catalog systems.

Essential Skills

AWSHadoopSparkPigHiveLucene/SOLRStorm/SamzaJavaPythonKotlinTypeScriptScalaRustAPI designdistributed system patternsservice-oriented architecturesdata structuresalgorithmsconcurrencysystem designlakehouse architecturesIcebergquery enginesAthenaTrinometadata/catalog systemsSpark Structured StreamingKafka/MSKKinesisLambda

Good to Have

agentic AI infrastructuretool registriesagent orchestrationLLM integrationagent state managementGenAI infrastructure patternsRAGembedding storesevaluation frameworks

Highlights

  • Actively hiring

More Details

RoleSoftware Development Engineer - Agentic Data Infrastructure, Partner Data And Analytics
IndustrySoftware Development
Employment TypeFull Time, Work from office

About the Company

Amazon logo

Amazon

Software Development