Software Development Engineer - Agentic Data Infrastructure, Partner Data And Analytics
Amazon
Amazon
Join the Data Services team within PDNA (Partner Data And Analytics) to build and manage the Agentic Data Platform. This platform is crucial for Amazon's Selling Partner analytics, serving over 900,000 Selling Partners and 30,000 Amazonians. We handle more than 4,000 standardized metrics and real-time streaming pipelines, validated at peak Black Friday scale. Our Curated Data Layer (CDL) is Amazon's strategic infrastructure for Selling Partner GenAI applications, including Seller Assistant and Supply Chain Expert. We are pioneering a future where intelligent agents autonomously manage data pipelines and deliver insights at scale.
Your role will involve constructing the essential infrastructure layer that bridges raw data and intelligent agents. Consider this the operating system for agentic data, encompassing MCP servers, query engines, schema contracts, streaming backbones, and validation services. This position is focused on systems engineering for agentic AI, building the foundational infrastructure that empowers these advanced agents.
Design and construct MCP servers and data access APIs. Engineer robust real-time streaming backbones using technologies like Spark Structured Streaming, Kafka/MSK, Kinesis, and Lambda. Develop agent orchestration infrastructure, including tool registries, execution planners, state stores, and handoff protocols for seamless agent coordination. Build trust and validation infrastructure to ensure the reliability of agent-generated outputs. Extend the lakehouse for agentic consumption by creating metadata contracts, schema registries, and catalog APIs on top of our Iceberg-based infrastructure. Take ownership of your systems end-to-end: design, build, test, deploy, operate, and document. Ensure your infrastructure is observable, self-healing, and resilient. Participate in on-call rotations, drive root-cause analysis, and permanently resolve operational issues.
A minimum of 3 years of experience with big data technologies such as AWS, Hadoop, Spark, Pig, Hive, Lucene/SOLR, or Storm/Samza. Possession of a Bachelor's degree in Computer Science, Computer Engineering, or equivalent practical experience. Demonstrated proficiency in at least one systems programming language, including Java, Python, Kotlin, TypeScript, Scala, or Rust. Proven experience in designing and operating production services, including API design, understanding of distributed system patterns (consistency, partitioning, fault tolerance), and service-oriented architectures. Strong foundational knowledge in data structures, algorithms, concurrency, and system design principles. Experience with data infrastructure components like lakehouse architectures (Iceberg), query engines (Athena, Trino), or metadata/catalog systems.
Amazon
Software Development