Seeking a Senior Data Engineer Consultant to architect and implement advanced data solutions. This role focuses on building robust data ingestion pipelines and leveraging cutting-edge NLP techniques to extract valuable information from diverse data sources.

Join our team to drive innovation in data processing and contribute to sophisticated analytics and search capabilities.

Develop and optimize scalable data ingestion pipelines using Python and PySpark within Databricks. Implement sophisticated tools like Unstructured.io and LlamaIndex for accurate text and table extraction from PDF documents. Construct high-accuracy Named Entity Recognition (NER) layers utilizing both open-source frameworks and cloud foundation models. Design, partition, and maintain optimal Delta Lake schemas for downstream analytics, reporting, and vector search indexing.

Proven experience with Apache Spark/PySpark engineering on Databricks and Delta Lake architecture. Expertise in Python-based ETL/ELT and scalable batch ingestion processes. Proficiency in PDF parsing and layout-aware extraction using Unstructured.io, LlamaIndex, and pdfplumber. Strong background in NLP/NER pipelines with spaCy, Hugging Face Transformers, and LayoutLM. Experience integrating with AWS Bedrock foundation models is essential.