Researchers own agent quality at Nauta. They sit inside the Head of FDE & Research’s team, next to the FDE: not in a separate org, which is what turns daily field feedback into a conversation instead of a ticket queue. Their job is to define what "good" looks like for every agent and for Hodor’s output, and to close the gap between what breaks in the field and what ships in the product.
This is applied, evaluation-driven work: the FDE’s daily feedback becomes evals, error analysis, and systematic fixes. Researchers don’t build the underlying infrastructure: the context brain is fuel they use and feed, not their main occupation. Only repeated agent patterns reach them; one-off platform bugs, document issues, and outages route elsewhere.
What you’ll do
- Turn daily field feedback from the FDE into evals, error analysis, and systematic fixes: one team, one standup, not a ticket queue
- Define "good": set and maintain the quality bar for agent behavior and for Hodor’s document-extraction output
- Build and maintain the eval suite that governs whether an agent’s behavior is acceptable: using tools like LangSmith or equivalent to trace, evaluate, and monitor AI workflows in production
- Investigate agent failures with real rigor: noisy inputs, edge cases, and incomplete data, and distinguish a genuine defect from an operational issue that belongs elsewhere
- Feed the context brain with what you learn, and pull from it to diagnose why an agent’s output looks wrong
- Convert repeated patterns across clients into specs for Product: quality is a gate, not a negotiation: when an eval regresses below the bar, the fix jumps the queue automatically
Who you are
- Strong foundation in evaluating and debugging LLM-powered systems in production: not just building them
- Comfortable with context engineering: prompt design, prompt pipelines, context window management, and tool/function calling
- Experience with the LangChain ecosystem (LangChain, LangGraph, LangSmith) or equivalent tooling for tracing and evaluating AI workflows
- Comfortable working with noisy, real-world, unstructured or semi-structured data: you don’t need a clean dataset to find the bug
- Strong Python proficiency and solid technical foundations
- High ownership and low ego: you track down the root cause of an agent failure and fix the systemic issue, not just the symptom
Nice to have
- Experience with document understanding workflows: classification, extraction, validation, structured data generation
- Background in data engineering or data pipelines: useful for diagnosing extraction issues at the source
- Interest in logistics, supply chain, or operations
About Nauta
Nauta is your AI partner, powered by an AI operational brain. Nauta reads how your company works: your email, your documents, your rules. Then AI agents do the daily work, like reading every document, checking invoices and chasing what’s missing. You stay in control.