Fresh Data for AI Agents with David Yaffe of Estuary

  • Jordi Torras
  • Podcast
  • English

What happens when an AI agent has a good model and an outdated view of the business? In this episode of the Torras AI Podcast, I speak with David Yaffe, co-founder and CEO of Estuary, about the data infrastructure behind useful AI. We discuss how quickly data needs to move, how to preserve its meaning as it moves, and how to give agents access to information while protecting production systems.

David co-founded Arbor, which was acquired by LiveRamp, and later served as LiveRamp's COO. His earlier product roles included Invite Media and DoubleClick Bid Manager. At Estuary, he works on connecting the systems where data originates with the systems that need to use it.

Watch the conversation

From advertising data to Estuary

David traces Estuary's origins to a practical problem at Arbor. In advertising, the value of a signal can change quickly. Someone considering a purchase may make a decision within minutes, so information that arrives much later can miss the opportunity to help. That led the team to build streaming infrastructure and an open-source system called Gazette, which became a foundation for Estuary.

We talk about making that infrastructure easier to use. Teams need to connect sources and destinations, keep data moving, and recover when something goes wrong. David describes Estuary's aim as making streaming pipelines accessible without requiring every customer to manage the underlying complexity.

Choosing the right time for the data

David explains Estuary's use of the phrase “right-time data.” The required speed depends on the work the data supports. An analytics team may want a consolidated view of a customer or a business. An application reacting to a new event may need the change immediately.

His example is a user receiving access to a new tenant. An event-driven application can act on that change as it happens. For another task, the useful representation may be an updated record that combines the user's current permissions. Choosing the timing also means choosing how consumers will interpret the data.

Freshness and governance both matter for agents

An agent acting for a business needs current information and appropriate access. David describes freshness and governance as separate concerns that both need attention. Fresh data can make an agent useful in a customer interaction, while excessive permissions can let it damage the systems it touches.

We also discuss data quality at the source. Analytics teams have long used transformations and models to work around imperfect source data. David argues that those workarounds become harder when an agent must take operational action quickly. Teams need to understand which records are reliable enough to drive a decision.

A transaction has to arrive with its meaning intact

One of the most concrete examples concerns a database transaction that updates two related tables. A new user and a new tenant might be created together. If downstream consumers see one update before the other, they can observe a partial state that never existed as a committed state at the source.

David explains how Estuary's newer runtime addresses the delivery of related changes together. He also describes buffering and compressing data on disk to support destinations that update less frequently. These are his account of the platform's design and capabilities, illustrated through the trade-offs we discuss in the interview.

Why agents should query a copy of production data

An agent exploring an unfamiliar database can issue many queries while working out how to answer a question. That exploration can add load to a production application. With broader permissions, mistakes can also affect the underlying data.

For analytics questions, David recommends giving the agent access to an appropriate replica or warehouse. For operational actions, he describes a more deliberate pipeline that reacts to relevant events. We explore how to keep the information useful while controlling the effect the agent can have on the application.

Reacting to events inside the pipeline

We then consider agents that respond to events as data moves. David uses suspicious account activity as an example. A new signup followed by unusually rapid API exploration could be a signal for a system to investigate or intervene. In that setting, response time has a direct connection to what the system can accomplish.

The example helps explain why some applications benefit from a continuous flow of events. The design still has to account for the action being taken and the consequences of an incorrect decision.

Building a connector is the beginning of the work

AI makes it easier to get an initial integration running. David points out that operating a pipeline also means dealing with schema changes, API changes, and failures over time. In his experience, maintenance accounts for much of the work.

We discuss how that changes the decision to build or buy. A small experiment may justify a simple pipeline built by the team. A business that depends on a continuous, reliable flow of data has to account for the ongoing engineering work as well as the initial implementation.

Data engineers and business value

David is cautious about predicting where AI will be several years from now. His current observation is that automation can let people spend more time on the business problems their systems serve. For data engineers, that includes checking accuracy, understanding what the business needs, and making decisions about the information that reaches it.

Our conversation returns to a practical question for anyone building with AI. What does the system need to know, how current must that knowledge be, and what should it be allowed to do with it?

He also points out a cost that can be easy to overlook. Reviewing agents' work creates its own burden, and he observes that people may be working harder as their responsibilities change. The cognitive load of checking the output remains part of the job.

Explore further

Learn more about Estuary, read David's articles, or explore the open-source Gazette project. David also mentions Estuary's public Slack community, accessible through its website.

Make AI work for you

Empower your vision with our expertise. Me and my team specialize in turning concepts into reality, delivering tailored solutions that redefine what's possible. Let's unlock the full potential of AI. Effectively.

Contact us