What does an AI engineer do? The day-to-day work

What does an AI engineer do? Agent tools, evals, guardrails, cost, data pipelines and deployment, how it differs from ML engineering, with real examples.

An AI engineer builds software that uses pretrained language models (like OpenAI's and Anthropic's) to do real work: answer from documents, query data, take actions in business systems. They don't train the model. They decide what the model can see and do, measure whether the output is right, and keep the whole thing running in production with cost and latency under control.

I'm an AI engineer and the CTO of DockPlus AI. Here is what the job actually involves, with examples from my own projects.

AI engineer vs. ML engineer, data scientist and software engineer

The four roles overlap, but each one has a different center.

• Data scientists answer questions with data: analysis, experiments, statistical models. The output is usually a finding, or a model someone else ships.
• ML engineers train, fine-tune and serve models. They own datasets, training pipelines, features, model deployment and drift monitoring.
• Software engineers build deterministic systems: the same input gives the same output, and a test tells you whether it's correct.
• AI engineers start from a model that's already trained and build the system around it. The catch is that the output isn't deterministic. The same question can produce two answers, and one of them can be confidently wrong.

In practice, an AI engineer is a software engineer who learned to work with a component that is sometimes wrong. Most of the job is setting limits, measuring, and giving the model the right context.

What does an AI engineer do day to day?

The work splits into a few tracks that show up in almost every project.

Designing tools for the agent

An agent is a model that can call functions. Designing those functions is a big part of the job: name, parameters, description, what it returns and what it must never do.

In LedgerJob, a SaaS for US construction contractors, the agent has 66 tools covering leads, quotes, projects, invoices and scheduling. One list backs both the in-app chat and an MCP server, so a new tool ships to both in the same deploy. Each tool's description is written for the model, and it matters as much as the code.

Setting guardrails

The model will try things it shouldn't. The AI engineer decides where the wall goes.

In LedgerJob, 16 of the 66 tools require human confirmation. Sending email or SMS, spending money with third parties or deleting data never runs inside the agent loop: it becomes a card the user has to approve. Identity comes from the API key, never from the model, so the agent can't act as another user.

In Achado de Leilão, a search engine for official auctions in Brazil, the MCP server includes a free-form SQL tool. The limits live in the database and the server, not in the prompt: a read-only session, a 15-second timeout and a 200-row cap per query.

In both cases the guardrail doesn't depend on the model following an instruction. It's enforced in code or in the database.

Managing cost and latency

Every model call costs tokens and time. An agent loop with no ceiling can burn a lot on a single question.

LedgerJob's loop is capped at 8 iterations per turn, and each tool result is truncated at 6,000 characters. A long invoice listing can't blow up the context or the bill. They're small numbers in the code, but they're the kind of decision that separates a prototype from a system people use every day.

Evaluating and observing

Without measurement you can't tell whether a prompt or model change made things better or worse. At DockPlus I use Langfuse to track latency, token cost and response quality.

On a RAG system over video lessons that I built for an online school, every retrieval stage (HyDE hypothetical answer, hybrid search, rerank, transcription) is logged as a Langfuse span. When an answer comes out bad, you can see which stage it started in. The same system flags the answer as low confidence when the best search score falls below 0.65, instead of answering with confidence it doesn't have.

Automated tests belong here too. The LedgerJob backend has 59 test files with 347 declared test cases. Business rules live in the service layer, not the HTTP route, because the agent and the MCP server skip the route validators. That way a rule holds on every path.

Preparing the data

An agent can only answer well about what it can look up. Often the project starts before the model, with a data pipeline.

In Cape Cod Permits, Massachusetts building permits are spread across six municipal systems with no common API. I wrote one collector per platform, all feeding a PostgreSQL database that now covers 74 towns. The MCP server is read-only and ships with a dataset guide the agent reads before analyzing, because fields aren't standardized across towns and a naive comparison leads to the wrong conclusion.

Integrating and shipping to production

The model is rarely the hard part. The hard part is wiring it into what the customer already runs. LedgerJob talks to Stripe, QuickBooks Online, Twilio, Google Calendar, EagleView, Hover and CompanyCam. Each integration has its own auth, rate limits and ways of failing.

Then comes deployment: containers, databases, queues, logs, alerts. I've worked in infrastructure since 2005, with Linux, high-availability databases and Ansible. None of that shows up in a demo, but it's what keeps an agent standing after the first month.

How I work: Claude Code and specs

I build with Claude Code. I design the system (in LedgerJob, the backend, the agent and the MCP server) and Claude Code writes the code with me. In AgentLab, my study lab for agent architectures, development started from 12 per-feature specs written before any code.

That changes an AI engineer's day in an interesting way: I use agents to build agents. The same questions apply on both sides. What can the agent do on its own, what needs review, and how do you know the result is right?

How to become an AI engineer: what to learn

There's no single path. This is the one I'd recommend, based on what I use every day.

1. Real programming. TypeScript or Python, well. REST APIs, databases, tests. Without this, nothing else holds up.
2. Model APIs. Tool calling, streaming, structured output, context limits. Build a small agent from scratch, with no framework, to understand the loop.
3. RAG. Embeddings, chunking, vector search with pgvector, hybrid search and reranking. Learn to say "I don't know" when retrieval finds nothing good.
4. MCP. The Model Context Protocol is an open protocol for exposing tools and data to agents, and clients like Claude already speak it. Writing a small MCP server teaches you a lot about tool design.
5. Evals and observability. Traces, cost per call, test sets for answers. Langfuse is a good place to start.
6. Frameworks, later. LangGraph and LangChain help when a flow has several stateful steps. Understanding the loop without them first makes you use them better.
7. Production. Docker, Linux, PostgreSQL, queues and logs. This is where a lot of AI projects stall.

Building one complete project teaches more than any course. Pick a real problem, ship it and measure it.

Summary

What an AI engineer does, in one line: turns a model that is sometimes wrong into a system you can trust. That means designing tools, putting limits outside the prompt, controlling cost, measuring quality, preparing data and running it all in production.

See my projects, my background on the about page, or my technical profile.

Hiring, or have a project in mind? get in touch