Building AI Agents From Scratch

Everyone is wrapping GPT-4 in a for-loop and calling it an agent. Here's what real autonomous tool-use looks like — the architecture, the failure modes, and the lessons from building Arora AI.

What an agent actually is

An agent is a system that takes a goal, breaks it into steps, executes those steps using tools, and adapts based on results. The loop is: observe → plan → act → observe again. That's it. But the details in each stage are where complexity lives.

The architecture of Arora AI

Arora uses a three-layer approach: a planning layer that decomposes goals into subtasks, a tool-use layer that executes individual actions (file reads, web searches, shell commands), and a memory layer that persists context across sessions.

The planning layer is the hardest part. LLMs are good at generating plausible-sounding plans. They're bad at generating executable ones. The key insight was adding a validation step — before executing a plan, we check each step for feasibility and surface ambiguities back to the user.

Failure modes nobody talks about

The most common failure isn't hallucination. It's over-confidence. The agent proceeds with a wrong assumption for three steps before something breaks. The fix is explicit uncertainty tracking — when confidence on a step drops below a threshold, pause and ask.

What I'd do differently

Start with memory. Everything else can be retrofitted. A stateless agent that forgets context between sessions is frustrating in a way that's hard to fix later once users have expectations set.