I Built a Knowledge Graph for Everything I Work On

Too many AI coding sessions start with the same dumb scavenger hunt. The agent reads the project instructions, searches for entry points, asks what an internal service name means, and slowly reconstructs decisions another agent understood yesterday.

The model isn’t forgetting. The system around it never kept the useful parts.

I built Scry because I was tired of paying for that reconstruction. It started as a fast code index. It could answer where a symbol was defined and who called it without making an agent run grep, open five files, and throw most of the result away.

That fixed one piece of the problem. Code is only one kind of knowledge.

A real project also has git history, database relationships, observed HTTP traffic, test coverage, architecture decisions, deployment state, and months of conversations with coding agents. Those things explain each other. Keeping them in separate tools meant the agent still had to do the joining itself.

Scry now puts those relationships behind one local service. I want the next agent to start with what the last one already figured out.

The knowledge projects leave behind

Most project knowledge never appears in a README. It is spread across places that answer different questions:

• Source code says what the program currently does.
• Git history says when it changed, who changed it, and what else moved with it.
• The database schema says what data can exist and how records relate.
• HTTP traffic says what the running application did, including requests that never matched the intended design.
• Coverage data says whether a function was exercised, not merely whether a test file exists nearby.
• Agent sessions contain the reasons, rejected options, machine names, operational incidents, and decisions that didn’t belong in code.

I originally built separate tools for several of these. Scry handled code. Tome handled schemas. Lore handled git. Flume captured HTTP traffic. They shared nearly all of their infrastructure: Go, Cobra, BadgerDB, a Unix socket, JSON-RPC, MCP, daemon lifecycle, and setup code.

Running four daemons to understand one project got old fast. It also forced the agent to know which tool owned each question, which is exactly the sort of bookkeeping the tool should handle. I folded them into Scry and added a graph over the domains.

Today one MCP server exposes 37 tools across code, git, schema, HTTP, graph, memory, and fleet rooms. I don’t care much about the tool count. I care that an agent doesn’t have to remember four product names before it can debug one request.

Two graphs, because the facts are different

Scry doesn’t put everything into one undifferentiated graph.

Each repository has a deterministic graph built from things a parser or local process can establish. Functions call other functions. Classes implement interfaces. Tables have foreign keys. A commit touches a migration. An observed endpoint was served by a handler. These facts come from SCIP indexes, git, database introspection, coverage files, and captured traffic.

There is no model guessing that OrderService.checkout calls PaymentGateway.charge. Either the index contains that edge or it doesn’t.

The global memory graph is different. It is built from past Claude, Codex, and autonomous-run transcripts. A small extraction model turns those conversations into episodes, entities, and time-stamped facts. That graph knows that a service moved to another machine, a particular architecture was chosen, or a project is blocked on a decision.

Those facts are useful, but they are inferred. Scry stores confidence and provenance with them. The agent can see the session that produced a fact and check the source when it matters.

Keeping the graphs separate makes the trust boundary obvious:

code + git + schema + HTTP + coverage
                  |
                  v
       deterministic repo graph

sessions + decisions + project state
                  |
                  v
          temporal memory graph

The memory graph links back to repository paths. That gives an agent a practical bridge: recall the project or decision globally, then pivot into the local repository graph for code-level detail.

What the graph changes

Without the graph, a question such as “why does this service run on the mini instead of the inference box?” isn’t really a code question. The answer may live in a session from three weeks ago about shell dependencies, resource contention, and a failed migration.

Scry can recall the service, the machines, and the decision. It returns the current facts plus the episodes that support them. The agent starts from the previous conclusion instead of inventing a new one.

A repository question works the same way at a smaller scale. “What connects this endpoint to this table?” can traverse an observed route, its handler, a call edge, and a database relationship. The agent may still open the handler before editing it, but it knows which handler and why.

This is the part I care about. The model may still make a bad edit, but at least it starts in the right handler with the old decision in view.

Time belongs in the data model

Project facts change. A service moves. A branch merges. A product name changes. A decision gets reversed after new evidence shows up.

Flat memory files are bad at this. The latest note overwrites the old one, or both statements survive and the next agent gets to choose which paragraph looks newer.

Scry facts have valid_from and invalid_at timestamps. When a new exclusive fact supersedes an old one, the old edge is invalidated rather than deleted. Normal recall returns the current state. An as_of query can reconstruct what the graph believed at an earlier point.

Most days this is boring database housekeeping. During an outage, it lets me ask what was deployed last month without treating an old fact as the current one.

The graph is local infrastructure

I didn’t want another hosted dashboard or a vector database that shipped every source file and transcript somewhere else.

Repository indexes live under ~/.scry/repos/ on the machine holding the code. The daemon communicates over a Unix socket. The memory domain is opt-in, redacts obvious secrets before extraction, and keeps its graph in BadgerDB.

My authoritative memory graph runs on a small machine that is always on. Development machines reach it through an SSH StreamLocalForward to another Unix socket. Code, schema, git, HTTP, and repository graph queries stay local. Memory is shared.

It isn’t a general distributed graph database. I don’t need one. I need the laptop and the agents running on it to agree about what “Hermes” means without putting a TCP port on the network.

What is still missing

The bridge between the global memory graph and repository graphs is currently explicit. A memory entity carries repository references. The agent recalls the entity, sees the relevant repo, and runs graph queries there.

Scry can’t yet answer one native path query that starts at “the decision to move this service” and ends at the exact function and deployment file changed because of it. The data model has the join points, but the traversal still crosses two stores.

That cross-store traversal is the next part I want to build. The agent should be able to start at a deployment decision and land on the function and config file that implemented it.

The older version of Scry made code lookup faster. The current version also keeps project context around between sessions, including details I have forgotten. The cross-graph join is still manual, and that is where the code is today.