Reflections on Hierarchical Memory and Local Contexts in LLMs with Self-Organizing AGENTS.md

Modern LLM agents often operate in complex project structures, where managing knowledge locally can become challenging. The AGENTS.md files, while simple in format, can serve as a form of local, organized context. They allow an LLM to reason consciously and reliably within the boundaries of a specific directory, while also supporting hierarchical structuring of knowledge across a project.

This article explores how such files can be thought of as layers of memory: upper levels providing general guidance, lower levels detailing specific operational rules, and the current directory capturing the immediate context. These are reflections and observations on how hierarchical contexts might support reasoning, learning, and the stabilization of experience for an LLM.

Creating AGENTS.md for the First Time

When an LLM first enters a directory, it performs a careful assessment of its contents without descending into subdirectories. From this initial observation, it forms hypotheses: What is the purpose of this area? What invariants are important? Which constraints must be respected? These insights are captured in an AGENTS.md file.

For example, in an authentication module, the file might look like:

# Context Hypothesis
This directory handles authentication logic, focusing on token lifecycle.
Do not hardcode secrets.
Changes here affect security boundaries.

Notice that the file focuses on the local area only — it does not attempt to capture the inner workings of subdirectories. Creating AGENTS.md is an act of first awareness: the LLM is effectively saying, “I understand the boundaries of this local area.”

Locality and Boundaries

Each AGENTS.md file is deliberately confined to its local area. The agent should not “peek” into subdirectories unless explicitly required, which preserves focus and prevents unnecessary context overload. This mirrors human cognition: when a person is working in a kitchen, they think about cooking, not the contents of every pot simultaneously.

Locality ensures that the agent’s attention is tightly coupled with the area where reasoning is actively needed. Every subdirectory has its own AGENTS.md, capturing its own context, while the parent directory provides broader guidelines.

The Hierarchy of AGENTS.md

The real power of AGENTS.md emerges when organized hierarchically. Upper-level files define project-wide rules, lower-level files refine these into area-specific guidelines, and the current directory’s file handles detailed operational instructions. This hierarchy achieves multiple goals simultaneously: it allows inheritance of rules, isolates changes, enables progressive refinement, and controls updates in a disciplined manner.

When an LLM operates in a nested directory, it constructs its effective context by merging files from the root down to the current directory:

AGENTS_context = merge(root/AGENTS.md, parent/AGENTS.md, current/AGENTS.md)

Conflicts are resolved in favor of the most local file: details and exceptions override global rules. Importantly, the agent never loads AGENTS.md from sibling directories, preserving the integrity of each local context.

Updating AGENTS.md

AGENTS.md are not updated on every file change. They respond only to meaningful signals: repeated friction, confirmed external validation (via tests, code reviews, or human guidance), and stabilized lessons. Each update should reduce cognitive load for the next interaction with this directory.

A practical structure might separate the current context from a short historical block of lessons learned:

# Current Context
This directory handles authentication logic. Focus on token lifecycle.

# Important Lessons (optional)
- Do not modify token expiration rules without updating tests.
- Always follow OAuth2 compliance. Learned from repeated issues.

The “Important Lessons” section captures only a small portion of past experience — enough to stabilize behavior — without overloading the agent with unnecessary history. Updates are applied only when experience has repeated or been confirmed, ensuring the AGENTS.md reflects stabilized knowledge rather than transient observations.

Managing Context Size

In real systems, hierarchical AGENTS.md may collectively exceed the LLM’s context limit. To address this, several strategies are employed:

• Local file is always fully loaded.
• Upper-level files are condensed into summaries of essential principles.
• Lower levels are ignored.

Advanced systems may implement chunking, retrieval-augmented strategies, or archive older rules to maintain both scalability and reasoning quality. The guiding principle remains: only the context relevant to the current task should be in memory.

Lifecycle of AGENTS.md

The life of an AGENTS.md file mirrors a cognitive process:

1. Emergence: Upon entering a directory, the agent identifies boundaries and constraints.
2. Evolution: Repeated friction or errors lead to updates capturing stabilized lessons.
3. Experience transfer: Confirmed, repeatable lessons may be promoted to parent directories.
4. Forgetting: Deleting a directory invalidates its AGENTS.md; the local context ceases to exist.

This lifecycle ensures that the agent maintains relevant, actionable knowledge while discarding obsolete context.

Transferring Experience Upwards

Not all lessons remain local. When a pattern or principle proves robust across multiple sub-areas, it can be elevated to a higher-level AGENTS.md, making it available to other areas. Transfer is done cautiously:

• Only repeated, validated experiences are promoted.
• Only invariant principles are moved — local exceptions remain in the original AGENTS.md.
• Promotion stabilizes knowledge without introducing noise into the global context.

For instance:

/AGENTS.md
- Always validate user input before database queries.
- Maintain OAuth2 compliance in all modules.

/backend/auth/AGENTS.md
- Focus on token lifecycle
- Do not hardcode secrets

Here, the validation and OAuth2 rules can live at the root level, becoming global invariants, while token lifecycle details stay local to authentication.

AGENTS.md as a System Context

Conceptually, AGENTS.md function like a dynamic system prompt, but with crucial differences:

• They are activated locally, not globally.
• Upper levels provide frameworks, lower levels add refinements, current level adds specific operational details.
• They guide reasoning rather than dictate output directly.

In short, the system prompt defines personality, AGENTS.md define situation.

Using Retrieval to Access Knowledge Across Directories

In complex projects, a local AGENTS.md file may not always contain all the knowledge an agent needs. Consider the scenario where a function from another module needs to be invoked. Performing a recursive traversal of all directories to locate the relevant context would be highly inefficient and could overwhelm the model with unnecessary information.

One way to address this is by introducing a retrieval-augmented mechanism as a complementary layer. Instead of scanning all directories, the LLM can perform a targeted search to identify which other local context may be relevant to the current task. This approach can be thought of as a form of “jumping” between cognitive areas when an action or decision requires knowledge from another domain.

How it Works

1. Identify the action or goal
   For example, the agent determines it needs to call a function calculateInterest() from the /finance/utils module.
2. Determine the relevant context
   The LLM uses semantic signals, function names, tags, or metadata to locate the appropriate AGENTS.md. Instead of recursively reading all files, it queries a retrieval layer that indexes all directories with their embeddings or summaries.
3. Load and use the relevant context
   Once the relevant AGENTS.md is identified, the agent can load it along with any parent AGENTS.md if necessary. The local reasoning then proceeds as if this external context were part of the immediate workspace.

Approaches to Retrieval

• Semantic retrieval:
  The agent embeds the query (“call calculateInterest()”) and searches a vector database of AGENTS.md embeddings to find the closest match. Only the relevant context is loaded.
  Principle: RAG / embeddings
  Advantage: Minimal context, flexible
  Limitation: Requires embedding/indexing
• Signal-driven triggers:
  AGENTS.md may include signals indicating external dependencies. For example: “If interest calculation is needed → see /finance/utils.” This allows the agent to resolve dependencies without performing full searches.
  Principle: AGENTS.md explicitly reference dependencies
  Advantage: Direct and explainable
  Limitation: Requires pre-defined signals
• Lightweight registry:
  A project-wide registry maps functions or topics to their respective directories. When a specific function is needed, the agent consults the registry to locate the relevant AGENTS.md, avoiding any unnecessary traversal.
  Principle: Centralized mapping of modules/functions
  Advantage: Fast, scalable
  Limitation: Needs upkeep

This allows reasoning to remain focused and efficient, preserves local context integrity, and supports the agent in connecting knowledge across multiple areas of a project.

AGENTS.md as an Experienced Developer

Think of a developer fully immersed in a project. They know the rules, dependencies, and nuances — but it takes hours or days of experience to internalize this knowledge.

An LLM with hierarchical AGENTS.md achieves a similar level of understanding instantly:

• Structured knowledge: Key goals, invariants, lessons, and constraints are all captured in AGENTS.md.
• Immediate context loading: The agent “jumps into the project” without needing a long adaptation period.
• RAG-assisted cross-directory access: Relevant knowledge from other areas can be fetched as needed, just like an experienced developer recalling details from memory.

Comparison:

• A human developer takes weeks to understand a project, while an LLM can load structured context instantly.
• Humans may forget rarely-used details, while LLMs preserve context and can access it on demand.
• Humans can make errors due to missed dependencies; LLMs minimize mistakes using local rules and retrieval.

In essence, hierarchical AGENTS.md gives the LLM the effect of an “experienced developer in situ,” capable of reasoning and acting across the project without the cognitive overhead.

Conclusion

AGENTS.md can be seen as a way to structure local knowledge in a repository, providing hierarchical context for reasoning agents. They allow the agent to focus on immediate tasks while remaining aware of broader constraints, accumulate lessons from repeated experience, and maintain a balance between local detail and global guidance.

These thoughts are offered as reflections on possible practices for organizing knowledge and context in projects, highlighting parallels with human cognition, memory, and learning. Whether these observations prove useful in practice depends on experimentation and further exploration.