File Loading

HTM can load text-based files (currently markdown) into long-term memory with automatic chunking, source tracking, and re-sync support. This is ideal for building knowledge bases from documentation, notes, or any text content.

Overview

The file loading system provides:

• Automatic chunking: Large files are split into semantically-aware chunks
• YAML frontmatter extraction: Metadata from file headers is preserved
• Source tracking: Files are tracked for re-sync when content changes
• Duplicate detection: Content hashing prevents duplicate chunks
• Soft delete: Unloading files uses soft delete for recovery

Quick Start

require 'htm'

htm = HTM.new(robot_name: "Document Loader")

# Load a single markdown file
result = htm.load_file("docs/guide.md")
# => { file_source_id: 1, chunks_created: 5, chunks_updated: 0, skipped: false }

# Load all markdown files from a directory
results = htm.load_directory("docs/", pattern: "**/*.md")
# => [{ file_path: "docs/guide.md", ... }, { file_path: "docs/api.md", ... }]

# Query nodes from a specific file
nodes = htm.nodes_from_file("docs/guide.md")

# Unload a file (soft deletes chunks)
htm.unload_file("docs/guide.md")

API Reference

load_file(path, force: false)

Loads a single file into long-term memory.

Parameter	Type	Default	Description
path	String	required	Path to the file
force	Boolean	false	Force reload even if file unchanged

Returns: Hash with keys: - file_source_id: ID of the FileSource record - chunks_created: Number of new chunks created - chunks_updated: Number of existing chunks updated - chunks_deleted: Number of chunks removed - skipped: Whether file was skipped (unchanged)

# Normal load - skips unchanged files
result = htm.load_file("docs/guide.md")

# Force reload even if file hasn't changed
result = htm.load_file("docs/guide.md", force: true)

load_directory(path, pattern: "**/*.md", force: false)

Loads all matching files from a directory.

Parameter	Type	Default	Description
path	String	required	Directory path
pattern	String	"**/*.md"	Glob pattern for files
force	Boolean	false	Force reload all files

Returns: Array of result hashes (one per file)

# Load all markdown files
results = htm.load_directory("docs/")

# Load only top-level markdown files
results = htm.load_directory("docs/", pattern: "*.md")

# Load specific subdirectory
results = htm.load_directory("docs/guides/", pattern: "**/*.md")

nodes_from_file(path)

Returns all nodes loaded from a specific file.

nodes = htm.nodes_from_file("docs/guide.md")
nodes.each do |node|
  puts "#{node.id}: #{node.content[0..50]}..."
end

unload_file(path)

Soft deletes all nodes from a file, along with their tag associations and robot links, and removes the file source. Both steps run in one transaction, so if either fails nothing changes. Loading the same content again restores them.

count = htm.unload_file("docs/guide.md")
puts "Removed #{count} chunks"

YAML Frontmatter

Files with YAML frontmatter have their metadata extracted and stored:

---
title: PostgreSQL Guide
author: HTM Team
tags:
  - database
  - postgresql
version: 1.2
---

# PostgreSQL Guide

Content starts here...

Access frontmatter via the FileSource model:

source = HTM::Models::FileSource.first(file_path: "docs/guide.md")
source.title            # => "PostgreSQL Guide"
source.author           # => "HTM Team"
source.frontmatter_tags # => ["database", "postgresql"]
source.frontmatter      # => { "title" => "...", "author" => "...", ... }

Chunking Strategy

HTM uses the Baran gem with MarkdownSplitter for intelligent chunking that respects markdown structure:

• Headers: Chunks break at header boundaries
• Code blocks: Code blocks are kept intact
• Horizontal rules: Natural section breaks
• Configurable size: Control chunk size and overlap

Configuration

# Global configuration
HTM.configure do |config|
  config.chunking.chunk_size = 1024    # Characters per chunk (default: 1024)
  config.chunking.chunk_overlap = 64   # Overlap between chunks (default: 64)
end

# Or via environment variables
# HTM_CHUNKING__CHUNK_SIZE=512
# HTM_CHUNKING__CHUNK_OVERLAP=50

Per-Loader Configuration

loader = HTM::Loaders::MarkdownLoader.new(
  htm,
  chunk_size: 512,
  chunk_overlap: 50
)
loader.load("docs/guide.md")

Re-Sync Behavior

The file loading system tracks file modification times for efficient re-syncing:

1. First load: Creates FileSource record and chunks
2. Subsequent loads: Compares mtime, skips unchanged files
3. Changed files: Re-chunks and updates nodes
4. Force reload: Bypasses mtime check

# First load - creates chunks
htm.load_file("docs/guide.md")
# => { skipped: false, chunks_created: 5 }

# Second load - skipped (unchanged)
htm.load_file("docs/guide.md")
# => { skipped: true }

# After editing file - re-syncs
htm.load_file("docs/guide.md")
# => { skipped: false, chunks_updated: 2, chunks_created: 1 }

# Force reload
htm.load_file("docs/guide.md", force: true)
# => { skipped: false, chunks_updated: 5 }

FileSource Model

The HTM::Models::FileSource tracks loaded files:

source = HTM::Models::FileSource.first(file_path: "docs/guide.md")

source.file_path       # Full path to file
source.mtime           # Last modification time
source.needs_sync?     # Check if file changed since load
source.chunks          # Associated nodes (ordered by position)
source.frontmatter     # Parsed YAML frontmatter
source.title           # Frontmatter title (convenience)
source.author          # Frontmatter author (convenience)
source.frontmatter_tags # Tags from frontmatter

Rake Tasks

HTM provides rake tasks for file management:

# Load a single file
rake 'htm:files:load[docs/guide.md]'

# Load directory
rake 'htm:files:load_dir[docs/]'
rake 'htm:files:load_dir[docs/,**/*.md]'

# List loaded files
rake htm:files:list

# Show file details
rake 'htm:files:info[docs/guide.md]'

# Unload a file
rake 'htm:files:unload[docs/guide.md]'

# Sync all files (reload changed)
rake htm:files:sync

# Show statistics
rake htm:files:stats

# Force reload with FORCE=true
FORCE=true rake 'htm:files:load[docs/guide.md]'

Best Practices

Organize Files Logically

# Load by category
htm.load_directory("docs/guides/", pattern: "**/*.md")
htm.load_directory("docs/api/", pattern: "**/*.md")
htm.load_directory("docs/tutorials/", pattern: "**/*.md")

Use Frontmatter for Metadata

---
title: API Authentication
category: api
tags:
  - security
  - authentication
priority: high
---

Tune Chunk Size for Your Content

# Smaller chunks for dense technical content
HTM.configure { |c| c.chunking.chunk_size = 512 }

# Larger chunks for narrative content
HTM.configure { |c| c.chunking.chunk_size = 2048 }

Regular Sync for Updated Content

# Sync all loaded files periodically (via rake task)
# rake htm:files:sync  # Re-checks all FileSource records

Example: Building a Knowledge Base

require 'htm'

# Initialize
htm = HTM.new(robot_name: "Knowledge Base")

# Configure chunking for technical docs
HTM.configure do |config|
  config.chunking.chunk_size = 768
  config.chunking.chunk_overlap = 100
end

# Load documentation
htm.load_directory("docs/", pattern: "**/*.md")
htm.load_directory("README.md")
htm.load_directory("CHANGELOG.md")

# Query the knowledge base
results = htm.recall(
  "How do I configure authentication?",
  strategy: :hybrid,
  limit: 5
)

results.each do |result|
  puts result['content']
  puts "---"
end

Related Documentation

• Adding Memories - Core memory operations
• Search Strategies - Querying loaded content
• API Reference: HTM - Complete API documentation
• Example: File Loading - Working example