HTM Rake Tasks Reference¶
Complete reference for all HTM rake tasks. HTM provides 44 rake tasks organized into five namespaces for managing the database, documentation, file loading, background jobs, and tag management.
Quick Reference¶
# List all HTM tasks
rake -T htm
# Most common tasks
rake htm:db:setup # First-time database setup
rake htm:db:migrate # Run pending migrations
rake test # Run all tests
rake htm:doc:all # Generate all documentation
Integrating Tasks into Your Application¶
Add HTM rake tasks to any Ruby application with a single line:
All HTM tasks will now be available. See Using Rake Tasks in Your Application for complete integration examples.
Database Tasks (htm:db:*)¶
Database management tasks for setup, migrations, maintenance, and inspection.
Setup and Schema¶
rake htm:db:setup¶
Sets up the HTM database schema and runs all migrations. Use for first-time setup or after dropping the database.
$ rake htm:db:setup
Connecting to database...
Host: localhost:5432
Database: htm_development
Verifying extensions...
✓ pgvector 0.8.1
✓ pg_trgm 1.6
Running migrations...
✓ 00001_create_robots.rb
✓ 00002_create_file_sources.rb
✓ 00003_create_nodes.rb
...
✓ HTM database setup complete
What it does:
- Connects to PostgreSQL using
HTM_DBURL - Verifies required extensions (pgvector, pg_trgm)
- Creates all HTM tables (robots, nodes, tags, file_sources, etc.)
- Runs all pending ActiveRecord migrations
- Creates database views and indexes
rake htm:db:create¶
Creates the database if it doesn't exist. Useful for CI/CD pipelines where the database may not be pre-created.
rake htm:db:migrate¶
Runs pending database migrations only. Use after pulling new code with migrations.
rake htm:db:schema:dump¶
Exports the current database schema to db/schema.sql. Run after migrations to keep the schema file in sync.
Status and Information¶
rake htm:db:status¶
Shows which migrations have been applied and which are pending.
$ rake htm:db:status
Migration Status
================================================================================
✓ 00001_create_robots.rb (applied: 2025-01-15 10:30:00)
✓ 00002_create_file_sources.rb (applied: 2025-01-15 10:30:01)
✓ 00003_create_nodes.rb (applied: 2025-01-15 10:30:02)
00009_add_new_column.rb (pending)
Summary: 8 applied, 1 pending
================================================================================
rake htm:db:info¶
Shows comprehensive database information including connection details, extensions, table row counts, and database size.
$ rake htm:db:info
HTM Database Information
================================================================================
Connection:
Host: localhost
Port: 5432
Database: htm_development
User: dewayne
PostgreSQL Version:
PostgreSQL 17.2 on darwin24.0.0
Extensions:
pg_trgm (1.6)
pgvector (0.8.1)
plpgsql (1.0)
HTM Tables:
robots: 3 rows
nodes: 1,542 rows
tags: 287 rows
file_sources: 12 rows
node_tags: 4,891 rows
robot_nodes: 892 rows
schema_migrations: 8 rows
Database Size: 48 MB
================================================================================
rake htm:db:verify¶
Verifies database connection and required extensions are installed.
$ rake htm:db:verify
Verifying HTM database...
✓ Connection successful
✓ pgvector extension installed (0.8.1)
✓ pg_trgm extension installed (1.6)
✓ Database verification complete
rake htm:db:stats¶
Shows detailed statistics about HTM data including node counts, tag usage, and embedding coverage.
$ rake htm:db:stats
HTM Database Statistics
================================================================================
Nodes:
Total: 1,542
With embeddings: 1,538 (99.7%)
With tags: 1,421 (92.2%)
Soft-deleted: 23
Tags:
Total: 287
Hierarchical depth: max 4 levels
Most used: "database:postgresql" (127 nodes)
Robots:
Total: 3
Most active: "research-bot" (892 nodes)
File Sources:
Total: 12
Total chunks: 1,542
================================================================================
Console and Testing¶
rake htm:db:console¶
Opens an interactive PostgreSQL console (psql) connected to your HTM database.
$ rake htm:db:console
psql (17.2)
Type "help" for help.
htm_development=> SELECT COUNT(*) FROM nodes;
count
-------
1542
(1 row)
htm_development=> \dt
List of relations
Schema | Name | Type | Owner
--------+-------------------+-------+---------
public | nodes | table | dewayne
public | robots | table | dewayne
public | tags | table | dewayne
...
htm_development=> \q
rake htm:db:test¶
Tests database connection by running test_connection.rb.
$ rake htm:db:test
Testing HTM database connection...
✓ Connected to PostgreSQL 17.2
✓ pgvector extension: 0.8.1
✓ pg_trgm extension: 1.6
✓ Connection test passed
rake htm:db:seed¶
Seeds the database with sample data for development and testing.
$ rake htm:db:seed
Seeding database...
Creating sample robot...
Creating sample nodes...
Creating sample tags...
✓ Seeded 3 robots, 10 nodes, 15 tags
Rebuild Tasks¶
rake htm:db:rebuild:embeddings¶
Clears and regenerates all vector embeddings via the configured LLM provider. Use when changing embedding models or dimensions.
Warning
This task can take significant time for large databases and will make API calls to your embedding provider.
$ rake htm:db:rebuild:embeddings
Rebuilding embeddings for 1,542 nodes...
Processing: 100% |████████████████████████████████| 1542/1542
✓ Rebuilt embeddings for 1,542 nodes
rake htm:db:rebuild:propositions¶
Extracts atomic propositions from all non-proposition nodes. Creates new proposition nodes with their own embeddings and tags.
$ rake htm:db:rebuild:propositions
Extracting propositions from 1,542 nodes...
Processing node 1/1542...
...
✓ Created 4,892 proposition nodes
Destructive Operations¶
Warning
These tasks delete data and cannot be undone! Use with extreme caution.
rake htm:db:drop¶
Drops all HTM tables, functions, triggers, and views. Requires confirmation.
$ rake htm:db:drop
Are you sure you want to drop all HTM tables? This cannot be undone!
Type 'yes' to confirm: yes
Dropping HTM tables...
✓ Dropped nodes
✓ Dropped tags
✓ Dropped robots
...
✓ All HTM tables dropped
rake htm:db:reset¶
Drops and recreates the entire database. Equivalent to drop + setup.
Documentation Tasks (htm:doc:*)¶
Tasks for generating API documentation, database diagrams, and building the documentation site.
All-in-One¶
rake htm:doc:all¶
Generates all documentation (database docs, YARD API docs), builds the MkDocs site, and starts the local preview server.
$ rake htm:doc:all
Generating database documentation...
✓ Created docs/database/public.nodes.md
✓ Created docs/database/schema.svg
...
Generating YARD documentation...
✓ Created docs/api/yard/
Running mkdocs build...
✓ Site built in site/
Starting preview server at http://127.0.0.1:8000/
Individual Documentation Tasks¶
rake htm:doc:db¶
Generates database documentation including table markdown files and ERD diagrams.
$ rake htm:doc:db
Generating database documentation...
Connecting to htm_development...
Generating table documentation...
✓ public.nodes.md
✓ public.robots.md
✓ public.tags.md
✓ public.file_sources.md
✓ public.node_tags.md
✓ public.robot_nodes.md
Generating diagrams...
✓ schema.svg (full ERD)
✓ public.nodes.svg
✓ public.robots.svg
...
✓ Database documentation generated in docs/database/
rake htm:doc:yard¶
Generates YARD API documentation from Ruby source code.
$ rake htm:doc:yard
Generating YARD documentation...
Parsing lib/**/*.rb...
Generating HTML...
✓ YARD documentation generated in docs/api/yard/
rake htm:doc:site¶
Builds the MkDocs documentation site.
$ rake htm:doc:site
Running mkdocs build...
INFO - Cleaning site directory
INFO - Building documentation to directory: site
INFO - Documentation built in 2.34 seconds
✓ Site built in site/
rake htm:doc:serve¶
Starts the MkDocs development server for live preview.
$ rake htm:doc:serve
Starting MkDocs server...
INFO - Building documentation...
INFO - Serving on http://127.0.0.1:8000/
# Press Ctrl+C to stop
rake htm:doc:clean¶
Removes generated documentation files.
$ rake htm:doc:clean
Cleaning generated documentation...
✓ Removed site/
✓ Removed docs/api/yard/
✓ Documentation cleaned
rake htm:doc:deploy¶
Deploys documentation to GitHub Pages.
$ rake htm:doc:deploy
Deploying to GitHub Pages...
Building site...
Pushing to gh-pages branch...
✓ Documentation deployed to https://madbomber.github.io/htm/
rake htm:doc:validate¶
Validates documentation for broken links and formatting issues.
$ rake htm:doc:validate
Validating documentation...
Checking internal links...
Checking code blocks...
Checking images...
✓ All 127 pages validated
rake htm:doc:changelog¶
Generates changelog from git history.
$ rake htm:doc:changelog
Generating changelog...
Analyzing commits since v0.5.0...
✓ CHANGELOG.md updated
File Loading Tasks (htm:files:*)¶
Tasks for loading text files into HTM long-term memory with automatic chunking and source tracking.
Loading Files¶
rake htm:files:load[path]¶
Loads a single file into HTM memory. Supports markdown files with YAML frontmatter.
$ rake 'htm:files:load[docs/guide.md]'
Loading docs/guide.md...
Extracting frontmatter...
Chunking content (1024 chars, 64 overlap)...
Creating nodes...
✓ Created 5 chunks from docs/guide.md
# Force reload even if unchanged
$ FORCE=true rake 'htm:files:load[docs/guide.md]'
rake htm:files:load_dir[path,pattern]¶
Loads all matching files from a directory recursively.
$ rake 'htm:files:load_dir[docs/]'
Loading files from docs/...
✓ docs/guide.md (5 chunks)
✓ docs/api/reference.md (12 chunks)
✓ docs/examples/basic.md (3 chunks)
✓ Loaded 3 files, 20 total chunks
# Custom glob pattern
$ rake 'htm:files:load_dir[content/,**/*.txt]'
File Management¶
rake htm:files:list¶
Lists all file sources loaded into HTM.
$ rake htm:files:list
Loaded File Sources
================================================================================
ID | Path | Chunks | Last Synced | Needs Sync?
---|---------------------------|--------|---------------------|------------
1 | docs/guide.md | 5 | 2025-01-15 10:30:00 | No
2 | docs/api/reference.md | 12 | 2025-01-15 10:30:01 | No
3 | docs/examples/basic.md | 3 | 2025-01-14 09:00:00 | Yes
Total: 3 files, 20 chunks
================================================================================
rake htm:files:info[path]¶
Shows detailed information about a loaded file.
$ rake 'htm:files:info[docs/guide.md]'
File Source: docs/guide.md
================================================================================
ID: 1
Path: docs/guide.md
Last Modified: 2025-01-15 10:29:45
Last Synced: 2025-01-15 10:30:00
Needs Sync: No
Frontmatter:
title: Getting Started Guide
author: Dewayne VanHoozer
tags: [guide, tutorial]
Chunks: 5
ID 101: "# Getting Started\n\nWelcome to HTM..." (1024 chars)
ID 102: "## Installation\n\nAdd HTM to your..." (987 chars)
ID 103: "## Basic Usage\n\nHere's how to..." (1024 chars)
ID 104: "### Adding Memories\n\nUse the..." (856 chars)
ID 105: "## Next Steps\n\nNow that you..." (432 chars)
================================================================================
rake htm:files:unload[path]¶
Removes a file from HTM memory (soft-deletes all chunks and removes file source).
$ rake 'htm:files:unload[docs/old-guide.md]'
Unloading docs/old-guide.md...
Soft-deleting 5 chunks...
Removing file source...
✓ File unloaded
rake htm:files:sync¶
Syncs all loaded files, reloading any that have changed on disk.
$ rake htm:files:sync
Syncing loaded files...
docs/guide.md - unchanged
docs/api/reference.md - unchanged
docs/examples/basic.md - CHANGED, reloading...
✓ Reloaded (3 chunks updated)
✓ Sync complete: 1 file updated
rake htm:files:stats¶
Shows file loading statistics.
$ rake htm:files:stats
File Loading Statistics
================================================================================
Total Files: 12
Total Chunks: 1,542
Average Chunks per File: 128.5
Chunk Size Distribution:
< 256 chars: 23 (1.5%)
256-512 chars: 156 (10.1%)
512-1024 chars: 1,363 (88.4%)
File Types:
.md: 12 files (100%)
Sync Status:
Up to date: 11 files
Needs sync: 1 file
================================================================================
Job Tasks (htm:jobs:*)¶
Tasks for managing HTM background jobs (embedding generation, tag extraction, proposition extraction).
Processing Jobs¶
rake htm:jobs:process¶
Processes all pending background jobs (embeddings, tags, propositions).
$ rake htm:jobs:process
Processing pending jobs...
Embedding jobs: 5 pending
Tag jobs: 8 pending
Proposition jobs: 0 pending
Processing...
✓ Processed 13 jobs (13 successful, 0 failed)
rake htm:jobs:process_embeddings¶
Processes only pending embedding generation jobs.
$ rake htm:jobs:process_embeddings
Processing embedding jobs...
5 pending jobs
Processing...
✓ Node 1542 - embedding generated (768 dimensions)
✓ Node 1543 - embedding generated (768 dimensions)
...
✓ Processed 5 embedding jobs
rake htm:jobs:process_tags¶
Processes only pending tag extraction jobs.
$ rake htm:jobs:process_tags
Processing tag jobs...
8 pending jobs
Processing...
✓ Node 1542 - extracted 3 tags
✓ Node 1543 - extracted 2 tags
...
✓ Processed 8 tag jobs
rake htm:jobs:process_propositions¶
Processes only pending proposition extraction jobs.
$ rake htm:jobs:process_propositions
Processing proposition jobs...
3 pending jobs
Processing...
✓ Node 1542 - extracted 5 propositions
✓ Node 1543 - extracted 3 propositions
...
✓ Processed 3 proposition jobs, created 12 proposition nodes
Job Status¶
rake htm:jobs:status¶
Shows status of background job queues.
$ rake htm:jobs:status
HTM Job Queue Status
================================================================================
Queue | Pending | Processing | Completed | Failed
----------------|---------|------------|-----------|-------
Embeddings | 5 | 0 | 1,537 | 0
Tags | 8 | 0 | 1,421 | 2
Propositions | 0 | 0 | 4,892 | 0
Last processed: 2025-01-15 10:30:00
================================================================================
rake htm:jobs:retry_failed¶
Retries all failed jobs.
$ rake htm:jobs:retry_failed
Retrying failed jobs...
2 failed jobs found
Retrying...
✓ Tag job for node 892 - success
✓ Tag job for node 1023 - success
✓ Retried 2 jobs (2 successful)
rake htm:jobs:clear¶
Clears all pending jobs from queues. Requires confirmation.
$ rake htm:jobs:clear
Are you sure you want to clear all pending jobs? (yes/no): yes
Clearing job queues...
✓ Cleared 5 embedding jobs
✓ Cleared 8 tag jobs
✓ Cleared 0 proposition jobs
✓ All queues cleared
Tag Tasks (htm:tags:*)¶
Tasks for managing and visualizing the hierarchical tag system.
Visualization¶
rake htm:tags:tree¶
Displays the tag hierarchy as a text tree.
$ rake htm:tags:tree
Tag Hierarchy
================================================================================
database
├── postgresql
│ ├── extensions
│ │ ├── pgvector
│ │ └── pg_trgm
│ └── performance
└── redis
└── caching
ai
├── embeddings
│ └── ollama
├── llm
│ ├── anthropic
│ └── openai
└── rag
================================================================================
# Filter by prefix
$ rake 'htm:tags:tree[database]'
database
├── postgresql
│ ├── extensions
│ │ ├── pgvector
│ │ └── pg_trgm
│ └── performance
└── redis
└── caching
rake htm:tags:mermaid¶
Exports tag hierarchy to Mermaid flowchart format in tags.md.
$ rake htm:tags:mermaid
Exporting tag hierarchy to Mermaid format...
✓ Exported to tags.md
# Filter by prefix
$ rake 'htm:tags:mermaid[ai]'
Output (tags.md):
```mermaid
graph TD
database["database"]
database_postgresql["postgresql"]
database --> database_postgresql
database_postgresql_extensions["extensions"]
database_postgresql --> database_postgresql_extensions
...
rake htm:tags:svg¶
Exports tag hierarchy to SVG diagram in tags.svg.
$ rake htm:tags:svg
Exporting tag hierarchy to SVG...
✓ Exported to tags.svg
# Filter by prefix
$ rake 'htm:tags:svg[web]'
rake htm:tags:export¶
Exports tag hierarchy to all formats (text, Mermaid, SVG).
$ rake htm:tags:export
Exporting tag hierarchy...
✓ tags.txt (text tree)
✓ tags.md (Mermaid flowchart)
✓ tags.svg (SVG diagram)
✓ Exported to 3 formats
# Filter by prefix
$ rake 'htm:tags:export[database]'
Tag Management¶
rake htm:tags:rebuild¶
Clears and regenerates all tags via LLM extraction.
Warning
This task clears all existing tags and re-extracts them using your configured LLM provider. It can take significant time for large databases.
$ rake htm:tags:rebuild
Are you sure you want to rebuild all tags? This will clear existing tags. (yes/no): yes
Rebuilding tags for 1,542 nodes...
Clearing existing tags...
Extracting tags...
Processing: 100% |████████████████████████████████| 1542/1542
✓ Rebuilt tags: 287 unique tags, 4,891 node-tag associations
Common Workflows¶
First-Time Setup¶
# Install dependencies
bundle install
# Set database URL
export HTM_DBURL="postgresql://user@localhost:5432/htm_development"
# Create and setup database
rake htm:db:create
rake htm:db:setup
# Verify setup
rake htm:db:verify
rake htm:db:info
# Optional: seed with sample data
rake htm:db:seed
Daily Development¶
# Check for pending migrations
rake htm:db:status
# Run migrations if needed
rake htm:db:migrate
# Run tests
rake test
# Process any pending background jobs
rake htm:jobs:process
# Open database console for debugging
rake htm:db:console
Documentation Updates¶
# Generate all documentation
rake htm:doc:all
# Or individually:
rake htm:doc:db # Database docs
rake htm:doc:yard # API docs
# Preview documentation locally
rake htm:doc:serve
# Deploy to GitHub Pages
rake htm:doc:deploy
Loading Content¶
# Load a single file
rake 'htm:files:load[docs/guide.md]'
# Load a directory
rake 'htm:files:load_dir[content/]'
# Check what's loaded
rake htm:files:list
# Sync changed files
rake htm:files:sync
# Process embeddings and tags
rake htm:jobs:process
Tag Management¶
# View tag hierarchy
rake htm:tags:tree
# Export for documentation
rake htm:tags:export
# Rebuild all tags (after changing LLM)
rake htm:tags:rebuild
Production Deployment¶
# NEVER use reset or drop in production!
# Run only migrations
rake htm:db:migrate
# Verify database state
rake htm:db:verify
rake htm:db:status
# Process any pending jobs
rake htm:jobs:process
Environment Variables¶
All tasks require database configuration. Set one of these:
| Variable | Description |
|---|---|
HTM_DBURL |
Full PostgreSQL connection URL (preferred) |
HTM_DBHOST |
Database host |
HTM_DBPORT |
Database port |
HTM_DBNAME |
Database name |
HTM_DBUSER |
Database username |
HTM_DBPASS |
Database password |
Example:
Troubleshooting¶
"Database configuration not found"¶
Set the HTM_DBURL environment variable:
"Extension not found"¶
Install required PostgreSQL extensions:
psql htm_development -c "CREATE EXTENSION IF NOT EXISTS vector;"
psql htm_development -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
Tasks not running¶
- Verify HTM gem is installed:
gem list htm - Verify tasks are loaded:
rake -T htm - Check for errors:
bundle exec rake htm:db:verify --trace
Slow job processing¶
- Check LLM provider status (Ollama, OpenAI, etc.)
- Increase batch size in configuration
- Run jobs in parallel:
rake htm:jobs:process WORKERS=4
See Also¶
- Using Rake Tasks in Your Application - Integration guide
- Setup Guide - Initial development setup
- Database Schema - Schema reference
- Testing Guide - Running tests