Importing the Legal Reference Datasets

How to load the five bundled legal datasets (CUAD, MAUD, ACORD, UNFAIR-ToS, LEDGAR) into your Lavern instance's knowledge base, so the agents — and the dashboard search — can retrieve them as reference material.

The import is driven by one script, scripts/seed-knowledge-base.ts. It downloads each dataset from its public source, chunks it, and indexes it as a global collection that every user on the instance can search.

1. What gets imported

ContractNLI is not included and cannot be seeded by this script. Its CC BY-NC-SA 4.0 license is non-commercial, incompatible with a commercial distribution. Passing --contractnli exits with an error by design. If you need it for personal, non-commercial use, fetch it from HuggingFace yourself and accept its terms separately. Do not bundle it into a commercial app.

See §6 Licensing before shipping these in a commercial product — all five are commercially usable, but BY-SA carries attribution + share-alike duties.

2. Prerequisites

• Node.js with the project's dev dependencies installed (npm install). The script runs via npx tsx (its shebang is #!/usr/bin/env npx tsx).
• The database. You don't need to pre-create it — the script calls initDatabase() itself, which runs the migrations and creates the kb_collections / kb_documents / kb_chunks tables and the kb_chunks_fts full-text index. The DB location follows your config (SHEM_DB_PATH, default ./data/lavern.db).
• Outbound network access to:
  • https://datasets-server.huggingface.co — MAUD, ACORD, UNFAIR-ToS, LEDGAR
  • https://github.com/TheAtticusProject/cuad — CUAD (downloaded as a zip; the HF Datasets Server can't serve CUAD because it runs custom Python).

  If you're behind a restricted egress policy (e.g. the Azure deployment's network rules), allow-list those two hosts, or run the seed once in an environment that can reach them and copy the resulting DB + cache.

• Disk for the download cache at ./data/seed-cache/ (raw JSON rows; CUAD also caches its zip + extraction there).

3. Running the import

From the repo root:

# Seed ALL five datasets (idempotent — skips any already imported)
npx tsx scripts/seed-knowledge-base.ts

# Re-seed everything from scratch (deletes existing collections first)
npx tsx scripts/seed-knowledge-base.ts --force

# Seed a single dataset
npx tsx scripts/seed-knowledge-base.ts --cuad
npx tsx scripts/seed-knowledge-base.ts --maud
npx tsx scripts/seed-knowledge-base.ts --acord
npx tsx scripts/seed-knowledge-base.ts --unfair-tos
npx tsx scripts/seed-knowledge-base.ts --ledgar

Behavior:

• No flags → seeds all five.
• Idempotent → if a collection already has data it's skipped (the script prints Already seeded. Use --force to re-seed.). Safe to re-run.
• --force → deletes the existing collection and re-imports it fresh.
• Per-dataset flags can be combined (e.g. --unfair-tos --ledgar).
• The full run takes a while — it paginates the HF Datasets Server 100 rows at a time with a 500 ms throttle between pages (to dodge HTTP 429s), and LEDGAR alone is 60K rows. First run is slow; subsequent runs use the cache and are fast.

Expected tail of a successful run:

═══════════════════════════════════════════════════════
Done. CUAD: <n>. MAUD: <n>. ACORD: <n>. UNFAIR-ToS: <n>. LEDGAR: <n>
Agents can now search with: search_knowledge_base("limitation of liability SaaS")

4. What happens under the hood

For each dataset the script:

1. initDatabase() — opens/migrates the DB.
2. ensureSystemUser() — creates a __system__ user (__system__@lavern.internal, no login) that owns the global collections.
3. ensureGlobalCollection(name, description, docType) — creates a row in kb_collections with is_global = 1, owned by __system__. Global means every user's search_knowledge_base can see it (it's not scoped to one user).
4. Fetch the rows:
   • HF datasets (MAUD/ACORD/UNFAIR-ToS/LEDGAR) via the paginated Datasets Server, cached to ./data/seed-cache/<name>-rows.json.
   • CUAD via the GitHub data.zip, extracted to ./data/seed-cache/cuad-extracted/ and parsed from train_separate_questions.json.
5. Chunk + index — groups annotations (by contract, deal point, or unfair label), then inserts kb_documents + kb_chunks rows in a single transaction. Each chunk carries metadata JSON recording its source (CUAD/MAUD/…) and dataset-specific fields (e.g. unfairType, clauseType, category).
6. FTS indexing is automatic — the kb_chunks_fts virtual table is kept in sync by triggers on kb_chunks, so no separate indexing step is needed.

Note: chunk content lands in kb_chunks; the per-document doc_type column on kb_documents is written as precedent by the shared insert helper, while the collection's doc_type reflects the table above (precedent vs regulation). Retrieval filters on either, so both are queryable.

Content + metadata stored per chunk

Every chunk persists both the searchable text and a structured metadata JSON blob, so you get content and relevant metadata for each dataset. The metadata is stored on kb_chunks.metadata and returned alongside results; the search-facing fields are heading (the clause/label name) and content (the text that gets full-text indexed).

Common to all: each chunk also carries chunk_index, word_count, the parent document_id/collection_id, and user_id = __system__. The parent kb_documents row records a synthetic filename (e.g. LEDGAR-<type>.txt, <contractTitle>.txt), word_count, and page_count.

If you want additional metadata captured (e.g. preserving CUAD's source contract category, MAUD's per-question answer rationale, or a jurisdiction value — currently left empty), that's a small change in the relevant seedX function: extend the JSON.stringify({ … }) block before insertChunk.run(...), and/or pass a non-empty jurisdiction to insertDoc. The retrieval layer already parses and returns metadata to callers, so anything you add there flows through to the agents and the API without further changes.

5. Verifying the import

Via the agent tool (what the agents use) — search_knowledge_base:

search_knowledge_base("limitation of liability SaaS")
search_knowledge_base("merger termination fee")
search_knowledge_base("unilateral change of terms")

Via the API / dashboard — the knowledge-base route (src/api/routes/knowledge-base.ts) exposes search and a collection listing to the UI.

Via SQL (sanity check):

SELECT c.name, COUNT(ch.id) AS chunks
FROM kb_collections c
LEFT JOIN kb_chunks ch ON ch.collection_id = c.id
WHERE c.is_global = 1
GROUP BY c.id;

You should see all five collections with non-zero chunk counts.

6. Licensing obligations

All five are safe for commercial use (none is non-commercial). But you take on obligations on distribution — handle these before shipping:

• Attribution (all five). Credit each dataset's creator (the Atticus Project for CUAD/MAUD/ACORD; LexGLUE for UNFAIR-ToS/LEDGAR), link the license, and indicate that you adapted it (the seeder reshapes/chunks the data). Surface this in a NOTICE file and/or an in-app "Data Sources" page.
• ShareAlike (UNFAIR-ToS, LEDGAR — CC BY-SA 4.0). Any adaptation of those datasets that you distribute must be licensed BY-SA 4.0. This does not infect your application code or your proprietary logic — software that queries the data is not an adaptation of it. Keep the datasets as separate collections (they already are) and don't merge BY-SA content into a combined work you want to keep proprietary.
• Gray area — derived artifacts. If you generate and distribute a derivative of the BY-SA datasets (a cleaned corpus export, or an embeddings / vector index of those specific datasets), it may count as an adaptation and need to be BY-SA. Keeping such derivatives server-side and non-distributed avoids the question. Get IP counsel to confirm for your distribution model.
• No extra restrictions / no DRM on the BY-SA material itself.

This is informational, not legal advice — confirm with counsel for a commercial launch.

7. Looking ahead: seeding under Postgres + pgvector

Today the seeder writes to SQLite and relies on the FTS5 triggers to index chunks for lexical search. Under the planned Azure migration (docs/azure-migration.md), the KB moves to Postgres + pgvector, which changes seeding in two ways:

1. Embeddings become part of seeding. Each chunk needs a vector (embedding vector(1536)) generated via Azure OpenAI text-embedding-3. seed-knowledge-base.ts becomes an embedding call site alongside src/knowledge-base/indexer.ts — embed each chunk (batched) before insert, and write content_tsv for the lexical half.
2. Backfill. Any rows seeded before the migration need a one-time pass to populate their embeddings.

Until then, the SQLite seeding flow above is unchanged. (The BY-SA embeddings caveat in §6 applies to the vectors you generate for UNFAIR-ToS and LEDGAR.)

8. Adding your own dataset

To import something beyond these five, mirror an existing seedX function in scripts/seed-knowledge-base.ts:

1. Add a collection name constant and a seedYourData(force) function.
2. Fetch your rows (reuse fetchAllHfRows for HF datasets, or your own loader).
3. ensureGlobalCollection(name, description, docType) — pick precedent, regulation, playbook, template, or prior_analysis for doc_type.
4. Insert kb_documents + kb_chunks via prepareInsertStatements(), setting heading, content, and a metadata JSON source tag. FTS indexing is automatic.
5. Wire a --your-flag into main().

Confirm the license permits your intended (commercial) use and redistribution before bundling it — that's exactly the check ContractNLI failed.

Quick reference

npx tsx scripts/seed-knowledge-base.ts                 # all five, idempotent
npx tsx scripts/seed-knowledge-base.ts --force         # wipe + re-import all
npx tsx scripts/seed-knowledge-base.ts --ledgar        # one dataset
# cache lives in ./data/seed-cache/  · DB at $SHEM_DB_PATH (default ./data/lavern.db)