KONI — Κοινή Online Nexus of Integration

An Open Canon Index, a Reader, and a Zero-Dependency Text-Reuse Engine

KONI turns scattered, human-only canon sources into one searchable, readable, research-grade text-reuse system — while being honest about what each piece is, where it came from, and what you may do with it.

    __ ______  _    ______
   / //_/ __ \/ | / /  _/
  / ,< / / / /  |/ // /
 / /| / /_/ / /|  // /
/_/ |_\____/_/ |_/___/

[ Engine: two-phase Flame | Reader + Engine deps: ZERO | Provenance: tracked, honestly ]

KONI is a lightweight workstation for classical philologists and digital humanists. It does three things: it builds a machine-readable, Linked-Open-Data index of the TLG (Thesaurus Linguae Graecae) canon from public sources; it serves a zero-dependency web reader for the openly-licensed Greek texts; and it runs a two-phase text-reuse engine ("Flame") that surfaces quotations, imitations, and formulaic echoes across ancient and Byzantine Greek — built for the realities of a heavily inflected, dialectally varied language.

The TLG canon — the index assigning every author a 4-digit ID (Homer = 0012) and every work a 3-digit ID (Iliad = 001) — is the backbone that links an identifier to an actual text. KONI assembles a normalized index (canon.json / canon.csv) by reconciling several public sources, and is honest about provenance: every author records which source(s) it came from, and every work carries a cts_confirmed flag (true = a real, openly-licensed text exists for it; false = attested in the canon but with no open text yet).

This index is built locally by the ETL pipeline on your machine. The project does not redistribute the canon assembled from restricted sources.

Why not just canonical-greekLit? CTS catalogues identify only works represented by available digital texts; they are not comprehensive authority registers for the TLG canon and do not provide a complete cross-reference between TLG identifiers and external authority systems (Wikidata / VIAF). Of the 8 816 works in the KONI canon, only ~1 622 have an open CTS text — the rest are attested in the canon but have no open edition yet. KONI indexes the full public canon and signals where the open infrastructure is still missing.

Coverage is the publicly reconstructable portion. The dataset represents the publicly reconstructable portion of the TLG canon. Newly catalogued authors and works available only within the subscription service cannot be incorporated until corresponding public metadata become available. KONI is therefore an open mirror of what is currently available, not a copy of the full subscription library — so coverage is intentionally never 100% complete.

Linked Open Data (JSON-LD). The canon is an authority list of entities, not a corpus of texts — so it fits Linked Open Data naturally. scripts/build_jsonld.py emits the index as JSON-LD (canon.jsonld) anchored on LAWD / SKOS / Dublin Core / PROV. Every author and work gets a stable koni: URI; authors carry skos:exactMatch links to Wikidata (wd:) and VIAF; confirmed works link to their dereferenceable Scaife text, and works with no open text yet are flagged koni:proposed — so the canon can also suggest the URIs that are still missing. The publishable @context lives at schema/context.jsonld, and the whole graph validates as RDF under a standard JSON-LD processor.

{
  "id": "koni:0059", "type": "Author",
  "name_grc": "Πλάτων",
  "exactMatch": "wd:Q859",                 // Wikidata (CC0), via TLG-id ↔ P3576
  "works": [{
    "id": "koni:0059.001", "type": "Work",
    "title_en": "Euthyphro",
    "cts_urn": "urn:cts:greekLit:tlg0059.tlg001",
    "ctsConfirmed": true,
    "exactMatch": "https://scaife.perseus.org/library/urn:cts:greekLit:tlg0059.tlg001/"
  }]
}

build_jsonld.py --links then writes the two publishable graphs, kept in separate files so their licences never mix:

• canon-links.nt — a CC0 cross-reference graph of nothing but koni: / Scaife ↔ public Wikidata / VIAF edges (identifiers and links only).
• canon-editions.nt — the per-work source-edition citations: the open TEI editions (CC BY-SA, from First1KGreek / Perseus) plus your own local:curated entries.

The full canon.jsonld carries TLG-derived names, epithets, and titles, so it stays local.

Flame is a two-phase pipeline — deliberately not a plagiarism string-matcher. It is tuned for oral-formulaic phrasing and historiographical imitation (e.g., Procopius reworking Herodotus) across inflectional and dialectal variation.

Phase 1 — candidate generation & ranking. Texts are chunked at citation-unit granularity (the deepest TEI textpart, e.g. 8.6.14), not whole books, so matches stay localized. A pure-Python BPE subword model (collections.Counter, re, unicodedata) feeds a TF-IDF cosine score that ranks how related two units are, while an inverted word-bigram index selects only the unit-pairs that actually share material — no dense, all-pairs scan over millions of combinations.

Phase 2 — local alignment. On the candidate pairs, Flame aligns the accent-flattened word lists with a hand-rolled Levenshtein-tolerant matcher: two words count as "the same" when their edit-ratio clears the Fuzzy threshold. This is what lets Ionic ποταμόν align with Attic ποταμὸς with no external library. Adjacent runs are fused across up to n_out non-matching words (leave-N-out), and strict core/chain filters discard isolated particle noise (τε, καὶ, δὲ), so only real reuse survives. (No agglomerative chaining — the filters do that work.)

Accent-insensitive by design. A pre-tokenization pass uses Unicode NFKD normalization to strip polytonic breathings, accents, and iota subscripts, so matching is robust across variant print editions; the original accented words are always kept for display.

Worked example (verified): comparing Procopius, de Bellis 8.6 with Herodotus, Histories 4.45 recovers the continents-naming formula (Nile / Phasis / Colchian / Tanais) as one long chain — exactly the kind of short, dialectally-shifted citation a global vector filter would bury.

• Canonical citation binding (CTS / URN). Every snippet stays bound to its metadata: author, work title (Latin / Greek), edition, section, and word range, hanging off a standard urn:cts:greekLit:… URN.
• Direct-comparison threshold bypass. For sweeps, Flame computes an automatic noise cutoff (mean + 1.2·σ); for a direct two-work comparison that auto-threshold is disabled and the cosine cutoff drops to 0.0, rescuing short 2–3-word micro-phrases instead of filtering them away.
• Streaming results + live progress. The comparison endpoint streams matches (NDJSON), strongest first, so the first hits render immediately while the rest load in the background — with an hourglass, a determinate progress bar, and a live "X matches · Y%" counter.
• Recompute on a button, not on every nudge. Server-side sliders (n-gram / n-out / min-chain / fuzzy) stage their change and light up an ↻ Recompute button, so you adjust several at once and run a single pass. The client-side filters (cosine, bridge-word colour) still apply live.
• Zero visual drift. Phase 2 matches directly on whole-word lists, so hit indices are word indices — highlights (Cardo / Noto Serif Greek) land on the exact words, and clicking a match scrolls its counterpart into view.
• High-fidelity export. One click compiles the current alignment into a publication-ready TSV with print-edition metadata, cosine scores, word ranges, and exact local snippets — ready for footnotes or a database.

ETL (canon build — runs locally)
  public sources ──► parse + merge + enrich ──► canon.json / canon.csv
       local/supplement.json ──► (curation overlay, tiered)  ┘   │
                                                 ├──► Diogenes export (CSV + SQLite)
                                                 └──► JSON-LD / LOD
                                                       canon.jsonld        (local; TLG-derived)
                                                       canon-links.nt      (publishable; CC0)
                                                       canon-editions.nt   (publishable; CC BY-SA + curated)
                                                       └─ firewall drops restricted:* tiers

Reader + Flame (runtime, stdlib only)
  openly-licensed TEI / CTS text ──► citation-unit chunking
        └──► Flame  Phase 1: BPE-subword TF-IDF + inverted bigram index (ranking)
                    Phase 2: Levenshtein word-block alignment (local match)
                        └──► streamed, highlighted alignments ──► TSV export
        Vanilla-JS SPA: search · author pages · two-pane reader · compare view

The Diogenes SQLite file is an export of the canon (for Diogenes/Perseus interop), not an input to the matching engine — the engine reads the TEI texts directly.

1) Install ingestion tools (one-time)

pip install -r requirements.txt

2) Fetch sources (one-time). The canon index is NOT shipped in git. Build it locally once from the public sources:

python scripts/fetch_sources.py

3) Build canon (one-time)

python scripts/build_canon.py

4) Launch the zero-dependency Flame engine & reader

python scripts/serve.py

In Flame · compare: pick two works with the autocomplete pickers, hit Compare, watch matches stream in, then tune the sliders and press ↻ Recompute. Click a highlighted word to jump to its counterpart; export as TSV.

5) Optional — export the canon as Linked Open Data (JSON-LD)

python scripts/build_canon.py            # build the canon index first (if not yet built)
python scripts/build_jsonld.py --links   # -> data/canon.jsonld + data/canon-links.nt

6) Optional — regenerate the open-backlog (gaps) report

python scripts/build_gaps_report.py       # -> reports/gaps_report.md (local, not tracked)

KONI is, in the long run, a modular Open Classical Authority Graph — a Linked-Open-Data authority layer for classical literature. The Greek TLG canon is the first module (KONI-GR); the architecture is intended to extend to further corpora as open sources allow. The natural next step is Latin (KONI-LA): Greek and Latin digital infrastructure today runs partly in parallel (TLG / CTS / Wikidata on one side, other Latin canons / CTS / Wikidata on the other), yet classical, late-antique, Byzantine-reception and humanist research constantly crosses the two. A feasibility mapping of open Latin sources (Perseus latinLit CTS, PHI, Wikidata Latin authors, VIAF) and of the schema changes a second module would need is part of the open backlog. Further corpora (Syriac, Coptic, medieval Latin) are a longer-term vision, not current work.

The open backlog. KONI is not only a cross-reference dataset, it is a signalling system for the community: it shows where open metadata, an open text, or an authority link is still missing. Running scripts/build_gaps_report.py produces reports/gaps_report.md — the authors with no Wikidata match, no VIAF link, no era, and the works that are attested in the canon but still have no open TEI text (flagged koni:proposed in the JSON-LD). Like Wikidata's red links, these gaps point independent researchers to where a new open TEI edition, a Wikidata P3576 statement, or a VIAF reconciliation adds the most value.

The aim is cooperative, not adversarial: the subscription TLG library has a legitimate model, and the Greek texts need not be opened. What KONI encourages is the release of structured metadata — TEI headers or their RDF-projectable parts (author, work, CTS URN, edition, date, language, related authority IDs) — without opening the texts themselves. If such metadata become public, the open research infrastructure enriches considerably and parts of KONI's reconciliation work become unnecessary — which is exactly the mark of a successful open-infrastructure project. Missing links and offered open TEI editions can be reported through the repository's issue tracker.

KONI bundles three different kinds of thing, and they are not under one license. ⚠️ Read this first: KONI touches sources with very different legal status (from PolyForm Noncommercial and CC BY-SA to copyrighted/restricted materials like the TLG). What it may redistribute, what it only builds locally, and what is off-limits are spelled out in the NOTICE file. That file is not an afterthought — it is the difference between a tool you can publish and one you can only run privately.

Licensing (mixed). KONI's own code is PolyForm Noncommercial 1.0.0 (see LICENSE) — free to use, change and share for any noncommercial purpose, including by educational, research and government institutions. Commercial use is not permitted. Note that this is a source-available license, not an OSI-approved open-source one. The Flame engine reimplements FLAME (Apache-2.0, same author). Loaded/bundled data is licensed separately — Greek texts CC BY-SA 4.0, Wikidata CC0; TLG-derived and unlicensed sources are not redistributed. Full breakdown in NOTICE.

The Linked-Open-Data outputs follow the same rule: the full canon.jsonld is built from the TLG-derived canon and stays local, while canon-links.nt — only the koni:/Scaife ↔ Wikidata (CC0) / VIAF cross-reference edges — is the publishable LOD subset.

• Reader + Flame engine — Python standard library only: http.server, collections, re, unicodedata, json, pathlib, csv, sqlite3, statistics, itertools, functools. No NumPy, PyTorch, Pandas, scikit-learn, tokenizers, or RapidFuzz — BPE and Levenshtein are hand-rolled.
• ETL pipeline only: requests, beautifulsoup4, lxml (scraping the public canon sources; not needed to run the reader or the engine).
• Linked Open Data: scripts/build_jsonld.py (pure stdlib) emits data/canon.jsonld (LAWD / SKOS / Dublin Core / PROV; skos:exactMatch to Wikidata/VIAF, Scaife @id for confirmed works, koni:proposed for the rest) and, with --links, the canon-links.nt cross-reference graph. The publishable context is schema/context.jsonld.
• Frontend: vanilla ECMAScript + CSS grid, responsive, dark mode.
• Validation: python scripts/validate_canon.py checks the canon against schema/canon.schema.json; every author records its source list and every work its cts_confirmed flag.