Local Rags RSS

Local Rags RSS is a derived resource built on existing national local-media datasets.

The outlet backbone comes from 3DLNews, which itself extends the Local Memory Project registry of local newspapers, television stations, and radio stations. We reuse that existing work — including outlet names, websites, geography, and supplied RSS metadata — as the starting point for this directory. The prior resources provide the registry and feed metadata; this project curates and operationalizes that inherited material.

Our contribution is downstream: we check those inherited feed candidates against the live web, discover current feeds when the supplied ones fail, reconcile publication identities, and select usable canonical feeds. The result is a current geographic source index connecting local newspapers to working RSS and Atom endpoints.

That index can serve as a routing layer for place-based news collection: resolve a place, identify the local publications that cover it, retrieve current feed items, and pass them to downstream search, mapping, or LLM/RAG systems.

The current resource contains 1,610 validated feed URLs covering all 50 states and Washington, DC: output/local_rags_rss.csv.

25 columns: source_id, outlet_id, outlet, outlet_aliases, state, state_abbr, census_region, census_division, county, fips, rucc_2023, metro_status, homepage, feed_url, feed_type, feed_title, discovery_method, freshness, latest_item_at, checked_at, source_record_ids, 3dl_source_lists, covered_outlet_count, covered_outlets, selection_status.

Each row also carries homepage, feed_title, latest_item_at, checked_at, upstream record IDs, and alias/coverage fields for reconciled outlets.

census_region and census_division follow the US Census Bureau regions and divisions. They are state-level classifications inherited by each county record.

rucc_2023 and metro_status come from the USDA 2023 Rural-Urban Continuum Codes. They join by county FIPS outside Connecticut. Because the 2023 codes use Connecticut's nine planning regions instead of its eight historical counties, Connecticut outlets are matched to a planning region by their source location.

County gaps inherited from the source registry are resolved through the auditable config/geography_overrides.csv table. Each correction is keyed to an upstream source record and retains its supporting URL, rationale, and review date. Geography overrides are applied before outlet reconciliation.

3DLNews extends the Local Memory Project (LMP) US local news media dataset, originally built to help communities collect stories from nearby publishers:

Nwala, A. C., Weigle, M. C., Ziegler, A. B., Aizman, A., & Nelson, M. L. (2017). Local Memory Project: Providing Tools to Build Collections of Stories for Local Events from Local Sources. JCDL 2017. https://doi.org/10.1109/JCDL.2017.7991576

LMP assembled a nationwide registry of local newspapers, TV, and radio outlets with websites, geo-coordinates, and RSS metadata. Those outlet and feed records form part of the inherited source material for this directory.

The outlet backbone and initial feed seeds come from 3DLNews, an MIT-licensed US local media resource built from an extended LMP registry:

Ariyarathne, G., & Nwala, A. C. (2024). 3DLNews: A Three-decade Dataset of US Local News Articles. arXiv:2408.04716. https://arxiv.org/abs/2408.04716 Resource: https://github.com/wm-newslab/3DLNews

3DLNews is a retrospective article corpus: nearly one million local-news URLs (with HTML) spanning 1996–2024, collected by scraping Google and Twitter (X) search results for more than 14,000 local newspapers, TV, and radio stations. The release also includes media-organization metadata, including RSS arrays that supply initial candidates for this project. Here, those inherited URLs are curated through live validation, discovery, identity reconciliation, and canonical selection.

In the paper, RSS appears mainly as contrast (NELA-Local collected articles from feeds). 3DLNews itself is not a maintained directory of live publisher feeds.

Article snapshots and live feed directories serve different jobs.

Researchers who need a historical sample of local coverage should use 3DLNews. Tools that need working feeds today need URLs that still resolve, parse as feeds, map cleanly to outlets, and can be rechecked when they rot.

Starting from 3DLNews newspaper records, Local Rags RSS builds a separate data product:

1. Live validation of supplied RSS/Atom candidates
2. Homepage discovery when supplied feeds are missing or unusable (HTML autodiscovery and common RSS paths)
3. Validation of discovered candidates before inclusion
4. Reconciliation of duplicate or variant outlet records
5. Canonical selection among multiple usable feeds (including legitimate section or edition feeds)
6. Rejection of comment feeds, irrelevant mappings, and duplicate canonical URLs
7. Auditable overrides for ambiguous identities (config/feed_overrides.csv)
8. Resumable national runs so interrupted checks do not start over

Other public directories were assessed and did not justify a second ingestion system. Values in 3dl_source_lists describe provenance inside 3DLNews; they do not mean those directories were independently crawled here.

• 9,439 3DL newspaper records ingested
• 4,639 supplied feed candidates checked (4,378 unique URLs)
• 8,823 publisher homepages checked where supplied feeds were absent or unusable
• 1,596 additional usable feeds found by discovery
• 7,738 reconciled publication identities
• 1,612 selected unique feeds → output/local_rags_rss.csv

Most selected feeds were found or confirmed on the live web rather than taken at face value from the upstream seed.

Rscript scripts/run-stage1.R
Rscript scripts/run-3dl-feeds.R --cores=4 --timeout=12 --batch-size=25
Rscript scripts/build-canonical.R

• run-stage1.R — download and normalize 3DL newspaper records and supplied feed URLs
• run-3dl-feeds.R — validate supplied feeds; discover and validate replacements from publisher homepages (checkpointed; --max-age-days=30 rechecks older results; omit to reuse every cached result)
• build-canonical.R — reconcile outlets, apply overrides, write the national CSV
• check-feed-rot.R — recheck the published feeds, retain observation history, and rediscover candidates after repeated failures

Outlet aliases and shared-feed coverage stay on the selected feed row rather than duplicating retrieval.

Ambiguous mappings remain in data-intermediate/reconciliation_review.csv and are not silently promoted. Refresh triage evidence (feed titles, recent item titles, homepage metadata; no article bodies):

Rscript scripts/collect-review-evidence.R --cores=4 --timeout=10 --batch-size=25

Optional structured LLM review (ellmer; reads OPENAI_API_KEY from the environment; never writes the key to project files):

Rscript scripts/run-llm-review.R --limit=5
Rscript scripts/run-llm-review.R
Rscript scripts/apply-llm-approvals.R   # only after inspecting high-confidence approvals
Rscript scripts/build-canonical.R

Recommendations are triage only. Durable decisions belong in config/feed_overrides.csv.

config/             3DL source, RSS discovery paths, reviewed overrides
R/                  ingestion, validation, discovery, reconciliation, review
scripts/            executable pipeline and review stages
tests/              executable checks
data-intermediate/  checkpoints, audits, and review tables (ignored)
output/              one published nationwide CSV

run-stage1.R downloads the upstream snapshot into data-raw/ (created on first run; ignored by git).

Maintainer handoff, unresolved review queue, retention rules, standalone-repo checklist, and textpress integration notes: PROJECT.md. Offline copy of the 3DLNews paper: 2408.04716v1.pdf.