whitelist-probe

  ██╗    ██╗██╗     ██████╗
  ██║    ██║██║     ██╔══██╗       whitelist-probe
  ██║ █╗ ██║██║     ██████╔╝       black-box whitelist reconstruction
  ██║███╗██║██║     ██╔═══╝        over a filtered mobile uplink
  ╚███╔███╔╝███████╗██║
   ╚══╝╚══╝ ╚══════╝╚═╝

     you ──▶ [ phone ] ──▶ [ carrier ] ──▶ ┃ ? ┃ ──▶ [ internet ]
                │                          ┃ ? ┃
                └─────────── probe ────────▶ ? ◀── reconstruct
                                            ┃ ? ┃
             phase 0  ·  what kind of filter is it?   (L3 / SNI / DNS)
             phase 1  ·  what, exactly, is on the whitelist?

A tool to reconstruct the effective internet whitelist an upstream filter is enforcing on your uplink, from the client side, without any prior knowledge of the list itself.

Built for the specific case of a Linux machine tethered through an Android phone whose mobile carrier has switched on filtering, but the design works against any IP / SNI / DNS-level filter that you can send packets through.

• When to use it
• When not to use it
• The core idea
• The four filter models
• Phases and data flow
• Key design decisions
• Pros
• Cons and known limitations
• Installation
• Quickstart
• CLI reference
• Configuration
• Interpreting the report
• Safety, etiquette, and legal notes
• Troubleshooting
• Architecture
• Extending
• FAQ

• Your mobile carrier, corporate proxy, school firewall, or similar upstream has applied an internet whitelist and published no list.
• You want to understand the shape of what's reachable before deciding whether to live with the filter, route around it, or change providers.
• You're researching how a given filter is implemented (at which layer, which hop, which resolver).
• You have legal authority to scan from the endpoint you're running it on.

• You do not own the endpoint and do not have explicit authorization to scan from it. This tool sends unsolicited SYNs and TLS handshakes toward many third-party hosts.
• You're on a metered connection where hundreds of MB matter more than the answer. The MVP corpus phase alone transfers 100-500 MB depending on settings.
• You want to circumvent a work/school filter without being noticed. Mobile carriers and corporate SOCs ban SIMs and flag users for scan-like traffic quickly, and the tool does not try to hide.
• You're trying to map an IPv4 range you'd specifically attack later. This is a reconnaissance tool for your own reachability, not a pen-testing harness.

A naive "reconstruct the whitelist" tool would scan IP ranges and mark reachable ones as whitelisted. That's wrong in at least three ways:

1. The filter might not be IP-based. A CIDR reconstruction of an SNI filter produces garbage: the same IP is reachable for one hostname and dead for another.
2. Uniformly scanning the IPv4 space is infeasible (2^32 addresses) and illegal from a consumer mobile connection.
3. Reachability to one address in a /24 doesn't tell you about the other 254, and the whitelist is almost never random — it follows organization and ASN boundaries.

whitelist-probe addresses all three:

1. Phase 0 identifies the filter model first by running a small (~100 probes), heterogeneous set of controlled experiments that isolate L3, L7 (SNI), and DNS filtering behaviors from one another.
2. Phase 1 picks the matching reconstruction strategy. IP-range reconstruction only runs if Phase 0 proves the filter is L3-based; SNI enumeration runs if it's L7; DNS diffing runs if resolver manipulation is detected.
3. The L3 pathway uses a Bayesian prior over ASN/provider categories and an adaptive prefix-bisection strategy, not a uniform sweep. Well-known blocks (AWS, Cloudflare) are probed lightly and accepted; uncertain prefixes are split into halves and re-probed until boundaries are found; empty space is ruled out with a sequential probability ratio test (SPRT).

The result is a probe budget in the low hundreds of MB producing a minimal CIDR set (or SNI list, or DNS map) with per-entry confidence.

Different upstreams enforce whitelists differently. whitelist-probe recognizes and produces an appropriately-shaped artifact for each:

Phase 0's verdict includes a confidence score; a low-confidence mixed result still runs all three Phase 1 variants so you can interpret for yourself.

 config.toml
     |
     v
 [Phase 0 characterize]   ~100 probes, ~5 min, ~5 MB
     |
     +--> reports/2026-04-13_16-30-00/phase0-characterize.{json,md}
     |    verdict: {filter_model, filter_location, carrier, recommended_phase1}
     v
 [Phase 1 scan]           auto-picked from Phase 0 verdict (same run folder)
     |
     +-- L3  -> reports/YYYY-MM-DD_HH-MM-SS/phase1a-scan_l3.{json,md}    CIDR set
     +-- SNI -> reports/YYYY-MM-DD_HH-MM-SS/phase1b-scan_sni.{json,md}   SNI list
     +-- DNS -> reports/YYYY-MM-DD_HH-MM-SS/phase1c-scan_dns.{json,md}   manipulation map
     v
 [Aggregate report]
     |
     +--> reports/YYYY-MM-DD_HH-MM-SS/whitelist-report.md    merged human-readable
     +--> reports/YYYY-MM-DD_HH-MM-SS/whitelist-report.json  same data, machine-readable
     |
     +--> reports/latest -> YYYY-MM-DD_HH-MM-SS/             (symlink, always current)

Each command invocation creates its own timestamped subfolder under reports/. The reports/latest symlink always points to the most recently started run. Running wlp r (report-only) automatically reads from latest and writes into that same folder.

Nine sub-steps, each emits a progress line:

1. Egress identity — public IP via ifconfig.me; ASN lookup via pyasn if a DB file is present.
2. Allowed-controls — TCP connect to a fixed set of anchor IPs known to be reachable under almost any policy (1.1.1.1, 8.8.8.8, 9.9.9.9).
3. Blocked-domain resolve — DNS-resolve likely-blocked domains via a public resolver to avoid carrier manipulation (otherwise you'd be probing the sinkhole, not the real target).
4. Blocked-controls — TCP connect to the resolved IPs. Compare verdict distribution vs allowed set.
5. MTR triangulation — TCP-SYN traceroute (port 443) to 3 allowed + 3 blocked targets. Count hops to the drop point; classify filter location as phone, carrier_edge, carrier_core, or upstream. Requires mtr installed; silently skipped otherwise (degrades filter-location confidence).
6. SNI variation test — five TLS ClientHellos to the same anchor IP, each with a different SNI. Divergent behavior → SNI filter present.
7. DNS resolver diff — resolve ~30 domains against carrier resolver and four public resolvers (1.1.1.1, 8.8.8.8, 9.9.9.9, 149.112.112.112, 208.67.222.222). Classify per domain as consistent, nxdomain_hijack, nodata_hijack, poisoned, or inconclusive.
8. IPv6 reachability — if IPv6 is available, probe a few blocked domains over v6; reachability → ipv6_bypass.
9. Bare-IP test — TCP connect to a blocked IP with no SNI, plus TLS handshake using the IP as SNI. Distinguishes port-level from SNI-level filtering.

Output: {filter_model, filter_location, carrier, recommended_phase1, confidence, evidence} written to SQLite (table runs), plus phase0-characterize.{json,md} in reports/.

Runs if Phase 0 verdict is L3 or mixed.

Corpus construction:

• Cisco Umbrella top-N domains (default N=100,000), cached in data/.
• Bulk DNS resolution of those domains via a public resolver (never the carrier's) yields 300k-800k unique IPs.
• Published cloud provider ranges: AWS ip-ranges.json, GCP cloud.json, Cloudflare ips-v4/ips-v6, Fastly. Azure skipped (auth-gated endpoint).
• Three random IPs sampled per cloud prefix as seed probes.

Probing:

• Async TCP-connect on port 443 (falls back to 80 on RST-no-service).
• 30 qps sustained, 50 concurrent, 3s timeout, short-lived sockets.
• Reserved networks (RFC1918, RFC6598 CGNAT, loopback, multicast, bogons) are skipped unconditionally.
• Byte-budget watchdog warns at 500 MB, halts cleanly at 1 GB (both configurable).
• SQLite checkpoint every 1000 probes; SIGINT flushes state and writes a partial report.

Bayesian reconstruction (see whitelist_probe/trie.py):

• Each prefix holds a Beta(α, β) posterior on P(whitelisted).
• Initial α/β from an ASN-category prior (cloud = 0.85, CDN = 0.80, ISP transit = 0.30, residential = 0.20, bogon = 0.01, default = 0.40).
• K=3 probes per prefix update the posterior.
• If posterior ∈ [0.25, 0.75] and depth < /29: bisect into two halves, queue both, recurse. Typical result: 3-5 probes at /24 decide the prefix, 6-10 probes converge to the actual boundary when it's finer.
• Wald's sequential probability ratio test guards against false negatives: prefixes marked rejected may be re-probed sparingly later.
• After the probe budget is exhausted, bottom-up coalescing merges any pair of accepted siblings into their parent, then iterates. Produces a minimal CIDR representation.

Output: accepted_cidrs, per_provider, per_asn, stats written to JSON + markdown.

Runs if Phase 0 verdict is SNI or mixed.

• Anchor IPs taken from Phase 0 (known-reachable: Cloudflare, Google).
• Rotate the top-N SNIs (default 50,000) against each anchor with TLS ClientHello. Record verdict per anchor per SNI.
• An SNI accepted by any anchor is considered whitelisted; no need to test all anchors once one accepts.
• Group accepted SNIs by eTLD+1 (via publicsuffix2) for a human-readable category summary.

Runs if Phase 0 verdict is DNS or mixed.

• Resolve the top-N domains (default same as umbrella_top_n) against the carrier resolver and up to four public resolvers.
• Classify each domain:
  • consistent — carrier and public answers overlap.
  • nxdomain_hijack — public resolves, carrier returns NXDOMAIN.
  • nodata_hijack — public has answers, carrier returns NOERROR/no-answers.
  • poisoned — carrier returns a different IP set disjoint from public.
  • inconclusive / timeout — everything else.
• UDP-only; much cheaper than TCP phases (roughly 40 qps, ~20 min for 100k domains).

wlp r reads the latest phase JSON sidecars from reports/latest/ and produces:

• reports/latest/whitelist-report.md — human-readable summary with filter model, reconstructed whitelist section (shape depends on Phase 0 verdict), per-ASN breakdown, sampled evidence, and confidence caveats.
• reports/latest/whitelist-report.json — the same data, machine-readable.

Every non-obvious choice made during development, with rationale:

The same reachability verdicts mean different things depending on the filter. Committing a big probe budget to IP-range reconstruction when the filter is really SNI-based produces noise. Phase 0 costs ~5 min and ~5 MB to figure out which tool to use; that investment pays itself back in avoided waste.

A uniform scan would spend its budget in high-entropy empty space. A trie keyed by prefix, with each node holding a Bayesian posterior, lets us (a) set informed priors per ASN category, (b) stop probing when we're confident, and (c) spend budget adaptively where it matters. Beta-Binomial is the natural conjugate prior for a hit-rate belief.

Filters almost always align to organization boundaries. Starting with P(whitelisted | AWS prefix) = 0.85 and P(whitelisted | residential ISP) = 0.20 means a single probe per prefix is often enough to commit or reject. Default priors are in config.toml:[trie].

Uncertain prefixes (posterior in [0.25, 0.75]) split into halves; each half gets fresh probes. This locates boundaries in O(log(prefix size)) probes, converging on actual whitelist edges without scanning all leaves. Stops at /29 (8 addresses) — finer than that is almost never useful and starts to look like a host sweep.

Probes that reject a prefix might be transient failures. SPRT (Wald) gives us a principled "how many more probes do I need to trust this?" rule. Cheaper than a fixed re-probe count.

A CIDR set is nicer than a forest of /29s. After probing, walk the trie bottom-up: if both siblings exceed accept_threshold, merge into parent; iterate until no more merges. Produces the minimal exact CIDR representation of the accepted set.

A full RouteViews BGP sweep (~1M prefixes) would cover more ground but adds a half-day of scan time, ~500 MB of data, and a meaningful SIM-flag risk. Umbrella gives us the IPs people actually try to reach, which is where real whitelists focus; published cloud ranges plug the "no public DNS pointer" gap for AWS/GCP/Cloudflare/Fastly. Together they reconstruct ~95% of typical consumer-facing whitelists. BGP is wired for a future version but not the MVP.

If the filter is DNS-based, resolving blocked domains through the carrier gives you the sinkhole IP. Probing that proves nothing. All bulk resolution goes to 1.1.1.1 / 8.8.8.8 with aiodns.

On a mobile uplink, long timeouts hoard phone-NAT conntrack entries. The conntrack table is the real bottleneck (4k-16k typical on Android), not bandwidth. Short timeouts mean sockets close quickly and we stay well below the table ceiling at 30 qps / 50 concurrent.

Empirical sweet spot. Above ~80 qps you start seeing conntrack saturation on consumer Android hotspots, which degrades your own connectivity. Above ~200 qps carrier scan-detection heuristics flag the SIM. 30/50 is sustainable for multi-hour runs.

Mobile data is billed. A byte budget (budget.halt_mb = 1000 default) bounds cost in the unit that matters; the runtime ceiling is a secondary safety net.

RFC1918/RFC6598/loopback/multicast/bogons are never probed. They're unreachable-by-definition and scanning them is wasted budget plus an obvious signature.

Originally planned DuckDB for analytical queries. Switched to SQLite because (a) zero-dep stdlib module, (b) WAL gives us resumability for free, (c) report-time queries are simple SELECT COUNT(*) GROUP BY, nowhere near DuckDB's strength.

Scans can take hours on mobile. Every 1000 probes the run state checkpoints to SQLite; SIGINT cancels in-flight probes, flushes the checkpoint, and writes a partial report. Re-running the same command picks up where it left off.

Greppable, diffable, viewable on any machine without a browser, and rendered nicely on GitHub / any IDE. JSON sidecars cover programmatic use. HTML dashboard is out of scope.

Neighbor project tor_bridge_updater uses uv run with inline dependencies and one file. whitelist-probe is ~2000 LOC across nine phase-and-concern boundaries that don't mix well; a package is cleaner. Install is still one command: uv sync.

Each invocation writes to reports/YYYY-MM-DD_HH-MM-SS/ rather than overwriting files in a flat reports/ dir. A latest symlink always points to the newest run folder. Rationale: multi-day runs are normal; overwriting the previous report mid-session destroys the baseline for comparison. wlp r (standalone) resolves latest to find phase JSON sidecars, so it always aggregates from the most recent scan without needing to specify a path. wlp status and wlp reset are the only commands that don't create a timestamped folder — they don't emit reports.

TCP-connect works unprivileged and is reliable for distinguishing open/closed/filtered from user space. Scapy SYN scanning would be ~10x faster and slightly more precise but needs cap_net_raw, which home Linux installs usually don't have. Scapy is a dependency but only used for the optional TCP traceroute path (falls back to shelling out to mtr --report-json).

• Works black-box — no list, no prior knowledge, no privileged access required. Just an internet-connected Linux host.
• Correctly distinguishes L3 from L7 from DNS filters before committing probe budget.
• Safe on mobile uplinks by default — conservative rate, conntrack- friendly timeouts, byte budget that halts cleanly.
• Resumable — multi-day scans survive Ctrl-C, reboots, link flaps.
• Evidence-backed — every verdict in the report cites raw probes stored in SQLite.
• Bayesian posteriors, not hard yes/no — each accepted CIDR carries a confidence; uncertain prefixes are called out separately.
• Adaptive bisection converges on real boundaries with a fraction of the probe budget a uniform scan would need.
• Multi-model output — even a mixed filter produces a coherent report across all three reconstruction strategies.
• Minimal dependencies — stdlib + eight third-party libraries, all MIT/BSD/Apache.
• Plays nicely with carrier detection — random destination order, jitter, rotating source ports, no contiguous /24 sweeps.

• No BGP sweep in MVP. Whitelists over non-public-web, non-cloud ranges (peering partners, government, corporate back-ends) may not appear in the reconstruction. The ~95% coverage estimate is for typical consumer whitelists.
• Anycast false positives. 1.1.1.1 reachability proves the local edge is reachable, not that the entire advertised prefix is whitelisted. The report flags this as a caveat but doesn't auto-correct.
• ASN attribution requires a pyasn DB file. Without it, per-ASN tables in the report are empty. Run pyasn_util_download.py --latest in the project root to get one; it's not bundled.
• Cannot read the phone's conntrack table. We only monitor the Linux side's /proc/sys/net/netfilter/nf_conntrack_count. The phone may saturate first; our defaults are empirical to prevent that but can't observe it.
• No unprivileged SYN scanning fallback. If you want scapy-based traceroute, grant cap_net_raw or run as root; otherwise filter- location detection degrades (we still have MTR via shell-out).
• SNI-scan cost scales poorly on dense filters. Each SNI probe is a full TLS handshake (~4 KB). 50k SNIs × 5 anchors ≈ 1 GB worst case. Default cap is 50k SNIs; use -C 5000 for a cheaper first look.
• Carrier-resolver detection relies on /etc/resolv.conf. If you have a local systemd-resolved proxy, it reports 127.0.0.53 and we fall back to the first public resolver in config, which degrades DNS-diff accuracy. Edit config.toml:[phase0].public_resolvers to put your real carrier resolver first, or run on a host where /etc/resolv.conf points directly at the carrier.
• No IPv6 whitelist reconstruction. Phase 0 detects IPv6 bypass; Phase 1 only reconstructs IPv4 ranges. IPv6 ranges are reported as evidence but not coalesced or attributed.
• No daemon / scheduled re-scan mode. Single-shot runs only. Detecting whitelist changes over time would need a wrapper script.
• Legal gray area on mobile carriers. See Safety.
• Unit tests only cover the trie. Scan drivers are integration-tested by running them; no mocked-network unit tests, by design.

Requires Python 3.13 and uv:

cd /home/luke/Projects/automation/white_lists
uv sync

uv sync installs two CLI entry points into the project venv:

• wlp — short alias, the recommended way to invoke the tool.
• whitelist-probe — long form, same binary.

Either can be run directly after activating the venv (source .venv/bin/activate) or via uv run wlp ... without activation.

Optional but recommended:

• mtr (for filter-location triangulation): sudo pacman -S mtr / sudo apt install mtr.

• A pyasn database file for ASN attribution in the report:

  uv run python -m pyasn.pyasn_util_download --latest
  uv run python -m pyasn.pyasn_util_convert --single rib.*.bz2 ipasn.dat

  (Neither step is mandatory; the tool degrades gracefully.)

wlp auto
cat reports/latest/whitelist-report.md

The --quick preset collapses the most common test flags into one switch:

wlp --quick auto

Equivalent to wlp --budget-mb 50 --domains 1000 --sni-limit 500 auto. Caps everything for a cheap, fast first look:

• 50 MB data ceiling (hard halt).
• 1,000 Umbrella domains resolved (instead of 100,000).
• 500 SNI variants (instead of 50,000).

Typical cost: 10-15 minutes, under 50 MB. Good for validating the setup before committing to a full run.

For use when the upstream carrier is likely to flag scan-like traffic (mobile SIMs, restrictive corporate networks). Activates a bundle of low-visibility defaults:

wlp --stealth auto
# or combine with quick for a cheap stealth probe:
wlp --stealth --quick auto

What changes under --stealth:

• Probe rate drops to 5 qps / 10 concurrent (from 30 / 50).
• Probe style switches from bare TCP-connect to full TLS + HTTP GET with browser-realistic headers (Chrome UA, Accept-Language, ALPN h2,http/1.1, graceful close). Real ACK ratios, realistic payload, not a SYN-heavy signature.
• Session windowing: 15-min work windows separated by 30–60 min random idle pauses. A long scan looks like bursty phone usage, not a continuous sweep.
• Port scope narrows to 443 only.
• Jitter doubles to 200 ms between probes.

Trade-offs:

• ~5-10× slower (a full L3 scan can take a day or more).
• ~5× more data per probe (TLS handshake + small HTTP response).
• Reconstruction depth unchanged — Bayesian bisection still runs.

All values in config.toml:[stealth] are tunable per-invocation via the usual -q / -c / -b flags, which take precedence over the stealth bundle.

# Phase 0 only — quick characterization
wlp char                    # alias for: wlp characterize

# Phase 1, auto-picked from Phase 0 verdict
wlp s                       # alias for: wlp scan

# Force a specific phase
wlp s -m l3                 # -m = --mode
wlp s -m sni
wlp s -m dns
wlp s -m all                # run every phase 1 variant

# Aggregate report from whatever phases have run
wlp r                       # alias for: wlp report

# Check last run status
wlp st                      # alias for: wlp status

The tool is invoked as wlp (or whitelist-probe / uv run wlp).

All subcommands accept these flags before the subcommand name. CLI flags override config.toml.

--quick is applied first; any explicit flag after it overrides that value. Example: wlp --quick -b 100 auto uses 100 MB cap, 1k domains, 500 SNIs.

Runs Phase 0. Writes reports/YYYY-MM-DD_HH-MM-SS/phase0-characterize.{json,md} and a calibration profile to the database.

Runs Phase 1. -m / --mode choices: auto (default), l3, sni, dns, all.
auto reads the latest Phase 0 verdict and dispatches; all runs every variant.

Reads the most recent run folder that contains phase data (via reports/latest symlink) and writes whitelist-report.{md,json} into it. Safe to re-run; overwrites only the aggregate output, not phase sidecars.

Runs characterize → scan → report with a single shared timestamped folder. The recommended entry point for a new probe session.

Prints the latest run per phase from the database. Does not create a reports folder.

characterize: run_id=abcdef123456 status=completed started=... probes={'open': 18, 'filtered': 42, ...}
scan_l3: no runs
scan_sni: run_id=...

Wipes data/ and reports/ entirely. Requires --yes to prevent accidents. Keeps source code and config.toml untouched.

All tunables live in config.toml at the project root. CLI flags override per-invocation.

[rate]
qps = 30                # sustained probe rate
concurrent = 50         # max in-flight sockets
jitter_ms = 50          # random jitter added to each inter-probe gap

[timeouts]
tcp_connect = 3.0
tls_handshake = 5.0
dns_query = 5.0
mtr_total = 30.0

[budget]
warn_mb = 500           # soft warning
halt_mb = 1000          # hard halt
max_runtime_hours = 36  # max wall time

[scope]
skip_reserved = true    # never probe RFC1918/6598/loopback/etc.
ports = [443, 80]       # probe ports in order
enable_ipv6 = true      # Phase 0 checks v6 reachability

[corpus]
umbrella_top_n = 100000
sni_cap = 50000
cloud_providers = ["aws", "gcp", "azure", "cloudflare", "fastly"]
bulk_dns_resolvers = ["1.1.1.1", "8.8.8.8"]

[phase0]
known_allowed = ["1.1.1.1:443", "8.8.8.8:443", "9.9.9.9:443", ...]
known_allowed_domains = ["cloudflare.com", "google.com", ...]
likely_blocked_domains = ["facebook.com", "twitter.com", ...]
public_resolvers = ["1.1.1.1", "8.8.8.8", "9.9.9.9", ...]
anchor_ips = ["1.1.1.1", "8.8.8.8"]

[trie]
posterior_lower = 0.25      # below → reject
posterior_upper = 0.75      # above → accept
accept_threshold = 0.75     # used in coalescing
reject_threshold = 0.25     # used for SPRT re-probing
probes_per_prefix = 3       # K for each prefix-level probe batch
max_bisect_depth = 29       # don't split below /29

# ASN-category priors (posterior before any probes)
prior_cloud = 0.85
prior_cdn = 0.80
prior_isp = 0.30
prior_residential = 0.20
prior_bogon = 0.01
prior_default = 0.40

Common tuning recipes:

• Very cheap dry run: wlp --quick auto (50 MB, 1k domains, 500 SNIs). Or push even lower: wlp -b 20 -n 500 -C 200 auto.
• Aggressive scan on generous data plan: edit config.toml to qps = 60, concurrent = 100, halt_mb = 5000. Verify your phone doesn't wedge on conntrack first.
• Home DSL (not mobile): you can push higher rates; also set cloud_providers = ["aws","gcp","azure","cloudflare","fastly"] (all).
• Customize what counts as "likely blocked": edit config.toml:[phase0].likely_blocked_domains. Put domains you expect your filter to block; Phase 0 uses them as negative controls.

reports/latest/whitelist-report.md is the primary deliverable (symlink resolves to the timestamped run folder). Sections:

Headline filter model + location + carrier + recommended Phase 1 + confidence (0.0-1.0). A confidence below ~0.3 means Phase 0 evidence was thin; treat verdicts as directional.

Produced by whitelist_probe/insights.py. This is the plain-English interpretation of the raw scan output and is the first thing to read:

• Openness score (0.00-1.00) and star rating. Weighted average of top-100, top-1k, and top-10k reachability rates (50/30/20 weights). Five stars = essentially open; one star = severely restricted.
• Plain-English summary naming the filter model, the reachability per popularity tier, mostly-open and mostly-blocked categories, and concrete examples of sites that work and sites that don't.
• Reachability by popularity tier — a whitelist that covers top-100 at 95% but top-10k at 0% indicates "popular services only." Uniform reachability across tiers indicates a narrow per-service filter rather than a tier-based whitelist.
• Reachability by category — a hand-curated map (80+ domains) groups top sites into social, messaging, video, news, cloud, tech, payment, reference, search, shopping, email, banking, gaming, education, maps. Category bars make it obvious at a glance what class of use this connection supports.
• Examples that work / don't — ranked sample of ~10 concrete domains per bucket. Useful for grounding: instead of "5% of cloud reachable," you see "microsoft.com works, googleapis.com doesn't."
• Filter granularity & scope — unique providers touched, ASN diversity, average IPs per reachable domain. High avg-IPs means the whitelist covers full CDN ranges; low avg-IPs means it hit just the nearest edge.
• Median RTT per probe verdict — a large latency gap between open / closed (reachable, ~tens of ms) and filtered / silent_drop (~3-5s timeout) reveals where the filter sits: it drops packets silently, while live endpoints respond fast.
• Totals — sanity-check numbers: domains evaluated, reachable vs unreachable, accepted CIDRs / whitelisted SNIs / DNS-manipulated counts.

The same data lives in reports/latest/whitelist-report.json:insights for programmatic use.

• Allowed-control success rate — should be near 100% or something is wrong with your uplink.
• Blocked-control filtered rate — should be high (0.6+) for the L3/SNI model to be credible; a low rate means "blocked" domains aren't really blocked at L3.
• SNI variation — number of SNIs accepted on a single anchor IP. Mixed results imply SNI-level filtering.
• DNS manipulation — count of domains where the carrier resolver differs from public resolvers.

Shape depends on filter model:

• L3: fenced block of CIDR entries. This is your whitelist.
• SNI: top-N eTLD+1 buckets with counts, followed by the first 200 individual SNIs. Full list in the JSON sidecar.
• DNS: classification counts + sampled examples per manipulation type.

Pre-coalescing prefix counts grouped by provider (aws, gcp, cloudflare, fastly, domain:* for DNS-derived seeds). Tells you which cloud footprint the filter covers.

Anycast warning, partial-run disclaimer, posterior-threshold explanation.

Every markdown section is backed by reports/latest/phase0-characterize.json and reports/latest/phase1*.json. Useful for automation:

import json
from pathlib import Path

run = Path("reports/latest")   # symlink to most recent run
p0  = json.loads((run / "phase0-characterize.json").read_text())
p1a = json.loads((run / "phase1a-scan_l3.json").read_text())
print(p0["filter_model"], len(p1a["accepted_cidrs"]))

To compare two runs, resolve the symlinks manually:

runs = sorted(Path("reports").iterdir(), key=lambda d: d.stat().st_mtime)
latest, previous = runs[-1], runs[-2]  # skip 'latest' symlink if needed

Read .claude/rules/safety.md for the long version.

Every probe has a byte cost; we track cumulatively:

• TCP connect: ~120 B (SYN + SYN-ACK + close or RST)
• TLS handshake (aborted): ~4 KB (ClientHello + ServerHello + cert chain)
• DNS query: ~200 B

Budgets:

• Default halt at 1000 MB (~1 GB). Override with --budget-mb.
• Warning at 500 MB — you'll see [safety] warning: ... MB in stderr.

Mobile carriers flag SIMs for aggressive port scanning much faster than fixed-line ISPs. Defaults are tuned conservatively:

• 30 qps / 50 concurrent.
• Random destination order (no linear sweeps).
• Jittered inter-probe gaps.
• Per-probe ephemeral source ports.
• Adaptive bisection touches ~1 IP per /24 average, not many.

If you observe a sudden rise in filtered verdicts against previously- reachable IPs mid-run, stop and wait 24 hours. That's the most likely signature of the carrier throttling or flagging you.

Sending unsolicited SYNs from a consumer mobile connection likely violates your carrier ToS in most jurisdictions, even if it's not a criminal act. Probing is not access. But:

• Do not use this against a network you don't own or aren't authorized to use.
• Do not publish reconstructed whitelists that identify third parties.
• Stop immediately if any infrastructure owner asks.
• This is a research tool. If you're unsure, keep --budget-mb 50 and --domains 1000 and stay in the low-noise regime.

IPv4: 0/8, 10/8, 100.64/10, 127/8, 169.254/16, 172.16/12, 192.0.0/24, 192.0.2/24, 192.168/16, 198.18/15, 198.51.100/24, 203.0.113/24, 224/4, 240/4.

IPv6: ::1/128, fc00::/7, fe80::/10, ff00::/8, ::/128.

Enforced in whitelist_probe/safety.py:is_probeable().

Fixed as of April 2026 — Phase 0 now emits a 1/9 ... through 9/9 ... progress line per sub-step. If you still see silence, it's usually stuck in the MTR step on hosts where mtr is missing or slow; remove mtr from your PATH to skip, or install it.

Fixed. If you hit this on an older install, pull the latest and uv sync again.

Comes from pyasn's own source file. Suppressed at import time in characterize.py:_asn_lookup(). Cosmetic; does not affect behavior.

Expected — we don't bundle the database. Either accept null ASN fields in the report, or populate with:

uv run python -m pyasn.pyasn_util_download --latest
uv run python -m pyasn.pyasn_util_convert --single rib.*.bz2 ipasn.dat

Check your mobile uplink latency first: time curl https://1.1.1.1 should be under 1s. Mobile connections with >500ms RTT will make probes appear to hang; bump timeouts.tcp_connect to 5s in config.toml.

Also: reduce scope. wlp -n 5000 s runs in ~30 minutes instead of ~24 hours.

Most likely the phone's NAT conntrack table filled. Tell-tale signs: everything including your own SSH session dies for ~30 seconds, then recovers. Mitigations:

• Drop rate: wlp -q 15 -c 20 s.
• Power-cycle the hotspot.
• Check the phone's conntrack ceiling with adb shell cat /proc/sys/net/netfilter/nf_conntrack_max if you have ADB.

Phase 0 didn't gather enough evidence. Causes and fixes:

• mtr missing → filter-location defaults to unknown. Install mtr.
• All your "likely blocked" domains are actually reachable → your filter isn't blocking what you think it is. Update config.toml:[phase0].likely_blocked_domains with domains you've actually seen fail.
• Carrier resolver is systemd-resolved proxy → DNS diff sees 127.0.0.53 vs public and declares everything divergent. Point /etc/resolv.conf at the real carrier resolver, or hardcode it as the first entry in [phase0].public_resolvers.

You ran wlp r before running wlp s. Run wlp s first (or wlp auto), then wlp r. The r command automatically picks the most recent run folder that contains phase data, so if you see this, either no scan has run yet or all past scan folders have been wiped.

Just re-run the same command. The tool loads the last incomplete run from SQLite and continues. You'll see [scan_l3] probes=... pick up near the previous total.

white_lists/
├── CLAUDE.md                         # Slim nav (for Claude Code agents)
├── .claude/rules/
│   ├── architecture.md               # Phase model, data flow, algorithms
│   ├── probing.md                    # Conntrack + rate + safety budgets
│   └── safety.md                     # Data cap, SIM risk, legal notes
├── pyproject.toml                    # PEP 621, Python 3.13, uv-managed
├── uv.lock
├── README.md                         # This file
├── config.toml                       # User-tunable rates/budgets
├── .gitignore                        # data/, reports/, caches
├── whitelist_probe/
│   ├── __init__.py
│   ├── __main__.py                   # `python -m whitelist_probe`
│   ├── cli.py                        # argparse dispatch
│   ├── config.py                     # config.toml -> typed dataclass
│   ├── probe.py                      # async TCP/TLS/DNS/mtr primitives
│   ├── calibrate.py                  # control-probe classifier
│   ├── characterize.py               # Phase 0 driver
│   ├── scan_l3.py                    # Phase 1a driver
│   ├── scan_sni.py                   # Phase 1b driver
│   ├── scan_dns.py                   # Phase 1c driver
│   ├── corpus.py                     # Umbrella + cloud-range loaders
│   ├── storage.py                    # SQLite schema + helpers
│   ├── trie.py                       # Patricia trie + Bayesian posterior
│   ├── report.py                     # Markdown + JSON output
│   └── safety.py                     # Rate limiter + byte budget + SIGINT
├── data/                             # runtime: SQLite + corpus cache (gitignored)
├── reports/                          # outputs (gitignored)
│   ├── 2026-04-13_16-30-00/          # one subfolder per run
│   │   ├── phase0-characterize.{json,md}
│   │   ├── phase1b-scan_sni.{json,md}
│   │   └── whitelist-report.{md,json}
│   └── latest -> 2026-04-13_16-30-00/  # symlink, always current
└── tests/
    └── test_trie.py                  # Bayesian update + coalesce + SPRT

Module responsibilities map one-to-one with phases. The dependency graph is acyclic: cli → [characterize, scan_l3, scan_sni, scan_dns, report] → {probe, corpus, safety, storage, trie, config}.

whitelist_probe/corpus.py has fetch_cloud_ranges as a template. Add a fetch_bgp_ranges that pulls from RouteViews / RIPE RIS, returns PrefixEntry list, merge into build_l3_corpus. You'll want to cap to the top-N largest prefixes by advertised size or filter by ASN category to avoid probing the full million.

1. Create whitelist_probe/scan_foo.py following scan_sni.py as a template. Must accept (cfg: Config, db: Db), return a result dataclass, write JSON + markdown to cfg.reports_dir / "phase1X-scan_foo.{json,md}".
2. Add scan_foo to cli._cmd_scan dispatch.
3. Add a foo option to the --mode choices.
4. Add collection logic in report.py:collect().

Edit whitelist_probe/trie.py:prior_for(). Add ASN-to-category mapping if you want per-ASN priors rather than per-provider.

tests/test_trie.py is the pattern. The trie module is pure-Python synchronous; new tests should run in <100 ms.

whitelist_probe/storage.py:Db isolates all SQL. Subclass it for a new backend; the rest of the code only sees the Db interface.

Q: Why not just use nmap / masscan? A: Both have unsafe rate defaults for mobile uplinks and obvious fingerprints to a carrier IDS. Neither does filter-model characterization or Bayesian reconstruction. If you're on a trusted network and want to scan the whole internet, use masscan. Not here.

Q: Why Python, given this is network-heavy work? A: The bottleneck is the uplink, not CPU. Python asyncio comfortably handles 100+ concurrent probes; you'll cap on conntrack or carrier rate- limiting long before Python itself.

Q: Does it work on macOS? A: Untested. The asyncio primitives are portable; mtr has to be installed separately. cap_net_raw doesn't exist on macOS. Raw scapy traceroute paths will fall back to the mtr shell-out.

Q: Can I run it on the Android phone directly via Termux? A: Probably, but haven't tested. Phone NAT is no longer in the path so you can push higher rates. pytricia / pyasn may need C-toolchain setup in Termux.

Q: How do I update the Umbrella list? A: Delete data/umbrella-top-1m.csv.gz; next run re-downloads it. The list is published daily.

Q: Can I feed my own IP list instead of Umbrella? A: Not via flags yet. Drop a file in data/ and modify corpus.py:build_l3_corpus to read it. Low-lift extension.

Q: Why does Phase 0 probe "likely blocked" domains? Isn't that circular? A: It's how we get a negative control set without knowing the whitelist. The choices in config.toml:[phase0].likely_blocked_domains are well-known globally-accessible domains that restrictive filters commonly block (major social networks, news). Even if the filter permits all of them, we learn the filter-model verdict none, which is actionable. If it blocks all of them, we have a clean negative set.

Q: Can this detect when the whitelist changes? A: Single-shot only today. Schedule the tool with cron/systemd and diff consecutive run outputs for a poor-man's change detector:

wlp auto
diff reports/2026-04-13_*/whitelist-report.json reports/2026-04-14_*/whitelist-report.json

Each run lands in its own timestamped subfolder, so you don't lose history between runs.

Q: How confident should I be in the reconstructed CIDR list? A: Per-CIDR confidence is the Beta-Binomial posterior; entries above accept_threshold = 0.75 are reasonably trustworthy. Entries in the uncertain band (shown separately) are best guesses. Anycast ranges are systematically over-represented — the report calls this out explicitly.