Baseline category labels use unkeyed SHA-1 and can be reversed with a dictionary

Summary

fd.build_baseline stores categorical frequencies under hashed labels so a baseline "cannot leak PII". The hash is plain SHA-1 of the value, truncated to 16 hex characters, with no key or salt. Anyone holding a baseline JSON can recover low-entropy labels, such as diagnosis codes, countries or product names, by hashing a guess list. They also get the exact frequency of each label.

Reproduction

import hashlib, json
import pandas as pd
import freshdata as fd

df = pd.DataFrame({"diagnosis": ["HIV", "HIV", "diabetes", "depression"] * 10})
stored = json.loads(json.dumps(fd.build_baseline(df, name="b").to_dict()))  # what save_baseline writes
freqs = stored["columns"]["diagnosis"]["frequencies"]
h = lambda s: "h:" + hashlib.sha1(s.encode()).hexdigest()[:16]
guesses = ["HIV", "diabetes", "depression", "asthma", "cancer"]
print({g: freqs[h(g)] for g in guesses if h(g) in freqs})

Expected

With the default include_samples=False, labels in a baseline cannot be recovered without a secret.

Actual

{'HIV': 0.5, 'diabetes': 0.25, 'depression': 0.25}

Impact

• Baselines are meant to be persisted and shared: JSON files committed next to pipelines and passed to CI.
• The module promises they are PII-free. Two docstrings make that claim:
  • The module docstring says baselines "never store raw sample values unless include_samples=True is passed explicitly, so a baseline cannot leak PII".
  • _hash_label is described as a "Stable, non-reversible category label for PII-safe baselines".
• Given a published or leaked baseline, an attacker needs only a dictionary of candidate values. They can recover which sensitive categories are present and their exact proportions, for example a diagnosis mix for a small cohort.
• No key is needed, and SHA-1 costs nanoseconds per guess, so enumerating low-entropy domains is trivial.
• The project already treats this pattern as unsafe elsewhere. The MaskingRule docs (src/freshdata/enterprise/config.py:52-56) say an unsalted hash "would otherwise let an attacker rainbow-table low-entropy PII".

Environment

freshdata 2.0.0 @ 55a8044. Reproduces on Python 3.9.6 / pandas 1.5.3 / numpy 1.26.4 and on Python 3.12.14 / pandas 2.3.3 / numpy 2.5.3.

Where

• src/freshdata/enterprise/contracts.py:91-93

  def _hash_label(value: str) -> str:
      """Stable, non-reversible category label for PII-safe baselines."""
      return "h:" + hashlib.sha1(value.encode("utf-8")).hexdigest()[:16]

• Used for frequencies and top values at contracts.py:723-729.
• The PII claim is at contracts.py:17-20.

Suggested fix

• Key the label hash with HMAC-SHA256, using a per-baseline secret (random by default, optionally supplied so baselines can be compared across runs). Store only a key identifier in the JSON, and require the same key in compare_to_baseline.
• Alternatively, when include_samples=False, store only an unlabeled, sorted frequency vector for categorical PSI (no label identifiers at all).
• Until fixed, soften the docstrings: labels are pseudonymous, not non-reversible.