Skip to content

feat(forward): add --skip-same to skip duplicate files - #1341

Open
lenny-ts wants to merge 2 commits into
iyear:masterfrom
lenny-ts:feat/forward-skip-same
Open

lenny-ts wants to merge 2 commits into
iyear:masterfrom
lenny-ts:feat/forward-skip-same

Conversation

@lenny-ts

Copy link
Copy Markdown
Contributor

Proposal

Fixes #1149.

When bulk forwarding from exported JSON files, the same media file may appear multiple times (overlapping exports, cross-posted documents, ...), and every occurrence gets forwarded again. This PR adds a --skip-same flag to tdl forward, mirroring the existing flag in tdl dl.

tdl forward --skip-same -f export.json

Behavior

  • A message is skipped if its media has already been forwarded during this run to the same destination peer/thread.
  • Media identity is fingerprinted as:
    • document → id + size
    • photo → id
  • Dedup is scoped per destination (peer id + thread + media key): routing the same file to two different chats is intentionally not affected.
  • Messages without file media are never deduplicated.
  • Skipped messages are logged at debug level and counted in a final summary log (skipped duplicated messages count=N).
  • Dedup also applies to --dry-run, so the preview matches what a real run would send.
  • Docs updated: "Skip Duplicates" section added to EN/ZH forward guide.

Design notes

  • Cross-run persistence was considered but intentionally not implemented: unlike dl (where the destination filesystem provides durable truth), forward would need local bookkeeping that can go stale (e.g., destination chat cleaned up), silently dropping legitimate re-forwards. Happy to discuss if maintainers prefer persistent KV-based dedup.
  • Grouped messages (albums): dedup applies to the primary message media; album children are not individually fingerprinted (same as upstream grouped handling).

Verification

  • go build, go vet, gofmt: clean
  • go test -race ./app/forward/...: green (new unit tests for media fingerprinting)
  • golangci-lint run app/forward/... cmd/...: 0 issues
  • tdl forward --help shows the new flag
  • Docs updated (EN + ZH guide)

Fixes iyear#1149.

When bulk forwarding from exported JSON, the same media file may appear
multiple times. Add a --skip-same flag that skips messages whose media
(document id+size or photo id) has already been forwarded in this run
to the same destination peer/thread.

Notes:
- dedup is scoped per destination, so routing the same file to two
  different chats is not affected
- messages without file media are never deduplicated
- works within a single run; cross-run persistence is intentionally not
  implemented (stale state would silently drop legitimate re-forwards)
- skipped messages are logged and counted in a final summary log
- add Skip Duplicates section to EN and ZH forward guide
- comment why albums are deduplicated as a whole and why dedup
  applies to dry-run too
@lenny-ts
lenny-ts requested a review from XMLHexagram as a code owner August 23, 2026 19:33

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feat] support the --skip-same in forward

1 participant