feat: contributors from crowd.dev git activity, refreshed daily (CM-1824) by mbani01 · Pull Request #4887 · linuxfoundation/crowd.dev

mbani01 · 2026-10-02T15:52:30Z

Summary

Second half of CM-1824, consuming the Tinybird snapshot shipped in #4886 (repo_commit_contributors_copy_ds, now live in lfx_insights).

Adds git_activity rows to repo_contributors: one commit-author row per repo and identity, for every repo crowd.dev tracks, with the CDP member id and the first/last commit timestamps (authored + co-authored). Bots, team members and organization profiles are already excluded upstream by the cleaned Tinybird source.

Changes

Migration V1790954467: nullable cdp_member_id UUID on repo_contributors, plus repo_contributors_sync_state (source PK, watermark, updated_at).
Sync module member-contributors/git-activity/: pages the Tinybird datasource in sorting-key order with a 4-column keyset cursor, maps channel to a packages-db repo (same canonicalisation as the governance sync) and username to an email or github-login identity, upserts with cdp_member_id; first_seen_at / last_seen_at use LEAST / GREATEST so channel spelling variants of one repo widen the window instead of overwriting it.
Workflow + schedules on the security-contacts-worker queue, next to the governance sync:
- git-activity-contributors-incremental-sync, 0 6 2-14,16-31 * *
- git-activity-contributors-full-sync, 0 6 1,15 * *
Tests for mapping, merging, watermark resolution, snapshot freshness, keyset paging and the sync paths.

How the incremental sync works

The Tinybird copy pipe rebuilds the snapshot nightly at 03:30 UTC. Each row carries lastUpdatedAt = max(activityRelations.updatedAt), which moves on ingestion and on member merges.
A run first reads max(computedAt) and throws if the snapshot is empty or not newer than the stored watermark, so a copy job that did not run is loud instead of silently reprocessing the same data.
Daily runs read only rows with lastUpdatedAt > watermark - 1 day and upsert them. Nothing is deleted.
The first run (no watermark) and the 1st/15th runs read the whole snapshot and reconcile: rows upserted get updated_at = NOW(), then git_activity rows with updated_at < run start are deleted. This closes the gaps a pure watermark cannot see (bot flag flips, repos disabled or re-enabled, identities absorbed by a merge, hard-deleted activities).
A full run refuses to reconcile when more than 20% of the existing git_activity rows would be deleted, which is what a truncated-but-non-empty snapshot (truncate + Sequin backfill during the copy) looks like.
The watermark is written only after the last page succeeded. A run that reads zero rows, reads rows but upserts none, or trips the reconcile guard throws and leaves state untouched.

Deployment

CROWD_TINYBIRD_BASE_URL and CROWD_TINYBIRD_ACTIVITIES_TOKEN must be present in the security-contacts-worker deployment, and the token needs read access to repo_commit_contributors_copy_ds. To be checked after deploy.
The first run is a full backfill of roughly 7M rows.

Depends on #4886 (merged).

…ibutors (CM-1824) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

…ark (CM-1824) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

…-1824) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

cursor · 2026-10-02T15:52:39Z

PR Summary

Medium Risk
Large-scale writes and periodic DELETE reconciliation on repo_contributors depend on Tinybird snapshot freshness; misconfiguration or a bad full run could remove or skew contributor data until caught by guards.

Overview
Adds a git activity contributor pipeline that reads crowd.dev commit-author data from Tinybird (repo_commit_contributors_copy_ds) and upserts git_activity rows into repo_contributors, including cdp_member_id and first/last commit windows.

Schema migration adds nullable cdp_member_id and repo_contributors_sync_state for per-source watermarks. The sync pages Tinybird with a keyset cursor, maps channels to packages-db repos (same URL canonicalization as governance), and upserts with LEAST/GREATEST on seen timestamps. Incremental runs filter by lastUpdatedAt (watermark minus one day) and never delete; full runs (no watermark, explicit full, or 1st/15th cron) reconcile by deleting untouched git_activity rows after upsert, with a 20% removal cap and guards for stale/empty snapshots and zero upserts.

Wires syncGitActivityContributors through Temporal on security-contacts-worker: daily incremental schedule plus bi-monthly full sync, alongside the existing governance contributors job. Includes broad unit tests for mapping, paging, watermarks, and reconcile safety.

^{Reviewed by Cursor Bugbot for commit 362bc8a. Bugbot is set up for automated code reviews on this repo. Configure here.}

CLAassistant · 2026-10-02T15:52:39Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Copilot

Copilot review overview

🟡 Changes recommended

Page-local aggregation can corrupt commit totals, and completed activity retries can fail due to the persisted watermark.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Prevent partial-count overwrites when source identities normalize together · New
Make freshness check idempotent after watermark commit · New

What changed in this PR

Adds daily Tinybird-to-packages-db synchronization for repository commit contributors.

Changes:

Adds contributor schema and watermark state.
Implements full/incremental synchronization and mapping.
Registers Temporal workflows, schedules, and tests.

File	Description
`backend/src/osspckgs/migrations/V1790954467__repo_contributors_git_activity.sql`	Adds contributor metrics and sync state.
`services/apps/packages_worker/src/activities.ts`	Exports the sync activity.
`services/apps/packages_worker/src/bin/security-contacts-worker.ts`	Registers the new schedules.
`services/apps/packages_worker/src/member-contributors/activities.ts`	Exposes the member-contributor activity.
`services/apps/packages_worker/src/member-contributors/git-activity/mapRows.ts`	Maps Tinybird rows to contributors.
`services/apps/packages_worker/src/member-contributors/git-activity/readCommitContributors.ts`	Implements Tinybird keyset paging.
`services/apps/packages_worker/src/member-contributors/git-activity/syncGitActivityContributors.ts`	Implements synchronization and reconciliation.
`services/apps/packages_worker/src/member-contributors/schedule.ts`	Defines full and incremental schedules.
`services/apps/packages_worker/src/member-contributors/workflows.ts`	Adds the Temporal workflow.
`services/apps/packages_worker/src/member-contributors/__tests__/syncGitActivityContributors.test.ts`	Tests mapping and synchronization behavior.
`services/apps/packages_worker/src/workflows/index.ts`	Exports the workflow.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2ccbe53. Configure here.}

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

…824) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

mbani01 · 2026-10-02T16:48:13Z

Review Feedback Addressed

Commits: 0628316, d0e97f5

Changes Made

syncGitActivityContributors.ts / mapRows.ts / migration: dropped commit_count. Not required by the ticket or any consumer, and not computable correctly from a snapshot keyed by raw channel + username: 5,899 canonical repos have more than one channel spelling in repos_to_channels, so page-local merging could not produce an all-time sum (per copilot-pull-request-reviewer, cursor)
syncGitActivityContributors.ts: first_seen_at / last_seen_at now use LEAST / GREATEST in the upsert so channel variants widen the window instead of overwriting it, independent of page order or which variant an incremental run re-reads (per copilot-pull-request-reviewer, cursor)

No Change Needed

syncGitActivityContributors.ts:96: the "not newer" retry failure is kept on purpose. The window is milliseconds at the end of the run, the data is already correct, and the failure self-heals at the next schedule. Treating computedAt == watermark as already applied would silently skip the more likely failure, a copy job that did not run (flagged by copilot-pull-request-reviewer)

Threads Resolved

3 of 3 unresolved threads addressed in this iteration.

Copilot

Copilot review overview

🟡 Changes recommended

Full reconciliation needs protection against deleting valid data when a nonempty but unexpectedly partial snapshot is received.

Review effort: Balanced
Findings: 1

Open (1)

Prevent destructive reconcile on unexpectedly truncated snapshots · New

Resolved since last review (2)

Prevent partial-count overwrites when source identities normalize together
Make freshness check idempotent after watermark commit

…snapshot (CM-1824) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

mbani01 · 2026-10-02T17:04:45Z

Review Feedback Addressed

Commit: 362bc8a

Changes Made

syncGitActivityContributors.ts: the 1st/15th full reconcile now refuses to delete when more than 20% of the existing git_activity rows were left untouched by the run, which is what a truncated-but-non-empty Tinybird snapshot looks like. The run throws with the counts, nothing is deleted, the watermark is not advanced. (per copilot[bot])
syncGitActivityContributors.test.ts: covers the guard passing on normal churn and refusing on an excessive drop.

Threads Resolved

1 of 1 unresolved thread addressed in this iteration.

Copilot

Copilot review overview

🔵 Needs a closer look

The initial 7M-row reconciliation and production Tinybird access require final human operational validation.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Prevent destructive reconcile on unexpectedly truncated snapshots

mbani01 added 4 commits October 2, 2026 15:22

feat: add member reference, commit count and sync state to repo contr…

c5a1a4f

…ibutors (CM-1824) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

feat: sync repo contributors from Tinybird git activity with a waterm…

3cb264d

…ark (CM-1824) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

feat: schedule daily and biweekly git-activity contributors syncs (CM…

95ea8df

…-1824) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

test: cover the git-activity contributors sync (CM-1824)

2ccbe53

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

Copilot AI balanced review requested due to automatic review settings October 2, 2026 15:52

Copilot started reviewing on behalf of mbani01 October 2, 2026 15:53 View session

mbani01 self-assigned this Oct 2, 2026

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Comment thread ...ces/apps/packages_worker/src/member-contributors/git-activity/syncGitActivityContributors.ts

Comment thread ...ces/apps/packages_worker/src/member-contributors/git-activity/syncGitActivityContributors.ts

cursor Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread ...ces/apps/packages_worker/src/member-contributors/git-activity/syncGitActivityContributors.ts

mbani01 added 2 commits October 2, 2026 16:46

refactor: drop the commit count from git-activity contributors (CM-1824)

0628316

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

fix: widen the git-activity seen window across channel variants (CM-1…

d0e97f5

…824) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

Copilot AI balanced review requested due to automatic review settings October 2, 2026 16:47

Copilot started reviewing on behalf of mbani01 October 2, 2026 16:48 View session

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Comment thread ...ces/apps/packages_worker/src/member-contributors/git-activity/syncGitActivityContributors.ts

fix: refuse the full git-activity reconcile on an unexpectedly small …

362bc8a

…snapshot (CM-1824) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

Copilot AI balanced review requested due to automatic review settings October 2, 2026 17:04

Copilot started reviewing on behalf of mbani01 October 2, 2026 17:04 View session

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Navigation Menu

feat: contributors from crowd.dev git activity, refreshed daily (CM-1824) - #4887

feat: contributors from crowd.dev git activity, refreshed daily (CM-1824)#4887
mbani01 wants to merge 7 commits into
mainfrom
feat/repo-contributors-git-activity-sync

mbani01 commented Oct 2, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Oct 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

mbani01 commented Oct 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

mbani01 commented Oct 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mbani01 commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

How the incremental sync works

Deployment

Uh oh!

cursor Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

CLAassistant commented Oct 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mbani01 commented Oct 2, 2026

Review Feedback Addressed

Changes Made

No Change Needed

Threads Resolved

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

mbani01 commented Oct 2, 2026

Review Feedback Addressed

Changes Made

Threads Resolved

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mbani01 commented Oct 2, 2026 •

edited

Loading

cursor Bot commented Oct 2, 2026 •

edited

Loading