The current system for managing the incoming test reports from testers has a few problems:

• Only one node is handling incoming test reports, which makes outages more likely
• The test reports are stored in the same MySQL database server as derived statistics, which increases the load on the database server and couples the two systems together
• The collection of 150,000,000 test reports (and increasing at 25,000/day at about 10 kB each totaling 81 GB/year) is above 1 TB, making every operation difficult
• 1 TB of storage is expensive. Combined with the CPU/memory required for the derived statistics reporting, it's cost-prohibitive.

To have a better separation of concerns, this proposal describes a "CPAN Testers Collector" system. The Collector provides only the following functionality:

• Receive incoming reports in a standard JSON format, described by a JSON schema
• Store incoming reports
• Respond to requests for report data by UUID
• Respond to requests to list reports based on the report timestamp

In this way, the Collector serves as the primary source of complete reports. Downstream systems can synchronize with the Collector periodically to derive their own data for specific functions like reporting and statistics.

Incoming Reports

The Collector will store reports in plain files, to reduce dependence on external systems. Report data will be stored in a directory path based on the report's UUID in hexadecimal format. The first and second pairs of UUID characters will be used for organizing the files into directories to prevent any single directory from being overly large.

# Given a UUID XXYY1234-1234-1234-1234-123456789012
# The main report JSON is:
./report/XX/YY/XXYY1234-1234-1234-1234-123456789012

To allow synchronization, the Collector will store lists of UUIDs in files organized using the date/time of the incoming report.

# Given a report UUID XXYY1234-1234-1234-1234-123456789012
# Reported at 2025-01-02T03:04:05
$ cat ./_meta/timestamp/2025/01/02/03/04/05
XXYY1234-1234-1234-1234-123456789012

The JSON schema URL should be added to the report in a $schema property. The schema URL should contain versioning for converting reports to any future JSON schema.

Ideally, all reporters would send JSON reports in the current, up-to-date format. In this case, the collector does not need to process the report and can quickly write it to disk. Unfortunately, this will not always be the case, so the collector will handle converting any prior report formats to the current, up-to-date format. In this way, the downstream systems which sync with the collector can have a single format to support for their derived data.

EDIT: Prior versions of this proposal had the Collector handling legacy Metabase formatting. Because the Metabase API requires saved state, the Collector will no longer do that. To implement the Metabase API requires keeping track of users by UUID so their data can be looked up when they submit a report. This is currently done in the MySQL database, and there's no reason to modify this if the preferred end-state is no reporter is using the Metabase API. Instead, the legacy Metabase shim currently in-place can forward the reports it gets to the Collector after doing its own transformation.

Storage Maintenance

The primary storage for reports will be a btrfs filesystem to allow for de-duplication and compression. As reports age, they are collected into squashfs files with increasing compression.

• Weekly files for 12 weeks
• Monthly files for 12 months
• Yearly files for as long as possible

Each squashfs file is mounted, layered, underneath the primary btrfs filesystem. Incoming reports are written to the btrfs filesystem, and a periodic job uses the timestamp indexes to add new reports to their first weekly squashfs file. Individual squashfs files can be mounted to combine into monthly and yearly files. For example:

/mnt/reports-all - Combined overlay
/mnt/reports-new - Writable btrfs with all metadata (for performance)
/mnt/reports-2025-W32 - SquashFS for a single week
/mnt/reports-2025-01 - SquashFS for a single month
/mnt/reports-2024 - SquashFS for a whole year

Every week, a new weekly /mnt/reports-2025-W33 is made from the reports in /mnt/reports-new. Once there are enough of those, they can be used to make a new monthly /mnt/reports-2025-07, and etc... for the yearly archive. There may be some temporary duplication of reports between the weekly and monthly archive files, but this should be no more than a few days' worth of data.

The current storage requirement is projected to be around 1TB for ~15 years of data (all the current system contains). Ideally, the largest SquashFS archive files will still be able to be synced between nodes for backups and replication.

Redundancy

To prevent downtime and protect against data loss, the Collector system and its storage should be replicated to 2 or 3 nodes in separate data centers, geographically distributed. Each node has its own direct DNS entry like collector-1, collector-2, etc... under the cpantesters.org domain.

A fastly configuration will be used to load balance incoming requests to active nodes. Nodes will cross-synchronize every few seconds using the timestamp files of the other nodes so that each node maintains a full copy of the data.

Synchronization API

The Collector will expose an HTTP API for listing report UUIDs uploaded during a specified time period. The API will be organized with date/time pieces as path parts. The final path part will also allow commas and ranges to specify multiple periods. The API will only ever return up to 100,000 IDs, sending back an HTTP link header with rel="next" for where to pick up the next set of results. The API will only ever accept requests with a days part specified: No complete months or years will be allowed.

GET /v1/timestamp/YYYY/MM/DD/HH/NN/SS
# Get all report UUIDs uploaded at exactly 2025-01-02 03:04:05
GET /v1/timestamp/2025/01/02/03/04/05

# Get all report UUIDs uploaded from 2025-01-02 03:04:00 through 2025-01-02 03:04:59
GET /v1/timestamp/2025/01/02/03/04

# Get all report UUIDs uploaded from 2025-01-02 00:00:00 through 2025-01-02 23:59:59
GET /v1/timestamp/2025/01/02

# Get all report UUIDs uploaded on either 2025-01-02 or 2025-01-03
GET /v1/timestamp/2025/01/02,03

# Error: Must specify days
GET /v1/timestamp/2025/01

# Error: Cannot use ranges in intermediate path parts
GET /v1/timestamp/2025/01/02,03/04

Reverse Proxy

A Varnish proxy will live in front of the collector HTTP service. Varnish is chosen because it is fast, can do live updates, and is also used by Fastly so the configuration can be moved to the CDN if needed. The reverse proxy will mostly be transparent, but may cache some static pages and handle whatever cache validation it can (to prevent load on the collector HTTP service).

The reverse proxy will also enable canary deployments and could potentially do real-time replication of incoming reports to other nodes in the cluster.

Documentation and Discovery

The main / page of the Collector will provide an informational page about submitting CPAN Testers reports: How to install a client, how to use that client to submit a report, etc... Additionally, it will link to API documentation for developing a CPAN Testers client. This documentation will be a versioned OpenAPI specification that consumes the separately-linked JSON schema that defines the report structure. The URL for these docs must contain the API version, and ideally we will expose an interactive application for testing the collector API.

Testing and Debugging

We should provide a collector-dev.cpantesters.org instance that users can use for manual or automated testing of CPAN Testers reporters. This non-production system also allows for our own canary deployments: The reverse proxy can copy traffic from production to the collector-dev to test it as needed.