Skip to content

Tags: google/clusterfuzz

Tags

v2.43.0

Toggle v2.43.0's commit message
[Mac] Add radamsa binary for ARM64 and update resolution logic

When executing macOS ARM64 Swarming tasks, ClusterFuzz invokes the radamsa
mutator binary. Previously, only an x86_64 macOS binary was bundled.

Changes:
* Add native macOS ARM64 binary (`bin/mac_arm64/radamsa`).
* Update `get_radamsa_path()` in `engine_common.py`:
  - Prefer `radamsa` from `APP_DIR` (fuzzer build directory) if present.
  - Detect ARM64 on macOS and select arm64 `radamsa` over x86_64 `radamsa`.
* Add unit tests for `get_radamsa_path()` in `engine_common_test.py`.

Tests:
* `python butler.py py_unittest -t core -p engine_common_test.py` (Succeed)
* `python butler.py py_unittest -t core -p environment_test.py` (Succeed)

Bug: 561690132
Signed-off-by: Manuel Briones <manuelbriones@google.com>

v2.42.0

Toggle v2.42.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
[Scheduling][Swarming] Retries tasks only on meant platform (#5495)

As found out in the parent PR:
#5476

In the scheduler, when a task fails to get scheduled in swarming, it
then tries to go on to other backends, so if for any reason(e.g. the
platform is not supported in batch per the batch config), then the whole
slice of batch tasks get rejected.

So for this PR, we make it so swarming tasks are only ever tried on
swarming, and other tasks continue to be tried on other backends.

Also when making this changes i thought of an edge case not considered
previously, what if there was a swarming task in the queue & then we
disable the flag? Then the scheduled would mark the task as not swarming
& would try it on other backends, and again this could show errors if
the platform is not supported there. So i changed the validations.

With this changes task now appear in swarming!
<img width="704" height="511" alt="image"
src="https://github.com/user-attachments/assets/df9665e9-7257-4db6-a38d-22384b6eedcc"
/>


## Changes
- `swarming/__init__.py`: Now we allow to validate if a given job is
swarming or not without considering the flag.
- `remote_task_gate.py`: Splits tasks in swarming and other, swarming
tasks get redirected to the swarming service and the unscheduled tasks
from this service are not added back again with the other tasks

## Tests
- A bunch of Tests + refactoring some tests to support this new edge
cases.

- Left this changes running in dev for 24 hours, and no longer saw the
error that canceled the whole slice of batch tasks if the job platform
didn't existed for batch,
I only saw this error:

<img width="1572" height="563" alt="image"
src="https://github.com/user-attachments/assets/cbbeb1a1-dad9-42a3-96fb-0da48ae7b7a2"
/>

At first sight i also thought that a swarming job ended up being
considered for batch, but we can ignore that because the real root cause
to even being it considered for batch is because the job doesn't exist!

So, this kind of errors doesn't appear because of a swarming task was
considered for batch, it appears because a non existen job was
considered for batch, im positive this error happens because of the
problems between the job-exporter and because that's a job that i
manually inserted/scheduled to test the parent ticket.

v2.41.2

Toggle v2.41.2's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Handle Firebase email verification failures and log auth rejections (#…

…5489)

### Summary

Follow-up to #5487, addressing three issues found while reviewing and
rolling it out.

**1. `sendEmailVerification()` failures were silent (`login.html`).**

The promise had no rejection handler, so any failure left the login page
doing
nothing at all: no message, no sign out, and `return false` had already
stopped
the sign-in flow. This is easy to hit in practice —
`auth/too-many-requests`
when a user retries the sign-in, or `auth/missing-email` if the provider
does
not expose an email. The user is now told what went wrong, and
`firebase.auth().signOut()` runs on both the success and failure paths.

**2. The verification link did not bring the user back (`login.html`).**

`sendEmailVerification()` is now called with a continue URL pointing at
the
login page, so clicking the link in the email returns the user to
ClusterFuzz
instead of a generic Firebase confirmation page.

Firebase rejects a continue URL whose domain is not among the project's
authorized domains, which is not guaranteed for deployments serving on a
custom
domain. To avoid turning "works, with an extra manual step" into "no
verification email at all",
`auth/{invalid,missing,unauthorized}-continue-uri`
falls back to a plain verification email.

**3. Rejections in `get_current_user()` were not logged (`auth.py`).**
An unverified email and a service account rejection both returned `None`
silently, which is indistinguishable from an ordinary access denial when
triaging a "why can't I log in" report. Both now log a warning. The
service
account check is also commented to record that it is defense-in-depth:
service
accounts authenticate through IAP or a bearer token, and nothing in the
codebase mints Firebase custom tokens for them today.

The rejection logic itself is unchanged; `if not email or
utils.is_service_account(email)` is only split so each branch can log
its own
reason.

### Testing
* The server side has no behavior change (logging and comments only), so
the
existing `auth_test.py` coverage still applies.

---------

Co-authored-by: Paulo Borges <paulovlb@google.com>

v2.41.1

Toggle v2.41.1's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Require email_verified for all Firebase providers and enforce Firebas…

…e email verification on login (#5487)

### Summary
- Remove the `sign_in_provider != 'github.com'` exemption in
`auth.get_current_user()` so `email_verified` is required for all
Firebase auth providers (and reject service account emails from Firebase
session cookies).
- Add defense-in-depth `user.email_verified` checks to
`auth.is_current_user_admin()`, `access.get_access()`, and
`helpers.get_user_email()`.
- Keep supporting GitHub sign-in by checking
`authResult.user.emailVerified` in `login.html` and calling
`authResult.user.sendEmailVerification()` when a user signs in with an
email not yet verified in Firebase.
- Add unit tests in `auth_test.py`, `access_test.py`, and
`helpers_test.py`.

v2.41.0

Toggle v2.41.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Fix untrusted remote batch fuzz tasks to pass signed URLs (#5458)

Ensure tworkers create signed URLs for fuzz tasks by changing
`is_remote_utask` to evaluate fuzz tasks specifically based on
architecture.

Blackbox fuzzers on linux which require data bundles like
`inferno_webbot` and `lokihardt_jshitter` are broken because they can't
access their data bundles. Before the batch migration was complete, they
fetched it as trusted linux bots through gsutil. Now, the tworker
doesn't generate a signed URL because the fuzz task is evaluating to
UtaskLocalExecutor and so it is skipped, leaving the untrusted linux
batch bot without the signed URLs for the data bundle

This is somewhat hardcoded to fuzz tasks, but the execution of fuzz
tasks is unique.

Fixes b/556617562

## Explanation

1. tworker picks up fuzz task.
[get_command_object()](https://github.com/google/clusterfuzz/blob/3a24291fbd47ba627f20d7703aaa89c23d808f1c/src/clusterfuzz/_internal/bot/tasks/commands.py#L201)
-> resolves to UTask
2. Tworker preprocess checks whether it should create a signed URL.
`setup.preprocess_get_data_bundles()` ->
`corpus_manager.get_proto_data_bundle_corpus()` ->
`task_main_runs_on_uworker()`-> `is_remote_utask('fuzz')`
    reads `COMMAND_TYPES= UTaskLocalExecutor` -> `False`
3. "Not getting signed data bundle URLs" and `corpus_urls = []`
4. Batch uworker runs utask_main and fetches data bundles:
`sync_data_bundle_corpus_to_disk()` -> `is_uworker()` -> "signed-URL
path"
5. `download_signed_urls([])`: empty data bundle dir -> `IndexError:
Cannot choose from an empty sequence`


## Testing

tested in dev

v2.40.2

Toggle v2.40.2's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
[Android] Adds guardrails for non apk fuzzing (#5470)

## Overview
In a previous stack of PRs (See
#5453) I made a ton of changes
for improving android fuzzing overall error detection when executing
test cases, but, as i got pointed out, my changes assumed that android
fuzzing work exclusively under apks, so some of the checks/validations
didn't take into account jobs where no APK nor PKG_NAME is provided.

This PR focus on tuning various checks/validations previously altered in
the stack of Prs above, so that if in android fuzzing with a binary(with
this i mean non-apk fuzzing) then we go back to the usual checks that we
had in place.

## Changes
- Bad build check in `tescase_manager.py` now only checks for an apk
process liveness if an apk package is found.
- The test cases directory in `app.py` defaults to the old (non
accesible by apks) dir if no apk is found.
- When clearing the directory in `device.py` we now clean using the
`find` command, as my previous command could ignore the hidden
files(e.g: `.<filename>`)
- Exit info is only checked if fuzzing with an apk, else we ignore this

## Tests
- Introduces a ton of unit tests for  the non apk test cases
Fuzzing is correctly executed on valid APKS:
<img width="2600" height="1432" alt="image"
src="https://github.com/user-attachments/assets/b9824a4a-441c-470d-9335-f3bfcfc2a1bf"
/>

And it continues to fail on invalid apks.

---------

Co-authored-by: André Nogueira Ribeiro <94117783+decoNR@users.noreply.github.com>

v2.40.1

Toggle v2.40.1's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Fix schedule fuzz return (#5467)

### Problem

After go/clusterfuzz-config-pr/369, `schedule_fuzz` is configured to run
every 5 minutes, but the logs show it running 4 times per 5 minute
window:

<img width="703" height="332" alt="image"
src="https://github.com/user-attachments/assets/147b7eb1-646b-4a8a-903c-2b8dba3686c0"
/>

Turns out `schedule_fuzz.main()` doesn't return anything. `run_cron.py`
does [`return 0 if task_module.main() else
1`](https://github.com/google/clusterfuzz/blob/master/src/python/bot/startup/run_cron.py#L68),
so `None` becomes exit code 1. The cron has always been "failing" from
Kubernetes' point of view.

The CronJob has `restartPolicy: OnFailure` and `backoffLimit: 3`, so
every tick turns into 1 attempt + 3 retries with the usual 10s/20s/40s
backoff. That matches the gaps above exactly.

The annoying part is that each retry re-runs the scheduling logic from
scratch, so we can be publishing up to 4x the intended number of fuzz
tasks per cycle.

### Solution
`main()` returns `True` now. 

So exit 1 means "something unexpected broke", not "nothing got
scheduled".

### Validation
After this deploys, the log cadence should drop from 4 lines per 5
minutes to 1.

v2.40.0

Toggle v2.40.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
[Android] Dinamically set test cases dir under app's specific storage (…

…#5439)

Bug: b/545195031

Since Android API level 30(and all apps targeting android 11+), apps
have scoped storage access, and we can only give it external storage
permissions at runtime [by a user facing
dialog](https://developer.android.com/about/versions/11/privacy/storage#permissions-target-any).
To avoid this, we now copy the testcases to the app's external
app-specific storage , which is always readable by our app's package.

We didn't noticed this issues because all test cases execute using `am
start` which almost the 99% of the time returns no error codes, hence we
though that the test case executed but that is not the case.

Example for chrome: Previously, if we tried to execute our test case,
the app would tell us it can't find the file(because of the read
external storage permissions):
<img width="2498" height="1393" alt="image"
src="https://github.com/user-attachments/assets/f0054420-c6cd-4894-b642-29e1cc2904b9"
/>

But after this changes the html renders successfully!
<img width="2566" height="1589" alt="image"
src="https://github.com/user-attachments/assets/28e67cc6-b372-4d33-a070-c1f555bc4881"
/>


Learn more:
- https://developer.android.com/about/versions/11/privacy/storage
- https://developer.android.com/training/data-storage#scoped-storage

#### Alternative
- The other alternative is to serve the testcases html/js/css files
trough a simple localhost server trough `adb reverse tcp:8000 tcp:8000`,
this is what we usually do in regular chrome tests, clusterfuzz already
support's this, and this is controlled by the fuzzer, not by the job, if
the fuzzer testcases have the 'http' word in the filename then
clusterfuzz will serve them trough http, but i think nevertheless we
need to make this change to keep our options open for any case we don't
want to serve them trough http.

## Changes
- Always pushes files to this directory
`/sdcard/Android/data/{PKG_NAME}/files/` which is always reable by the
given app.
- Dynamically calculate the Test case directory based off the app's pkg
- no longer allows the job to override DEVICE_TESTCASES_DIR
- Now at fuzz session setup we clean from
`/sdcard/Android/data/{PKG_NAME}/files/*`.

v2.39.2

Toggle v2.39.2's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Reverts task short circuit for duplicate test cases (#5444)

This was added in #5405, but in b/555138064 we found that this is a use
case of Clusterfuzz to upload a better repro of an existing testcase and
reupload it manually.

v2.39.1

Toggle v2.39.1's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Harden DEPS parsing and sanitize revision metadata handling (#5438)

- Replace dynamic evaluation in DEPS parsing with a restricted AST-based
parser.
- Add revision string format validation and template-based URL
formatting.
- Sanitize issue metadata across uworker post-processing tasks before
persistence.
- Escape revision identifiers in testcase detail fallback rendering.

b/534004999

TAG=agy
CONV=6fa9c8c8-6088-4c02-b27d-ab873dda019f