Tags: google/clusterfuzz
Tags
[Mac] Add radamsa binary for ARM64 and update resolution logic When executing macOS ARM64 Swarming tasks, ClusterFuzz invokes the radamsa mutator binary. Previously, only an x86_64 macOS binary was bundled. Changes: * Add native macOS ARM64 binary (`bin/mac_arm64/radamsa`). * Update `get_radamsa_path()` in `engine_common.py`: - Prefer `radamsa` from `APP_DIR` (fuzzer build directory) if present. - Detect ARM64 on macOS and select arm64 `radamsa` over x86_64 `radamsa`. * Add unit tests for `get_radamsa_path()` in `engine_common_test.py`. Tests: * `python butler.py py_unittest -t core -p engine_common_test.py` (Succeed) * `python butler.py py_unittest -t core -p environment_test.py` (Succeed) Bug: 561690132 Signed-off-by: Manuel Briones <manuelbriones@google.com>
[Scheduling][Swarming] Retries tasks only on meant platform (#5495) As found out in the parent PR: #5476 In the scheduler, when a task fails to get scheduled in swarming, it then tries to go on to other backends, so if for any reason(e.g. the platform is not supported in batch per the batch config), then the whole slice of batch tasks get rejected. So for this PR, we make it so swarming tasks are only ever tried on swarming, and other tasks continue to be tried on other backends. Also when making this changes i thought of an edge case not considered previously, what if there was a swarming task in the queue & then we disable the flag? Then the scheduled would mark the task as not swarming & would try it on other backends, and again this could show errors if the platform is not supported there. So i changed the validations. With this changes task now appear in swarming! <img width="704" height="511" alt="image" src="https://github.com/user-attachments/assets/df9665e9-7257-4db6-a38d-22384b6eedcc" /> ## Changes - `swarming/__init__.py`: Now we allow to validate if a given job is swarming or not without considering the flag. - `remote_task_gate.py`: Splits tasks in swarming and other, swarming tasks get redirected to the swarming service and the unscheduled tasks from this service are not added back again with the other tasks ## Tests - A bunch of Tests + refactoring some tests to support this new edge cases. - Left this changes running in dev for 24 hours, and no longer saw the error that canceled the whole slice of batch tasks if the job platform didn't existed for batch, I only saw this error: <img width="1572" height="563" alt="image" src="https://github.com/user-attachments/assets/cbbeb1a1-dad9-42a3-96fb-0da48ae7b7a2" /> At first sight i also thought that a swarming job ended up being considered for batch, but we can ignore that because the real root cause to even being it considered for batch is because the job doesn't exist! So, this kind of errors doesn't appear because of a swarming task was considered for batch, it appears because a non existen job was considered for batch, im positive this error happens because of the problems between the job-exporter and because that's a job that i manually inserted/scheduled to test the parent ticket.
Handle Firebase email verification failures and log auth rejections (#… …5489) ### Summary Follow-up to #5487, addressing three issues found while reviewing and rolling it out. **1. `sendEmailVerification()` failures were silent (`login.html`).** The promise had no rejection handler, so any failure left the login page doing nothing at all: no message, no sign out, and `return false` had already stopped the sign-in flow. This is easy to hit in practice — `auth/too-many-requests` when a user retries the sign-in, or `auth/missing-email` if the provider does not expose an email. The user is now told what went wrong, and `firebase.auth().signOut()` runs on both the success and failure paths. **2. The verification link did not bring the user back (`login.html`).** `sendEmailVerification()` is now called with a continue URL pointing at the login page, so clicking the link in the email returns the user to ClusterFuzz instead of a generic Firebase confirmation page. Firebase rejects a continue URL whose domain is not among the project's authorized domains, which is not guaranteed for deployments serving on a custom domain. To avoid turning "works, with an extra manual step" into "no verification email at all", `auth/{invalid,missing,unauthorized}-continue-uri` falls back to a plain verification email. **3. Rejections in `get_current_user()` were not logged (`auth.py`).** An unverified email and a service account rejection both returned `None` silently, which is indistinguishable from an ordinary access denial when triaging a "why can't I log in" report. Both now log a warning. The service account check is also commented to record that it is defense-in-depth: service accounts authenticate through IAP or a bearer token, and nothing in the codebase mints Firebase custom tokens for them today. The rejection logic itself is unchanged; `if not email or utils.is_service_account(email)` is only split so each branch can log its own reason. ### Testing * The server side has no behavior change (logging and comments only), so the existing `auth_test.py` coverage still applies. --------- Co-authored-by: Paulo Borges <paulovlb@google.com>
Require email_verified for all Firebase providers and enforce Firebas… …e email verification on login (#5487) ### Summary - Remove the `sign_in_provider != 'github.com'` exemption in `auth.get_current_user()` so `email_verified` is required for all Firebase auth providers (and reject service account emails from Firebase session cookies). - Add defense-in-depth `user.email_verified` checks to `auth.is_current_user_admin()`, `access.get_access()`, and `helpers.get_user_email()`. - Keep supporting GitHub sign-in by checking `authResult.user.emailVerified` in `login.html` and calling `authResult.user.sendEmailVerification()` when a user signs in with an email not yet verified in Firebase. - Add unit tests in `auth_test.py`, `access_test.py`, and `helpers_test.py`.
Fix untrusted remote batch fuzz tasks to pass signed URLs (#5458) Ensure tworkers create signed URLs for fuzz tasks by changing `is_remote_utask` to evaluate fuzz tasks specifically based on architecture. Blackbox fuzzers on linux which require data bundles like `inferno_webbot` and `lokihardt_jshitter` are broken because they can't access their data bundles. Before the batch migration was complete, they fetched it as trusted linux bots through gsutil. Now, the tworker doesn't generate a signed URL because the fuzz task is evaluating to UtaskLocalExecutor and so it is skipped, leaving the untrusted linux batch bot without the signed URLs for the data bundle This is somewhat hardcoded to fuzz tasks, but the execution of fuzz tasks is unique. Fixes b/556617562 ## Explanation 1. tworker picks up fuzz task. [get_command_object()](https://github.com/google/clusterfuzz/blob/3a24291fbd47ba627f20d7703aaa89c23d808f1c/src/clusterfuzz/_internal/bot/tasks/commands.py#L201) -> resolves to UTask 2. Tworker preprocess checks whether it should create a signed URL. `setup.preprocess_get_data_bundles()` -> `corpus_manager.get_proto_data_bundle_corpus()` -> `task_main_runs_on_uworker()`-> `is_remote_utask('fuzz')` reads `COMMAND_TYPES= UTaskLocalExecutor` -> `False` 3. "Not getting signed data bundle URLs" and `corpus_urls = []` 4. Batch uworker runs utask_main and fetches data bundles: `sync_data_bundle_corpus_to_disk()` -> `is_uworker()` -> "signed-URL path" 5. `download_signed_urls([])`: empty data bundle dir -> `IndexError: Cannot choose from an empty sequence` ## Testing tested in dev
[Android] Adds guardrails for non apk fuzzing (#5470) ## Overview In a previous stack of PRs (See #5453) I made a ton of changes for improving android fuzzing overall error detection when executing test cases, but, as i got pointed out, my changes assumed that android fuzzing work exclusively under apks, so some of the checks/validations didn't take into account jobs where no APK nor PKG_NAME is provided. This PR focus on tuning various checks/validations previously altered in the stack of Prs above, so that if in android fuzzing with a binary(with this i mean non-apk fuzzing) then we go back to the usual checks that we had in place. ## Changes - Bad build check in `tescase_manager.py` now only checks for an apk process liveness if an apk package is found. - The test cases directory in `app.py` defaults to the old (non accesible by apks) dir if no apk is found. - When clearing the directory in `device.py` we now clean using the `find` command, as my previous command could ignore the hidden files(e.g: `.<filename>`) - Exit info is only checked if fuzzing with an apk, else we ignore this ## Tests - Introduces a ton of unit tests for the non apk test cases Fuzzing is correctly executed on valid APKS: <img width="2600" height="1432" alt="image" src="https://github.com/user-attachments/assets/b9824a4a-441c-470d-9335-f3bfcfc2a1bf" /> And it continues to fail on invalid apks. --------- Co-authored-by: André Nogueira Ribeiro <94117783+decoNR@users.noreply.github.com>
Fix schedule fuzz return (#5467) ### Problem After go/clusterfuzz-config-pr/369, `schedule_fuzz` is configured to run every 5 minutes, but the logs show it running 4 times per 5 minute window: <img width="703" height="332" alt="image" src="https://github.com/user-attachments/assets/147b7eb1-646b-4a8a-903c-2b8dba3686c0" /> Turns out `schedule_fuzz.main()` doesn't return anything. `run_cron.py` does [`return 0 if task_module.main() else 1`](https://github.com/google/clusterfuzz/blob/master/src/python/bot/startup/run_cron.py#L68), so `None` becomes exit code 1. The cron has always been "failing" from Kubernetes' point of view. The CronJob has `restartPolicy: OnFailure` and `backoffLimit: 3`, so every tick turns into 1 attempt + 3 retries with the usual 10s/20s/40s backoff. That matches the gaps above exactly. The annoying part is that each retry re-runs the scheduling logic from scratch, so we can be publishing up to 4x the intended number of fuzz tasks per cycle. ### Solution `main()` returns `True` now. So exit 1 means "something unexpected broke", not "nothing got scheduled". ### Validation After this deploys, the log cadence should drop from 4 lines per 5 minutes to 1.
[Android] Dinamically set test cases dir under app's specific storage (… …#5439) Bug: b/545195031 Since Android API level 30(and all apps targeting android 11+), apps have scoped storage access, and we can only give it external storage permissions at runtime [by a user facing dialog](https://developer.android.com/about/versions/11/privacy/storage#permissions-target-any). To avoid this, we now copy the testcases to the app's external app-specific storage , which is always readable by our app's package. We didn't noticed this issues because all test cases execute using `am start` which almost the 99% of the time returns no error codes, hence we though that the test case executed but that is not the case. Example for chrome: Previously, if we tried to execute our test case, the app would tell us it can't find the file(because of the read external storage permissions): <img width="2498" height="1393" alt="image" src="https://github.com/user-attachments/assets/f0054420-c6cd-4894-b642-29e1cc2904b9" /> But after this changes the html renders successfully! <img width="2566" height="1589" alt="image" src="https://github.com/user-attachments/assets/28e67cc6-b372-4d33-a070-c1f555bc4881" /> Learn more: - https://developer.android.com/about/versions/11/privacy/storage - https://developer.android.com/training/data-storage#scoped-storage #### Alternative - The other alternative is to serve the testcases html/js/css files trough a simple localhost server trough `adb reverse tcp:8000 tcp:8000`, this is what we usually do in regular chrome tests, clusterfuzz already support's this, and this is controlled by the fuzzer, not by the job, if the fuzzer testcases have the 'http' word in the filename then clusterfuzz will serve them trough http, but i think nevertheless we need to make this change to keep our options open for any case we don't want to serve them trough http. ## Changes - Always pushes files to this directory `/sdcard/Android/data/{PKG_NAME}/files/` which is always reable by the given app. - Dynamically calculate the Test case directory based off the app's pkg - no longer allows the job to override DEVICE_TESTCASES_DIR - Now at fuzz session setup we clean from `/sdcard/Android/data/{PKG_NAME}/files/*`.
Harden DEPS parsing and sanitize revision metadata handling (#5438) - Replace dynamic evaluation in DEPS parsing with a restricted AST-based parser. - Add revision string format validation and template-based URL formatting. - Sanitize issue metadata across uworker post-processing tasks before persistence. - Escape revision identifiers in testcase detail fallback rendering. b/534004999 TAG=agy CONV=6fa9c8c8-6088-4c02-b27d-ab873dda019f
PreviousNext