Android buildbot failures after updating to API level 35 #142387
Description
Activity
The following SDK packages were installed on the x86_64 buildbot:
Path | Version | Description | Location ------- | ------- | ------- | ------- build-tools;34.0.0 | 34.0.0 | Android SDK Build-Tools 34 | build-tools/34.0.0 build-tools;35.0.0 | 35.0.0 | Android SDK Build-Tools 35 | build-tools/35.0.0 cmake;3.22.1 | 3.22.1 | CMake 3.22.1 | cmake/3.22.1 emulator | 35.1.21 | Android Emulator | emulator ndk;26.2.11394342 | 26.2.11394342 | NDK (Side by side) 26.2.11394342 | ndk/26.2.11394342 ndk;27.1.12297006 | 27.1.12297006 | NDK (Side by side) 27.1.12297006 | ndk/27.1.12297006 ndk;27.2.12479018 | 27.2.12479018 | NDK (Side by side) 27.2.12479018 | ndk/27.2.12479018 ndk;27.3.13750724 | 27.3.13750724 | NDK (Side by side) 27.3.13750724 | ndk/27.3.13750724 platform-tools | 35.0.2 | Android SDK Platform-Tools | platform-tools platforms;android-34 | 3 | Android SDK Platform 34 | platforms/android-34 platforms;android-35 | 2 | Android SDK Platform 35 | platforms/android-35 system-images;android-34;aosp_atd;x86_64 | 2 | AOSP ATD Intel x86_64 Atom System Image | system-images/android-34/aosp_atd/x86_64 system-images;android-35;aosp_atd;x86_64 | 1 | AOSP ATD Intel x86_64 Atom System Image | system-images/android-35/aosp_atd/x86_64At first it seemed like some of the tests involving pythontest.net were passing, as there were some "fetching" lines in the log in places such as test_hashlib. But it turns out these tests were actually being skipped when the download failed.
Upgrading
emulatorto version 36.3.10 was enough to fix the problem on this worker.As of today, the aarch64 buildbot had been up for 222 days, and had the following SDK packages installed:
Path | Version | Description | Location ------- | ------- | ------- | ------- build-tools;34.0.0 | 34.0.0 | Android SDK Build-Tools 34 | build-tools/34.0.0 build-tools;35.0.0 | 35.0.0 | Android SDK Build-Tools 35 | build-tools/35.0.0 cmake;3.22.1 | 3.22.1 | CMake 3.22.1 | cmake/3.22.1 emulator | 35.6.11 | Android Emulator | emulator ndk;27.2.12479018 | 27.2.12479018 | NDK (Side by side) 27.2.12479018 | ndk/27.2.12479018 ndk;27.3.13750724 | 27.3.13750724 | NDK (Side by side) 27.3.13750724 | ndk/27.3.13750724 platform-tools | 35.0.2 | Android SDK Platform-Tools | platform-tools platforms;android-34 | 3 | Android SDK Platform 34 | platforms/android-34 platforms;android-35 | 2 | Android SDK Platform 35 | platforms/android-35 system-images;android-34;aosp_atd;arm64-v8a | 2 | AOSP ATD ARM 64 v8a System Image | system-images/android-34/aosp_atd/arm64-v8a system-images;android-35;aosp_atd;arm64-v8a | 1 | AOSP ATD ARM 64 System Image | system-images/android-35/aosp_atd/arm64-v8a system-images;android-35;default;arm64-v8a | 2 | ARM 64 v8a System Image | system-images/android-35/default/arm64-v8aCheck disk space, since that's been a problem on this machine before
There was 37 GB free, much more than the 12 GB where it's had issues before.
Move the SDK and .android directories out of the way and recreate them from scratch.
Removed .android and allowed it to be recreated. Made no difference.
Removed android-sdk and recreated it from commandlinetools-mac-14742923_latest.zip, downloaded from https://developer.android.com/studio. Made no difference.
Reboot the machine
Made no difference, although it did unintentionally cause the buildbot worker to start.
The message is related to OpenGL, so maybe it'll make a difference to log in graphically.
Made no difference, regardless of whether I logged in as the admin user (
ec2-user), buildbot user (android), or ran android.py in a terminal within theandroiduser's graphical session – though that was probably reusing a Gradle daemon started in the SSH session, so it wouldn't have made any difference anyway.Check for OS updates
Updating macOS from 15.3.1 to 15.7.3 allowed the emulator to start! So my best guess is that there was some issue with that macOS version which wasn't exposed until we re-ran the Gradle-managed emulator snapshot creation process, which we probably hadn't done since the previous OS update.
However, the emulator was unable to access the internet, causing the same test failures as seen in the x86_64 buildbot above. All the emulator versions mentioned above had the same problem, but reducing the system image API level to 34 or lower made the problem go away.
So I think we'd better just treat the aosp_atd image as untrustworthy on API level 35. I'll make a PR to reduce it to 33, which while not as small as 35, is still much smaller than 34:
4249 /Users/android/android-sdk/system-images/android-33/aosp_atd 8333 /Users/android/android-sdk/system-images/android-34/aosp_atd 1688 /Users/android/android-sdk/system-images/android-35/aosp_atdReacted by Hugo van KemenadeSince this PR was merged, the API level was reduced again to 32 in #142912.
While testing #148054, I deleted the existing AVD on the aarch64 buildbot, and it failed to recreate with the same "EGLContext creation failed" errors as above. However, the problem isn't specific to that PR; it also happens on the main branch now that the old emulator is gone.
I don't know why the emulator is so fragile on this machine; I've never had such problems on my own Mac.
Although the system image was reinstalled, it's still at version 1, which I assume is the only version. The emulator version is also unchanged:
emulator | 35.6.11 | Android Emulator | emulator system-images;android-32;aosp_atd;arm64-v8a | 1 | AOSP ATD ARM 64 v8a System Image | system-images/android-32/aosp_atd/arm64-v8aThe macOS version is also unchanged at 15.7.3. However, the
uptimecommand showed that the machine was last rebooted 7 days ago (March 30). Logging in graphically gave me an alert that the machine had restarted because of a problem:panic(cpu 1 caller 0xfffffe0027584ee0): IOKit Daemon (kernelmanagerd) stall[0], (60s): registry root held busy, kernelmanagerd has not checked in @IOService.cpp:5829Details
Debugger message: panic Memory ID: 0x6 OS release type: User OS version: 24G419 Kernel version: Darwin Kernel Version 24.6.0: Wed Nov 5 21:33:59 PST 2025; root:xnu-11417.140.69.705.2~1/RELEASE_ARM64_T8112 Fileset Kernelcache UUID: 8EF81EBFB2DA156B9DB72F21B3FBCE3A Kernel UUID: 2BDF559B-BD91-31AC-8F09-345BD5A73C49 Boot session UUID: 4A8AA60F-DF67-4BF7-B95C-75A2D7D46F51 iBoot version: iBoot-11881.140.96 iBoot Stage 2 version: iBoot-13822.61.10 secure boot?: YES roots installed: 0 Paniclog version: 14 KernelCache slide: 0x000000001e640000 KernelCache base: 0xfffffe0025644000 Kernel slide: 0x000000001e648000 Kernel text base: 0xfffffe002564c000 Kernel text exec slide: 0x000000001fdcc000 Kernel text exec base: 0xfffffe0026dd0000 mach_absolute_time: 0x76a71d50 Epoch Time: sec usec Boot : 0x69ca527a 0x0009c958 Sleep : 0x00000000 0x00000000 Wake : 0x00000000 0x00000000 Calendar: 0x69ca52c9 0x0004dfcc Zone info: Zone map: 0xfffffe1016000000 - 0xfffffe3616000000 . VM : 0xfffffe1016000000 - 0xfffffe15e2000000 . RO : 0xfffffe15e2000000 - 0xfffffe187c000000 . GEN0 : 0xfffffe187c000000 - 0xfffffe1e48000000 . GEN1 : 0xfffffe1e48000000 - 0xfffffe2414000000 . GEN2 : 0xfffffe2414000000 - 0xfffffe29e0000000 . GEN3 : 0xfffffe29e0000000 - 0xfffffe2fac000000 . DATA : 0xfffffe2fac000000 - 0xfffffe3616000000 Metadata: 0xfffffec8fc010000 - 0xfffffec905810000 Bitmaps : 0xfffffec905810000 - 0xfffffec909e24000 Extra : 0 - 0 CORE 0 recently retired instr at 0xfffffe0026f94028 CORE 1 recently retired instr at 0xfffffe0026f92784 CORE 2 recently retired instr at 0xfffffe0026f94028 CORE 3 recently retired instr at 0xfffffe0026f94028 CORE 4 recently retired instr at 0xfffffe0026f94028 CORE 5 recently retired instr at 0xfffffe0026f94028 CORE 6 recently retired instr at 0xfffffe0026f94028 CORE 7 recently retired instr at 0xfffffe0026f94028 TPIDRx_ELy = {1: 0xfffffe241651a7b8 0: 0x0000000000000001 0ro: 0x000000016af0f0e0 } CORE 0 PVH locks held: None CORE 1 PVH locks held: None CORE 2 PVH locks held: None CORE 3 PVH locks held: None CORE 4 PVH locks held: None CORE 5 PVH locks held: None CORE 6 PVH locks held: None CORE 7 PVH locks held: None CORE 0: PC=0xfffffe0026e6849c, LR=0xfffffe0026e6849c, FP=0xfffffeca0ae6bed0 CORE 1 is the one that panicked. Check the full backtrace for details. CORE 2: PC=0xfffffe0026e6849c, LR=0xfffffe0026e6849c, FP=0xfffffeca0ba47ed0 CORE 3: PC=0xfffffe002a131dbc, LR=0xfffffe002a131e08, FP=0xfffffeca0b8a2c10 CORE 4: PC=0xfffffe0026e6849c, LR=0xfffffe0026e6849c, FP=0xfffffeca0a967ed0 CORE 5: PC=0xfffffe0026e684a0, LR=0xfffffe0026e6849c, FP=0xfffffeca0b63fed0 CORE 6: PC=0xfffffe0026e684a0, LR=0xfffffe0026e6849c, FP=0xfffffeca0ba23ed0 CORE 7: PC=0xfffffe0026e684a0, LR=0xfffffe0026e6849c, FP=0xfffffeca0a17bed0 Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space Panicked task 0xfffffe2eabf68e38: 248 pages, 4 threads: pid 110: watchdogd Panicked thread: 0xfffffe241651a7b8, backtrace: 0xfffffeca0b777490, tid: 2149 lr: 0xfffffe0026e2c72c fp: 0xfffffeca0b777520 lr: 0xfffffe0026f8a7dc fp: 0xfffffeca0b777590 lr: 0xfffffe0026f889b0 fp: 0xfffffeca0b777650 lr: 0xfffffe0026dd78b0 fp: 0xfffffeca0b777660 lr: 0xfffffe0026e2ca68 fp: 0xfffffeca0b777a30 lr: 0xfffffe00276bbfc4 fp: 0xfffffeca0b777a50 lr: 0xfffffe0027584ee0 fp: 0xfffffeca0b777b30 lr: 0xfffffe0026f51f70 fp: 0xfffffeca0b777b60 lr: 0xfffffe0026e33578 fp: 0xfffffeca0b777c00 lr: 0xfffffe0026e0a22c fp: 0xfffffeca0b777c60 lr: 0xfffffe0026e1dbc0 fp: 0xfffffeca0b777cb0 lr: 0xfffffe0026e1dfd8 fp: 0xfffffeca0b777da0 lr: 0xfffffe0026f7d954 fp: 0xfffffeca0b777e50 lr: 0xfffffe0026f88e80 fp: 0xfffffeca0b777f10 lr: 0xfffffe0026dd78b0 fp: 0xfffffeca0b777f20 lr: 0x0000000190c35c34 fp: 0x0000000000000000 last started kext at 981389383: com.apple.driver.AppleUSBODD 556 (addr 0xfffffe0025c6a390, size 1941) loaded kexts: com.apple.driver.AppleUSBODD 556 com.apple.iokit.SCSITaskUserClient 500.120.2 com.apple.driver.AppleUSBMassStorageInterfaceNub 556 com.apple.driver.CoreKDL 1 com.apple.driver.AppleBiometricServices 1 com.apple.driver.DiskImages.ReadWriteDiskImage 493.0.0 com.apple.driver.DiskImages.UDIFDiskImage 493.0.0 com.apple.driver.DiskImages.RAMBackingStore 493.0.0 com.apple.driver.DiskImages.FileBackingStore 493.0.0 com.apple.iokit.AppleBCM5701Ethernet 11.0.0 com.apple.driver.SEPHibernation 1 com.apple.driver.AppleUSBDeviceNCM 5.0.0 com.apple.driver.AppleThunderboltIP 4.0.3 com.apple.driver.AppleFileSystemDriver 3.0.1 com.apple.nke.l2tp 1.9 com.apple.filesystems.tmpfs 1 com.apple.driver.AppleSmartBatteryManager 161.0.0 com.apple.filesystems.nfs 1 com.apple.filesystems.lifs 1 com.apple.filesystems.apfs 2332.140.13 com.apple.IOTextEncryptionFamily 1.0.0 com.apple.filesystems.hfs.kext 683.120.3 com.apple.security.BootPolicy 1 com.apple.BootCache 40 com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0 com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1 com.apple.driver.AppleAOPVoiceTrigger 440.4 com.apple.driver.AppleTypeCRetimer 1.0.0 com.apple.AppleEmbeddedSimpleSPINORFlasher 1 com.apple.driver.AppleCS42L84Audio 840.26 com.apple.driver.ApplePMP 1 com.apple.driver.AppleSmartIO2 1 com.apple.driver.AppleSN012776Amp 840.26 com.apple.driver.AppleT8020SOCTuner 1 com.apple.driver.AppleT8112CLPC 1 com.apple.driver.AppleM68Buttons 1.0.0d1 com.apple.driver.ApplePMPFirmware 1 com.apple.AGXG14G 329.2 com.apple.driver.AppleProResHW 475.10 com.apple.driver.AppleAVD 865 com.apple.driver.AppleAVE2 803.73.1 com.apple.driver.AppleJPEGDriver 7.6.9 com.apple.driver.AppleMobileDispH14G-DCP 140.0 com.apple.driver.usb.AppleSynopsysUSB40XHCI 1 com.apple.driver.AudioDMAController-T8112 460.8.1 com.apple.driver.AppleSPIMC 1 com.apple.driver.AppleSamsungSerial 1.0.0d1 com.apple.driver.AppleSerialShim 1 com.apple.driver.AppleEventLogHandler 1 com.apple.driver.AppleS5L8960XNCO 1 com.apple.driver.AppleT8112PMGR 1 com.apple.driver.AppleS8000AES 1 com.apple.driver.AppleS5L8920XPWM 1.0.0d1 com.apple.driver.AppleS8000DWI 1.0.0d1 com.apple.driver.AppleInterruptControllerV2 1.0.0d1 com.apple.driver.AppleBluetoothModule 1 com.apple.driver.AppleBCMWLANBusInterfacePCIe 1 com.apple.driver.AppleT8110DART 1 com.apple.driver.AppleS5L8940XI2C 1.0.0d2 com.apple.driver.AppleT8110 1 com.apple.iokit.IOUserEthernet 1.0.1 com.apple.driver.usb.AppleUSBUserHCI 1 com.apple.iokit.IOKitRegistryCompatibility 1 com.apple.iokit.EndpointSecurity 1 com.apple.driver.AppleDiskImages2 385.140.4 com.apple.AppleSystemPolicy 2.0.0 com.apple.kec.InvalidateHmac 1 com.apple.kec.AppleEncryptedArchive 1 com.apple.iokit.IOSCSIMultimediaCommandsDevice 500.120.2 com.apple.iokit.IOBDStorageFamily 1.8 com.apple.iokit.IODVDStorageFamily 1.8 com.apple.iokit.IOCDStorageFamily 1.8 com.apple.driver.usb.cdc 5.0.0 com.apple.driver.AppleUSBAudio 750.4 com.apple.iokit.IOAudioFamily 600.2 com.apple.vecLib.kext 1.2.0 com.apple.driver.AppleConvergedIPCOLYBTControl 1 com.apple.driver.AppleConvergedPCI 1 com.apple.driver.AppleBluetoothDebug 1 com.apple.driver.AppleMesaSEPDriver 100.99 com.apple.iokit.IOBiometricFamily 1 com.apple.driver.DiskImages.KernelBacked 493.0.0 com.apple.driver.AppleUVDMDriver 1.0.0 com.apple.driver.AppleUVDM 1.0.0 com.apple.driver.AppleThunderboltPCIUpAdapter 4.1.1 com.apple.driver.AppleThunderboltDPOutAdapter 8.5.1 com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI 1 com.apple.driver.usb.AppleUSBXHCIPCI 1.2 com.apple.driver.AppleXsanScheme 3 com.apple.driver.AppleSEPHDCPManager 1.0.1 com.apple.driver.AppleTrustedAccessory 1 com.apple.iokit.AppleSEPGenericTransfer 1 com.apple.driver.usb.networking 5.0.0 com.apple.driver.AppleThunderboltDPInAdapter 8.5.1 com.apple.driver.AppleThunderboltDPAdapterFamily 8.5.1 com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4 com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1 com.apple.driver.AppleBTM 1.0.1 com.apple.nke.ppp 1.9 com.apple.driver.AppleBSDKextStarter 3 com.apple.kext.triggers 1.0 com.apple.driver.IOHIDPowerSource 1 com.apple.driver.AppleCallbackPowerSource 1 com.apple.filesystems.hfs.encodings.kext 1 com.apple.driver.AppleSyntheticGameController 12.6.1 com.apple.driver.usb.AppleUSBHostPacketFilter 1.0 com.apple.driver.AppleDCPDPTXProxy 1.0.0 com.apple.driver.DCPDPFamilyProxy 1 com.apple.driver.AppleDiagnosticDataAccessReadOnly 1.0.0 com.apple.driver.AppleHPM 3.4.4 com.apple.driver.AppleCSEmbeddedAudio 840.26 com.apple.driver.AppleAOPAudio 440.12 com.apple.driver.AppleEmbeddedAudio 840.26 com.apple.iokit.AppleARMIISAudio 440.17 com.apple.driver.IISAudioIsolatedStreamECProxy 440.17 com.apple.driver.ExclavesAudioKext 240.34 com.apple.driver.ApplePassthroughPPM 3.0 com.apple.driver.AppleA7IOP-MXWrap-v1 1.0.2 com.apple.driver.AppleT8103PCIeCPIODMA 1 com.apple.driver.AppleThunderboltNHI 7.2.81 com.apple.iokit.IONVMeFamily 2.1.0 com.apple.driver.AppleNANDConfigAccess 1.0.0 com.apple.driver.AppleMobileApNonce 1 com.apple.driver.AppleSPU 1 com.apple.AGXFirmwareKextG14GRTBuddy 1 com.apple.AGXFirmwareKextRTBuddy64 329.2 com.apple.driver.AppleMCA2-T8112 940.3 com.apple.driver.AppleEmbeddedAudioLibs 420.3 com.apple.iokit.IOGPUFamily 104.6.3 com.apple.driver.AppleH11ANEInterface 8.600.2 com.apple.driver.DCPAVFamilyProxy 1 com.apple.iokit.IOMobileGraphicsFamily-DCP 343.0.0 com.apple.driver.AppleDCP 1 com.apple.driver.AppleFirmwareKit 1 com.apple.driver.AppleFirmwareUpdateKext 1 com.apple.iokit.IOMobileGraphicsFamily 343.0.0 com.apple.driver.AppleM2ScalerCSCDriver 265.0.0 com.apple.driver.AppleStockholmControl 1.0.0 com.apple.driver.AppleSPMIPMU 1.0.1 com.apple.driver.AppleDialogPMU 1.0.1 com.apple.driver.AppleSPMI 1.0.1 com.apple.driver.AppleUSBXDCIARM 1.0 com.apple.driver.AppleUSBXDCI 1.0 com.apple.iokit.IOUSBDeviceFamily 2.0.0 com.apple.driver.usb.AppleSynopsysUSBXHCI 1 com.apple.driver.usb.AppleUSBXHCI 1.2 com.apple.driver.AppleEmbeddedUSBHost 1 com.apple.driver.usb.AppleUSBHub 1.2 com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2 com.apple.driver.AppleT8112TypeCPhy 1 com.apple.driver.AppleT8103TypeCPhy 1 com.apple.driver.AppleDisplayCrossbar 1.0.0 com.apple.iokit.IODisplayPortFamily 1.0.0 com.apple.driver.AppleTypeCPhy 1 com.apple.driver.AppleSART 1 com.apple.driver.ApplePMGR 1 com.apple.driver.AppleA7IOP-ASCWrap-v4 1.0.2 com.apple.driver.AppleARMWatchdogTimer 1 com.apple.driver.AppleT8112PCIe 1 com.apple.driver.AppleT8103PCIeC 1 com.apple.iokit.IOThunderboltFamily 9.3.3 com.apple.iokit.IOPortFamily 1.0 com.apple.driver.AppleBluetoothDebugService 1 com.apple.driver.AppleBCMWLANCore 1.0.0 com.apple.iokit.IO80211Family 1200.13.1 com.apple.driver.IOImageLoader 1.0.0 com.apple.driver.AppleOLYHAL 1 com.apple.iokit.IOAVBFamily 1320.3 com.apple.plugin.IOgPTPPlugin 1340.12 com.apple.driver.AppleGPIOICController 1.0.2 com.apple.driver.AppleFireStormErrorHandler 1 com.apple.driver.AppleMultiFunctionManager 1 com.apple.driver.corecapture 1.0.4 com.apple.driver.AppleEmbeddedPCIE 1 com.apple.driver.ApplePIODMA 1 com.apple.iokit.IOTimeSyncFamily 1340.12 com.apple.driver.DiskImages 493.0.0 com.apple.iokit.IOGraphicsFamily 599 com.apple.iokit.IOBluetoothFamily 9.0.0 com.apple.driver.AppleUIO 1 com.apple.driver.AppleSSE 1.0 com.apple.driver.AppleSEPKeyStore 2 com.apple.driver.AppleUSBTDM 556 com.apple.iokit.IOUSBMassStorageDriver 259.100.1 com.apple.iokit.IOPCIFamily 2.9 com.apple.iokit.IOUSBHostFamily 1.2 com.apple.driver.AppleUSBHostMergeProperties 1.2 com.apple.driver.usb.AppleUSBCommon 1.0 com.apple.driver.AppleSMC 3.1.9 com.apple.driver.RTBuddy 1.0.0 com.apple.driver.AppleEmbeddedTempSensor 1.0.0 com.apple.driver.AppleARMPMU 1.0 com.apple.iokit.IOAccessoryManager 1.0.0 com.apple.driver.AppleOnboardSerial 1.0 com.apple.iokit.IOSerialFamily 11 com.apple.iokit.IOSCSIBlockCommandsDevice 500.120.2 com.apple.iokit.IOSCSIArchitectureModelFamily 500.120.2 com.apple.driver.AppleRSMChannel 1 com.apple.iokit.IORSMFamily 1 com.apple.driver.AppleLockdownMode 1 com.apple.driver.AppleIPAppender 1.0 com.apple.iokit.IOSkywalkFamily 1.0 com.apple.driver.mDNSOffloadUserClient 1.0.1b8 com.apple.iokit.IONetworkingFamily 3.4 com.apple.driver.AppleFDEKeyStore 28.30 com.apple.driver.AppleEffaceableStorage 1.0 com.apple.driver.AppleCredentialManager 1.0 com.apple.driver.AppleSEPManager 1.0.1 com.apple.driver.IODARTFamily 1 com.apple.driver.AppleA7IOP 1.0.2 com.apple.driver.IOSlaveProcessor 1 com.apple.driver.AppleBiometricSensor 2 com.apple.iokit.IOHIDFamily 2.0.0 com.apple.AUC 1.0 com.apple.iokit.IOSurface 372.5.2 com.apple.iokit.IOAVFamily 1.0.0 com.apple.iokit.IOHDCPFamily 1.0.0 com.apple.iokit.IOCECFamily 1 com.apple.iokit.IOAudio2Family 1.0 com.apple.driver.AppleIISController 440.2 com.apple.driver.AppleAudioClockLibs 420.3 com.apple.driver.FairPlayIOKit 72.13.0 com.apple.driver.AppleARMPlatform 1.0.2 com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0 com.apple.iokit.IOReportFamily 47 com.apple.security.quarantine 4 com.apple.security.sandbox 300.0 com.apple.iokit.IOStorageFamily 2.1 com.apple.kext.AppleMatch 1.0.0d1 com.apple.driver.AppleMobileFileIntegrity 1.0.5 com.apple.iokit.CoreAnalyticsFamily 1 com.apple.security.AppleImage4 7.0.0 com.apple.kext.CoreTrust 1 com.apple.iokit.IOCryptoAcceleratorFamily 1.0.1 com.apple.kec.pthread 1 com.apple.kec.Libm 1 com.apple.kec.Compression 1.0 com.apple.kec.corecrypto 14.0 ** Stackshot Succeeded ** Bytes Traced 169019 (Uncompressed 412912) **Rebooting made no difference, but updating the emulator from version 35.6.11 to 36.5.10 fixed the problem.
On the x86_64 buildbot, the tests are running to completion, but giving errors about not being able to contact pythontest.net.
These recently appeared intermittently on the aarch64 buildbot, on the main and 3.15 branches only. There are several network-related failures, but the most common ones are:
====================================================================== FAIL: test_ftp_error (test.test_urllib2.HandlerTests.test_ftp_error) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/user/0/org.python.testbed/files/python/lib/python3.15/urllib/request.py", line 1531, in ftp_open host = socket.gethostbyname(host) socket.gaierror: [Errno 7] No address associated with hostname During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/data/user/0/org.python.testbed/files/python/lib/python3.15/test/test_urllib2.py", line 815, in test_ftp_error urlopen("ftp://www.pythontest.net/") ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/user/0/org.python.testbed/files/python/lib/python3.15/urllib/request.py", line 489, in open response = self._open(req, data) File "/data/user/0/org.python.testbed/files/python/lib/python3.15/urllib/request.py", line 506, in _open result = self._call_chain(self.handle_open, protocol, protocol + '_open', req) File "/data/user/0/org.python.testbed/files/python/lib/python3.15/urllib/request.py", line 466, in _call_chain result = func(*args) File "/data/user/0/org.python.testbed/files/python/lib/python3.15/urllib/request.py", line 1533, in ftp_open raise URLError(msg) urllib.error.URLError: <urlopen error [Errno 7] No address associated with hostname> During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/data/user/0/org.python.testbed/files/python/lib/python3.15/test/test_urllib2.py", line 817, in test_ftp_error self.assertEqual(raised.reason, ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ f"ftp error: {exception.args[0]}") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: gaierror(7, 'No address associated with hostname') != 'ftp error: 500 OOPS: cannot change directory:/nonexistent' ---------------------------------------------------------------------- ... ====================================================================== ERROR: test_getcode (test.test_urllibnet.urlopenNetworkTests.test_getcode) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/user/0/org.python.testbed/files/python/lib/python3.15/test/test_urllibnet.py", line 107, in test_getcode self.assertEqual(e.exception.code, 404) ^^^^^^^^^^^^^^^^ AttributeError: 'URLError' object has no attribute 'code' ----------------------------------------------------------------------These errors first appeared on 05-08, followed by 06-05, 06-09 and 06-11. On 06-15 there was a different network-related failure. After that, it started failing in various ways in about 1 run out of every 5.
Upgrading
emulatorto version 36.3.10 was enough to fix the problem on [the x86_64] worker.The aarch64 worker was already on 36.5.10 as a result of the issues in April above. There have been no macOS version updates or reboots since then, or any other changes on the machine as far as I know.
I tried updating the emulator to the current version 36.6.11. At first, it failed to start up about half of the time, and when it did start it was recreating the snapshot every time. After removing the
gradle-manageddirectory and allowing it to be recreated, things now appear to be normal. Testing with the following script:( while true; do if ! python Platforms/Android test --managed maxVersion -- -c 'from test import support; print(support.open_urlresource("http://www.pythontest.net/hashlib/blake2b.txt").read()[:50])'; then exit 1; fi; done )It passed for 15 iterations in a row, and never gave the "This may be transient, so continuing to wait" message which indicates that the snapshot is being recreated.
After reactivating the buildbot with emulator 36.6.11, it failed 1 out of the next 7 runs. I was able to reproduce this manually by running the above script for 65 iterations on the 3.15 branch, and then again after 11 iterations. I also reproduced it by running test_ftp_error for 45 and 29 iterations, so there doesn't appear to be any difference between the two methods. In both cases, each iteration took about 20 seconds. Rebooting made no difference.
I also tried emulator 36.3.10, which has been working fine on the x86_64 buildbot for the last 6 months, but it failed to start with the same EGLContext errors as above. Rebooting made no difference.
Going back to emulator 36.6.11, I tried the system image
android-35/aosp_atd, which previously failed consistently on this machine. But now, it succeeds consistently, reaching 200 iterations of the script. So I'll make a PR to switch to this image.- added 4 commits that reference this issue
on Jun 21, 2026
Bug report
Bug description:
This update was done in #142289 because the API level 34 system image was much larger, and was filling up the disk on GitHub Actions. It works fine in GitHub Actions, but caused different problems in the two Android buildbots:
On the aarch64 buildbot, the new system image failed to start, with its logs showing a repeated loop of system services crashing and restarting:
EGL-related messages leading up to the crash (from a different run):
I guessed this was caused by the emulator not being compatible with the emulator system image, so I installed the current emulator version 36.3.10, but it made no difference. Neither did 36.2.12, which is what I'd used to test the update on my own aarch64 Mac, or 35.6.11, which is the last in the 35 series. Unfortunately I didn't keep a note of what the previously installed version was.
The same thing happens with several other system images, even the API level 34 one we were previously using, either from the existing snapshot or a freshly created one.
I've taken the aarch64 buildbot offline until this is fixed. Next things to try:
On the x86_64 buildbot, the tests are running to completion, but giving errors about not being able to contact pythontest.net. It's possible this is caused by an emulator / system image incompatibility, but be careful in case the same thing happens here.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Other
Linked PRs