QPC: All platforms are unstable

Ref: https://ci.adoptopenjdk.net/job/QEMUPlaybookCheck/229/

With the latest QPC run, all platforms have failed to some degree. 4 have failed due calling buildJDK.sh in an incorrect way (incorrect as of #1962 ).

The risc-v platform is still blocked by #1483

And the arm32 platform seems to be running out of space during the playbook execution.

Changes made to qemuPlaybookCheck.sh have been made to use the correct arguments (willsparker/2123_1). Testing on QPC at: https://ci.adoptopenjdk.net/job/QEMUPlaybookCheck/230/

The above PR should fix the buildJDK.sh issues. Once this has been merged, I'll look at running the arm32 box, to determine if there's still issues with the space.

The PR to fix the arguments has been merged, and it appears to be working on all platforms. For the S390x and ppc64le architectures, the linux.sh script fails for JDK8, as it is unable to find JDK7 on the machines (as Zulu-7 is not installed). I thought that they were meant to default then to using JDK8, but apparently not. This is not the case for JDK11, where I have had the build start on both machines (though currently, I've only managed a QPC run that had ppc64le fail (maybe due to the wrong version of gcc..?), and s390x that core dumped (maybe due to a bad jdk-10 install..?))

I'm going to look into that platform-specific-configuration script, to see if it's meant to set the boot jdk to the build jdk in cases wehre the boot jdk can't be found. If it isn't, I'll look into install zulu-7 on those platforms.

It can always be overriden with JDK7_BOOT_DIR if needed, but I know we've tried to let the autodetection work.
My preference is for it not to fall back if possible, since you then run the risk of being unsure whether from one build to the next it's used a JDK7 or JDK8 as the boot JDK. Our "real" CentOS/RHEL7 ppc64le and s390x build machines all use a true JDK7.

Okay! Thanks for the guidance. Looks like I'll have to adapt the Zulu-7 ansible task to include those platforms :-)

EDIT: Turns out no, this can't be done (see: https://adoptium.slack.com/archives/C53GHCXL4/p1619777906294700). Best case is having those platforms use JDK-8 to build

With #2176 (comment) , I think the last part of fixing the non RISC-V platforms is to figure out how to extend the build images, as they run out of space halfway through a build.

QEMU qcow2 images can be resized easily enough with
qemu-img resize $IMG +10G (if you wanted to add 10GB to it).
However, the partitions still need to be extended, which will be specific to the platform, and is likely to be a massive pain. Helpfully, I think all of the platforms that fail due to disk issues will fail in the build. Therefore, we could mount it as a new partition and just build on that one if it's that difficult.

EDIT: AH hang on, I've already documented all of this. Okay, just going to extend them by 10GB, and create a new partition on /home/linux that will have that extra 10GB. This method should be fine for all images that don't have a swap partitions. I'll create backups of the current images, just in case :-)

@sxa

List of what I did to extend each of the images. After extending partitions, I'll re-compress the image with xz -z, and move the images into /home/jenkins/qemu_base_images/resized_images/, that way we still have backups of the old images

Debian10.aarch64.dsk :
Image is qcow2, so extend by running qemu img resize $IMAGE +10G

When logging into the machine, it automatically extended the root partition, which was lovely to see. So that has 25GB on it now.

Debian11.riscv.dsk:
qcow2 image, so same command as above.
This one has 10GB mounted on the root partition, and 5GB on /home/linux. So, I'll delete the /home/linux partition and create a new one in the same place, of size 15GB:

root@debian:~# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    254:0    0  25G  0 disk 
|-vda1 254:1    0  10G  0 part /
`-vda2 254:2    0   5G  0 part /home/linux
root@debian:~# fdisk /dev/vda
Command (m for help): d
Partition number (1,2, default 2): 2

Partition 2 has been deleted.

Command (m for help): n
Partition number (2-128, default 2): 2
First sector (20971393-52428766, default 20971520): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (20971520-52428766, default 52428766): 

Created a new partition 2 of type 'Linux filesystem' and of size 15 GiB.
Partition #2 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: Y

The signature will be removed by a write command.
Command (m for help): w
The partition table has been altered.
Syncing disks.

root@debian:~# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    254:0    0  25G  0 disk 
|-vda1 254:1    0  10G  0 part /
`-vda2 254:2    0  15G  0 part /home/linux

Debian8.Arm32.dsk:
Still a qcow2 image, so, as above

This ones a bit different, there's a lot more partitions:

$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    254:0    0   30G  0 disk 
├─vda1 254:1    0  243M  0 part /boot
├─vda2 254:2    0  5.3G  0 part /
├─vda3 254:3    0    1K  0 part 
├─vda5 254:5    0  502M  0 part [SWAP]
└─vda6 254:6    0   14G  0 part /home

Unfortunately, I can't seem to find a way of extending the root partition, without the following error / issue on startup

[  155.371104] FS-Cache: Netfs 'nfs' registered for caching
Welcome to emergency mode! After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" to try again
to boot into default mode.
[  155.698568] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Give root password for maintenance
(or type Control-D to continue):

I tried removing partitions 2-6, and recreating the / and /home partition, but the same issue occurs. I tried just removing the second and third partition, and it caused the rest of them to disappear as well. I'm going to skip this one for now, in the hopes that somebody else knows how to do this ( @sxa ...? 👀 )

Ubuntu1804.arm64.dsk:

The disk format is raw. I was able to extend it with qemu-img resize <> +10G though, and it is showing up in lsblk:

NAME                             MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                              252:0    0   30G  0 disk 
├─vda1                           252:1    0  512M  0 part /boot/efi
└─vda2                           252:2    0 19.5G  0 part 
  ├─ubuntu--18--arm64--vg-root   253:0    0 18.6G  0 lvm  /
  └─ubuntu--18--arm64--vg-swap_1 253:1    0  976M  0 lvm  [SWAP]

However, whenever I run any fdisk commands, I get
GPT PMBR size mismatch (41943039 != 62914559) will be corrected by w(rite)., which fails when I exit, with fdisk: failed to write disklabel: Invalid argument. For some reason, fdisk can't fix this, but parted can. Running parted -l will prompt a fix/ignore check. After this, I followed the instructions from this link - created a new partition with fdisk (called /dev/vda3) with the remaining space, created the physical volume pvcreate /dev/vda3, extended the volume group with vgextend /dev/ubuntu-18-arm64-vg /dev/vda3. From there, I extended the LV that was needed lvextend +2568 /dev/ubuntu-18-arm64-vg/root and resized it to the file system : resize2fs /dev/ubuntu-18-arm64-vg/root.
The result is:

root@ubuntu-18-arm64:~# lsblk
NAME                             MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                              252:0    0   30G  0 disk 
├─vda1                           252:1    0  512M  0 part /boot/efi
├─vda2                           252:2    0 19.5G  0 part 
│ ├─ubuntu--18--arm64--vg-root   253:0    0 28.6G  0 lvm  /
│ └─ubuntu--18--arm64--vg-swap_1 253:1    0  976M  0 lvm  [SWAP]
└─vda3                           252:3    0   10G  0 part 
  └─ubuntu--18--arm64--vg-root   253:0    0 28.6G  0 lvm  /

Which looks alright to me 👍

Ubuntu18.ppc64le.dsk:

Also uses LVM, and a raw disk image, so I'll do the same process as before, hopefully. Initial lsblk shows (after extending the disk):

NAME                 MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                    8:0    0   25G  0 disk 
├─sda1                 8:1    0    7M  0 part 
└─sda2                 8:2    0   15G  0 part 
  ├─linux--vg-root   253:0    0 14.3G  0 lvm  /
  └─linux--vg-swap_1 253:1    0  676M  0 lvm  [SWAP]
sr0                   11:0    1 1024M  0 rom

I wonder if sr0 there is going to cause issues ..
Nope, didn't seem to. Having followed the same instructions as before (and instructions on how to extend the LVM here):

root@linux:~# lsblk
NAME                 MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                    8:0    0   25G  0 disk 
├─sda1                 8:1    0    7M  0 part 
├─sda2                 8:2    0   15G  0 part 
│ ├─linux--vg-root   253:0    0 24.3G  0 lvm  /
│ └─linux--vg-swap_1 253:1    0  676M  0 lvm  [SWAP]
└─sda3                 8:3    0   10G  0 part 
  └─linux--vg-root   253:0    0 24.3G  0 lvm  /
sr0                   11:0    1 1024M  0 rom

Ubuntu18.S390x.dsk :

This one is a QCOW2 image, so same command as all of them. This one is also LVM, but for some reason, didn't require parted -l.

root@ubuntu:~# lsblk
NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                   252:0    0   25G  0 disk 
└─vda1                252:1    0   15G  0 part 
  ├─ubuntu--vg-root   253:0    0   14G  0 lvm  /
  └─ubuntu--vg-swap_1 253:1    0  964M  0 lvm  [SWAP]

After the same above process:

NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                   252:0    0   25G  0 disk 
├─vda1                252:1    0   15G  0 part 
│ ├─ubuntu--vg-root   253:0    0 24.1G  0 lvm  /
│ └─ubuntu--vg-swap_1 253:1    0  964M  0 lvm  [SWAP]
└─vda2                252:2    0   10G  0 part 
  └─ubuntu--vg-root   253:0    0 24.1G  0 lvm  /

I'm glad I/whoever setup the Ubuntu Images, had the foresight to make the partitions LVM :-)

With all but the Debian.ARM32 image extended, I'll swap the original images with the resized images to test on the vagrant server. I'm aware that now QPC can run on build-equinix-ubuntu2004-armv8-1, so I'll quickly reconfigure the job to just use the infra-ibmcloud-vagrant* machines for the time being. If all goes well, I'll move the images over :-)

See:
QPC#264

EDIT: Looks like the Debian ones failed - it looks like debian10/aarch64 didn't boot in time, and something else wrong with debian11/riscv64. Not too worried about RISCV, as it is currently running fine in QPC#263

Nope, I messed up the RISCV one - this happened 3 times now:

08:46:58 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

08:46:59 /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

08:47:00 mkdir: cannot create directory '.ssh': Permission denied

I would assume in the process of extending the partition, the Linux user no longer owns their own home directory. Redid that, and tested here: QPC#267

It still looks like the Debian10/aarch64 one isn't booting in time. I'll try a local test where I extend the boot time to 180s, to see if this fixes it.

Bright side, both Ubuntu18/ppc64le and Ubuntu18/aarch64 seem to be fine after the image resize, in QPC#264- ppc64le actually passed, and aarch64 failed the test, though it was able to fully complete a build, which is nice.

Ubuntu18/s390x appears to be failing on apt-get upgrade :

Processing triggers for initramfs-tools (0.130ubuntu3.12) ...
update-initramfs: Generating /boot/initrd.img-4.15.0-74-generic
Using config file '/etc/zipl.conf'
Run /lib/s390-tools/zipl_helper.device-mapper /boot
Error: Unsupported setup: Directory '/boot' is located on a multi-target device-mapper device
Error: Script could not determine target parameters
run-parts: /etc/initramfs/post-update.d//zz-zipl exited with return code 1
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 linux-firmware
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

Presumably this is an issue with the root file system now being on a several partitions. Interesting that this wasn't an issue with the other Ubuntu machines. I could apt hold the linux-firmware and initramfs-tools packages to 'fix' the issue, but that feels wrong ...

Navigation Menu

Repository navigation

QPC: All platforms are unstable #2121

63 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

QPC: All platforms are unstable #2121

Description

Activity

Willsparker commented on Apr 13, 2021

Willsparker commented on Apr 15, 2021

Willsparker commented on Apr 27, 2021

sxa commented on Apr 27, 2021

Willsparker commented on Apr 29, 2021

Willsparker commented on May 10, 2021

Willsparker commented on May 13, 2021

Willsparker commented on May 17, 2021

Willsparker commented on May 18, 2021

Willsparker commented on May 19, 2021

Willsparker commented on May 20, 2021

63 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions