Add tests for WebAgg using Playwright - #23540

I'm not sure what to do about the screenshot test, as I really want to confirm that the figure is sent to the browser, but browsers differ in how they show the resize grip, which causes te comparison to fail.

Can you mask the resize grip by drawing a white rectangle on top using imagemagic?

result = next(iter(done)).result()
E       playwright._impl._api_types.Error: Executable doesn't exist at /home/vsts/.cache/ms-playwright/chromium-1015/chrome-linux/chrome
E       ╔════════════════════════════════════════════════════════════╗
E       ║ Looks like Playwright was just installed or updated.       ║
E       ║ Please run the following command to download new browsers: ║
E       ║                                                            ║
E       ║     playwright install                                     ║
E       ║                                                            ║
E       ║ <3 Playwright Team                                         ║
E       ╚════════════════════════════════════════════════════════════╝

looks like we need some extra steps in the CI setup?

I'm not sure what to do about the screenshot test, as I really want to confirm that the figure is sent to the browser, but browsers differ in how they show the resize grip, which causes te comparison to fail.

Can you mask the resize grip by drawing a white rectangle on top using imagemagic?

Actually, I forgot we have access to which browser is running the test, so I could just have two test images... I'll see what the diff looks like.

So this is nearly working on GitHub Actions. The current failure is on macOS. For some reason, the canvas gets a border and it's black on Linux, but blue on macOS. There's also some slight difference in the resize grip size.

AppVeyor only fails one test, so it's likely a similar deal.

Azure passes on Linux, but fails on macOS and Windows, but on several more tests. I'm not sure why that is yet.

Current status:

1. Firefox renders the resize grip differently across platforms; I've added a simple edit with Pillow to smooth out those differences.
2. WebKit seems to be broken on Ubuntu (it generates an empty screenshot, and resizing fails), but the build that Playwright ships requires very old libraries, so I can't test it on Fedora.
3. WebKit fails on Windows, for interactive tests. The screenshot does appear to pass.
4. The macOS 14 build seems to get stuck for currently unknown reasons.

On all other platforms, Chromium and Firefox pass all tests.

On web-based projects it is common to do functional tests across the whole matrix of OS and web browser, but only take screenshots on one (linux chrome) to minimise the churn in test images caused by regular browser updates. It should be less of a problem here as the screenshot images are really those generated by Agg, but it is an option.

It might be worth adding some real plotted data to the screenshots, maybe just a line plot, to prove it works.

We did have a case where things broke on a specific browser #18142, so I would like to check all of them if possible. That was of course due to something that was eventually standardized and not likely to be something to worry about in the future, but who knows what else we might use later that is inconsistent.

I did limit the screenshot to just the canvas part that should come from us, to avoid most browser-specific UI (other than the resize grip which has been edited out.)

OK, I managed to set up a toolbox container in order to try out WebKit locally. It seems rather annoying:

1. Though caniuse.com says resize:both works on desktop Safari, it also doesn't work on iOS Safari. The build of WebKit that Playwright uses seems to follow the latter; I even tried the example from MDN and there was no resize grip and no reaction to the cursor anywhere near the corner. But it does seem to work on real macOS according to the XPASS in CI?
2. There seems to be some kind of timing issue with the canvas or the screenshot tool. I modified the test a bit so that we should have Playwright wait until after it's sent (it's a bit roundabout, (as we wait for an ack from the browser in order to change the title in the browser, so that we can have Playwright wait for the browser), but I couldn't find a way for Playwright to just wait for a callback on our side.) Even though both with and without the waits work fine in Firefox and Chrome, in WebKit we sometimes just get an empty figure. This even occurs on CI here. Locally, I can run with --slowmo=200 (or even --tracing=on/--video=on) and that makes it pass. I understand loading an image can be inherently async, but I moved the ack to the onload for the image just for that so it should occur after we've actually updated the canvas.

Based on microsoft/playwright#31017, I've decided to only check WebKit on macOS. Other platforms are marked as XFAIL.