Evaluating Element Timing for images

By Peter Hedenskog, Senior Software Engineer, Wikimedia Performance Team

Background

One of the tasks we in the Performance Team have been struggling with is finding better metrics that can tell us more about the user experience than the technical metrics we usually get out of browsers.

We started out 2015 trying to find a way to know when images are displayed for the user. We tried out the latest patterns at that moment in T115600. We used our WebPageTest instance to record a video of the browser loading the Obama page, and followed state of the art technology at that moment using a User Timing mark to fire when the image was displayed.

The results were very disappointing. The mark was at 2.0 seconds, but as you can see in the screenshot, the image was displayed at 4.8 seconds. It was off by 2.8 seconds 🙁 We did multiple tests and we got the same result multiple times. We tried the state-of-the-art technique people where talking about and it was clearly completely wrong. This taught us the important lesson the reliability of new RUM metrics we decide to collect need to be verified in synthetic testing using a video recording of the browser.

The next attempt to measure when images appear was when WebPageTest added support for visual element metrics (meaning analyzing a video and getting metrics for specific elements), but that only helps us with synthetic testing. We also want better metrics collected directly from our users.

Element timings

@Gilles has been working on enabling origin trials for Chrome for us to verify the effectiveness and usefulness of upcoming performance APIs. Recently we enabled the Trial for Element Timing for Images on Russian Wikipedia. The goal of this API is to report exactly what we had been looking for: when an image is actually displayed to the user.

Let’s verify the accuracy of this new metric and see if it works better than old approximations marked with user timings.

Evaluating element timings

Using Browsertime we record a video of the screen and run some extra JavaScript to collect the new metric. Then we compare the metric we get from JavaScript with the one we get from the video.

The first large image in an article is named thumbnail-high, so we know which one to use. The following JavaScript snippet is what allows us to get the Element Timing metric just for that element:

(function() {
    const elements = performance.getEntriesByType('element');
    for (let element of elements) {
        if (element.name === 'thumbnail-high') {
            return element.startTime;
        }
    }
})();

This is passed to Browsertime, which runs it after the page has loaded. Visual Elements are enabled, which analyses the video and gives us a timing corresponding to when the largest image within the viewport is displayed (which for most articles, is the thumbnail-high image).

$ docker run --rm -v "$(pwd)":/browsertime sitespeedio/browsertime:4.6.0 --script thumbnail-high.js https://ru.wikipedia.org/wiki/Древесные_стрижи -n 11 --visualElements

This was run on two different connectivity types and 11 times in a row. Then we keep the median for both metrics and we get the following:

The video recording performed by browsertime is done at 30 frames per second. Which means each frame lasts 1000/30 = 33.333ms. This indicates that the differences seen between Element Timing and the video analytics are within one frame. Element Timing might very well be the more accurate one, since it’s not constrained by the video recording’s 30fps cadence.

That looks really promising and very accurate, particularly compared to old workarounds. We tested a couple more URLs that you can see in T219231 and they showed the same result.

For our content, it looks like the Element Timing API finally provides a way for us to know accurately when images are really displayed to users!

About this post

This post was originally published on the Wikimedia Performance Team Phame blog.

Featured image credit: Barack Obama hand on Resolute Desk, Pete Souza, public domain