I believe it's due to the encryption of github tokens that only work when the PR is done by someone who is member of the Piwik ORG. @diosmosis would there be a way to have artifacts uploaded for all PRs including external contributors?

Not for external contributors w/o making the artifacts password public (ie, unencrypted).

It would be helpful to have a solution for this. Maybe we can make the password public and instead add some security on the uploads server to make sure only expected things are uploaded (images etc)?
I just had a look at a PR with failing UI tests and had to run UI tests manually. Maybe there are other ideas?

instead add some security on the uploads server to make sure only expected things are uploaded (images etc)?

Should already be in place. Issue is w/ people potentially uploading lots of images or DDoS-ing. If there is some way to verify the request comes from travis-ci, then we don't need the password.

Not sure if that's possible as possibly pretty much anything can be changed/faked if someone wants to

Just thought of something: try removing the environment variables from .travis.yml and setting them here: https://travis-ci.org/piwik/piwik/settings . Maybe these will be used by PRs.

That's cool, haven't seen it before. I presume it's exactly for such cases maybe.

Actually in that case should make sure it's only the artifacts password that's used there. The autoupdate token shouldn't be forwarded to other repos.

Adding to 2.15.1 as it would be awesome to get UI tests artifacts for third party contributed Pull requests 👍

I will try it now

Actually, I just read this:

By default, the value of these new environment variables is hidden from the export line in the logs. This corresponds to the behavior of encrypted variables in your .travis.yml.

Similarly, we do not provide these values to untrusted builds, triggered by pull requests from another repository.

So it won't work as pull requests could read the secret token otherwise. It should work though if we made the token visible. In this case it would as well work to define it in .travis.yml in plain text. But then anyone could ddos again and we cannot verify request is made from travis

Got screenshots to upload for a PR (build: https://travis-ci.org/piwik/piwik/jobs/99020321, screens: http://builds-artifacts.piwik.org/piwik/piwik/master/17369/).

Works for plugins as well: (build: https://travis-ci.org/piwik/plugin-TasksTimetable/builds/99023364, screens: http://builds-artifacts.piwik.org/piwik/plugin-TasksTimetable/master/72/).

See ui-tests-viewer for relevant changes.

Closing this issue.

👍 Awesome

It doesn't seem to work for System tests. I've created a new issue: #9685