UI tests multiple browsers

[individual-it]

Description

This makes every UI test run in multiple browser.

What tests are run now:

• Chrome 48 on Linux, the newest Chrome that Saucelabs supports on Linux

• latest Chrome on Windows 10 (currently 57)

• latest-1 Chrome on Windows 10 (currently 56)

• oldest available Chrome (26) on Windows 10, as oC claims to run on Chrome 18+

• Firefox 47 on Windows 10
  webdriver for current Firefox is actually not ready, so we need to test on 47.0. see JWP actions endpoints do not get translated to W3C by standalone SeleniumHQ/selenium#3693 & https://stackoverflow.com/questions/38746148/behat-mink-force-selenium-driver-to-use-chrome-instead-of-firefox#38764599

• oldest available Firefox (14) on Windows 10, as oC claims to run on FF 14+

• IE 11
  IE has a problem with one test, getBoundingClientRect() throws an error. This is a known bug but I don't expect MS to fix it :-)
  so I just skip this test for IE

Because the tests are run now multiple times it takes them much longer to pass. This time will increase again as we add UI tests.
to discuss

1. what browser/OS combinations do we want to run.
   e.g. do we really want to support ancient browsers like FF 14 or Chrome 26? Or should the documentation just be changed to support only the latest versions of browsers, versions on ESR and maybe latest-1
2. do we want to run all UI tests on every PR?
   we could run only a part of the tests (limited by browser and/or test suits) on PRs and then run all tests on "master" or periodically. Then in case somebody thinks a special change should run all UI tests there can be a test-special branch that runs all tests.

Side note: The first problem, and that took me a while to figure out, was the fact that the setting of env. variables does not work as expected in travis when separating them by a semicolon. They were somehow not correctly set. Need to separate them by a space (according to travis documentation)

Motivation and Context

Currently all UI tests are only run inside Chrome 48 on Linux, its better to test multiple browsers to catch bugs that only happen in a certain browser

How Has This Been Tested?

It is the test :-) http://www.commitstrip.com/en/2017/01/03/developer-bedtime-paradox/

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

[DeepDiver1975]

tests are visible here: https://saucelabs.com/beta/builds/aaa2abaa35f84426adbe54c166035101

There seems to be an issue with the success state - can you please have a look? @individual-it THX

[individual-it]

from the saucelabs documentation:

After you've run your tests, you may notice in your Sauce Labs dashboard that the test status is Complete, rather than Passed/Failed. This is because Selenium doesn't have a way to interpret test results, only whether the test ran to completion or not. You have a few ways to set the Pass/Fail status of your tests.

You must use the the update_job method in the Sauce Labs REST API after the test has run.

I will check and implement it

[individual-it]

the status should now be reported to saucelabs and also travis will report a link to the saucelabs builds. So everybody can look at the video, screenshots etc.

But travis somehow doe not accept the job any-more. Any setting changed in the night?

[felixboehm]

1. make a reasonable choice
2. yes, on commits only a certain browser x suite should run. @DeepDiver1975 what do you think?
   We will setup nightly builds and tests to cover a larger matrix.

• Please document: How to run the tests with only one, two or all browsers from commandline?

[individual-it]

wrote documentation and made a PR here: owncloud-archive/documentation#3049

[individual-it]

reduced the amount of browsers, left the browsers that I personally would test for every PR

[DeepDiver1975]

the status should now be reported to saucelabs and also travis will report a link to the saucelabs builds. So everybody can look at the video, screenshots etc.

But travis somehow doe not accept the job any-more. Any setting changed in the night?

this is not working - no idea why - the travis console shows no sign of the command being executed

[individual-it]

working now

you should be able to get the badges now

[DeepDiver1975]

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.