I woke up this morning and like a lot of people, started going through emails on my phone. The first email I read was a test failure on one of the UATs that had run at midnight. I had been working on this specific test for a while now because I kept receiving random errors. Here is how I fixed it.
We have a user acceptance testing (UAT) job that runs every time we deploy to a new environment and it also runs nightly.
clickOnAButton(); wait(10); // seconds assertValue();
The problem with the above, is that if the action performed by "clickOnAButton" takes longer than 10 seconds, the assertion will fail. On a "fast" server it will pass, but if the server takes a bit longer it will fail.
We obviously had to improve our tests. What we did is introduce waits for actual UI elements instead of x amount of time. The code above became:
clickOnAButton(); waitForLoadingToFinish(); assertValue();
The "waitForLoadingToFinish();" function checked that the loading indicator element was no longer visible THEN it would execute the assertValue() function.
This worked great for a while. However this specific test kept having "random" failures. After it failed yet another time last night I decided to spend some time on it.
The issue was that the "clickOnAButton()" function was doing some calculations and after it finished, it would make a server call and show the loading indicator.
WebdriverJS is incredibly fast at executing test code thus whenever the clickOnAButton function would take too long, the waitForLoadingToFinish() function would return true because the loading indicator wasn't showing in the first place.
After realizing this classic mistake, I refactored the test code into the following:
clickOnAButton(); waitForLoadingToShow(); waitForLoadingToFinish(); assertValue();
The waitForLoadingToShow function will mostly always return true right away, but for those cases where it is taking a while, it will actually tell the selenium driver to slow down and wait for the loading to show first then wait for it to hide.
Bottom line, don't expect the selenium driver to go at the same speed as a human, it will be much much faster and thus you need prepare your test code to slow it down when it needs to.