Solving the Mystery Behind Alternating Failed Tests in QUnit
One of the strange problems I faced with QUnit is a strange phenomenon where a custom written plugin fails the test on every alternate invocations of the test.
You cannot imagine the frustration of debugging the code and the testing framework for one whole day. Everything looks fine. So when I ran it the first time, the test fails. But when I refresh the page, the test passes. So I refresh the page again and the test fails! But press F5 (refresh), voila, it passes!
So, being the stubborn fella that I am, I refuse to accept this as a reality. After taking a good hard look, I finally accept that my code is fine. So the issue must lie with the testing framework. Of course searching for “QUnit fails on alternate invocations” yields no straight answers.
Going back to good ol’ research, I began looking at each page to look for some clues as to the cause. Funny enough, my breakthrough came from a Microsoft page ( http://msdn.microsoft.com/en-us/magazine/gg749824.aspx) of all places.
Of course the first few sections didn’t have any thing that I didn’t already know. Then I reached the section “More Efficient TDD”. Under it, I found out that QUnit is designed such that on a page refresh it runs the failed test first in an attempt to be more efficient in testing.
A simple concept. But it is also the part of the reason why my test fails and passes on alternate page refreshes.
To explain why, I need to first explain what I’m testing. I’ve written a plugin that automatically binds to existing elements on the page. So when the file containing the plugin is included in the page, the elements are binded automatically.
If so, then why does it work on alternate invocations? Ah… therein lies the magic. To make tests more efficient, QUnit will run the failed tests before other tests upon page refresh. So, when the failed tests are executed before other tests, the elements in the test harness that are already present are binded, and thus, the tests PASS! But because these tests now pass, when the page is refreshed they go back to the same order as they originally was, and thus they now FAIL!
Of course with this realization, it prompted me to rewrite the plugin more efficiently. Hope this self-discovery is helpful to someone out there! Have fun testing!