Moving from PhantomJS to Chrome

We used PhantomJS for automated UI testing for some time. It successfully helped us to be sure that our Polymer-component based pages behaved correctly. At least that’s what we thought.

We used Mocha for testing with mocha-phantomjs package. Our tests looked pretty much the same: we created some components, triggered events and evaluated the results. Some of the tests were asynchronous, some not. With each feature the amount of tests got higher, and we made sure all of them passed.

Migration from PhantomJS to headless chrome was very seamless. In the bottom line, all I did was changing mocha-phantomjs to mocha-headless-chrome in a couple of places. The interesting part came later, when I saw the results of the same tests we ran before.

The first interesting thing I found was that PhantomJS did not mention the assertion failure in the below code:

it('should fail with false condition', function() {
    const condition = false;
    setTimeout(function() {
        assert(condition);
    }, 10);
});
// PhantomJS tests
//     ✓ should fail with false condition
//
//   1 passing (4ms));

Of course, you say, it’s an asynchronous test, you missed the done() callback! And you are 100% percent right. But… This code slipped into production and was left there unnoticed. Until we made the move:

// PhantomJS tests
//   ✓ should fail with false condition
//
// 1 passing (12ms)
//
// [Error: AssertionError]

Did you notice the AssertionError on the last line? That was a clear sign that something was wrong. After being pointed directly to a bad test, it was a matter of minutes to figure out it was broken, and fix it:

it('should fail with false condition', function(done) {
    const condition = false;
    setTimeout(function() {
        assert(condition);
        done();
    }, 10);
});
//     1) should fail with false condition
// [Error: AssertionError]
//
//   0 passing (23ms)
//   1 failing
//
//   1) PhantomJS tests should fail with false condition:
//      Uncaught Error: Uncaught AssertionError: false == true

Another issue, more pleasant, was ES6 support. It was really frustrating to discover that I accidentally used let or Object.assign or shorthand Object initializer somewhere inside a component’s code. In PhantomJS, it broke the entire test silently, without showing related errors.

<!-- component code -->
<dom-module id="meh-meh">
    <template>
        <div>
            {{ item }}
        </div>
    </template>
    <script type="text/javascript">
        Polymer({
            is: 'meh-meh',

            properties: {
                item: {
                    value: function() {
                    // note the 'let' (es2015 syntax)
                        let thing = 'thing';
                        return thing;
                    },
                },
            },
        });
    </script>
</dom-module>

// test code
it.only('should have el.$ object', function() {
    const el = document.createElement('meh-meh');
    el['item'] = 'an item';
    document.body.appendChild(el);

    // $ is created by Polymer
    const condition = el.$ !== undefined;
    assert(condition);
});

In PhantomJS this test failed. In headless chrome, it worked fine.

The move from PhantomJS to headless chrome for UI testing enabled us to consider using ES6 for our client side code. It gave us an indication that something was wrong with our existing tests. It made client side tests faster. And, of course, it just felt good to improve our system’s build process with a small change in package.json.