Web Apocrypha

For some time it has been possible to build and run QtWebKit on Linux using Google's V8 JavaScript engine instead of the default JavaScriptCore. I thought it would be good to see some numbers comparing the runtime performance of the two engines in the same environment and also measuring the performance of the browser bindings.

The Machine

Zoltan was kind to let me use his dedicated benchmarking machine, which he used previously in his memory allocator benchmarks. The machine has an x86 dual-core 2.33GHz CPU and runs Slackware 13.1.0 on an SMP kernel (Goodbye to poor old Debian Lenny). The measurements were done with the r73358 revision of the Linux-Qt port of WebKit using Qt version 4.7.1.

The Benchmarks

The overall in-browser performance was measured using our JavaScript-based page load benchmarking engine called Methanol, which soon will serve as our continuous page loading test (PLT) and its results will be published in the Benchmark section.

Before quickly introducing Methanol and presenting the benchmark results I'll show the WindScorpion, SunSpider and V8 results, which measure the JavaScript performance itself. To ease the comparison and avoid confusion these JavaScript test suites were converted to run inside the SunSpider framework using the SunSpider/make-hosted script from WebKit trunk. I ran the tests five times on the machine, the results were stable, so I chose to depict the charts of the overall median runtime.

WindScorpion

First let's check out the WindScorpion benchmark, which is our own compilation of tests:

The numbers on the chart correspond to the following tests:

bubbleSort:    1
des:           2
dicePoker:     3
email:         4
factor:        5
floyd:         6
formatNumber:  7
genetic:       8
huffman:       9
IEEE754Conv:  10
longFact:     11
quickSort:    12
redBlackTree: 13
solve:        14
xmlParser:    15

As you can see on the above bar chart, QtTestBrowser with V8 is more than twice as fast than with JSC, on average. However, the interesting parts here are the des (Data Encryption Standard) test (#2 on the chart: 228.4 ms for V8 compared to 157.3 ms for JSC) and the IEEE754Conv test (#10: the result for V8 was 1645.0 ms compared to 1612.3 ms for JSC), which converts between decimal representation of numbers and the binary IEEE 754 floating point format. On these tests V8 seems to be slightly slower. On the other hand, among others, the huffman (#9: 42.0 ms for V8 and 544.8 ms for JSC), floyd (#6: 133.1 ms for V8 and 739.8 ms for JSC) and the xmlParser (#15: 780.1 ms for V8 compared to 4325.1 ms for JSC) tests show a giant speedup compared to JSC.

SunSpider

Now let's look at the following SunSpider results:

As you can see on SunSpider too, QtTestBrowser with V8 is approximatelly ~1.28x faster than JSC. But it also shows some interesting slowdowns in the 3d-cube, bitops-bitwise-and, crypto-md5, crypto-sha1 tests. Here the most significant speedup is in access/binary-trees which is 4x faster and string/base64 which is 2.43x faster with V8 than with JSC.

V8

Here follow the V8 results:

As expected on the V8 test suite QtTestBrowser with V8 confidently defeats JSC on all tests, so no interesting parts here.

Testing the new operator

By investigating the JavaScript benchmarks more closely, we had the feeling that object allocation might be a reason for the difference between the two JS engines. Gabor suggested the following test to check the performance of the new operator:

  var a = function() {}
  var dateDiff = new Date();
  for(var i = 0; i < 1024 * 1024 * 1024; ++i)
    new a;
  document.write(new Date() - dateDiff);

The results of this test were:

The new operator seems to be twice as fast on V8 compared to JSC, as the results of Gabor's test show. This is on par with the overall WindScorpion and V8 suite results.

Methanol

Now that we are confident enough that QtWebKit with V8 is faster than with JSC let us look at the Methanol results. Methanol uses QtTestBrowser to load locally mirrored webpages as test cases (currently the top 20 most popular sites of Alexa Top 500, downloaded at the end of April, 2010), and measures the overall page layouting and rendering time of these front pages one by one. First, it loads a page to fill the caches, then loads the same page five times in sequence and measures the average page loading time. Methanol of course does not measure the JavaScript engine performance itself, instead it lets us conclude something about how the engine is tied to the browser.

To get a statistically relevant data set I ran the test suite 50 times, determined the minimum, maximum, lower and upper quartile and the median runtimes of each individual pageload as depicted in the per-page results below.

Let's see the summarised results first:

As you can see V8 doesn't perform as well in the page loading test as it does in JavaScript tests. V8 was at about 5% slower than JSC. The next graph shows the maximum, minimum and median results of the test sessions measured during the fifty runs.

The following box-and-whisker diagram shows the per-page results.

As you can see there are at least 5 pages out of 20 which are conclusively faster with V8 but this is not enough to win the overall comparison. Further more there is an extreme deviation on test #11 with V8, and there are some tight results, where the difference is insignificant and inconclusive.

An interesting detail is that the deviation of measurements for V8 is always at least slightly bigger than for JSC, and twice as big in some cases (average deviation without test #11 was below 2% for JSC and at about 3.5% for V8).
This motivated us to take a closer look at the deviation of JavaScript benchmark results as well, and we found that V8 produces somewhat bigger deviation than JSC in those tests too.

Conclusion

Conflicting outcome of JavaScript benchmark results and the page loading test results indicate that there is still some work to do to enhance the performance of QtWebKit-V8. In the JavaScript benchmarks the V8 engine is the clear winner, which is a good argument to use it in the QtScript and QtDeclarative modules, but the page loading test shows, that JSC has still its benefits when it comes to browsing and bindings performance.

Moreover, it would be also interesting to see some memory benchmarks and some results on ARM, but this should be the subject of another blog post.