Understanding the JetStream 3 Benchmark Suite: A Q&A on WebAssembly Performance Evolution

The JetStream 3 benchmark suite marks a major step forward in measuring browser engine performance, especially for WebAssembly. Developed collaboratively by WebKit, Google, and Mozilla, it addresses the limitations of its predecessor, JetStream 2, where ultra-fast startup times rendered benchmarks meaningless. This Q&A explores the challenges, solutions, and innovations behind JetStream 3.

What is JetStream 3 and why was it developed?

JetStream 3 is a cross-browser benchmark suite that evaluates the performance of modern web applications, emphasizing WebAssembly (Wasm) workloads. It was developed as a joint effort by browser vendors, including the WebKit team at Apple, Google Chrome, and Mozilla Firefox, to replace JetStream 2. The primary motivation was that JetStream 2 had become outdated — browser engines had optimized Wasm startup so aggressively that certain benchmarks produced a score of infinity, making them useless. JetStream 3 introduces a fundamentally new approach to measuring performance, focusing on the real-world impact of Wasm in critical paths like libraries, image decoders, and UI frameworks.

How did WebAssembly benchmarking evolve from JetStream 2 to 3?

In JetStream 2, WebAssembly performance was measured in two distinct phases: Startup and Runtime. This design assumed that large C/C++ applications (like games) would tolerate a long one-time startup cost in exchange for high throughput. However, as browser engines dramatically reduced startup times, the scoring formula Score = 5000 / Time led to mathematical infinities when times dropped below 1 millisecond. JetStream 3 shifts away from this phase separation, integrating Wasm workloads into a continuous measurement that better reflects how Wasm is used today — as part of the critical path for page loads rather than isolated monolithic apps. This change ensures benchmarks remain meaningful even as engines continue to improve.

What was the "infinity problem" in JetStream 2?

The infinity problem arose because JetStream 2 used Date.now() to measure each iteration's time, which rounds down any sub-1-ms value to 0 ms. The scoring formula, Score = 5000 / Time, then divided by zero, producing an infinite score. While a score of infinity sounds impressive, it actually made the benchmark useless for comparing performance — a 0 ms sub-score would overshadow all other metrics. This flaw was temporarily patched in JetStream 2.2 by clamping the score to a maximum of 5000, but it signaled that the suite needed a fundamental redesign to accommodate engines that could instantiate Wasm modules in near-zero time.

Why did JetStream 2's Wasm startup time reach zero?

Browser engines, particularly WebKit's JavaScriptCore, optimized WebAssembly instantiation so thoroughly that for small workloads, startup time effectively fell to zero milliseconds. This was a result of relentless micro-optimizations: shaving off fractions of a millisecond from what was once a 100 ms process. As engines reduced the initial instantiation to just 2 ms, a further 0.1 ms improvement suddenly represented a 5% gain. In JetStream 2's measurement system, any time below 1 ms was rounded down to 0 by Date.now(), causing the infinite score. This success in optimization ironically broke the benchmark, proving that the test was no longer challenging enough to measure state-of-the-art performance.

How does JetStream 3 improve upon JetStream 2?

JetStream 3 introduces several key improvements over its predecessor. First, it abandons the separate Startup and Runtime phases for WebAssembly, integrating Wasm tests into a unified workload that better mirrors real-world usage — where Wasm is often part of libraries or frameworks used on every page load. Second, the suite uses higher-precision timing methods to avoid the rounding issues that caused infinities. Third, it includes more diverse and modern workloads that challenge browsers in areas like image decoding, UI rendering, and data processing. Finally, JetStream 3 is designed to be extensible, allowing easy addition of new tests as the web evolves. These changes ensure the benchmark remains useful for driving and measuring performance improvements in all major browser engines.

What role does WebKit's JavaScriptCore play in JetStream 3?

WebKit's JavaScriptCore (JSC) was instrumental in both identifying the need for JetStream 3 and implementing the optimizations that made previous benchmarks obsolete. The JSC team aggressively optimized Wasm startup paths, leading to the zero-time scenario that broke JetStream 2. For JetStream 3, JSC engineers collaborated with Google and Mozilla to design benchmarks that accurately represent modern Wasm usage. They also contributed engineering insights to ensure the new suite measures performance in a way that encourages general optimizations rather than benchmark-specific tweaks. WebKit's deep involvement ensures JetStream 3 reflects the latest advancements in JSC, including improvements in just-in-time compilation and module instantiation, while keeping the suite fair across all browsers.

What are the key features of JetStream 3's WebAssembly testing?

• Unified workflow: No separation between startup and runtime; Wasm is tested as part of cohesive tasks.
• High-precision timing: Uses methods more precise than Date.now() to avoid rounding errors.
• Real-world scenarios: Includes workloads from libraries, image decoders, and UI frameworks where Wasm is on the critical path.
• Scalable complexity: Tests range from small interactive modules to larger data processing tasks.
• Cross-browser fairness: Designed to avoid favoring any specific engine's optimization strategy.
• Easily extensible: Architecture allows adding new Wasm subtests as the ecosystem grows.

These features ensure that JetStream 3 measures meaningful performance differences and continues to drive browser engine improvement without the pitfalls of its predecessor.