Jekyll2023-07-27T07:12:23+00:00https://axel.isouard.fr/feed.xmlAxel IsouardAxel Isouard - Software EngineerAxel Isouardaxel@isouard.frMeet WebTransport: The Future of WebSockets2023-07-20T00:00:00+00:002023-07-20T00:00:00+00:00https://axel.isouard.fr/blog/2023/07/20/webtransport-future-of-websockets<p>During many years, <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_client_applications">WebSockets</a> have been the only way to transmit
arbitrary data in a bidirectional way between a client and a server. However,
we’ve encountered some limitations making them not suitable for some use cases.</p>
<p>For example, WebSockets are not multiplexed, so if you want to send multiple
messages at the same time, you need to implement your own multiplexing layer on
top of WebSockets.</p>
<p>Furthermore, WebSockets can fall short in scenarios demanding low latency, <a href="https://news.ycombinator.com/item?id=13266692">such
as video gaming,</a> due to their reliance on TCP. As a protocol
designed for reliability rather than speed, TCP may not always be the optimal
choice for time-sensitive applications where a delay can drastically impact user
experience.</p>
<p>This has left <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API">WebRTC</a> as the primary alternative for transmitting fast
and unreliable data thanks to the <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Using_data_channels">datachannels</a> feature provided
by UDP. However, while it can facilitate high-speed data transfer, WebRTC
presents its own set of complexities and may not be the most feasible solution
in all contexts.</p>
<p>This is where <a href="https://webtransport.day/">WebTransport</a> steps in, offering a promising way to
address these challenges.</p>
<h1 id="what-is-webtransport">What is WebTransport?</h1>
<p><a href="https://webtransport.day/">WebTransport</a> represents a significant step forward in the realm
of web communication. Developed as part of the <a href="https://datatracker.ietf.org/doc/draft-ietf-webtrans-overview/">IETF (Internet Engineering Task
Force)</a> standard, WebTransport is designed to provide flexible, efficient,
and secure communication between browsers and servers.</p>
<p>This technology, already available in the latest version of <a href="https://blog.chromium.org/2021/11/chrome-97-webtransport-new-array-static.html">Chrome</a> and
<a href="https://www.mozilla.org/en-US/firefox/114.0/releasenotes/">Firefox</a>, addresses the limitations of WebSockets and WebRTC by
offering multiplexed and unordered delivery of messages, as well as low latency,
thanks to its use of <a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol">UDP</a>.</p>
<p>We know that the use of <a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol">UDP</a> can be a double-edged sword. On the one hand,
it allows us to send data faster, but on the other hand, it can be unreliable,
due to a higher probability of packet loss than <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a>.</p>
<p>The main advantage of using UDP is that we don’t need to establish a connection
before sending data. If you can picture TCP being a strong and solid network
cable, UDP would be a radio antenna.</p>
<p>Thanks to <a href="https://www.chromium.org/quic">QUIC</a>, a transport protocol built on top of UDP, we can have
the best of both worlds. <a href="https://www.chromium.org/quic">QUIC</a> provides a reliable and secure connection,
taking care of retransmitting lost packets, guaranteeing the order of the
messages, and automatically encrypting the data for us.</p>
<p>That’s right, we won’t need to worry about UDP’s issues. But as the protocol is
different, we’ll need to use it in a very different way than we’re used to with
WebSockets.</p>
<h1 id="getting-started-with-webtransport">Getting started with WebTransport</h1>
<p>Enough theory, let’s see how we can use that marvel in practice. As the
technology is still in its early stages at the time of writing this article, it
could be very difficult to create a WebTransport server from scratch.</p>
<p>The only implementation I’ve found is the <a href="https://github.com/BiagioFesta/wtransport">wtransport crate</a> written
in <a href="https://www.rust-lang.org/">Rust</a>. It could be a great opportunity to finally learn the Rust
language. But you have to keep in mind that the web browser will require the
WebTransport server to have a valid TLS certificate.</p>
<p>Unfortunately, self-signed certificates won’t work that time, as you’ll require
the browser to trust your certificate via command-line. Our easiest way to get
started would be using the demo WebTransport server available at
<a href="https://webtransport.day/">https://webtransport.day</a>.</p>
<h2 id="establishing-a-connection">Establishing a connection</h2>
<p>The first step is to establish a connection with the server. This process will
be very similar to the one we’re used to with WebSockets:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">url</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">https://echo.webtransport.day</span><span class="dl">'</span>
<span class="kd">const</span> <span class="nx">transport</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">WebTransport</span><span class="p">(</span><span class="nx">url</span><span class="p">)</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">Initiating connection</span><span class="dl">'</span><span class="p">)</span>
<span class="k">try</span> <span class="p">{</span>
<span class="k">await</span> <span class="nx">transport</span><span class="p">.</span><span class="nx">ready</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">error</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="dl">'</span><span class="s1">Connection failed</span><span class="dl">'</span><span class="p">,</span> <span class="nx">error</span><span class="p">)</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">Connection ready</span><span class="dl">'</span><span class="p">)</span>
</code></pre></div></div>
<p>As you can see, we have to wait for the <code class="language-plaintext highlighter-rouge">ready</code> promise to be resolved before
going any further. This is due to our WebTransport client sending a connection
request to the server through the UDP protocol. This is the same as setting up
a radio antenna and having to wait for a signal to be received.</p>
<div id="1-webtransport-console" style="width: 100%; height: 200px; border: solid lightgray 1px; margin-bottom: 8px; overflow: scroll">
</div>
<p><button id="1-webtransport-connect">Connect</button>
<button id="1-webtransport-disconnect" disabled="">Disconnect</button></p>
<script type="text/javascript">
(function() {
var connectButton;
var disconnectButton;
var webtransportConsole;
var transport;
function webtransportLog(message) {
var line = document.createElement('p');
line.style.margin = '0';
line.textContent = message;
webtransportConsole.appendChild(line);
}
function onConnectClick() {
connectButton.disabled = true;
var url = 'https://echo.webtransport.day';
transport = new WebTransport(url);
webtransportLog('Connecting to https://echo.webtransport.day ...');
transport.ready
.then(() => {
webtransportLog('Connected!');
disconnectButton.disabled = false;
})
.catch((error) => {
webtransportLog('Connection failed: ' + error);
});
transport.closed
.then(() => {
webtransportLog('Connection closed normally');
disconnectButton.disabled = true;
connectButton.disabled = false;
})
.catch((error) => {
webtransportLog('Connection closed abruptly: ' + error);
disconnectButton.disabled = true;
connectButton.disabled = false;
});
}
function onDisconnectClick() {
webtransportLog('Disconnecting');
transport.close();
}
window.addEventListener('load', function () {
connectButton = document.getElementById('1-webtransport-connect');
connectButton.addEventListener('click', onConnectClick);
disconnectButton = document.getElementById('1-webtransport-disconnect');
disconnectButton.addEventListener('click', onDisconnectClick);
webtransportConsole = document.getElementById('1-webtransport-console');
});
})();
</script>
<p>Handling disconnection is also considered best practice and can be done by
listening waiting for the <code class="language-plaintext highlighter-rouge">closed</code> promise to be fullfilled:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">transport</span><span class="p">.</span><span class="nx">closed</span>
<span class="p">.</span><span class="nx">then</span><span class="p">(()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">Connection closed normally</span><span class="dl">'</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">.</span><span class="k">catch</span><span class="p">((</span><span class="nx">error</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="dl">'</span><span class="s1">Connection closed abruptly</span><span class="dl">'</span><span class="p">,</span> <span class="nx">error</span><span class="p">)</span>
<span class="p">})</span>
</code></pre></div></div>
<h1 id="sending-and-receiving-data">Sending and receiving data</h1>
<p>Now that we have a connection, we can start sending and receiving data. However,
this process is very different from what we’re used to with WebSockets. You
would think that sending data would be as simple as calling a <code class="language-plaintext highlighter-rouge">send</code> method on
our <code class="language-plaintext highlighter-rouge">transport</code> object, but it’s not that simple.</p>
<p>There are currently three possible ways to send data with WebTransport. Each
way have something in common: they use <em>streams</em>:</p>
<p><strong>Datagram streams</strong>: they are similar to UDP packets, they are unordered and
can be lost. They are the fastest way to send data, but they are not
guaranteed to be received. Perfect use cases for datagrams would be sending
the position of a player in a video game or sending the current position of a
user’s mouse cursor.</p>
<p><strong>Unidirectional streams</strong>: they are similar to WebSockets, they are ordered
and guaranteed to be received. Take note that they are unidirectional, they
should be used when you only need to send or receive data in one direction.
A good example would be a file transfer, where you only need to receive data
from the server to the browser.</p>
<p><strong>Bidirectional streams</strong>: exactly like unidirectional streams, but they can
be used to send and receive data in both directions. They should be used when
you need to wait for a response from the server after sending a message, like
when you would like to authenticate a user.</p>
<h2 id="using-datagrams">Using datagrams</h2>
<p>Datagrams are the fastest way to send data, making the process way easier than
using unidirectional or bidirectional streams. We can send data by retrieving
the <code class="language-plaintext highlighter-rouge">datagrams</code> stream from our <code class="language-plaintext highlighter-rouge">transport</code> object:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">stream</span> <span class="o">=</span> <span class="nx">transport</span><span class="p">.</span><span class="nx">datagrams</span>
<span class="kd">const</span> <span class="nx">writer</span> <span class="o">=</span> <span class="nx">stream</span><span class="p">.</span><span class="nx">writable</span><span class="p">.</span><span class="nx">getWriter</span><span class="p">()</span>
<span class="k">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="k">new</span> <span class="nx">TextEncoder</span><span class="p">().</span><span class="nx">encode</span><span class="p">(</span><span class="dl">'</span><span class="s1">Hello there!</span><span class="dl">'</span><span class="p">))</span>
</code></pre></div></div>
<p>That’s right, as simple as that.</p>
<p>On every stream object, such as <code class="language-plaintext highlighter-rouge">datagrams</code>, you will find a <code class="language-plaintext highlighter-rouge">readable</code> and a
<code class="language-plaintext highlighter-rouge">writable</code> property, being each one an instance of a <code class="language-plaintext highlighter-rouge">ReadableStream</code> and a
<code class="language-plaintext highlighter-rouge">WritableStream</code> respectively.</p>
<p>You can then use the <code class="language-plaintext highlighter-rouge">getReader()</code> and <code class="language-plaintext highlighter-rouge">getWriter()</code> methods to receive and send
data respectively. In our example, we’re sending a string, but you can send any
kind of data, such as a <code class="language-plaintext highlighter-rouge">Uint8Array</code> or a <code class="language-plaintext highlighter-rouge">Blob</code>.</p>
<p>However, we still need to receive incoming data, and that won’t be as simple as
listening for a <code class="language-plaintext highlighter-rouge">message</code> event like we’re used to with WebSockets. Instead, we
need to use the <code class="language-plaintext highlighter-rouge">read()</code> method from the <code class="language-plaintext highlighter-rouge">readable</code> property.</p>
<p>This method will return an object with two properties: one named <code class="language-plaintext highlighter-rouge">value</code>, which
will contain the data received, and a boolean named <code class="language-plaintext highlighter-rouge">done</code>, which will be <code class="language-plaintext highlighter-rouge">true</code>
when the stream is closed.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">async</span> <span class="kd">function</span> <span class="nx">receiveDatagrams</span><span class="p">(</span><span class="nx">transport</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">stream</span> <span class="o">=</span> <span class="nx">transport</span><span class="p">.</span><span class="nx">datagrams</span>
<span class="kd">const</span> <span class="nx">reader</span> <span class="o">=</span> <span class="nx">stream</span><span class="p">.</span><span class="nx">readable</span><span class="p">.</span><span class="nx">getReader</span><span class="p">()</span>
<span class="k">while</span> <span class="p">(</span><span class="kc">true</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">value</span><span class="p">,</span> <span class="nx">done</span> <span class="p">}</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">reader</span><span class="p">.</span><span class="nx">read</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">done</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">Datagram stream closed by the server</span><span class="dl">'</span><span class="p">)</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">Received</span><span class="dl">'</span><span class="p">,</span> <span class="nx">value</span><span class="p">.</span><span class="nx">length</span><span class="p">,</span> <span class="dl">'</span><span class="s1">bytes:</span><span class="dl">'</span><span class="p">,</span> <span class="k">new</span> <span class="nx">TextDecoder</span><span class="p">().</span><span class="nx">decode</span><span class="p">(</span><span class="nx">value</span><span class="p">))</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Now this can start to be confusing.</p>
<p>This infinite loop will keep reading incoming data from the server until the
stream is closed. But wouldn’t it be more convenient just checking for a
<code class="language-plaintext highlighter-rouge">message</code> event? Or call the blocking <code class="language-plaintext highlighter-rouge">read()</code> method only when some data is
available?</p>
<p>Well, there’s a simple way to execute the function above:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">receiveDatagrams</span><span class="p">(</span><span class="nx">transport</span><span class="p">)</span>
</code></pre></div></div>
<p>That’s right, we just need to call the function, ignoring the fact that it’s
a <code class="language-plaintext highlighter-rouge">Promise</code>, that it could be executed only if we defined a <code class="language-plaintext highlighter-rouge">.then()</code> callback.</p>
<p>But in that case, it’s really enough just calling the function without blocking
it with an <code class="language-plaintext highlighter-rouge">await</code> keyword. It will just be executed in parallel with the rest
of our code.</p>
<p>Feel free to see that by yourself, using the demo below:</p>
<div id="2-webtransport-console" style="width: 100%; height: 200px; border: solid lightgray 1px; margin-bottom: 8px; overflow: scroll">
</div>
<div style="margin-bottom: 8px">
<button id="2-webtransport-connect">Connect</button>
<button id="2-webtransport-disconnect" disabled="">Disconnect</button>
</div>
<div style="display: flex; margin-bottom: 8px">
<input id="2-webtransport-input" type="text" placeholder="Datagram..." style="margin: 0; width: 100%; border: solid lightgray 1px" disabled="" />
<button id="2-webtransport-send" disabled="">Send</button>
</div>
<script type="text/javascript">
(function() {
var connectButton;
var disconnectButton;
var sendButton;
var inputText;
var webtransportConsole;
var transport;
function webtransportLog(message) {
var line = document.createElement('p');
line.style.margin = '0';
line.textContent = message;
webtransportConsole.appendChild(line);
}
function onConnectClick() {
connectButton.disabled = true;
var url = 'https://echo.webtransport.day';
transport = new WebTransport(url);
webtransportLog('Connecting to https://echo.webtransport.day ...');
transport.ready
.then(() => {
webtransportLog('Connected!');
disconnectButton.disabled = false;
inputText.disabled = false;
sendButton.disabled = false;
receiveDatagrams();
})
.catch((error) => {
webtransportLog('Connection failed: ' + error);
});
transport.closed
.then(() => {
webtransportLog('Connection closed normally');
disconnectButton.disabled = true;
connectButton.disabled = false;
inputText.disabled = true;
sendButton.disabled = true;
})
.catch((error) => {
webtransportLog('Connection closed abruptly: ' + error);
disconnectButton.disabled = true;
connectButton.disabled = false;
inputText.disabled = true;
sendButton.disabled = true;
});
}
function onDisconnectClick() {
webtransportLog('Disconnecting');
transport.close();
}
async function onSendClick() {
const value = inputText.value;
if (value.length < 1) {
return;
}
inputText.value = '';
const stream = transport.datagrams;
const writer = stream.writable.getWriter();
await writer.write(new TextEncoder().encode(value));
webtransportLog('You: ' + value);
}
async function receiveDatagrams() {
const stream = transport.datagrams;
const reader = stream.readable.getReader();
while (true) {
const { value, done } = await reader.read();
if (done) {
webtransportLog('Datagram stream closed by the server');
return;
}
webtransportLog('Server: ' + new TextDecoder().decode(value));
}
}
window.addEventListener('load', function () {
connectButton = document.getElementById('2-webtransport-connect');
connectButton.addEventListener('click', onConnectClick);
disconnectButton = document.getElementById('2-webtransport-disconnect');
disconnectButton.addEventListener('click', onDisconnectClick);
sendButton = document.getElementById('2-webtransport-send');
sendButton.addEventListener('click', onSendClick);
inputText = document.getElementById('2-webtransport-input');
webtransportConsole = document.getElementById('2-webtransport-console');
});
})();
</script>
<h2 id="using-streams">Using streams</h2>
<p>Now that we know how to send and receive datagrams, let’s see how to use
unidirectional and bidirectional streams.</p>
<p>First of all, it is important to understand that streams in WebTransport are
designed to not be left open forever. This stands in contrast to the persistent
connections of WebSockets or the constant opening and closing of HTTP polling.</p>
<p>Stream are more similar to a phone call, where you open a connection,
communicate, and then hang up when you’re done. The great thing about streams is
that it can go both ways, meaning that you can also receive a phone call from
the server side.</p>
<h3 id="creating-an-unidirectional-stream">Creating an unidirectional stream</h3>
<p>Let’s start by creating a unidirectional stream. This is the simplest type of
stream, where only one side can send data.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">stream</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">transport</span><span class="p">.</span><span class="nx">createUnidirectionalStream</span><span class="p">()</span>
<span class="kd">const</span> <span class="nx">writer</span> <span class="o">=</span> <span class="nx">stream</span><span class="p">.</span><span class="nx">writable</span><span class="p">.</span><span class="nx">getWriter</span><span class="p">()</span>
<span class="k">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="k">new</span> <span class="nx">TextEncoder</span><span class="p">().</span><span class="nx">encode</span><span class="p">(</span><span class="dl">"</span><span class="s2">Hello!</span><span class="dl">"</span><span class="p">))</span>
<span class="k">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">close</span><span class="p">()</span>
</code></pre></div></div>
<p>Note the importance of waiting for the stream to be created before sending a
message. This is because we need to wait for the server to accept our request
to open it.</p>
<p>Also remember that you can only use the <code class="language-plaintext highlighter-rouge">writable</code> property of the stream to
send data, the <code class="language-plaintext highlighter-rouge">readable</code> property is only available on the server’s side and
only if you’ve received that stream from the server instead of creating it by
yourself.</p>
<h3 id="creating-a-bidirectional-stream">Creating a bidirectional stream</h3>
<p>This step is very similar to the previous one, just with a small difference:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">stream</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">transport</span><span class="p">.</span><span class="nx">createBidirectionalStream</span><span class="p">()</span>
<span class="kd">const</span> <span class="nx">writer</span> <span class="o">=</span> <span class="nx">stream</span><span class="p">.</span><span class="nx">writable</span><span class="p">.</span><span class="nx">getWriter</span><span class="p">()</span>
<span class="k">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="k">new</span> <span class="nx">TextEncoder</span><span class="p">().</span><span class="nx">encode</span><span class="p">(</span><span class="dl">"</span><span class="s2">Hello!</span><span class="dl">"</span><span class="p">))</span>
</code></pre></div></div>
<p>We can also receive data from the server, with the same way we can receive a
datagram:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">async</span> <span class="kd">function</span> <span class="nx">receiveBidirectionalData</span><span class="p">(</span><span class="nx">stream</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">reader</span> <span class="o">=</span> <span class="nx">stream</span><span class="p">.</span><span class="nx">readable</span><span class="p">.</span><span class="nx">getReader</span><span class="p">()</span>
<span class="kd">const</span> <span class="nx">writer</span> <span class="o">=</span> <span class="nx">stream</span><span class="p">.</span><span class="nx">writable</span><span class="p">.</span><span class="nx">getWriter</span><span class="p">()</span>
<span class="k">while</span> <span class="p">(</span><span class="kc">true</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">value</span><span class="p">,</span> <span class="nx">done</span> <span class="p">}</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">reader</span><span class="p">.</span><span class="nx">read</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">done</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">Datagram stream closed by the server</span><span class="dl">'</span><span class="p">)</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">Received</span><span class="dl">'</span><span class="p">,</span> <span class="nx">value</span><span class="p">.</span><span class="nx">length</span><span class="p">,</span> <span class="dl">'</span><span class="s1">bytes:</span><span class="dl">'</span><span class="p">,</span> <span class="k">new</span> <span class="nx">TextDecoder</span><span class="p">().</span><span class="nx">decode</span><span class="p">(</span><span class="nx">value</span><span class="p">))</span>
<span class="k">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="k">new</span> <span class="nx">TextEncoder</span><span class="p">().</span><span class="nx">encode</span><span class="p">(</span><span class="dl">"</span><span class="s2">Message received!</span><span class="dl">"</span><span class="p">))</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h3 id="receiving-incoming-streams">Receiving incoming streams</h3>
<p>Up until now we’ve only seen how to create streams, but what would happen if the
server wanted to send us a reliable message?</p>
<p>The answer is that we need to wait for the server to send us a stream, and then
we can use it to communicate with it. There’s actually two special streams that
we can read from, in order to receive incoming streams:
<code class="language-plaintext highlighter-rouge">incomingBidirectionalStreams</code> and <code class="language-plaintext highlighter-rouge">incomingUnidirectionalStreams</code> attributes.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">async</span> <span class="nx">listenIncomingStreams</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">reader</span> <span class="o">=</span> <span class="nx">transport</span><span class="p">.</span><span class="nx">incomingBidirectionalStreams</span><span class="p">.</span><span class="nx">getReader</span><span class="p">()</span>
<span class="k">while</span> <span class="p">(</span><span class="kc">true</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// value will be a bidirectional stream</span>
<span class="c1">// like if we have used `createBidirectionalStream`</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">value</span><span class="p">,</span> <span class="nx">done</span> <span class="p">}</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">reader</span><span class="p">?.</span><span class="nx">read</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">done</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">'</span><span class="s1">Incoming bidirectional streams closed by the server</span><span class="dl">'</span><span class="p">)</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="kd">const</span> <span class="nx">stream</span> <span class="o">=</span> <span class="nx">value</span>
<span class="nx">receiveBidirectionalData</span><span class="p">(</span><span class="nx">stream</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h1 id="quick-summary">Quick summary</h1>
<p>In any case you might be confused about the proper use of WebTransport, here’s a
small summary:</p>
<ul>
<li>Use <a href="#using-datagrams">datagrams</a> for small messages that don’t need to be
reliable. These are similar to UDP packets, and are useful for things like
real-time coordinates.</li>
<li>Use <a href="#creating-an-unidirectional-stream">createUnidirectionalStream</a> for
sending reliable messages to the server. Switch to
<a href="#creating-a-bidirectional-stream">createBidirectionalStream</a> whenever you
need to receive a reply from it.</li>
<li>Use <a href="#receiving-incoming-streams">incomingUnidirectionalStreams and incomingBidirectionalStreams</a>
to receive incoming streams from the server.
This would prevent the requirement of polling the server for new messages, or
leaving a stream open forever.</li>
</ul>
<h1 id="conclusion">Conclusion</h1>
<p>WebTransport is a revolutionary technology that provides a powerful and flexible
framework for bidirectional communication between a client and a server.</p>
<p>It resolves the limitations of WebSockets and WebRTC, providing developers
a solution that is multiplexed, efficient, and suitable for low-latency
applications.</p>
<p>Getting started with that feature can be challenging due to its infancy and lack
of widespread implementation.</p>
<p>However, as it continues to mature and become more mainstream, it has the
potential to redefine the way we build real-time, high-performance web
applications.</p>Axel Isouardaxel@isouard.frDeep dive into how WebTransport can overcome the limitations of WebSockets and WebRTC, discussing its unique features like the use of datagrams, unidirectional and bidirectional streams, and the revolutionary QUIC protocolBuilding a Virtual Reality 3D Chat with Three.js, React and WebTransport2023-07-18T00:00:00+00:002023-07-18T00:00:00+00:00https://axel.isouard.fr/blog/2023/07/18/threejs-react-3d-chat-with-webtransport<p>15 years ago, my friends and I were captivated by Active Worlds, a 3D chat
platform that allowed users to effortlessly create new experiences in existing
worlds, even establishing their own.</p>
<p>However, our enjoyment was limited by a tight budget. After exploring
alternatives such as Blaxxun, which used web browsers and VRML technologies
to connect people, I decided to create my own solution, intending to offer
more freedom and possibilities to unleash our imagination.</p>
<p>At the time, my programming knowledge was restricted to mIRC scripting, TCL,
and Visual Basic 6. Languages like C, C++, and Java seemed quite complex, Unity
wasn’t around, and Unreal Engine was still popularly known as
Unreal Tournament 2004. Consequently, I had to put this idea on hold.</p>
<p>Fast forward 15 years, and I’ve mastered the C programming language, primarily
by delving deep into the Quake 3 source code. With the evolution of technology,
creating 3D games has become simpler than ever, and web browsers no longer need
Flash, ActiveX, or NPAPI plug-ins to run incredible software effortlessly,
without any cumbersome installations.</p>
<p>A month ago, Mozilla released Firefox 114, introducing the WebTransport feature.
This innovation was the missing piece I needed to enable players to transmit
their movements smoothly within 3D environments.</p>
<p>Employing HTTP3 and UDP, this technology is clearly the successor to WebSockets,
which relied solely on TCP. With this, WebRTC will no longer be necessary for
unreliable and fast data transmission.</p>
<p>Now, with all the required tools in hand, I can finally set this idea in motion
and see where it leads.</p>
<p>For this project, I plan to use Three.js and React for the front-end and Rust
for the back-end, as wtransport was the only library with a working
implementation of a WebTransport server at the time of writing this post.</p>
<p>I’ll be sharing more details as progress unfolds. I look forward to reading any
feedback you might have throughout this exhilarating journey.</p>Axel Isouardaxel@isouard.frLeveraging my experience with programming languages and cutting-edge technologies like WebTransport, I aim to create an environment where imagination is the only limit.Visual Regression Testing with Cypress and Storybook2023-05-20T00:00:00+00:002023-05-20T00:00:00+00:00https://axel.isouard.fr/blog/2023/05/20/react-visual-regression-testing-cypress-storybook<p>When we were introduced to unit testing with Jest, we’ve been told about
quitting comparing the existence, attributes and values of our DOM elements
with the ones we expect in our components.</p>
<p>These assertions have been replaced by snapshots, which are a representation of
the DOM at a given time.</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span> <span class="nx">expect</span><span class="p">,</span> <span class="nx">test</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">vitest</span><span class="dl">"</span><span class="p">;</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">render</span><span class="p">,</span> <span class="nx">screen</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">@testing-library/react</span><span class="dl">"</span><span class="p">;</span>
<span class="k">import</span> <span class="nx">userEvent</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">@testing-library/user-event</span><span class="dl">"</span><span class="p">;</span>
<span class="k">import</span> <span class="nx">App</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">./App.tsx</span><span class="dl">"</span><span class="p">;</span>
<span class="nx">test</span><span class="p">(</span><span class="dl">"</span><span class="s2">displays initial counter</span><span class="dl">"</span><span class="p">,</span> <span class="k">async</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">render</span><span class="p">(<</span><span class="nc">App</span> <span class="p">/>);</span>
<span class="c1">// expect(screen.getByRole("button").textContent).toEqual("count is 0");</span>
<span class="nx">expect</span><span class="p">(</span><span class="nx">screen</span><span class="p">.</span><span class="nx">getByRole</span><span class="p">(</span><span class="dl">"</span><span class="s2">button</span><span class="dl">"</span><span class="p">)).</span><span class="nx">toMatchSnapshot</span><span class="p">();</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Running the test above will create a snapshot file in the <code class="language-plaintext highlighter-rouge">src/__snapshots__</code>
folder, named <code class="language-plaintext highlighter-rouge">App.test.tsx.snap</code>.</p>
<p>This file contains the DOM representation of the component at the time the test
was run, with the expected values.</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html</span>
<span class="nx">exports</span><span class="p">[</span><span class="s2">`displays initial counter 1`</span><span class="p">]</span> <span class="o">=</span> <span class="s2">`
<button>
count is
0
</button>
`</span><span class="p">;</span>
</code></pre></div></div>
<p>The test would succeed if the snapshot file doesn’t exist, or if the DOM
representation matches the one in the existing snapshot file.</p>
<p>Modifying the component would fail the test, and the snapshot file would have to
be updated with the new DOM representation.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> FAIL src/App.test.tsx > displays initial counter
Error: Snapshot `displays initial counter 1` mismatched
❯ src/App.test.tsx:11:38
9|
10| // expect(screen.getByRole("button").textContent).toEqual("count is 0");
11| expect(screen.getByRole("button")).toMatchSnapshot();
| ^
12| });
13|
- Expected - 1
+ Received + 1
`<button>␊
- count is ␊
+ Count value is ␊
0␊
</button>`
</code></pre></div></div>
<p>You could just run <code class="language-plaintext highlighter-rouge">npm test -- -u</code> to confirm that your modification was
intentional, update the snapshot file and make the test pass again.</p>
<p>It is now guaranteed that our component is <strong>rendered</strong> as expected.</p>
<p>Despite the apparent convenience and reliability of snapshots, one issue
remains: they guarantee that our component is rendered as expected, but not that
it will look as expected. For that, we need a different testing approach.</p>
<h1 id="enter-visual-regression-testing">Enter Visual Regression Testing</h1>
<p>Visual Regression Testing is a technique that allows you to compare the rendered
component with a reference image, and fail the test if the two images don’t
match.</p>
<p>Instead of taking a snapshot of the DOM representation of the component, we
want to take a snapshot of the rendered component and store it as an image,
like if we were taking a screenshot.</p>
<figure>
<a href="/assets/images/posts/react-visual-regression-testing-cypress-storybook/original.jpg">
<img src="/assets/images/posts/react-visual-regression-testing-cypress-storybook/original.jpg" />
</a>
<figcaption>Original component</figcaption>
</figure>
<p>Then our <code class="language-plaintext highlighter-rouge">toMatchSnapshot</code> assertion would compare the rendered component with
the reference image, fail the test if the two images don’t match and show us
the difference between the two images.</p>
<figure>
<a href="/assets/images/posts/react-visual-regression-testing-cypress-storybook/diff.jpg">
<img src="/assets/images/posts/react-visual-regression-testing-cypress-storybook/diff.jpg" />
</a>
<figcaption>Comparison between the original and modified versions</figcaption>
</figure>
<p>How can we do this magic, knowing that our tests are running inside a terminal
and simulated DOM environment thanks to <code class="language-plaintext highlighter-rouge">jsdom</code>?</p>
<p>We will have to use a web browser to render our component, which is good news
since most of our co-workers have at least Chrome or Firefox installed on their
computer.</p>
<p>However, the bad news is that our co-workers and other developers around the
world have different screen resolutions, operating systems and typography
settings which can greatly impact on the way our component is rendered and give
us false positives.</p>
<p>In order to tackle this issue, we will have to use a single web browser,
with a fixed screen resolution and same fonts installed to run our tests. The
best solution we have at the time of writing this article is to use a Docker
image with a headless browser installed.</p>
<p>You have to keep in mind that the resulting docker image might be quite heavy
to download and store inside your private docker registry, but it’s still worth
it if you want to avoid the design of your application to be broken by a
single modification in a global CSS file.</p>
<p>Also having a whole web browser stored inside your docker image makes sense
since that same web browser is your target platform, as your users will be
using it to access your application.</p>
<h2 id="docker-image">Docker image</h2>
<p>Once you’ve convinced your infra team to store a (maximum of) 2 GB docker image,
for dependencies caching purposes, in their private docker registry, you can
start working on the docker image itself.</p>
<p>Fortunately for us, there are already <a href="https://github.com/cypress-io/cypress-docker-images">some docker images</a>
with Cypress, Node and the Google Chrome web browser pre-installed.</p>
<p>Create a new <code class="language-plaintext highlighter-rouge">Dockerfile</code> at the root of your application with the following
contents:</p>
<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> cypress/browsers:node18.12.0-chrome107</span>
<span class="k">WORKDIR</span><span class="s"> /app</span>
<span class="k">COPY</span><span class="s"> package.json package-lock.json /app/</span>
<span class="k">RUN </span>npm ci
<span class="k">COPY</span><span class="s"> . /app</span>
<span class="k">RUN </span>npm run build
</code></pre></div></div>
<p>Now you can build this image by running the following command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker build <span class="nt">-t</span> my-app <span class="nb">.</span>
</code></pre></div></div>
<p>And execute Cypress inside the container:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">-it</span> my-app npm run <span class="nb">test</span>:e2e
</code></pre></div></div>
<p>Feel free to mount some directories to store the screenshots and videos in
case you would like to try failing some tests:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">-it</span> <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/cypress/screenshots:/app/cypress/screenshots <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/cypress/videos:/app/cypress/videos <span class="se">\</span>
my-app sh <span class="nt">-c</span> <span class="s2">"npm run test:e2e ; chown -R </span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-u</span><span class="si">)</span><span class="s2">:</span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-g</span><span class="si">)</span><span class="s2"> cypress"</span>
</code></pre></div></div>
<p>Our docker image is now ready to be used in our CI pipeline. Let’s see how we
can perform visual regression testing with Cypress.</p>
<h2 id="cypress">Cypress</h2>
<p>You will find commercial options available to setup visual regression testing
in a seamless way <a href="https://docs.cypress.io/plugins#Visual%20Testing">on the official website</a>, but let’s focus on the
open-source ones.</p>
<p>I’ve been using <code class="language-plaintext highlighter-rouge">cypress-image-snapshot</code> during the last few years, but it seems
unmaintained at the time of writing this article. After digging a bit, I found
a fork of this plugin called
<a href="https://github.com/FRSOURCE/cypress-plugin-visual-regression-diff">cypress-plugin-visual-regression-diff</a>,
made by the folks at <a href="https://www.frsource.org/">FRSource</a>.</p>
<p>The installation instructions are very straightforward with very low effort
required. First, you have to get the package:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm <span class="nb">install</span> <span class="nt">--save-dev</span> @frsource/cypress-plugin-visual-regression-diff
</code></pre></div></div>
<p>Then initialize it inside your <code class="language-plaintext highlighter-rouge">cypress.config.ts</code> file:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> import { defineConfig } from "cypress";
<span class="gi">+import { initPlugin } from "@frsource/cypress-plugin-visual-regression-diff/plugins";
</span>
export default defineConfig({
e2e: {
baseUrl: "http://localhost:3000",
setupNodeEvents(on, config) {
// implement node event listeners here
<span class="gi">+ initPlugin(on, config);
</span> },
},
component: {
devServer: {
framework: "react",
bundler: "vite",
},
<span class="gi">+ setupNodeEvents(on, config) {
+ initPlugin(on, config);
+ },
</span> },
});
</code></pre></div></div>
<p>Register the commands inside your <code class="language-plaintext highlighter-rouge">cypress/support/commands.ts</code> file:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="dl">"</span><span class="s2">@frsource/cypress-plugin-visual-regression-diff</span><span class="dl">"</span><span class="p">;</span>
</code></pre></div></div>
<p>That’s it. Inside your test file, located at <code class="language-plaintext highlighter-rouge">cypress/e2e/App.cy.ts</code> you can use
the <code class="language-plaintext highlighter-rouge">cy.matchImage();</code> method to take a snapshot of the rendered component and
compare it with the reference image automatically.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> describe("default vite react app", () => {
it("increments the counter", () => {
cy.visit("/");
cy.get("button").should("have.text", "count is 0");
cy.get("button").click().should("have.text", "count is 1");
<span class="gi">+ cy.matchImage();
</span> });
});
</code></pre></div></div>
<p>The next time you’ll run your tests with <code class="language-plaintext highlighter-rouge">npm run test:e2e</code>, the following
output will be displayed:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> (Screenshots)
- /home/unnamedcoder/git/my-app/cypress/e2e/__image_snapshots__/default vite react app inc (1000x660)
rements the counter #0.actual.png
</code></pre></div></div>
<p>As you can see, we’ve just created an image snapshot of our whole page. The
file located inside the <code class="language-plaintext highlighter-rouge">cypress/e2e/__image_snapshots__</code> directory should be
committed to your git repository.</p>
<p>But before we are doing such thing, we need to understand why we’ve created
a docker image previously, by running the same tests with docker this time:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker build <span class="nt">-t</span> my-app <span class="nb">.</span>
<span class="nv">$ </span>docker run <span class="nt">-it</span> <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/cypress/e2e/__image_snapshots__:/app/cypress/e2e/__image_snapshots__ <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/cypress/screenshots:/app/cypress/screenshots <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/cypress/videos:/app/cypress/videos <span class="se">\</span>
my-app sh <span class="nt">-c</span> <span class="s2">"npm run test:e2e ; chown -R </span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-u</span><span class="si">)</span><span class="s2">:</span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-g</span><span class="si">)</span><span class="s2"> cypress"</span>
</code></pre></div></div>
<p>A similar error to the following one below should be outputted:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 1) default vite react app
increments the counter:
Error: Image diff factor (0.989%) is bigger than maximum threshold option 0.01.
at Context.eval (webpack:///./node_modules/@frsource/cypress-plugin-visual-regression-diff/dist/support.js:154:0)
</code></pre></div></div>
<p>This means that the rendered component is different from the reference image by
more than 1%. This is due to the fact that the docker image is using a different
operating system, screen resolution and fonts than your local machine.</p>
<p>Since that docker image will be used by our CI pipeline, we need to make sure
that the reference image is the same as the one generated by the docker image.
And tell our co-workers to run the tests exclusively with docker to avoid
having different reference images.</p>
<p>Therefore, we need to update the reference images generated by ourselves, with
the ones generated by the docker image.</p>
<p>Since there’s actually no need to remove it by hand, we need to add the
following lines into the <code class="language-plaintext highlighter-rouge">scripts</code> section of our <code class="language-plaintext highlighter-rouge">package.json</code> file:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> "scripts": {
"dev": "vite",
"build": "tsc && vite build",
// ...
"test:e2e-start": "cypress run --e2e",
"test:e2e": "start-server-and-test dev http-get://localhost:3000 test:e2e-start",
<span class="gi">+ "test:e2e-update-start": "cypress run --e2e --env pluginVisualRegressionUpdateImages=true",
+ "test:e2e-update": "start-server-and-test dev http-get://localhost:3000 test:e2e-update-start",
</span> // ...
},
</code></pre></div></div>
<p>And run our new <code class="language-plaintext highlighter-rouge">test:e2e-update</code> script we’ve just added:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker build <span class="nt">-t</span> my-app <span class="nb">.</span>
<span class="nv">$ </span>docker run <span class="nt">-it</span> <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/cypress/e2e/__image_snapshots__:/app/cypress/e2e/__image_snapshots__ <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/cypress/screenshots:/app/cypress/screenshots <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/cypress/videos:/app/cypress/videos <span class="se">\</span>
my-app sh <span class="nt">-c</span> <span class="s2">"npm run test:e2e-update ; chown -R </span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-u</span><span class="si">)</span><span class="s2">:</span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-g</span><span class="si">)</span><span class="s2"> cypress"</span>
</code></pre></div></div>
<p>Our tests should have passed and the reference image should have been updated.</p>
<h2 id="what-about-storybook-">What about Storybook ?</h2>
<p>Visual regression testing with Storybook is pretty straightforward. I’ve been
using Loki for a while now but the latest version seems to have trouble with
Storybook 7 and React 18.</p>
<p>Which is why we’re going to switch to Storycap for generating the Storybook
snapshots, with the help of reg-cli for comparing them.</p>
<p>Storycap can be installed with the following command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm <span class="nb">install </span>storycap <span class="nt">--save-dev</span>
</code></pre></div></div>
<p>Snapshot generation will be done with the command below:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>storycap <span class="nt">--serverCmd</span> <span class="s2">"storybook dev -p 9001"</span> http://localhost:9001
info Wait <span class="k">for </span>connecting storybook server http://localhost:9001.
info Executable Chromium path: /usr/bin/google-chrome-stable
info Storycap runs with simple mode
info Found 1 stories.
info Screenshot stored: __screenshots__/App/Default.png <span class="k">in </span>573 msec.
info Screenshot was ended successfully <span class="k">in </span>10826 msec capturing 1 PNGs.
</code></pre></div></div>
<p>As stated in the output above, the generated snapshot is located inside the
<code class="language-plaintext highlighter-rouge">__screenshots__</code> directory.</p>
<p>Let’s install reg-cli to perform the visual regression testing:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm <span class="nb">install </span>reg-cli <span class="nt">--save-dev</span>
</code></pre></div></div>
<p>reg-cli will take as first parameter the path to the folder containing the
actual images we will generate during the test (in our case, <code class="language-plaintext highlighter-rouge">__screenshots__</code>),
the folder containing our reference images as second parameter and the folder
where the diff images will be stored as third parameter.</p>
<p>It can also generate a report in HTML format with the <code class="language-plaintext highlighter-rouge">-R</code> parameter, which is
very convenient for debugging purposes when our tests are failing.</p>
<p>First, we can ask reg-cli to store our generated snapshots as reference images:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>reg-cli ./__screenshots__ ./expected <span class="nt">-U</span>
✔ pass __screenshots__/App/Default.png
All images are updated.
✨ your expected images are updated ✨
✔ 1 file<span class="o">(</span>s<span class="o">)</span> passed.
</code></pre></div></div>
<p>Then, we can run our tests with the following command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>reg-cli ./__screenshots__ ./expected ./diff
✔ pass __screenshots__/App/Default.png
✔ 1 file<span class="o">(</span>s<span class="o">)</span> passed.
</code></pre></div></div>
<p>As a reminder, these commands should be run through Docker as well, to make
sure that the reference images are the same as the ones generated by our CI
pipeline.</p>
<p>You can add the following scripts inside your <code class="language-plaintext highlighter-rouge">package.json</code> file:</p>
<p>Since there’s actually no need to remove it by hand, we need to add the
following lines into the <code class="language-plaintext highlighter-rouge">scripts</code> section of our <code class="language-plaintext highlighter-rouge">package.json</code> file:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> "scripts": {
"dev": "vite",
"build": "tsc && vite build",
// ...
"test:e2e-start": "cypress run --e2e",
"test:e2e": "start-server-and-test dev http-get://localhost:3000 test:e2e-start",
<span class="gi">+ "test:visual": "reg-cli ./__screenshots__ ./expected ./diff",
+ "test:visual-capture": "storycap --serverCmd \"storybook dev -p 9001\" http://localhost:9001",
+ "test:visual-update": "reg-cli ./__screenshots__ ./expected -U",
</span> // ...
},
</code></pre></div></div>
<p>Run the following command to (re)generate the reference images:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker build <span class="nt">-t</span> my-app <span class="nb">.</span>
<span class="nv">$ </span>docker run <span class="nt">-it</span> <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/__screenshots__:/app/__screenshots__ <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/expected:/app/expected <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/diff:/app/diff <span class="se">\</span>
my-app sh <span class="nt">-c</span> <span class="s2">"npm run test:visual-capture && npm run test:visual-update ; chown -R </span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-u</span><span class="si">)</span><span class="s2">:</span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-g</span><span class="si">)</span><span class="s2"> __screenshots__ expected"</span>
</code></pre></div></div>
<p>And run our new <code class="language-plaintext highlighter-rouge">test:visual</code> script we’ve just added to make sure our design
remains intact:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">-it</span> <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/__screenshots__:/app/__screenshots__ <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/expected:/app/expected <span class="se">\</span>
<span class="nt">-v</span> <span class="sb">`</span><span class="nb">pwd</span><span class="sb">`</span>/diff:/app/diff <span class="se">\</span>
my-app sh <span class="nt">-c</span> <span class="s2">"npm run test:visual ; chown -R </span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-u</span><span class="si">)</span><span class="s2">:</span><span class="si">$(</span><span class="nb">id</span> <span class="nt">-g</span><span class="si">)</span><span class="s2"> __screenshots__ expected diff"</span>
</code></pre></div></div>
<h1 id="wrapping-up">Wrapping up</h1>
<p>In conclusion, visual regression testing represents a significant step forward
in ensuring the integrity of our application’s design. By embracing this method,
we can avoid unwanted changes in design caused by minor alterations in global
CSS files or dependencies updates.</p>
<p>Despite certain challenges, such as maintaining large Docker images and
adjusting to local machine differences, the overall benefits, particularly in
maintaining UI consistency, are well worth the investment.</p>
<p>Therefore, developers and teams should consider visual regression testing as an
indispensable component in their testing toolkit.</p>Axel Isouardaxel@isouard.frImprove your React application's user experience by ensuring visual consistency across every update, using Cypress and Storybook with advanced techniques such as visual regression testing.Bootstrapping a new React app in 20232023-05-18T00:00:00+00:002023-05-18T00:00:00+00:00https://axel.isouard.fr/blog/2023/05/18/new-react-app-in-2023<p>In 2023, starting a new <a href="https://reactjs.org">React</a> app involves embracing cutting-edge tools
and techniques that not only boost productivity but also enhance code quality.</p>
<p>Whether you’re a seasoned developer or just getting started, this comprehensive
guide will walk you through the process of bootstrapping a new React app with
the minimum recommended to create your app with confidence.</p>
<p>We’ll cover important steps that are both essential, including:</p>
<ul>
<li>Bootstrapping the app with the lightning-fast <a href="https://vitejs.dev">Vite</a> instead of the
traditional <code class="language-plaintext highlighter-rouge">create-react-app</code> utility.</li>
<li>Installing a CSS framework like <a href="https://tailwindcss.com">Tailwind CSS</a>,</li>
<li>Making a robust foundation for your code through effective linting and
formatting with <a href="https://eslint.org">ESLint</a> and <a href="https://prettier.io">Prettier</a>.</li>
<li>Ensuring code stability and functionality with comprehensive unit, end-to-end,
and component tests using tools like <a href="https://vitest.dev">Vitest</a> and <a href="https://www.cypress.io">Cypress</a>.</li>
<li>Incorporating <a href="https://storybook.js.org">Storybook</a> for isolated and focused component
development,</li>
<li>Implementing a CI/CD pipeline with <a href="https://github.com/features/actions">GitHub Actions</a>.</li>
</ul>
<p>By following these steps, you will be able to ensure a solid foundation for your
upcoming React projects.</p>
<h2 id="create-a-git-repository">Create a git repository</h2>
<p>Setting up a proper version control system for your project is still crucial
before going any further. This would allow you to save your modifications and
revert them when needed, but also to share your progress and collaborate with
other developers around the world.</p>
<figure>
<a href="/assets/images/posts/new-react-app-in-2023/github-new-repo.jpg">
<img src="/assets/images/posts/new-react-app-in-2023/github-new-repo.jpg" />
</a>
</figure>
<p>Assuming that you’ve created a new repository on
<a href="https://github.com/new">GitHub</a> called <code class="language-plaintext highlighter-rouge">my-app</code>, under the account
<code class="language-plaintext highlighter-rouge">unnamedcoder</code>, you can initialize a new git repository locally by following the
provided instructions.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">mkdir </span>my-app <span class="o">&&</span> <span class="nb">cd </span>my-app
<span class="nv">$ </span>git init
<span class="nv">$ </span>git remote add origin git@github.com:unnamedcoder/my-app.git
</code></pre></div></div>
<h2 id="use-vite-to-bootstrap-your-app">Use Vite to bootstrap your app</h2>
<p>We’ve been using create-react-app for years, but it’s time to move on. New
alternatives like <a href="https://vitejs.dev">Vite</a> offer several advantages, including faster
performance, improved configuration flexibility, native support for modern web
standards and compatibility with other popular frontend frameworks.</p>
<p>Bootstrapping a new <a href="https://reactjs.org">React</a> application using <a href="https://www.typescriptlang.org">TypeScript</a>
with <a href="https://vitejs.dev">Vite</a> is as simple as running the following command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm init vite@latest <span class="nb">.</span> <span class="nt">--</span> <span class="nt">--template</span> react-ts
Need to <span class="nb">install </span>the following packages:
create-vite@4.3.1
Ok to proceed? <span class="o">(</span>y<span class="o">)</span> y
Scaffolding project <span class="k">in</span> /home/unnamedcoder/my-app...
Done. Now run:
npm <span class="nb">install
</span>npm run dev
</code></pre></div></div>
<p>As stated above, we just have to execute <code class="language-plaintext highlighter-rouge">npm install</code> to have our dependencies
ready, then <code class="language-plaintext highlighter-rouge">npm run dev</code> to run our fresh new app.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm run dev
VITE v4.3.7 ready <span class="k">in </span>326 ms
➜ Local: http://localhost:5173/
➜ Network: use <span class="nt">--host</span> to expose
➜ press h to show <span class="nb">help</span>
</code></pre></div></div>
<p>Open your favorite web browser, then head to the URL mentioned in the output.
You should see the default React app generated by Vite running flawlessly.</p>
<figure>
<a href="/assets/images/posts/new-react-app-in-2023/vite-react.png">
<img src="/assets/images/posts/new-react-app-in-2023/vite-react.png" />
</a>
</figure>
<p>Everything is already configured, including the hot module replacement allowing
the app to reload automatically every time you do a modification. You won’t
need to execute <code class="language-plaintext highlighter-rouge">npm run eject</code> anymore.</p>
<h2 id="install-a-css-framework">Install a CSS framework</h2>
<p>Writing pure CSS code is fun, but also time-consuming and error-prone. It looks
like writing assembly code to me. But thanks to <a href="https://vitejs.dev">Vite</a>, we can directly
write SASS, SCSS or LESS code in our React components.</p>
<p>However, you may want to use a CSS framework to speed up your development such
as <a href="https://tailwindcss.com">Tailwind CSS</a>, now is your chance to finally take a look at
this marvel.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm <span class="nb">install</span> <span class="nt">-D</span> tailwindcss postcss autoprefixer
<span class="nv">$ </span>npx tailwindcss init <span class="nt">-p</span>
</code></pre></div></div>
<p>Add the following lines at the bottom of the <code class="language-plaintext highlighter-rouge">src/index.css</code> file:</p>
<div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">@tailwind</span> <span class="n">base</span><span class="p">;</span>
<span class="k">@tailwind</span> <span class="n">components</span><span class="p">;</span>
<span class="k">@tailwind</span> <span class="n">utilities</span><span class="p">;</span>
</code></pre></div></div>
<p>Configure Tailwind CSS to look for <code class="language-plaintext highlighter-rouge">index.html</code> and every kind of source code
inside the <code class="language-plaintext highlighter-rouge">src</code> folder by applying the following modifications inside the
<code class="language-plaintext highlighter-rouge">tailwind.config.js</code> file:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> /** @type {import('tailwindcss').Config} */
export default {
<span class="gd">- content: [],
</span><span class="gi">+ content: [
+ "./index.html",
+ "./src/**/*.{js,ts,jsx,tsx}",
+ ],
</span> theme: {
extend: {},
},
plugins: [],
}
</code></pre></div></div>
<p>Now you can use <a href="https://tailwindcss.com">Tailwind CSS</a> classes in your React components
like in the following example:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><h1</span> <span class="na">className=</span><span class="s">"text-3xl font-bold underline"</span><span class="nt">></span>
Hello world!
<span class="nt"></h1></span>
</code></pre></div></div>
<figure>
<a href="/assets/images/posts/new-react-app-in-2023/tailwind.gif">
<img src="/assets/images/posts/new-react-app-in-2023/tailwind.gif" />
</a>
</figure>
<p>Feel free to explore <a href="https://tailwindcss.com">their website</a> for more documentation,
configuration guides and best practice tips. You can also check out
<a href="https://tailwindui.com">Tailwind UI</a> which offers a curated collection of beautifully
designed components and templates to give you more inspiration.</p>
<h2 id="setup-code-linting">Setup code linting</h2>
<p>Always assume that you are working with other people, even if you’re still a
beginner or working on a personal project. By setting up a code linter, you will
ensure consistency across your codebase, avoid common mistakes and ease the
onboarding of new developers by helping them sending pull requests.</p>
<p>We’re lucky that Vite already comes with a built-in <a href="https://eslint.org">ESLint</a>
configuration, but there are still more useful rules to apply:</p>
<ul>
<li><a href="https://github.com/import-js/eslint-plugin-import">eslint-plugin-import</a> to check import/export syntax
and prevent issues with misspelling of file paths and import names</li>
<li><a href="https://github.com/jsx-eslint/eslint-plugin-jsx-a11y">eslint-plugin-jsx-a11y</a> to check accessibility rules
in JSX elements to ensure inclusive, user-friendly applications</li>
<li><a href="https://github.com/jsx-eslint/eslint-plugin-react">eslint-plugin-react</a> to enforce React-specific coding
patterns and best practices, helping to maintain consistent and high-quality
code</li>
</ul>
<p>You can apply these rules by installing the following packages:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm <span class="nb">install</span> <span class="nt">-D</span> eslint-plugin-import eslint-plugin-jsx-a11y eslint-plugin-react
</code></pre></div></div>
<p>And update your <code class="language-plaintext highlighter-rouge">.eslintrc.cjs</code> file with the following content:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> module.exports = {
env: { browser: true, es2020: true },
extends: [
'eslint:recommended',
'plugin:@typescript-eslint/recommended',
'plugin:react-hooks/recommended',
<span class="gi">+ 'plugin:react/recommended',
+ 'plugin:import/recommended',
+ 'plugin:jsx-a11y/recommended',
</span> ],
parser: '@typescript-eslint/parser',
parserOptions: { ecmaVersion: 'latest', sourceType: 'module' },
plugins: ['react-refresh'],
<span class="gi">+ settings: {
+ react: {
+ version: 'detect',
+ },
+ 'import/resolver': {
+ node: {
+ paths: ['src', 'public'],
+ extensions: ['.js', '.jsx', '.ts', '.tsx'],
+ },
+ },
+ },
</span> rules: {
'react-refresh/only-export-components': 'warn',
},
}
</code></pre></div></div>
<p>Running the <code class="language-plaintext highlighter-rouge">npm run lint -- --fix</code> command should output the following:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm run lint <span class="nt">--</span> <span class="nt">--fix</span>
<span class="o">></span> my-app@0.0.0 lint
<span class="o">></span> eslint src <span class="nt">--ext</span> ts,tsx <span class="nt">--report-unused-disable-directives</span> <span class="nt">--max-warnings</span> 0 <span class="nt">--fix</span>
/home/unnamedcoder/my-app/src/App.tsx
3:22 error Unable to resolve path to module <span class="s1">'/vite.svg'</span> import/no-unresolved
✖ 1 problem <span class="o">(</span>1 error, 0 warnings<span class="o">)</span>
</code></pre></div></div>
<p>This error still occurs because of the <code class="language-plaintext highlighter-rouge">vite.svg</code> file imported in the <code class="language-plaintext highlighter-rouge">App.tsx</code>
component. I couldn’t find a way to fix this issue, so we need to disable this
rule exceptionally for this import by adding the following comment at the top of
it:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> import { useState } from 'react'
import reactLogo from './assets/react.svg'
<span class="gi">+// eslint-disable-next-line import/no-unresolved
</span> import viteLogo from '/vite.svg'
import './App.css'
</code></pre></div></div>
<p>Running <code class="language-plaintext highlighter-rouge">npm run lint</code> again should produce no output, thus confirming that
everything is working as expected.</p>
<h2 id="setup-code-formatting">Setup code formatting</h2>
<p>We’ve been expressing concerns about the misuse of ESLint, which was primarily
designed for detecting syntax errors and enforcing code rules. It was often
utilized for code formatting as well, a task for which it was not specifically
tailored, leading to a lot of confusion and frustration.</p>
<p>The introduction of <a href="https://prettier.io">Prettier</a> as a complementary tool to ESLint has
addressed this issue. <a href="https://prettier.io">Prettier</a> is a dedicated code formatter that
automatically formats code according to a consistent style, allowing ESLint to
focus on its core purpose of identifying and preventing potential coding issues.</p>
<p>You can install <a href="https://prettier.io">Prettier</a> by running the following command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm <span class="nb">install</span> <span class="nt">-D</span> prettier eslint-config-prettier
</code></pre></div></div>
<p>And disable ESLint rules that might conflict with Prettier by updating the
<code class="language-plaintext highlighter-rouge">extends</code> section of your <code class="language-plaintext highlighter-rouge">.eslintrc.cjs</code> file:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> extends: [
'eslint:recommended',
'plugin:@typescript-eslint/recommended',
'plugin:react-hooks/recommended',
'plugin:react/recommended',
'plugin:import/recommended',
'plugin:jsx-a11y/recommended',
<span class="gi">+ 'plugin:prettier/recommended',
</span> ],
// ...
rules: {
<span class="gi">+ 'prettier/prettier': 'warn',
</span> 'react-refresh/only-export-components': 'warn',
},
</code></pre></div></div>
<p>Next time you’ll run the <code class="language-plaintext highlighter-rouge">npm run lint</code> command, you shall see a lot of warnings
about code formatting. You can fix them by running the <code class="language-plaintext highlighter-rouge">npm run lint -- --fix</code>
or <code class="language-plaintext highlighter-rouge">prettier -c -w .</code> commands.</p>
<figure>
<a href="/assets/images/posts/new-react-app-in-2023/prettier.gif">
<img src="/assets/images/posts/new-react-app-in-2023/prettier.gif" />
</a>
</figure>
<h2 id="write-unit-tests">Write unit tests</h2>
<p>This step is highly recommended to keep your application working as intended and
avoid breaking it when you will be adding new features or fixing bugs.</p>
<p>You don’t have to take snapshots, mock states, effects or internal functions of
your components anymore. Unit testing became as simple as imitating user’s
behavior and interactions over your components, thanks to the
<a href="https://testing-library.com/docs/react-testing-library/intro">React Testing Library</a>.</p>
<p><a href="https://vitest.dev">Vitest</a> also simplifies the process by providing a built-in
configuration for Jest, eliminating the struggle of setting up the testing
environment, babel config, file transforms, … with the following steps:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm <span class="nb">install</span> <span class="nt">-D</span> vitest @vitest/ui @vitest/coverage-c8 <span class="se">\</span>
@testing-library/react <span class="se">\</span>
@testing-library/jest-dom <span class="se">\</span>
@testing-library/user-event <span class="se">\</span>
@types/react @types/react-dom jsdom
</code></pre></div></div>
<p>Add the following entries inside the <code class="language-plaintext highlighter-rouge">scripts</code> section of your <code class="language-plaintext highlighter-rouge">package.json</code>
file:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> "scripts": {
"dev": "vite",
"build": "tsc && vite build",
"lint": "eslint src --ext ts,tsx --report-unused-disable-directives --max-warnings 0",
<span class="gd">- "preview": "vite preview"
</span><span class="gi">+ "preview": "vite preview",
+ "test": "vitest",
+ "test:coverage": "vitest --coverage",
+ "test:ui": "vitest --ui"
</span> },
</code></pre></div></div>
<p>Update your <code class="language-plaintext highlighter-rouge">vite.config.ts</code> by adding the testing configuration:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gi">+/// <reference types="vitest" />
</span>
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
// https://vitejs.dev/config/
export default defineConfig({
plugins: [react()],
<span class="gi">+ test: {
+ globals: true,
+ environment: 'jsdom',
+ },
</span> })
</code></pre></div></div>
<p>We know that the default application generated by Vite is a simple counter which
can be incremented by clicking on a button. We want to ensure that the button
works as intented.</p>
<p>The unit test for this component will be as simple as the following:</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// src/App.test.tsx</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">expect</span><span class="p">,</span> <span class="nx">test</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">vitest</span><span class="dl">'</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">render</span><span class="p">,</span> <span class="nx">screen</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@testing-library/react</span><span class="dl">'</span>
<span class="k">import</span> <span class="nx">userEvent</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@testing-library/user-event</span><span class="dl">'</span>
<span class="k">import</span> <span class="nx">App</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">./App.tsx</span><span class="dl">'</span>
<span class="nx">test</span><span class="p">(</span><span class="dl">'</span><span class="s1">displays initial counter and increments upon click</span><span class="dl">'</span><span class="p">,</span> <span class="k">async</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">render</span><span class="p">(<</span><span class="nc">App</span> <span class="p">/>)</span>
<span class="k">await</span> <span class="nx">userEvent</span><span class="p">.</span><span class="nx">click</span><span class="p">(</span><span class="nx">screen</span><span class="p">.</span><span class="nx">getByText</span><span class="p">(</span><span class="dl">'</span><span class="s1">count is 0</span><span class="dl">'</span><span class="p">))</span>
<span class="nx">expect</span><span class="p">(</span><span class="nx">screen</span><span class="p">.</span><span class="nx">getByRole</span><span class="p">(</span><span class="dl">'</span><span class="s1">button</span><span class="dl">'</span><span class="p">).</span><span class="nx">textContent</span><span class="p">).</span><span class="nx">toEqual</span><span class="p">(</span><span class="dl">'</span><span class="s1">count is 1</span><span class="dl">'</span><span class="p">)</span>
<span class="p">})</span>
</code></pre></div></div>
<p>You can run the test by running <code class="language-plaintext highlighter-rouge">npm test</code>. You should see the following output:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DEV v0.31.0 /home/unnamedcoder/my-app
✓ src/App.test.tsx <span class="o">(</span>1<span class="o">)</span>
Test Files 1 passed <span class="o">(</span>1<span class="o">)</span>
Tests 1 passed <span class="o">(</span>1<span class="o">)</span>
Start at 11:12:44
Duration 1.29s <span class="o">(</span>transform 208ms, setup 0ms, collect 398ms, tests 79ms, environment 282ms, prepare 158ms<span class="o">)</span>
</code></pre></div></div>
<p>You can also execute <code class="language-plaintext highlighter-rouge">npm run test:ui</code> to open a user interface and enjoy a more
interactive experience. It’s also possible to run <code class="language-plaintext highlighter-rouge">npm run test:coverage</code> to
check the coverage of your tests.</p>
<p>Add the following line at the bottom of your <code class="language-plaintext highlighter-rouge">.gitignore</code> file to avoid pushing
the coverage report to your repository:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>coverage
</code></pre></div></div>
<h2 id="write-end-to-end-tests">Write end-to-end tests</h2>
<p>Everytime I’ve been hearing about end-to-end testing, I was immediately
remembering about the struggles I had with Selenium, telling my teammates to
download the latest version of ChromeDriver, and the pain of writing tests in
a very locked down environment.</p>
<p>I believe that you’ve also felt the same way, our prayers have been answered
explaining the reason why <a href="https://www.cypress.io">Cypress</a> has been created.</p>
<figure>
<a href="/assets/images/posts/new-react-app-in-2023/cypress.gif">
<img src="/assets/images/posts/new-react-app-in-2023/cypress.gif" />
</a>
</figure>
<p><a href="https://www.cypress.io">Cypress</a> is still a wonderful option for end-to-end testing, also
providing us a great way to develop our application without depending on the
backend, thus focusing on the frontend development in an isolated way.</p>
<p>You can install <a href="https://www.cypress.io">Cypress</a> by running the following command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm <span class="nb">install</span> <span class="nt">-D</span> cypress typescript start-server-and-test
</code></pre></div></div>
<p>Add the following entry inside the <code class="language-plaintext highlighter-rouge">scripts</code> section of your <code class="language-plaintext highlighter-rouge">package.json</code>:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> "scripts": {
"dev": "vite",
"build": "tsc && vite build",
"lint": "eslint src --ext ts,tsx --report-unused-disable-directives --max-warnings 0",
"preview": "vite preview",
"test": "vitest",
"test:coverage": "vitest --coverage",
<span class="gd">- "test:ui": "vitest --ui"
</span><span class="gi">+ "test:ui": "vitest --ui",
+ "test:e2e-start": "cypress open --e2e",
+ "test:e2e": "start-server-and-test dev http-get://localhost:3000 test:e2e-start"
</span> },
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">test:e2e</code> script will execute the <code class="language-plaintext highlighter-rouge">dev</code> script, wait for the application
to be ready on the port <code class="language-plaintext highlighter-rouge">3000</code> before executing <code class="language-plaintext highlighter-rouge">test:e2e-start</code> which will
open <a href="https://www.cypress.io">Cypress</a> in interactive mode.</p>
<p>It is therefore necessary to assign a specific port instead of a random one to
our development server by updating our <code class="language-plaintext highlighter-rouge">vite.config.ts</code> file:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> export default defineConfig({
plugins: [react()],
test: {
globals: true,
environment: 'jsdom',
},
<span class="gi">+ server: {
+ host: true,
+ port: 3000,
+ },
</span> })
</code></pre></div></div>
<p>Then run the following command to open Cypress for the first time and generate
the initial configuration files:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cypress open
</code></pre></div></div>
<p>Once you’ve selected “E2E Testing”, you can close the window and create the
first test inside the <code class="language-plaintext highlighter-rouge">cypress/e2e/App.cy.ts</code> with the following content:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">describe</span><span class="p">(</span><span class="dl">'</span><span class="s1">default vite react app</span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">it</span><span class="p">(</span><span class="dl">'</span><span class="s1">increments the counter</span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">cy</span><span class="p">.</span><span class="nx">visit</span><span class="p">(</span><span class="dl">'</span><span class="s1">/</span><span class="dl">'</span><span class="p">)</span>
<span class="nx">cy</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">button</span><span class="dl">'</span><span class="p">).</span><span class="nx">should</span><span class="p">(</span><span class="dl">'</span><span class="s1">have.text</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">count is 0</span><span class="dl">'</span><span class="p">)</span>
<span class="nx">cy</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">button</span><span class="dl">'</span><span class="p">).</span><span class="nx">click</span><span class="p">().</span><span class="nx">should</span><span class="p">(</span><span class="dl">'</span><span class="s1">have.text</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">count is 1</span><span class="dl">'</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">})</span>
</code></pre></div></div>
<p>We also need to specify the base url of our application in the file to match the
port we’ve set into the Vite configuration file, by adding the following line
inside the <code class="language-plaintext highlighter-rouge">cypress.config.ts</code> file:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> import { defineConfig } from "cypress";
export default defineConfig({
e2e: {
<span class="gi">+ baseUrl: "http://localhost:3000",
</span> setupNodeEvents(on, config) {
// implement node event listeners here
},
},
});
</code></pre></div></div>
<p>Then execute it by running <code class="language-plaintext highlighter-rouge">npm run test:e2e</code>. You should now be able to see the
default Vite welcome page, with the tests passing.</p>
<p>Your IDE might have troubles while resolving <code class="language-plaintext highlighter-rouge">describe</code>, <code class="language-plaintext highlighter-rouge">it</code> and <code class="language-plaintext highlighter-rouge">cy</code> objects.
This can be fixed by creating a <code class="language-plaintext highlighter-rouge">tsconfig.json</code> file inside the <code class="language-plaintext highlighter-rouge">cypress</code> folder
with the following contents:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"compilerOptions"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"target"</span><span class="p">:</span><span class="w"> </span><span class="s2">"es5"</span><span class="p">,</span><span class="w">
</span><span class="nl">"lib"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"es5"</span><span class="p">,</span><span class="w"> </span><span class="s2">"dom"</span><span class="p">],</span><span class="w">
</span><span class="nl">"types"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"cypress"</span><span class="p">,</span><span class="w"> </span><span class="s2">"node"</span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"include"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"**/*.ts"</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h2 id="write-component-tests">Write component tests</h2>
<p>Jest is still popular to run unit tests for our components. Its popularity
is also due to its execution speed thanks to the jsdom dependency which allows
it to execute the tests inside a minimal browser and fake DOM environment.</p>
<p>While it’s still a good choice for basic features, it can be a bit limited
when you want to check the visibility of an element, from its CSS attribute
or its dynamic position on the screen.</p>
<p>You would also have to spend more time mocking unsupported APIs, in order to
retrieve some data from cookies of local storage, which is why it would be
better to run the tests inside a real browser like the ones your end-users will
use.</p>
<p>This is where Cypress saves the day again, by providing us a way to run our
component tests inside a real browser.</p>
<p>First, add the following entry inside the <code class="language-plaintext highlighter-rouge">scripts</code> section of your
<code class="language-plaintext highlighter-rouge">package.json</code> file:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> "scripts": {
"dev": "vite",
"build": "tsc && vite build",
"lint": "eslint src --ext ts,tsx --report-unused-disable-directives --max-warnings 0",
"preview": "vite preview",
"test": "vitest",
"test:coverage": "vitest --coverage",
"test:ui": "vitest --ui",
"test:e2e-start": "cypress open --e2e",
<span class="gd">- "test:e2e": "start-server-and-test dev http-get://localhost:3000 test:e2e-start"
</span><span class="gi">+ "test:e2e": "start-server-and-test dev http-get://localhost:3000 test:e2e-start",
+ "test:component": "cypress open --component"
</span> },
</code></pre></div></div>
<p>Create a new file <code class="language-plaintext highlighter-rouge">src/App.cy.tsx</code> with the following content:</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nx">App</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">./App</span><span class="dl">'</span>
<span class="nx">describe</span><span class="p">(</span><span class="dl">'</span><span class="s1"><App /></span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">it</span><span class="p">(</span><span class="dl">'</span><span class="s1">displays initial counter and increments upon click</span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">cy</span><span class="p">.</span><span class="nx">mount</span><span class="p">(<</span><span class="nc">App</span> <span class="p">/>)</span>
<span class="nx">cy</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">button</span><span class="dl">'</span><span class="p">).</span><span class="nx">should</span><span class="p">(</span><span class="dl">'</span><span class="s1">have.text</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">count is 0</span><span class="dl">'</span><span class="p">)</span>
<span class="nx">cy</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">button</span><span class="dl">'</span><span class="p">).</span><span class="nx">click</span><span class="p">().</span><span class="nx">should</span><span class="p">(</span><span class="dl">'</span><span class="s1">have.text</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">count is 1</span><span class="dl">'</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">})</span>
</code></pre></div></div>
<p>Then run the following command to open Cypress for the first time and run the
component test:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm run <span class="nb">test</span>:component
</code></pre></div></div>
<p>Select React.js as the front-end framework, and Vite as the bundler. At the time
of writing this article, Cypress will expect you to have Typescript 4 installed
instead of 5. You can safely click on the “Skip” button to continue.</p>
<p>Once you’ve chose your favorite browser and started the test, you should see
the component being displayed with the tests passing.</p>
<p>Which framework should you pick? It depends on your needs. In my opinion, I
would stick to Jest for very logical code and utility functions not involving
anything about React, even if that part is not needed anymore since I would
prioritize component functionality.</p>
<p>For any other tests needing a web browser, I would keep using Cypress for both
component and E2E tests, while avoiding jsdom at all costs.</p>
<p>By the way, your IDE might have troubles while resolving <code class="language-plaintext highlighter-rouge">describe</code>, <code class="language-plaintext highlighter-rouge">it</code> and
<code class="language-plaintext highlighter-rouge">cy</code> objects. This can be fixed by adding the <code class="language-plaintext highlighter-rouge">cypress</code> item inside the
<code class="language-plaintext highlighter-rouge">include</code> array at the bottom of the root <code class="language-plaintext highlighter-rouge">tsconfig.json</code> file:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> // ...
},
- "include": ["src"],
<span class="gi">+ "include": ["src", "cypress"],
</span> "references": [{ "path": "./tsconfig.node.json" }]
}
</code></pre></div></div>
<h2 id="adopt-storybook">Adopt Storybook</h2>
<p>This step is not required at all, but again, recommended. Incorporating
<a href="https://storybook.js.org">Storybook</a> into your development workflow offers a valuable
advantage, as it allows isolated component development without the need for a
backend to test various states such as success, loading and error.</p>
<figure>
<a href="/assets/images/posts/new-react-app-in-2023/storybook.gif">
<img src="/assets/images/posts/new-react-app-in-2023/storybook.gif" />
</a>
</figure>
<p>Storybook can also be used as a true replacement of Jest’s snapshot testing
feature, which was only checking the rendered HTML of a component, instead of
its true appearance by taking in account the CSS styles applied to it.</p>
<p>Storybook can be installed with its dependencies by running the following
command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npx storybook@latest init
</code></pre></div></div>
<p>You can run it by executing <code class="language-plaintext highlighter-rouge">npm run storybook</code>, your favorite web browser will
open the served Storybook instance at <code class="language-plaintext highlighter-rouge">http://localhost:6006</code>.</p>
<p>Default stories were created during the installation inside the <code class="language-plaintext highlighter-rouge">src/stories</code>
folder. It can be removed safely, as we will create the story for our <code class="language-plaintext highlighter-rouge">App</code>
component inside the <code class="language-plaintext highlighter-rouge">src/App.stories.tsx</code> file with the following content:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="kd">type</span> <span class="p">{</span> <span class="nx">Meta</span><span class="p">,</span> <span class="nx">StoryObj</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">@storybook/react</span><span class="dl">'</span><span class="p">;</span>
<span class="k">import</span> <span class="nx">App</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">./App</span><span class="dl">'</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">meta</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">title</span><span class="p">:</span> <span class="dl">'</span><span class="s1">App</span><span class="dl">'</span><span class="p">,</span>
<span class="na">component</span><span class="p">:</span> <span class="nx">App</span><span class="p">,</span>
<span class="na">tags</span><span class="p">:</span> <span class="p">[</span><span class="dl">'</span><span class="s1">autodocs</span><span class="dl">'</span><span class="p">],</span>
<span class="p">}</span> <span class="nx">satisfies</span> <span class="nx">Meta</span><span class="o"><</span><span class="k">typeof</span> <span class="nx">App</span><span class="o">></span><span class="p">;</span>
<span class="k">export</span> <span class="k">default</span> <span class="nx">meta</span><span class="p">;</span>
<span class="kd">type</span> <span class="nx">Story</span> <span class="o">=</span> <span class="nx">StoryObj</span><span class="o"><</span><span class="k">typeof</span> <span class="nx">meta</span><span class="o">></span><span class="p">;</span>
<span class="k">export</span> <span class="kd">const</span> <span class="nx">Default</span><span class="p">:</span> <span class="nx">Story</span> <span class="o">=</span> <span class="p">{};</span>
</code></pre></div></div>
<p>Then you can see the <code class="language-plaintext highlighter-rouge">App</code> component being displayed inside Storybook, with the
ability to interact with it by clicking on the counter button.</p>
<p>Now you are able to focus on the component development without the need to
start the development server and navigate to the page where it is displayed.</p>
<h2 id="add-a-cicd-pipeline">Add a CI/CD pipeline</h2>
<p>We’ve set up up code linting and unit testing to our application, now we can
make sure that our code is properly written and still works as expected before
pushing it to our remote repository.</p>
<p>But there is still an issue, as we have to keep in mind that we are not coding
alone, other developers might be willing to contribute to your project by
sending bugfixes, new features or enhancements with pull requests.</p>
<p>There is no guarantee that the code they will send will be linted and tested
accordingly to your standards, which is why we need to automate this process
by adding a CI/CD pipeline.</p>
<figure>
<a href="/assets/images/posts/new-react-app-in-2023/pipeline.gif">
<img src="/assets/images/posts/new-react-app-in-2023/pipeline.gif" />
</a>
</figure>
<p>There are many CI/CD providers available, but I will use GitHub Actions for
this article.</p>
<p>Create a new file <code class="language-plaintext highlighter-rouge">.github/workflows/main.yml</code> with the following content:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Main</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">push</span><span class="pi">:</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">*'</span>
<span class="na">pull_request</span><span class="pi">:</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">*'</span>
</code></pre></div></div>
<p>This part of the file will tell GitHub Actions to run the pipeline on every
push and pull request made to any branch of the repository.</p>
<p>Then we need to add jobs to the pipeline, which will be responsible for running
the linting and unit testing tasks after preparing the testing environment.</p>
<p>This can be done by adding the following content to the file:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">jobs</span><span class="pi">:</span>
<span class="na">lint</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout repository</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Node.js</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/setup-node@v2</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">node-version</span><span class="pi">:</span> <span class="m">18</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install dependencies</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">npm ci</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Run linter</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">npm run lint</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Run unit tests</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">npm test</span>
</code></pre></div></div>
<h2 id="wrapping-up">Wrapping up</h2>
<p>We’ve seen how to set up a React application from scratch, with the ability to
write code in Typescript, lint it, test it through a CI/CD pipeline, which will
run the linting and testing tasks on every push and pull request made to any
branch of the repository.</p>
<p>This is a good starting point for any React application, but there are still
many things to do, such as setting-up an automated release process, monitoring
the errors in real-time, implementing visual regression testing, …</p>
<p>Unfortunately, these steps would have made this article too long, but I promise
that I will write about them in the future.</p>
<p>Feel free to write a comment below if you have any question or suggestion, also
subscribe to my <a href="https://twitter.com/aisouard">Twitter</a>, <a href="https://instagram.com/axl.is">Instagram</a> or
<a href="https://www.linkedin.com/in/axelisouard">LinkedIn</a> accounts to get notified when I will publish a new post.</p>Axel Isouardaxel@isouard.frA look at the state of the React ecosystem in 2023 and how to bootstrap a new React app.Image Recognition in Video Games with OpenCV2020-07-13T00:00:00+00:002020-07-13T00:00:00+00:00https://axel.isouard.fr/blog/2020/07/13/endless-lake-image-recognition-video-games-opencv-python<p>A few months ago, a friend of mine challenged me to beat him on a video game called <a href="https://www.agame.com/game/endless-lake">Endless Lake</a> on Facebook. It took me a long time to almost reach his high score manually like a human being.</p>
<p>But this also made me realize that it was an opportunity to finally get my hands on Machine Learning, by creating a software that would beat the game, ideally until the counter explodes.</p>
<p>This post is the first in a series on making an artificial intelligence for the <a href="https://www.agame.com/game/endless-lake">Endless Lake</a> video game, based on deep learning and other machine learning techniques. This paragraph will be updated while I’ll discover new and different techniques to make this AI more efficient.</p>
<p>In the meantime, you can take a look at my project inside the <a href="https://github.com/aisouard/endless-fake">endless-fake</a> repository, which is a toolset helping me to benchmark image recognition and Machine Learning techniques quickly and comfortably.</p>
<h1 id="game-overview">Game Overview</h1>
<p>Before we go further, let’s take a quick look at the video game itself. It’s a web-browser based one, which goal results in jumping into three different ways to avoid falling into the lake.</p>
<p><img src="/assets/images/posts/endless-fake-1/quick_jumps.gif" alt="Quick jump" class="align-left" />
The quick jump, requiring you to perform a single tap, is more than enough for a short gap and recommended for almost any kind of situation, especially if we encounter a tiny platform right after the gap.</p>
<p><img src="/assets/images/posts/endless-fake-1/double_jumps.gif" alt="Double jump" class="align-right" />
The double jump, which could be performed with a double-tap, would be necessary to get over wider gaps than usual.</p>
<p><img src="/assets/images/posts/endless-fake-1/long_jumps.gif" alt="Long jump" class="align-left" />
Then comes the long jump, which could be done with a single tap, followed by another one after a short pause while the character is still floating in the air. That one is very rarely used over extremely wide gaps.</p>
<p>Now that we’ve seen every action possible, including the ability to just
wait before the next gap. We’ve also realized that our decisions will depend on
these three factors:</p>
<ul>
<li>Distance remaining until we reach the next gap</li>
<li>Length of that same gap</li>
<li>Length of the next platform we’ll land on</li>
</ul>
<figure class="">
<img src="/assets/images/posts/endless-fake-1/overview.png" alt="Project's overview" /><figcaption>
Overview of our game analysis
</figcaption></figure>
<p>All we need to do is retrieving these three values before passing them to our Machine Learning model so it can determine the best decision to take depending on these numbers.</p>
<h1 id="capturing-the-image">Capturing the image</h1>
<p>Let’s see how we can play the game and take screenshots before analyzing them. For the sake of simplicity, we’ll use the Python programming language with the OpenCV image processing library.</p>
<p>First, we need to execute the game inside a web browser. Here are two
methods for simply doing this task.</p>
<h2 id="using-chromium-embedded-framework">Using Chromium Embedded Framework</h2>
<p>This solution is the one I would recommend at a first glance, using the <a href="https://github.com/cztomczak/cefpython">cefpython package</a>. It consists of running a standalone process of the Google Chromium web browser and control it entirely.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">cefpython3</span> <span class="kn">import</span> <span class="n">cefpython</span> <span class="k">as</span> <span class="n">cef</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="k">class</span> <span class="nc">BrowserHandler</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">parent</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">parent</span> <span class="o">=</span> <span class="n">parent</span>
<span class="bp">self</span><span class="p">.</span><span class="n">image</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">def</span> <span class="nf">OnPaint</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">browser</span><span class="p">,</span> <span class="n">element_type</span><span class="p">,</span> <span class="n">paint_buffer</span><span class="p">,</span> <span class="o">**</span><span class="n">_</span><span class="p">):</span>
<span class="nb">buffer</span> <span class="o">=</span> <span class="n">paint_buffer</span><span class="p">.</span><span class="n">GetString</span><span class="p">(</span><span class="n">mode</span><span class="o">=</span><span class="s">"bgra"</span><span class="p">,</span>
<span class="n">origin</span><span class="o">=</span><span class="s">"top-left"</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">image</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">fromstring</span><span class="p">(</span><span class="nb">buffer</span><span class="p">,</span> <span class="n">np</span><span class="p">.</span><span class="n">uint8</span><span class="p">)</span>
<span class="p">.</span><span class="n">reshape</span><span class="p">((</span><span class="mi">640</span><span class="p">,</span> <span class="mi">360</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span>
<span class="n">browser</span> <span class="o">=</span> <span class="n">cef</span><span class="p">.</span><span class="n">CreateBrowserSync</span><span class="p">(</span><span class="n">url</span><span class="o">=</span><span class="s">"http://127.0.0.1"</span><span class="p">,</span> <span class="n">settings</span><span class="o">=</span><span class="p">{</span>
<span class="s">"windowless_rendering_enabled"</span><span class="p">:</span> <span class="bp">True</span>
<span class="p">})</span>
<span class="n">browser</span><span class="p">.</span><span class="n">SetClientHandler</span><span class="p">(</span><span class="n">BrowserHandler</span><span class="p">())</span>
</code></pre></div></div>
<p>But I’ve been through some side effects, such as the requirement to execute it inside a separate thread since it will block your whole process. Its windowless rendering feature allowing us to retrieve directly the raw pixels of the image is a very neat one, but it might not be working properly under Linux being my favorite operating system.</p>
<p>If you can manage to overcome these issues, then it would become the easiest solution and the most complete one fitting your needs.</p>
<h2 id="using-selenium">Using Selenium</h2>
<p>My favorite option, a combination of the <a href="https://github.com/SeleniumHQ/selenium">selenium package</a> with a native way to capture screenshots continuously using my Python package called <a href="https://github.com/aisouard/nativecap">nativecap</a>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">nativecap</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="kn">from</span> <span class="nn">selenium</span> <span class="kn">import</span> <span class="n">webdriver</span>
<span class="kn">from</span> <span class="nn">selenium.webdriver.chrome.options</span> <span class="kn">import</span> <span class="n">Options</span>
<span class="n">chrome_options</span> <span class="o">=</span> <span class="n">Options</span><span class="p">()</span>
<span class="n">chrome_options</span><span class="p">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s">"--mute-audio"</span><span class="p">)</span>
<span class="n">chrome_options</span><span class="p">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s">"--allow-file-access-from-files"</span><span class="p">)</span>
<span class="n">chrome_options</span><span class="p">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s">"--no-proxy-server"</span><span class="p">)</span>
<span class="n">driver</span> <span class="o">=</span> <span class="n">webdriver</span><span class="p">.</span><span class="n">Chrome</span><span class="p">(</span><span class="n">executable_path</span><span class="o">=</span><span class="s">"./chromedriver"</span><span class="p">,</span>
<span class="n">chrome_options</span><span class="o">=</span><span class="n">chrome_options</span><span class="p">)</span>
<span class="n">driver</span><span class="p">.</span><span class="n">set_window_rect</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">y</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">width</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span> <span class="n">height</span><span class="o">=</span><span class="mi">640</span> <span class="o">+</span> <span class="mi">133</span><span class="p">)</span>
<span class="n">driver</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"http://127.0.0.1"</span><span class="p">)</span>
<span class="c1"># ...
</span>
<span class="n">position</span> <span class="o">=</span> <span class="n">driver</span><span class="p">.</span><span class="n">get_window_position</span><span class="p">()</span>
<span class="nb">buffer</span> <span class="o">=</span> <span class="n">nativecap</span><span class="p">.</span><span class="n">capture</span><span class="p">(</span><span class="n">position</span><span class="p">[</span><span class="s">'x'</span><span class="p">],</span> <span class="n">position</span><span class="p">[</span><span class="s">'y'</span><span class="p">],</span> <span class="mi">360</span><span class="p">,</span> <span class="mi">640</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">ctypeslib</span><span class="p">.</span><span class="n">as_array</span><span class="p">(</span><span class="nb">buffer</span><span class="p">)</span>
<span class="n">image</span> <span class="o">=</span> <span class="n">data</span><span class="p">.</span><span class="n">reshape</span><span class="p">(</span><span class="mi">640</span><span class="p">,</span> <span class="mi">360</span><span class="p">,</span> <span class="mi">4</span><span class="p">)[:,</span> <span class="p">:,</span> <span class="p">:</span><span class="mi">3</span><span class="p">]</span>
</code></pre></div></div>
<p>Selenium is normally intended to be used as a functional testing tool for web development, it will open your already existing web browser in testing mode, then allow you to simulate clicks or keyboard typing directly from the same Python script. All of this in a non-blocking manner.</p>
<h1 id="finding-the-player">Finding the player</h1>
<p>After taking a look at this image, we can notice a little shadow right under the character. Let’s filter it with the OpenCV’s <code class="language-plaintext highlighter-rouge">inRange</code> method, which takes the minimum and maximum RGB values as parameters.</p>
<figure class="">
<img src="/assets/images/posts/endless-fake-1/filter-shadows.png" alt="Filtering the player's shadow" /><figcaption>
Filtering the player’s shadow
</figcaption></figure>
<p>Now we can see exactly where our player is, but we also might encounter some cases where these same colors are used in unexpected places of the image, mostly in some situations where the player would be cloned multiple times.</p>
<p>We can safely apply morphing techniques (erosion and dilatation) to get rid off tiny pixels, blurring, then only keep the biggest groups of pixels.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">kernel</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> <span class="n">np</span><span class="p">.</span><span class="n">uint8</span><span class="p">)</span>
<span class="n">mask</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">morphologyEx</span><span class="p">(</span><span class="n">mask</span><span class="p">,</span> <span class="n">cv2</span><span class="p">.</span><span class="n">MORPH_CLOSE</span><span class="p">,</span> <span class="n">kernel</span><span class="p">)</span>
<span class="n">mask</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">medianBlur</span><span class="p">(</span><span class="n">mask</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span>
<span class="n">cnts</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">findContours</span><span class="p">(</span><span class="n">mask</span><span class="p">,</span> <span class="n">cv2</span><span class="p">.</span><span class="n">RETR_TREE</span><span class="p">,</span>
<span class="n">cv2</span><span class="p">.</span><span class="n">CHAIN_APPROX_SIMPLE</span><span class="p">)</span>
<span class="n">filtered</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="k">lambda</span> <span class="n">c</span><span class="p">:</span> <span class="n">cv2</span><span class="p">.</span><span class="n">contourArea</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="o">></span> <span class="mf">80.0</span><span class="p">,</span> <span class="n">cnts</span><span class="p">))</span>
</code></pre></div></div>
<figure class="">
<img src="/assets/images/posts/endless-fake-1/filter-shadows-2.png" alt="Cleaning up by applying morphology and blurring techniques" /><figcaption>
Cleaning up by applying morphology and blurring techniques
</figcaption></figure>
<p>To make sure that we don’t mistake a player’s shadow for something else, I decided to retrieve the white pixels belonging to the player, draw a vertical line starting from the shadow directing to the top, then count the white pixels overlapping with that same line.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">player_mask</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">inRange</span><span class="p">([</span><span class="mi">208</span><span class="p">,</span> <span class="mi">208</span><span class="p">,</span> <span class="mi">208</span><span class="p">],</span> <span class="p">[</span><span class="mi">255</span><span class="p">,</span> <span class="mi">255</span><span class="p">,</span> <span class="mi">255</span><span class="p">])</span>
<span class="n">line_mask</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">zeros</span><span class="p">((</span><span class="mi">640</span><span class="p">,</span> <span class="mi">360</span><span class="p">),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="p">.</span><span class="n">uint8</span><span class="p">)</span>
<span class="n">cv2</span><span class="p">.</span><span class="n">line</span><span class="p">(</span><span class="n">line_mask</span><span class="p">,</span> <span class="n">pos</span><span class="p">,</span> <span class="p">(</span><span class="n">pos</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">pos</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">-</span> <span class="mi">128</span><span class="p">),</span> <span class="mi">255</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span>
<span class="n">overlap</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">bitwise_and</span><span class="p">(</span><span class="n">line_mask</span><span class="p">,</span> <span class="n">player_mask</span><span class="p">)</span>
<span class="k">return</span> <span class="n">cv2</span><span class="p">.</span><span class="n">countNonZero</span><span class="p">(</span><span class="n">overlap</span><span class="p">)</span> <span class="o">></span> <span class="mi">75</span>
</code></pre></div></div>
<figure class="">
<img src="/assets/images/posts/endless-fake-1/filter-shadows-3.png" alt="Confirming that the shadow actually belongs to our playing, by finding a sufficient amount of white pixels above" /><figcaption>
Confirming that the shadow actually belongs to our playing, by finding a sufficient amount of white pixels above
</figcaption></figure>
<h1 id="determining-the-direction">Determining the direction</h1>
<p>Before retrieving our first value, being the remaining distance to our next gap, we must determine our player’s direction which could be left, right, or bottom.</p>
<figure class="">
<img src="/assets/images/posts/endless-fake-1/directions.png" alt="Showing the player's possible directions: Left, Bottom, Right" /><figcaption>
Showing the player’s possible directions: Left, Bottom, Right
</figcaption></figure>
<p>In the beginning, I’ve been struggling a lot by simply filtering the platform’s pixels. That method ended up being very messy since we can encounter obstacles like some birds flying randomly over your path, or little arches obstructing our view.</p>
<p>Then I just figured out that it could be quite simpler! If we take a closer look at the platform’s edges, we can notice that each one of these has a unique color. All we need to do is filtering these, then grab the longest and closest one to the player.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">left_mask</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">inRange</span><span class="p">([</span><span class="mi">142</span><span class="p">,</span> <span class="mi">177</span><span class="p">,</span> <span class="mi">231</span><span class="p">],</span> <span class="p">[</span><span class="mi">169</span><span class="p">,</span> <span class="mi">209</span><span class="p">,</span> <span class="mi">255</span><span class="p">])</span>
<span class="n">right_mask</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">inRange</span><span class="p">([</span><span class="mi">112</span><span class="p">,</span> <span class="mi">146</span><span class="p">,</span> <span class="mi">196</span><span class="p">],</span> <span class="p">[</span><span class="mi">148</span><span class="p">,</span> <span class="mi">186</span><span class="p">,</span> <span class="mi">234</span><span class="p">])</span>
</code></pre></div></div>
<figure class="">
<img src="/assets/images/posts/endless-fake-1/all-edges.png" alt="Retrieving all visible edges: Left direction in blue, Right direction in orange" /><figcaption>
Retrieving all visible edges: Left direction in blue, Right direction in orange
</figcaption></figure>
<p>That way, we can not only determine the player’s direction but also the opposite edge defining the platform’s end, where a jump would be required before falling into the lake.</p>
<p>To achieve that, we need to draw two lines, both going respectively to the left and right direction. Retrieve the overlapping edges, keep the closest ones, then the longest one will be defined as the player’s direction.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">draw_line</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">angle</span><span class="p">,</span> <span class="n">length</span><span class="p">):</span>
<span class="n">x</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">start</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="n">length</span> <span class="o">*</span> <span class="n">math</span><span class="p">.</span><span class="n">cos</span><span class="p">(</span><span class="n">angle</span> <span class="o">*</span> <span class="n">math</span><span class="p">.</span><span class="n">pi</span> <span class="o">/</span> <span class="mf">180.0</span><span class="p">))</span>
<span class="n">y</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">start</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="n">length</span> <span class="o">*</span> <span class="n">math</span><span class="p">.</span><span class="n">sin</span><span class="p">(</span><span class="n">angle</span> <span class="o">*</span> <span class="n">math</span><span class="p">.</span><span class="n">pi</span> <span class="o">/</span> <span class="mf">180.0</span><span class="p">))</span>
<span class="n">cv2</span><span class="p">.</span><span class="n">line</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">coords</span><span class="p">,</span> <span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">x</span><span class="p">),</span> <span class="mi">255</span><span class="p">,</span> <span class="n">thickness</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
<span class="n">left_sample</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">zeros</span><span class="p">((</span><span class="mi">640</span><span class="p">,</span> <span class="mi">360</span><span class="p">),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="p">.</span><span class="n">uint8</span><span class="p">)</span>
<span class="n">right_sample</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">zeros</span><span class="p">((</span><span class="mi">640</span><span class="p">,</span> <span class="mi">360</span><span class="p">),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="p">.</span><span class="n">uint8</span><span class="p">)</span>
<span class="n">draw_line</span><span class="p">(</span><span class="n">left_sample</span><span class="p">,</span> <span class="n">coords</span><span class="p">,</span> <span class="o">-</span><span class="mf">56.5</span><span class="p">,</span> <span class="mi">300</span><span class="p">)</span>
<span class="n">draw_line</span><span class="p">(</span><span class="n">right_sample</span><span class="p">,</span> <span class="n">coords</span><span class="p">,</span> <span class="mf">56.5</span><span class="p">,</span> <span class="mi">300</span><span class="p">)</span>
<span class="n">left_overlap</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">bitwise_and</span><span class="p">(</span><span class="n">left_sample</span><span class="p">,</span> <span class="n">left_mask</span><span class="p">)</span>
<span class="n">right_overlap</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">bitwise_and</span><span class="p">(</span><span class="n">right_sample</span><span class="p">,</span> <span class="n">right_mask</span><span class="p">)</span>
</code></pre></div></div>
<figure class="">
<img src="/assets/images/posts/endless-fake-1/determining-direction.png" alt="Determining the player's direction by looking for the closest and longest edge" /><figcaption>
Determining the player’s direction by looking for the closest and longest edge
</figcaption></figure>
<h1 id="retrieving-our-values">Retrieving our values</h1>
<p>We’ve got our direction, that’s great! The distance remaining until the gap can be determined by calculating the one between the player’s position and the edge’s center.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_closest_edge</span><span class="p">(</span><span class="n">edge_mask</span><span class="p">,</span> <span class="n">start</span><span class="p">):</span>
<span class="n">edges</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">cnts</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">findContours</span><span class="p">(</span><span class="n">edge_mask</span><span class="p">,</span>
<span class="n">cv2</span><span class="p">.</span><span class="n">RETR_TREE</span><span class="p">,</span> <span class="n">cv2</span><span class="p">.</span><span class="n">CHAIN_APPROX_SIMPLE</span><span class="p">)</span>
<span class="n">cnts</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span>
<span class="k">lambda</span> <span class="n">c</span><span class="p">:</span> <span class="n">cv2</span><span class="p">.</span><span class="n">contourArea</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="o">></span> <span class="mf">10.0</span><span class="p">,</span> <span class="n">cnts</span><span class="p">))</span>
<span class="k">for</span> <span class="n">cnt</span> <span class="ow">in</span> <span class="n">cnts</span><span class="p">:</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">cv2</span><span class="p">.</span><span class="n">moments</span><span class="p">(</span><span class="n">cnt</span><span class="p">)</span>
<span class="n">cx</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">m</span><span class="p">[</span><span class="s">"m10"</span><span class="p">]</span> <span class="o">/</span> <span class="n">m</span><span class="p">[</span><span class="s">"m00"</span><span class="p">])</span>
<span class="n">cy</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">m</span><span class="p">[</span><span class="s">"m01"</span><span class="p">]</span> <span class="o">/</span> <span class="n">m</span><span class="p">[</span><span class="s">"m00"</span><span class="p">])</span>
<span class="n">edges</span><span class="p">.</span><span class="n">append</span><span class="p">((</span><span class="n">cv2</span><span class="p">.</span><span class="n">norm</span><span class="p">(</span><span class="n">start</span><span class="p">,</span> <span class="p">(</span><span class="n">cx</span><span class="p">,</span> <span class="n">cy</span><span class="p">)),</span>
<span class="p">(</span><span class="n">start</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">start</span><span class="p">[</span><span class="mi">1</span><span class="p">]),</span>
<span class="p">(</span><span class="n">cx</span><span class="p">,</span> <span class="n">cy</span><span class="p">)))</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">edges</span><span class="p">)</span> <span class="o"><</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="k">return</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">edges</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
</code></pre></div></div>
<p>As for the gap’s length, we can still draw a line from the player’s position pointing to its direction until the end of the image, then see how many pixels are overlapping the platform’s. That way we would obtain the next platform’s length by overlapping on its inverted mask.</p>
<figure class="">
<img src="/assets/images/posts/endless-fake-1/getting-values.png" alt="Retrieving the remaining values. Next platform's length will overlap the platform's mask. Gap's length will overlap the inverted one." /><figcaption>
Retrieving the remaining values. Next platform’s length will overlap the platform’s mask. Gap’s length will overlap the inverted one.
</figcaption></figure>
<h1 id="wrapping-up">Wrapping Up</h1>
<p>That’s all for the image recognition process! As you may have figured out, I planned to use these values for a neural network model. More details will come into another blog post once I successfully deal with this.</p>
<p>I’m completely aware that there are certainly better ways to deal with this kind of problem, such as the use of CNNs (Convolutional Neural Networks), which would just require us to send images to our Machine Learning model.</p>
<p>These methods will be covered as soon as I can get my hands on them. Feel free to leave a comment below if you have better ideas!</p>Axel Isouardaxel@isouard.frBuilding an Artificial Intelligence to beat the Endless Lake video game. Let's gather some data with the image recognition part with OpenCV and Python, before giving these to our future neural network!Streaming a video with Media Source Extensions2016-05-24T00:00:00+00:002016-05-24T00:00:00+00:00https://axel.isouard.fr/blog/2016/05/24/streaming-webm-video-over-html5-with-media-source<p>Back in the 2000’s, playing videos on a web page was truly a mess. The audience
needed to install plugins such as RealPlayer, or just cross fingers while
waiting for the video to show up inside a Windows Media Player canvas. YouTube
has proven us that Adobe Flash was great for it’s time to stream media. Since
then, embedding videos became a less painful task than ever.</p>
<p>Today, the HTML5 specification allows us to embed a video on a web page, like
if we want to embed a picture.</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><video</span> <span class="na">width=</span><span class="s">"640"</span> <span class="na">src=</span><span class="s">"video.webm"</span><span class="nt">></video></span></code></pre></figure>
<p>
<video width="640" controls="">
<source src="/assets/images/posts/webm-streaming/mov_bbb.mp4" type="video/mp4" />
<source src="/assets/images/posts/webm-streaming/mov_bbb.ogg" type="video/ogg" />
Your browser does not support HTML5 video.
</video>
</p>
<p>But the URL specified inside the <code class="language-plaintext highlighter-rouge">src</code> attribute must lead to a complete video
file. What if we’d want to stream a video by directly injecting it’s binary
data right inside the player, just like that ?</p>
<p>
<button id="addNextSegmentBtn" class="btn btn--disabled">Loading...</button>
</p>
<p>
<video id="player" style="background-color: black" width="640" controls="">
</video>
</p>
<script type="text/javascript">
var button;
var player;
var sourceBuffer;
var mediaSource;
var currentSegment = 0;
window.addEventListener('load', function () {
button = document.getElementById('addNextSegmentBtn');
button.addEventListener('click', onAddNextSegmentBtnClick);
mediaSource = new MediaSource();
mediaSource.addEventListener('sourceopen', mediaSourceOpen);
player = document.getElementById('player');
player.src = window.URL.createObjectURL(mediaSource);
});
function onAddNextSegmentBtnClick() {
ga('send', {
hitType: 'event',
eventCategory: 'MediaSourceDemo',
eventAction: 'AddNextSegmentBtnClick',
eventValue: currentSegment
});
if (currentSegment > 4) {
return;
}
button.className = 'btn btn--disabled';
button.innerHTML = 'Please wait...';
fetchNextSegment('media_000' + currentSegment++ + '.segment', function (bytes) {
sourceBuffer.appendBuffer(bytes);
});
}
function mediaSourceOpen() {
var mimeType = 'video/webm; codecs="vorbis,vp9"';
ga('send', {
hitType: 'event',
eventCategory: 'MediaSourceDemo',
eventAction: 'MediaSourceOpen'
});
sourceBuffer = mediaSource.addSourceBuffer(mimeType);
sourceBuffer.addEventListener('updateend', function () {
ga('send', {
hitType: 'event',
eventCategory: 'MediaSourceDemo',
eventAction: 'MediaSourceUpdateEnd',
eventValue: currentSegment
});
if (currentSegment > 4) {
mediaSource.endOfStream();
button.className = 'btn btn--disabled';
button.innerHTML = 'Loading complete';
return;
}
button.className = 'btn';
button.innerHTML = 'Click to add the next segment (' + currentSegment +
' of 5)';
});
fetchNextSegment('init.segment', function (bytes) {
sourceBuffer.appendBuffer(bytes);
});
}
function fetchNextSegment(filename, cb) {
var req = new XMLHttpRequest();
req.open('get', '/assets/images/posts/webm-streaming/' + filename, true);
req.onload = function (e) {
cb(new Uint8Array(req.response));
};
req.responseType = 'arraybuffer';
req.send();
}
</script>
<p>According to the <a href="https://developer.mozilla.org/en-US/Apps/Fundamentals/Audio_and_video_delivery/Live_streaming_web_audio_and_video#Streaming_Protocols">MDN website</a>, the <code class="language-plaintext highlighter-rouge">src</code> attribute supports the <em>RTMP</em>
protocol, which was used by <em>Adobe Flash</em> to stream videos on YouTube. That
wasn’t enough for my graduation project involving video streaming over a
peer-to-peer network between web browsers.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// Because I was looking for something THAT simple...</span>
<span class="kd">var</span> <span class="nx">player</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">'</span><span class="s1">player</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">data</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Uint8Array</span><span class="p">([</span><span class="mh">0x1F</span><span class="p">,</span> <span class="mh">0x6B</span><span class="p">,</span> <span class="p">...]);</span>
<span class="nx">player</span><span class="p">.</span><span class="nx">appendBytes</span><span class="p">(</span><span class="nx">data</span><span class="p">);</span></code></pre></figure>
<h2 id="create-a-media-source-object">Create a Media Source object</h2>
<p>The HTML5 Media element does not provide a way to do it directly. YouTube uses
the <em>Media Source Extensions (MSE)</em> API to replace the behavior of their old
flash video player. It simply takes the place of the file URL as the <code class="language-plaintext highlighter-rouge">src</code>
attribute on your <code class="language-plaintext highlighter-rouge"><video></code> element and gives you the possibility to inject the
media’s data like that:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">var</span> <span class="nx">mediaSource</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">MediaSource</span><span class="p">();</span>
<span class="nx">mediaSource</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">'</span><span class="s1">sourceopen</span><span class="dl">'</span><span class="p">,</span> <span class="nx">mediaSourceOpen</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">player</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="dl">'</span><span class="s1">player</span><span class="dl">'</span><span class="p">);</span>
<span class="nx">player</span><span class="p">.</span><span class="nx">src</span> <span class="o">=</span> <span class="nb">window</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">createObjectURL</span><span class="p">(</span><span class="nx">mediaSource</span><span class="p">);</span></code></pre></figure>
<h2 id="add-a-source-buffer">Add a Source Buffer</h2>
<p>Your video’s <code class="language-plaintext highlighter-rouge">src</code> attribute has been replaced with a generated URL, the web
browser will no longer try to retrieve the video file remotely, but it will
listen to the attached <em>Source Buffers</em> instead. These buffers are appended to
the <em>Media Source</em> object and filled with binary data from the video we want to
stream. A <em>Source Buffer</em> must be created upon initialization by telling which
media format and codecs we’re going to use.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">var</span> <span class="nx">sourceBuffer</span><span class="p">;</span>
<span class="kd">function</span> <span class="nx">mediaSourceOpen</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">data</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Uint8Array</span><span class="p">([</span><span class="mh">0x1F</span><span class="p">,</span> <span class="mh">0x9A</span><span class="p">,</span> <span class="mh">0x3B</span><span class="p">,</span> <span class="p">...]);</span>
<span class="kd">var</span> <span class="nx">mimeType</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">video/webm; codecs="vorbis,vp9"</span><span class="dl">'</span><span class="p">;</span>
<span class="nx">sourceBuffer</span> <span class="o">=</span> <span class="nx">mediaSource</span><span class="p">.</span><span class="nx">addSourceBuffer</span><span class="p">(</span><span class="nx">mimeType</span><span class="p">);</span>
<span class="nx">sourceBuffer</span><span class="p">.</span><span class="nx">appendBuffer</span><span class="p">(</span><span class="nx">data</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<h2 id="find-the-right-data">Find the right data</h2>
<p>The video should play, but it doesn’t. Everything here is correct, except the
most important thing: the data we are transmitting. You might expect that we
could just split the video file as we want, but if you take a look at the
<a href="https://w3c.github.io/media-source/index.html">Media Source Extensions</a> specification, you’ll notice that this API
is very fragile and will break on the first mistake it will find
inside the data you’ll transmit.</p>
<p>Our Source Buffer will take two kinds of data as input:</p>
<ul>
<li>Initialization Segment, the first slice of video we have to send, this
tells our player about the video resolution, duration, bitrate, …</li>
<li>Media Segment, the next slices containing the images and audio data to
play.</li>
</ul>
<p>These segments must be structured accordingly to the
<a href="https://w3c.github.io/media-source/byte-stream-format-registry.html#registry">MSE Byte Stream Format Registry</a>. Only a few media formats and audio/video
codecs are supported. We will work with the <em>WebM</em> media format as it’s more
used and easier to work with than the <em>MP4</em> one, which I will write about
later in another post.</p>
<figure>
<a href="/assets/images/posts/webm-streaming/youtube-webm-chunk.png">
<img src="/assets/images/posts/webm-streaming/youtube-webm-chunk.png" />
</a>
<figcaption>Receiving a WebM Media Segment from YouTube</figcaption>
</figure>
<h2 id="encode-the-video">Encode the video</h2>
<p>MSE supports the <em>WebM</em> format with <em>VP8</em> and <em>VP9</em> video codecs and Vorbis
audio codec. We’ll use the popular <em>FFmpeg</em> tool in order to encode the video
properly, then we’ll have to remux it so we can retrieve the needed segments.</p>
<p>Here’s the command line I’ve been using all the time to convert a video to the
WebM format with the <em>VP9</em> video codec and the <em>Vorbis</em> audio codec:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>ffmpeg <span class="nt">-i</span> movie.avi <span class="nt">-c</span>:v libvpx-vp9 <span class="nt">-b</span>:v 8000k <span class="nt">-tile-columns</span> 4 <span class="se">\</span>
<span class="nt">-frame-parallel</span> 1 <span class="nt">-keyint_min</span> 90 <span class="nt">-g</span> 90 <span class="nt">-f</span> webm <span class="nt">-dash</span> 1 <span class="se">\</span>
<span class="nt">-c</span>:a libvorbis <span class="nt">-b</span>:a 64K movie.webm</code></pre></figure>
<p>You should keep the <code class="language-plaintext highlighter-rouge">-tile-columns</code> and <code class="language-plaintext highlighter-rouge">-frame-parallel</code> flags as they are,
to enable multi-threaded encoding. The <code class="language-plaintext highlighter-rouge">-keyint_min</code> and <code class="language-plaintext highlighter-rouge">-g</code> flags define the
length of your media segments. They represent the “GOP” which stands for Group
Of Pictures.</p>
<p>Assuming that the video’s framerate is set to 30 frames per seconds, we’ll
end up with media segments lasting 3 seconds each. That’s the only way to
define each media segment length, since it’s the encoder’s job to split the
video for the Media Source Extensions API.</p>
<p>Once the encoding process is finished, we need to generate the seeks table
(which is similar to a <em>table of contents</em>) at the beginning of the file
(since FFmpeg won’t go back and rewrite the beginning of the file after the
process). It must be present inside the initialization segment, in order to
let the player know the which media segment should be played at the time we
are seeking.</p>
<p>For that, you have to grab and compile the <a href="https://github.com/webmproject/libwebm">sample_muxer tool</a> and run the
following command:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>sample_muxer <span class="nt">-i</span> movie.webm <span class="nt">-o</span> movie_muxed.webm <span class="nt">-output_cues</span> 1 <span class="se">\</span>
<span class="nt">-cues_before_clusters</span> 1</code></pre></figure>
<h2 id="read-the-data">Read the data</h2>
<p>Our movie is now ready to be streamed over the web. It’s time to read the
binary data inside and extract the segments.</p>
<p>Before that, you need to learn about your video’s file format. WebM files are
similar to MKV files, they are both using the EBML format which is short for
Extensible Binary Meta Language. The structure is a little like XML, except
that this time, it’s not a human-readable format.</p>
<p>The following code shows how it would look like if it was:</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="c"><!-- begin of the Initialization Segment --></span>
<span class="nt"><Ebml</span> <span class="na">offset=</span><span class="s">"0"</span> <span class="na">ebml-id=</span><span class="s">"0x1A45DFA3"</span> <span class="na">size=</span><span class="s">"43"</span> <span class="na">data-size=</span><span class="s">"31"</span><span class="nt">></span>
<span class="nt"><Version</span> <span class="na">offset=</span><span class="s">"12"</span> <span class="na">ebml-id=</span><span class="s">"0x4286"</span> <span class="na">size=</span><span class="s">"4"</span> <span class="na">data-size=</span><span class="s">"1"</span><span class="nt">></span>1<span class="nt"></Version></span>
<span class="nt"><ReadVersion</span> <span class="na">offset=</span><span class="s">"16"</span> <span class="na">ebml-id=</span><span class="s">"0x42F7"</span> <span class="na">size=</span><span class="s">"4"</span> <span class="na">data-size=</span><span class="s">"1"</span><span class="nt">></span>1<span class="nt"></ReadVersion></span>
<span class="nt"><MaxIDLength</span> <span class="na">offset=</span><span class="s">"20"</span> <span class="na">ebml-id=</span><span class="s">"0x42F2"</span> <span class="na">size=</span><span class="s">"4"</span> <span class="na">data-size=</span><span class="s">"1"</span><span class="nt">></span>4<span class="nt"></MaxIDLength></span>
<span class="nt"><MaxSizeLength</span> <span class="na">offset=</span><span class="s">"24"</span> <span class="na">ebml-id=</span><span class="s">"0x42F3"</span> <span class="na">size=</span><span class="s">"4"</span> <span class="na">data-size=</span><span class="s">"1"</span><span class="nt">></span>8<span class="nt"></MaxSizeLength></span>
<span class="nt"><DocType</span> <span class="na">offset=</span><span class="s">"28"</span> <span class="na">ebml-id=</span><span class="s">"0x4282"</span> <span class="na">size=</span><span class="s">"7"</span> <span class="na">data-size=</span><span class="s">"4"</span><span class="nt">></span>webm<span class="nt"></DocType></span>
<span class="nt"><DocTypeVersion</span> <span class="na">offset=</span><span class="s">"35"</span> <span class="na">ebml-id=</span><span class="s">"0x4287"</span> <span class="na">size=</span><span class="s">"4"</span> <span class="na">data-size=</span><span class="s">"1"</span><span class="nt">></span>2<span class="nt"></DocTypeVersion></span>
<span class="nt"><DocTypeReadVersion</span> <span class="na">offset=</span><span class="s">"39"</span> <span class="na">ebml-id=</span><span class="s">"0x4285"</span> <span class="na">size=</span><span class="s">"4"</span> <span class="na">data-size=</span><span class="s">"1"</span><span class="nt">></span>2<span class="nt"></DocTypeReadVersion></span>
<span class="nt"></Ebml></span>
<span class="nt"><Segment</span> <span class="na">offset=</span><span class="s">"43"</span> <span class="na">ebml-id=</span><span class="s">"0x18538067"</span> <span class="na">size=</span><span class="s">"1448"</span> <span class="na">data-size=</span><span class="s">"1436"</span><span class="nt">></span>
...
<span class="nt"><Void</span> <span class="na">offset=</span><span class="s">"124"</span> <span class="na">ebml-id=</span><span class="s">"0x6C"</span> <span class="na">size=</span><span class="s">"158"</span> <span class="na">data-size=</span><span class="s">"149"</span><span class="nt">></span>(149 bytes)<span class="nt"></Void></span>
<span class="nt"><Info</span> <span class="na">offset=</span><span class="s">"282"</span> <span class="na">ebml-id=</span><span class="s">"0x1549A966"</span> <span class="na">size=</span><span class="s">"81"</span> <span class="na">data-size=</span><span class="s">"69"</span><span class="nt">></span>...<span class="nt"></Info></span>
<span class="nt"><Tracks</span> <span class="na">offset=</span><span class="s">"363"</span> <span class="na">ebml-id=</span><span class="s">"0x1654AE6B"</span> <span class="na">size=</span><span class="s">"133"</span> <span class="na">data-size=</span><span class="s">"121"</span><span class="nt">></span>...<span class="nt"></Tracks></span>
<span class="c"><!-- end of the Initialization Segment --></span>
<span class="c"><!-- begin of the first Media Segment --></span>
<span class="nt"><Cluster</span> <span class="na">offset=</span><span class="s">"700"</span> <span class="na">ebml-id=</span><span class="s">"0x1F43B675"</span> <span class="na">size=</span><span class="s">"766"</span> <span class="na">data-size=</span><span class="s">"754"</span><span class="nt">></Cluster></span>
<span class="c"><!-- end of the first Media Segment --></span>
<span class="c"><!-- begin of the second Media Segment --></span>
<span class="nt"><Cluster</span> <span class="na">offset=</span><span class="s">"1466"</span> <span class="na">ebml-id=</span><span class="s">"0x1F43B675"</span> <span class="na">size=</span><span class="s">"766"</span> <span class="na">data-size=</span><span class="s">"754"</span><span class="nt">></Cluster></span>
<span class="c"><!-- end of the second Media Segment --></span>
...
<span class="nt"></Ebml></span></code></pre></figure>
<p>Reading the <a href="https://w3c.github.io/media-source/webm-byte-stream-format.html#webm-init-segments">WebM Byte Stream Format</a> is also necessary to understand which
kind of data we should append to the Media Source buffer. The specification
tells us that the <em>Initialization Segment</em> must start with an <em>EBML</em> element
followed by a <em>Segment</em> element, which must include a <em>Segment Information</em>
and <em>Tracks</em> element.</p>
<p>Instead of writing a full-featured parser, we can simply read as many bytes as
necessary until we stumble upon a <em>Cluster</em> element header, which is the
beginning of our first <em>Media Segment</em>.</p>
<p>Let’s make that process easy by opening the video file with a hexadecimal
editor. <a href="https://wiki.gnome.org/Apps/Ghex">GHex</a> is my favourite one under Linux, <a href="https://mh-nexus.de/en/hxd">HxD</a> is also a good one
under Windows and <a href="http://ridiculousfish.com/hexfiend">Hex Fiend</a> should be more than enough for your Mac.</p>
<figure>
<a href="/assets/images/posts/webm-streaming/webm-hex-header.png">
<img src="/assets/images/posts/webm-streaming/webm-hex-header.png" />
</a>
<figcaption>WebM file inside a hexadecimal editor</figcaption>
</figure>
<p>By reading the <a href="https://www.matroska.org/technical/specs/index.html">EBML specification</a>, we can figure out that the first
bytes <code class="language-plaintext highlighter-rouge">1A 45 DF A3</code> represent the <em>EBML</em> tag. This tag is the beginning of our
Initialization Segment, the binary data is likely to be the same after every
encoding process, we shouldn’t bother parsing the tag’s contents and skip it
instead by moving 31 bytes forward as written in the tag’s length.</p>
<figure>
<a href="/assets/images/posts/webm-streaming/webm-hex-segment.png">
<img src="/assets/images/posts/webm-streaming/webm-hex-segment.png" />
</a>
<figcaption>Segment element</figcaption>
</figure>
<p>The next tag is the Segment tag, containing some valuable information for our
video player, such as the video’s resolution, duration, bitrate and
codec-related stuff. It also contains every <em>Cluster</em> tags, which happen to be
our Media Segments until the very end of the file.</p>
<p>It would have been much easier if the <em>Cluster</em> tags could be located outside
that <em>Segment</em> one, but this time, we will have to dig inside it and skip
everything until we hit the first <em>Cluster</em> tag.</p>
<p>Reading EBML tags is not straightforward, <a href="https://gist.github.com/aisouard/4ffc0bd2b992cf65e432e86d8471b83c">here’s the function</a> I’ve been
using to parse every EBML tags and get their length. <em>I’m currently finishing
my EBML parser written in JavaScript with the Node.js framework, feel free to
<a href="https://twitter.com/aisouard">follow me on Twitter</a> to get an update once it’s done.</em></p>
<h2 id="send-the-segments">Send the segments</h2>
<p>Once we got everything we need, we can finally proceed to the most exciting
part, which is seeing everything working perfectly. We just have to take the
source code at the top of this blog post, and inject our Initialization Segment.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">var</span> <span class="nx">initSegment</span> <span class="o">=</span> <span class="nx">retrieveInitSegment</span><span class="p">();</span>
<span class="kd">var</span> <span class="nx">sourceBuffer</span><span class="p">;</span>
<span class="kd">function</span> <span class="nx">mediaSourceOpen</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">data</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Uint8Array</span><span class="p">(</span><span class="nx">initSegment</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">mimeType</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">video/webm; codecs="vorbis,vp9"</span><span class="dl">'</span><span class="p">;</span>
<span class="nx">sourceBuffer</span> <span class="o">=</span> <span class="nx">mediaSource</span><span class="p">.</span><span class="nx">addSourceBuffer</span><span class="p">(</span><span class="nx">mimeType</span><span class="p">);</span>
<span class="nx">sourceBuffer</span><span class="p">.</span><span class="nx">appendBuffer</span><span class="p">(</span><span class="nx">data</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>Depending on your web browser, the video player might be blank and show a
black picture, waiting for the first Media Segment to be appended. But you
can’t append one segment after another, otherwise, the Source Buffer will
throw an error and the media player will crash.</p>
<p>The Source Buffer emits an event before and after updating, you have to wait
until it fires the <code class="language-plaintext highlighter-rouge">updateend</code> event before appending another Media Segment.</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">var</span> <span class="nx">mediaSegments</span> <span class="o">=</span> <span class="p">[];</span>
<span class="kd">function</span> <span class="nx">addMediaSegment</span><span class="p">(</span><span class="nx">bytes</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">mediaSegments</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="nx">bytes</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">onUpdateEnd</span><span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">mediaSegments</span><span class="p">.</span><span class="nx">length</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">sourceBuffer</span><span class="p">.</span><span class="nx">appendBuffer</span><span class="p">(</span><span class="nx">mediaSegments</span><span class="p">.</span><span class="nx">shift</span><span class="p">());</span>
<span class="p">}</span>
<span class="nx">sourceBuffer</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">'</span><span class="s1">updateend</span><span class="dl">'</span><span class="p">,</span> <span class="nx">onUpdateEnd</span><span class="p">);</span>
<span class="nx">sourceBuffer</span><span class="p">.</span><span class="nx">appendBuffer</span><span class="p">(</span><span class="nx">initSegment</span><span class="p">);</span></code></pre></figure>
<h2 id="whats-next">What’s next</h2>
<p>This article showed you how to stream a video with the HTML5 Media Player
instead of the Adobe Flash Player. Now you have some basic tips to help you
creating a video player that is similar to YouTube’s, I really hope they were
useful to you.</p>
<p>You can improve it by handling the seeking behavior. You’ll have to retrieve
the current time position, compare to each <em>Cluster</em>’s absolute time code, then
inject the right <em>Media Segment</em> into the <em>Source Buffer</em>.</p>
<p>Or even better, you could implement an algorithm that would switch to different
video resolutions and bitrates, depending to the current viewer’s downloading
bandwidth. But that stuff is pretty complicated and depends on many factors.
Rather than writing everything yourself, you should take a look at
<a href="https://github.com/Dash-Industry-Forum/dash.js">the Dash.js library</a> for a cutting-edge MPEG DASH implementation.</p>
<p>Stay tuned for future articles about video streaming and other HTML5 &
JavaScript tips. I’m also in fond of video game development so I will surely
write about that topic later. Feel free to send me any feedback or suggestion
via the comments or <a href="https://twitter.com/aisouard">Twitter</a>!</p>Axel Isouardaxel@isouard.frEmbedding a video inside a web page has been simple since the release of the HTML5 specification. This article will show you how to inject a partial video inside the player.