P2P Web communication
P2P Web communication
netcode.io is a simple network protocol that lets clients securely connect to dedicated servers and communicate over UDP. It’s connection oriented and encrypts and signs packets, and provides authentication support so only authenticated clients can connect to dedicated servers.
It’s designed to shunt players off from the main website to a number of dedicated server instances, each with some maximum number of players (up to 256 players per-instance in the reference implementation).
A client authenticates with the web backend using standard authentication techniques (eg. OAuth).
Once a client is authenticated they request to play a game by making a REST call. The REST call returns a connect token to that client encoded as base64 over HTTPS.
A connect token has two parts:
When connecting to a dedicated server the client sends a connection request packet repeatedly over UDP. This packet contains the private connect token data, plus some additional application data.
Provided the connect token is valid, it is decrypted. Internally it contains a list of dedicated server addresses that the connect token is valid for, stopping malicious clients going wide with one connect token and using it to connect to all available dedicated servers.
The server also checks if the connect token has already been used by searching a short history of connect token HMACs, and ignores the connection request if a match is found. This prevents one connect token from being used to connect multiple clients.
Additionally, the server enforces that only one client with a given IP address and port may be connected at any time, and only one client by unique client id may be connected at any time, where client id is a 64 bit integer that uniquely identifies a client that has been authenticated by the web backend.
After the connect token is validated:
Next, the server checks if there is room for the client on the server. Each server supports some maximum number of clients, for example a 64 player game has 64 slots for clients to connect to. If the server is full, it responds with a connection request denied packet. This lets clients quickly know to move on to the next server in the list when a server is full.
If there is room for the client, the server doesn’t yet assign the client to that slot, but instead stores the address + HMAC for the connect token for that client as a potential client. The server then responds with a connection challenge packet, which contains a challenge token which is a block of data encrypted with a random key rolled when the server is started.
This key randomization ensures there is not a security problem when the same sequence number is used to encrypt challenge tokens across multiple servers (the servers do not coordinate). Also, the connection challenge packet is significantly smaller than the connection request packet by design, to eliminate the possibility of the protocol being used as part of a DDoS amplification attack.
The client receives the connection challenge packet over UDP and switches to a state where it sends connection response packets to the server. Connection response packets simply reflect the challenge token back to the dedicated server, establishing that the client is actually able to receive packets on the source IP address they claim they are sending packets from. This stops clients with spoofed packet source addresses from connecting.
When the server receives a connection response packet it looks for a matching pending client entry, and if one exists, it searches once again for a free slot for the client to connect to. If there isn’t one, it replies with a connection request denied packet since there may have been a slot free when the connection request was first received that is no longer available.
Alternatively, the server assigns the client to a free slot and replies back with a connection keep-alive packet, which tells the client which slot it was assigned on the server. This is known as a client index. In multiplayer games, this is typically used to identify clients connected to a server. For example, clients 0,1,2,3 in a 4 player game correspond to players 1,2,3 and 4.
The server now considers the client connected and is able to send connection payload packets down to that client. These packets wrap game specific data and are delivered unreliable-ordered. The only caveat is that since the client needs to first receive a connection keep-alive before it knows its client index and considers itself to be fully connected, the server tracks on a per-client slot basis whether that client is confirmed.
The confirmed flag per-client is initially set to false, and flips true once the server has received a keep-alive or payload packet from that client. Until a client is confirmed, each time a payload packet is sent from the server to that client, it is prefixed with a keep-alive packet. This ensures the client is statistically likely to know its client index and be fully connected prior to receiving the first payload packet sent from the server, minimizing the number of connection establishment round-trips.
Now that the client and server are fully connected they can exchange UDP packets bidirectionally. Typical game protocols sent player inputs from client to server at a high rate like 60 times per-second, and world state from the server to client at a slightly lower rate, like 20 times per-second. However more recent AAA games are increasing the server update rate.
If the server or client don’t exchange a steady stream of packets, keep-alive packets are automatically generated so the connection doesn’t time out. If no packets are received from either side of the connection for a short amount of time like 5 seconds, the connection times out.
If either side of the connection wishes to cleanly disconnect, a number of connection disconnect packets are fired across redundantly, so that statistically these packets are likely to get through even under packet loss. This ensures that clean disconnects happen quickly, without the other side waiting for time out.
Interactive Connectivity Establishment (ICE) is a technique used in computer networking to find ways for two computers to talk to each other as directly as possible in peer-to-peer networking.
You want to avoid communicating through a central server (which would slow down communication, and be expensive), but direct communication between client applications on the Internet is very tricky due to network address translators (NATs), firewalls, and other network barriers.
ICE is developed by the Internet Engineering Task Force MMUSIC working group and is published as RFC-8445.
ICE provides a framework with which a communicating peer may discover and communicate its public IP address so that it can be reached by other peers.
RTC(real-time communication).
With WebRTC, you can add real-time communication capabilities to your application that works on top of an open standard.
The technologies behind WebRTC are implemented as an open web standard and available as regular JavaScript APIs in all major browsers.
For native clients, like Android and iOS applications, a library is available that provides the same functionality.
The WebRTC standard covers, on a high level, two different technologies: media capture devices and peer-to-peer connectivity.
These devices are commonly referred to as Media Devices and can be accessed with JavaScript through the navigator.mediaDevices object, which implements the MediaDevices interface.
The communication between peers can be video, audio or arbitrary binary data (for clients supporting the RTCDataChannel API).
In order to discover how two peers can connect, both clients need to provide an ICE Server configuration. This is either a STUN or a TURN-server, and their role is to provide ICE candidates to each client which is then transferred to the remote peer. This transferring of ICE candidates is commonly called signaling.
Signaling is needed in order for two peers to share how they should connect. Usually this is solved through a regular HTTP-based Web API (i.e., a REST service or other RPC mechanism) where web applications can relay the necessary information before the peer connection is initiated.
Signaling can be implemented in many different ways, and the WebRTC specification doesn't prefer any specific solution.
Kurento is an Open source WebRTC media server.
The WebRTC specification is very clear about what video encoding formats (also known as codecs) are supported, any complying implementation must be able to support both VP8 and H.264 video codecs.(See RFC7742 / 5. Mandatory-to-Implement Video Codec).
The point of WebRTC is to actually be able to serve video in a secure, reliable and effective manner, WebRTC supports a variety of feedback mechanisms that allow any viewer to "inform" about current network conditions to the sender of the video:
By introducing a decoder + encoder pair(transcoding), the gateway is able to solve both the codec compatibility and the network reliability issues.
The agnosticbin element inside Kurento is again the component in charge of building an encoding tree that covers codec requirements for all consumers, without doing the same work twice:
If Kurento Media Server, its Application Server, or any of the clients are located behind a NAT, you need to use a STUN or a TURN server in order to achieve NAT traversal. In most cases, STUN is effective in addressing the NAT issue with most consumer network devices (routers). However, it doesn’t work for many corporate networks, so a TURN server becomes necessary.
Apart from that, you need to open all UDP ports in your system configuration, as STUN will use any random port from the whole [0-65535] range.
NGINX (pronounced engine x) is a popular lightweight web server application you can install on the Raspberry Pi to allow it to serve web pages.
Like Apache, NGINX can serve HTML files over HTTP, and with additional modules can serve dynamic web pages using scripting languages such as PHP.
By default, NGINX puts a test HTML file in the web folder.
Nginx RTMP is an Nginx module which allows you to add RTMP and HLS streaming to your media server.
Install the RTMP module so Nginx can handle your media stream:
To set up RTMP support you need to add rtmp{} section to the configuration file "/etc/nginx/nginx.conf":
-t duration
(input/output). When used as an input option (before "-i"), limit the duration of data read from the input file. When used as an output option (before an output url), stop writing the output after its duration reaches duration. There are two accepted syntaxes for expressing time duration. -timestamp date
(output). Set the recording timestamp in the container. The accepted syntax is: -an
Disable audio recording.
Web Players/Javascript Libraries:
To generate a .m3u8 list:
netcode.io
netcode.io is a simple network protocol that lets clients securely connect to dedicated servers and communicate over UDP. It’s connection oriented and encrypts and signs packets, and provides authentication support so only authenticated clients can connect to dedicated servers.
It’s designed to shunt players off from the main website to a number of dedicated server instances, each with some maximum number of players (up to 256 players per-instance in the reference implementation).
- the web backend performs authentication
- when a client wants to play, it makes a REST call to obtain a connect token which is passed to the dedicated server as part of the connection handshake over UDP. Connect tokens are short lived and rely on a shared private key between the web backend and the dedicated server instances.
How it works
A client authenticates with the web backend using standard authentication techniques (eg. OAuth).
Once a client is authenticated they request to play a game by making a REST call. The REST call returns a connect token to that client encoded as base64 over HTTPS.
A connect token has two parts:
- A private portion Encrypted and signed by the shared private key. This cannot be read, modified or forged by the client.
- A public portion Information the client needs to connect, like encryption keys for UDP packets and the list of server addresses to connect to, along with some other information
When connecting to a dedicated server the client sends a connection request packet repeatedly over UDP. This packet contains the private connect token data, plus some additional application data.
Provided the connect token is valid, it is decrypted. Internally it contains a list of dedicated server addresses that the connect token is valid for, stopping malicious clients going wide with one connect token and using it to connect to all available dedicated servers.
The server also checks if the connect token has already been used by searching a short history of connect token HMACs, and ignores the connection request if a match is found. This prevents one connect token from being used to connect multiple clients.
Additionally, the server enforces that only one client with a given IP address and port may be connected at any time, and only one client by unique client id may be connected at any time, where client id is a 64 bit integer that uniquely identifies a client that has been authenticated by the web backend.
After the connect token is validated:
- not expired
- decrypted successfully
- the dedicated server’s public IP is in the list of server addresses
Next, the server checks if there is room for the client on the server. Each server supports some maximum number of clients, for example a 64 player game has 64 slots for clients to connect to. If the server is full, it responds with a connection request denied packet. This lets clients quickly know to move on to the next server in the list when a server is full.
If there is room for the client, the server doesn’t yet assign the client to that slot, but instead stores the address + HMAC for the connect token for that client as a potential client. The server then responds with a connection challenge packet, which contains a challenge token which is a block of data encrypted with a random key rolled when the server is started.
This key randomization ensures there is not a security problem when the same sequence number is used to encrypt challenge tokens across multiple servers (the servers do not coordinate). Also, the connection challenge packet is significantly smaller than the connection request packet by design, to eliminate the possibility of the protocol being used as part of a DDoS amplification attack.
The client receives the connection challenge packet over UDP and switches to a state where it sends connection response packets to the server. Connection response packets simply reflect the challenge token back to the dedicated server, establishing that the client is actually able to receive packets on the source IP address they claim they are sending packets from. This stops clients with spoofed packet source addresses from connecting.
When the server receives a connection response packet it looks for a matching pending client entry, and if one exists, it searches once again for a free slot for the client to connect to. If there isn’t one, it replies with a connection request denied packet since there may have been a slot free when the connection request was first received that is no longer available.
Alternatively, the server assigns the client to a free slot and replies back with a connection keep-alive packet, which tells the client which slot it was assigned on the server. This is known as a client index. In multiplayer games, this is typically used to identify clients connected to a server. For example, clients 0,1,2,3 in a 4 player game correspond to players 1,2,3 and 4.
The server now considers the client connected and is able to send connection payload packets down to that client. These packets wrap game specific data and are delivered unreliable-ordered. The only caveat is that since the client needs to first receive a connection keep-alive before it knows its client index and considers itself to be fully connected, the server tracks on a per-client slot basis whether that client is confirmed.
The confirmed flag per-client is initially set to false, and flips true once the server has received a keep-alive or payload packet from that client. Until a client is confirmed, each time a payload packet is sent from the server to that client, it is prefixed with a keep-alive packet. This ensures the client is statistically likely to know its client index and be fully connected prior to receiving the first payload packet sent from the server, minimizing the number of connection establishment round-trips.
Now that the client and server are fully connected they can exchange UDP packets bidirectionally. Typical game protocols sent player inputs from client to server at a high rate like 60 times per-second, and world state from the server to client at a slightly lower rate, like 20 times per-second. However more recent AAA games are increasing the server update rate.
If the server or client don’t exchange a steady stream of packets, keep-alive packets are automatically generated so the connection doesn’t time out. If no packets are received from either side of the connection for a short amount of time like 5 seconds, the connection times out.
If either side of the connection wishes to cleanly disconnect, a number of connection disconnect packets are fired across redundantly, so that statistically these packets are likely to get through even under packet loss. This ensures that clean disconnects happen quickly, without the other side waiting for time out.
Interactive Connectivity Establishment (ICE)
Interactive Connectivity Establishment (ICE) is a technique used in computer networking to find ways for two computers to talk to each other as directly as possible in peer-to-peer networking.
You want to avoid communicating through a central server (which would slow down communication, and be expensive), but direct communication between client applications on the Internet is very tricky due to network address translators (NATs), firewalls, and other network barriers.
ICE is developed by the Internet Engineering Task Force MMUSIC working group and is published as RFC-8445.
ICE provides a framework with which a communicating peer may discover and communicate its public IP address so that it can be reached by other peers.
- Session Traversal Utilities for NAT (STUN) is a standardized protocol for such address discovery including NAT classification.
- Traversal Using Relays around NAT (TURN) places a third-party server to relay messages between two clients when direct media traffic between peers is not allowed by a firewall.
WebRTC
RTC(real-time communication).
With WebRTC, you can add real-time communication capabilities to your application that works on top of an open standard.
The technologies behind WebRTC are implemented as an open web standard and available as regular JavaScript APIs in all major browsers.
For native clients, like Android and iOS applications, a library is available that provides the same functionality.
WebRTC APIs
The WebRTC standard covers, on a high level, two different technologies: media capture devices and peer-to-peer connectivity.
- Media capture devices
- For cameras and microphones, we use navigator.mediaDevices.getUserMedia() to capture MediaStreams.
- For screen recording, we use navigator.mediaDevices.getDisplayMedia().
- The peer-to-peer connectivity The RTCPeerConnection interface is the central point for establishing and controlling the connection between two peers in WebRTC.
Getting started with media devices
These devices are commonly referred to as Media Devices and can be accessed with JavaScript through the navigator.mediaDevices object, which implements the MediaDevices interface.
Getting started with peer connections
The communication between peers can be video, audio or arbitrary binary data (for clients supporting the RTCDataChannel API).
In order to discover how two peers can connect, both clients need to provide an ICE Server configuration. This is either a STUN or a TURN-server, and their role is to provide ICE candidates to each client which is then transferred to the remote peer. This transferring of ICE candidates is commonly called signaling.
Signaling is needed in order for two peers to share how they should connect. Usually this is solved through a regular HTTP-based Web API (i.e., a REST service or other RPC mechanism) where web applications can relay the necessary information before the peer connection is initiated.
Signaling can be implemented in many different ways, and the WebRTC specification doesn't prefer any specific solution.
Kurento
Kurento is an Open source WebRTC media server.
Kurento as a WebRTC gateway for IP cameras
The WebRTC transport
The WebRTC specification is very clear about what video encoding formats (also known as codecs) are supported, any complying implementation must be able to support both VP8 and H.264 video codecs.(See RFC7742 / 5. Mandatory-to-Implement Video Codec).
The point of WebRTC is to actually be able to serve video in a secure, reliable and effective manner, WebRTC supports a variety of feedback mechanisms that allow any viewer to "inform" about current network conditions to the sender of the video:
By introducing a decoder + encoder pair(transcoding), the gateway is able to solve both the codec compatibility and the network reliability issues.
Using Kurento Media Server
- Acquiring a video stream from a variety of sources, together with the optional transcoding of the media, is performed by the PlayerEndpoint.
- Then everything related to the WebRTC communications is handled by the aptly named WebRtcEndpoint.
The agnosticbin element inside Kurento is again the component in charge of building an encoding tree that covers codec requirements for all consumers, without doing the same work twice:
Installation Guide
If Kurento Media Server, its Application Server, or any of the clients are located behind a NAT, you need to use a STUN or a TURN server in order to achieve NAT traversal. In most cases, STUN is effective in addressing the NAT issue with most consumer network devices (routers). However, it doesn’t work for many corporate networks, so a TURN server becomes necessary.
Apart from that, you need to open all UDP ports in your system configuration, as STUN will use any random port from the whole [0-65535] range.
Setting up an NGINX web server on a Raspberry Pi
NGINX (pronounced engine x) is a popular lightweight web server application you can install on the Raspberry Pi to allow it to serve web pages.
Like Apache, NGINX can serve HTML files over HTTP, and with additional modules can serve dynamic web pages using scripting languages such as PHP.
Install NGINX
sudo apt-get update --fix-missing sudo apt-get install nginx sudo /etc/init.d/nginx start
Test the web server
By default, NGINX puts a test HTML file in the web folder.
/var/www/html/index.nginx-debian.htmlThe configuration file is:
/etc/nginx/nginx.conf
- Configuration File’s Structure
- A simple directive consists of the name and parameters separated by spaces and ends with a semicolon ";"
- A block directive consists of the name, pameraters, and a set of additional instructions surrounded by braces "{" and "}".
- Serving Static Content This will require editing of the configuration file and setting up of a server block inside the http block with location blocks. The URI specified in the request’s header will be compared against the parameters of the location directives defined inside the server block. For ex.,
server {
location / {
root /data/www;
}
location /images/ {
root /data;
}
}
- In response to requests with URIs starting with /images/, the server will send files from the /data/images directory.
- Requests with URIs not starting with /images/ will be mapped onto the /data/www directory.
Setup the Live Streaming Server with Real-Time Messaging Protocol (RTMP)
Nginx RTMP is an Nginx module which allows you to add RTMP and HLS streaming to your media server.
Install the RTMP module so Nginx can handle your media stream:
sudo apt-get install libnginx-mod-rtmp
To set up RTMP support you need to add rtmp{} section to the configuration file "/etc/nginx/nginx.conf":
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
rtmp {
server {
listen 1935;
ping 30s;
notify_method get;
application live {
live on;
# sample HLS
hls on;
hls_fragment 6s;
hls_playlist_length 60s;
hls_path /tmp/hls;
hls_sync 100ms;
}
}
}
http {
sendfile off;
tcp_nopush on;
directio 512;
default_type application/octet-stream;
server {
listen 80;
location /hls {
# Serve HLS fragments
types {
application/vnd.apple.mpegurl m3u8;
video/mp2t ts;
}
root /tmp/;
add_header Cache-Control no-cache;
add_header 'Access-Control-Allow-Origin' '*';
}
location / {
root html;
index index.html index.htm;
}
}
}
We need to also create the directory where our video stream is stored,sudo mkdir -p /tmp/hlsRestart Nginx with its new configuration:
sudo /etc/init.d/nginx start sudo systemctl restart nginxNow that your server is ready to accept your video streams.
Set up your streaming software
sudo apt-get install ffmpeg ffmpeg -f video4linux2 -i /dev/video0 -c:v libx264 -an -f flv rtmp://192.168.4.1/live/camera
- -f fmt (input/output). Force input or output file format. The format is normally auto detected for input files and guessed from the file extension for output files, so this option is not needed in most cases.
- -i url (input). input file url
- -c[:stream_specifier] codec (input/output,per-stream). Select an encoder (when used before an output file) or a decoder (when used before an input file) for one or more streams. codec is the name of a decoder/encoder or a special value "copy" (output only) to indicate that the stream is not to be re-encoded. For ex.,
ffmpeg -i INPUT -map 0 -c:v libx264 -c:a copy OUTPUTencodes all video streams with libx264 and copies all audio streams.
[-][:]:[....]
- 55 55 seconds
- 12:03:45 12 hours, 03 minutes and 45 seconds
[(YYYY-MM-DD|YYYYMMDD)[T|t| ]]((HH:MM:SS[.m...]]])|(HHMMSS[.m...]]]))[Z]
now
If the value is "now" it takes the current time.
Viewing your stream
ffplay rtmp://192.168.4.1/live/cameraor
vlc http://jltw.ddns.net/hls/camera.m3u8For a web browser (or web video player) may support HLS playback natively:
<video src="myHLS.m3u8"></video>
- To let Google Chrome browser to play m3u8 list Download the Native HLS Playback extension.
- Apple Safari and Microsoft Edge browsers This is the only way to use HLS (and adaptive streaming in general) on Mobile Safari, the iOS browser.
Web Players/Javascript Libraries:
- hls.js JavaScript library specifically for playing HLS with the <video> element in browsers that support MediaSource. (No player chrome)
- video.js Free and open source web video player, that supports HLS via plugins.
- Video.js HTTP Streaming
- videojs-contrib-hls (being replaced by HTTP Streaming)
To generate a .m3u8 list:
ffmpeg -i camera.mp4 -g 60 -hls_time 2 out.m3u8
留言