Node.js developer tools. Mqtt protocol for working with web sockets
Web socket technology allows sending messages from a server to a client in a web application or in a mobile application, which cannot be done using the REST-API. To work with web sockets, they often use the socket.io library, or developers work with native browser web socket objects. In this post, I will try to show that both ways do not solve all the problems, and it is much better to use specialized servers, for example, the mqtt server (earlier it was called the mqtt broker) for working with web sockets.
In fairness, and to avoid unnecessary disputes, I note that in addition to the mqtt server, a number of other servers can be used, for example rabbitmq.
Developing applications using web sockets seems very simple, until we come across a reality in which connection breaks often occur. The first task that we have to solve is to track connection breaks and restore them. The situation is complicated by the fact that during the disconnection and reconnection, the client continues to send new messages, as well as new messages can be sent to the client that are more likely to be lost.
It is necessary at the application level to monitor the receipt of messages, and to implement their repeated delivery. I would like to pay particular attention to the phrase “at the application level” (but I would like it to be at the protocol level).
As soon as we added logic for tracking message delivery, all messages began to arrive, but it immediately became clear that there were duplicate messages, since the message could have been received, and confirmation of this fact was lost due to a disconnected connection. And you need to double the amount of program code to exclude duplicate messages.
Along with the complication of the program code, its effectiveness also decreases, for which they often criticize the socket.io library. It, of course, is less effective than working with native web sockets, in particular, due to the presence of reconnection logic and message delivery confirmation (I immediately notice that re-delivery logic is not implemented in socket.io).
A more reliable and efficient way would be to bring this logic to the protocol level. And such a protocol exists - it is mqtt. The first version of the mqtt protocol was developed by Andy Stanford-Clark (IBM) and Arlene Nipper (Arcom) in 1999. The MQTT 3.1.1 specification was standardized by the OASIS consortium in 2014.
The mqtt protocol has a “quality of service” (qos) parameter that can take values:
0 - message is delivered if possible;
1 - the message is delivered guaranteed, but there may be duplicates;
2 - the message is delivered guaranteed and guaranteed once.
That is, the mqtt protocol solves the problem with guaranteed message delivery, and this issue is being removed from the agenda. But not only this question.
Performance and scaling.
When working with web sockets, all connected clients leave open connections to the server, even if there is no real messaging. This load is inherently different from the load in the REST-API, which is determined by the flow of requests. The burden of open connections on web sockets is difficult to emulate during the testing phase. Therefore, an erroneous assumption is often made about the sufficient performance of the application by the number of messages sent and received, without taking into account the burden of maintaining a large number of open connections with clients.
If we transfer all work with web sockets to a specialized mqtt server, then our nodejs application opens only one connection on a web socket (or tcp, because mqtt supports both protocols) with an mqtt server, and we can scale our application by connecting multiple nodejs instances to the mqtt server.
If the resources of one mqtt server are exhausted, you can organize a cluster of mqtt servers without affecting applications on nodejs.
Now let's move on to the example.
The mqtt server or broker, as it was called in the previous specifications, works according to the model of sending messages / subscribing to messages. Each message is sent to the topic. The recipient subscribes to the topic of messages. Both the sender and receiver have two identifiers: clientId (device identifier) and userName (username).
The device identifier is important, as it is associated with the subscription and messages will be sent to it. The username, unlike the device identifier, does not play a decisive role in message delivery, and is used to differentiate access to topics.
To work with the mqtt protocol on the client side, the github.com/eclipse/paho.mqtt.javascript library is used. There are several server implementations, including free ones. In this example, we will use the emqx server, which runs through docker-compose (see github.com/apapacy/tut-mqtt ).
For testing, create a document in which we will set clientId, userName and message text:
Sending messages is implemented in the message.js file:
To check, open the index.html file in the browser, set clientId, userName, message text, and send several messages (you can read them in the console, as the client sends messages to the test topic and is subscribed to this topic itself).
Now open another browser or another browser tab, and join with another (this is important) clientId. Send a few more messages from the first browser, and make sure that they come to both clients, since they have different clientId and they are both subscribed to the topic test.
Now close the second browser (or second browser tab), and send a few more messages. After that, reopen the second browser, and join with the same clientId. Make sure in the console logs that you have received all messages that were sent during the period when the second browser (second tab) was closed. This happened because:
Sample code can be downloaded from the repository .
apapacy@gmail.com
September 29, 2019
In fairness, and to avoid unnecessary disputes, I note that in addition to the mqtt server, a number of other servers can be used, for example rabbitmq.
Developing applications using web sockets seems very simple, until we come across a reality in which connection breaks often occur. The first task that we have to solve is to track connection breaks and restore them. The situation is complicated by the fact that during the disconnection and reconnection, the client continues to send new messages, as well as new messages can be sent to the client that are more likely to be lost.
It is necessary at the application level to monitor the receipt of messages, and to implement their repeated delivery. I would like to pay particular attention to the phrase “at the application level” (but I would like it to be at the protocol level).
As soon as we added logic for tracking message delivery, all messages began to arrive, but it immediately became clear that there were duplicate messages, since the message could have been received, and confirmation of this fact was lost due to a disconnected connection. And you need to double the amount of program code to exclude duplicate messages.
Along with the complication of the program code, its effectiveness also decreases, for which they often criticize the socket.io library. It, of course, is less effective than working with native web sockets, in particular, due to the presence of reconnection logic and message delivery confirmation (I immediately notice that re-delivery logic is not implemented in socket.io).
A more reliable and efficient way would be to bring this logic to the protocol level. And such a protocol exists - it is mqtt. The first version of the mqtt protocol was developed by Andy Stanford-Clark (IBM) and Arlene Nipper (Arcom) in 1999. The MQTT 3.1.1 specification was standardized by the OASIS consortium in 2014.
The mqtt protocol has a “quality of service” (qos) parameter that can take values:
0 - message is delivered if possible;
1 - the message is delivered guaranteed, but there may be duplicates;
2 - the message is delivered guaranteed and guaranteed once.
That is, the mqtt protocol solves the problem with guaranteed message delivery, and this issue is being removed from the agenda. But not only this question.
Performance and scaling.
When working with web sockets, all connected clients leave open connections to the server, even if there is no real messaging. This load is inherently different from the load in the REST-API, which is determined by the flow of requests. The burden of open connections on web sockets is difficult to emulate during the testing phase. Therefore, an erroneous assumption is often made about the sufficient performance of the application by the number of messages sent and received, without taking into account the burden of maintaining a large number of open connections with clients.
If we transfer all work with web sockets to a specialized mqtt server, then our nodejs application opens only one connection on a web socket (or tcp, because mqtt supports both protocols) with an mqtt server, and we can scale our application by connecting multiple nodejs instances to the mqtt server.
If the resources of one mqtt server are exhausted, you can organize a cluster of mqtt servers without affecting applications on nodejs.
Now let's move on to the example.
The mqtt server or broker, as it was called in the previous specifications, works according to the model of sending messages / subscribing to messages. Each message is sent to the topic. The recipient subscribes to the topic of messages. Both the sender and receiver have two identifiers: clientId (device identifier) and userName (username).
The device identifier is important, as it is associated with the subscription and messages will be sent to it. The username, unlike the device identifier, does not play a decisive role in message delivery, and is used to differentiate access to topics.
To work with the mqtt protocol on the client side, the github.com/eclipse/paho.mqtt.javascript library is used. There are several server implementations, including free ones. In this example, we will use the emqx server, which runs through docker-compose (see github.com/apapacy/tut-mqtt ).
For testing, create a document in which we will set clientId, userName and message text:
<script src="/paho-mqtt.js"></script> <script src="/messages.js"></script> <form name="sender" onsubmit="return false"> <input type="text" name="user"> <input type="text" name="client"> <input type="text" name="message"> <input type="button" onclick="connect()" value="connect"> <input type="button" onclick="send()" value="send"> </form>
Sending messages is implemented in the message.js file:
var client; var connectOptions = { timeout: 30, reconnect: true, cleanSession: false, mqttVersion: 4, keepAliveInterval: 10, onSuccess: onConnect, onFailure: onFailure } function connect() { try { client = new Paho.Client('localhost', 8083, '/mqtt', document.forms.sender.client.value); connectOptions.userName = document.forms.sender.user.value; client.connect(connectOptions); } catch (ex) { console.log(ex); } } function onConnect() { console.log('on connect'); client.onMessageArrived = function(message) { console.log("onMessageArrived: " + message.payloadString); } client.subscribe("test", { qos: 2 }); } function onFailure(err) { console.log('on failure', JSON.stringify(err)); } function send() { var message = new Paho.Message(document.forms.sender.message.value); message.destinationName = "test"; message.qos = 2; client.send(message); }
To check, open the index.html file in the browser, set clientId, userName, message text, and send several messages (you can read them in the console, as the client sends messages to the test topic and is subscribed to this topic itself).
Now open another browser or another browser tab, and join with another (this is important) clientId. Send a few more messages from the first browser, and make sure that they come to both clients, since they have different clientId and they are both subscribed to the topic test.
Now close the second browser (or second browser tab), and send a few more messages. After that, reopen the second browser, and join with the same clientId. Make sure in the console logs that you have received all messages that were sent during the period when the second browser (second tab) was closed. This happened because:
- When sending a message, the quality level qos = 2 was set;
- You have previously joined the same topic with the same clientId by setting qos = 2;
- The connection options are set to cleanSession: false.
Sample code can be downloaded from the repository .
apapacy@gmail.com
September 29, 2019
All Articles