MQTT Protocol: Conceptual Immersion
The Message Queuing Telemetry Transport (MQTT) protocol has been used for many years, but now it is especially relevant due to the explosive growth of IoT: both consumer and industrial devices implement distributed networks and edge computing, and devices with continuous data transmission are becoming part of everyday of life.
This means that lightweight, open, and affordable protocols will become even more important over time. This article provides a conceptual immersion in MQTT: how it works, how it is used now and how it will be used in the future.
MQTT is a publisher-subscriber-based messaging protocol (pub / sub). The initial version in 1999 was published by Andy Stanford-Clark of IBM and Arlene Nipper of Cirrus Link. They viewed MQTT as a way to maintain communication between machines in networks with limited bandwidth or unpredictable communications. One of the first options for its use was to ensure that fragments of the pipeline were in contact with each other and with the central links via satellites.
Given the harsh operating conditions, the protocol is made small and light. It is ideal for low power devices and with limited battery life. These now include the ubiquitous smartphones, and the ever-growing number of sensors and connected devices.
Thus, MQTT has become a protocol for streaming data between devices with limited CPU power and / or battery life, as well as for networks with expensive or low bandwidth, unpredictable stability or high latency. That is why MQTT is known as the ideal vehicle for IoT. It is built on the TCP / IP protocol, but there is an MQTT-SN branch for working via Bluetooth, UDP, ZigBee and other IoT networks other than TCP / IP.
MQTT is not the only real-time pub / sub messaging protocol of its kind, but it has already gained widespread acceptance in various environments that depend on machine-to-machine communications. His peers include Web Application Messaging Protocol , Streaming Text-Oriented Messaging Protocol, and Alternative Message Queueing Protocol .
MQTT is the logical choice for developers who want to create applications with reliable functionality and broad compatibility with devices and applications connected to the Internet, including browsers, smartphones and IoT devices.
The communication system built on MQTT consists of a publisher server, a broker server, and one or more clients. The publisher does not require any settings for the number or location of subscribers who receive messages. In addition, subscribers do not need to tune to a specific publisher. There may be several message brokers in the system.
MQTT provides a way to create a hierarchy of communication channels - a kind of branch with leaves. Whenever the publisher has new data for distribution to customers, the message is accompanied by a note on delivery control. Higher-level clients can receive each message, while lower-level clients can receive messages related to only one or two basic channels, “branches” at the bottom of the hierarchy. This facilitates the exchange of information ranging in size from two bytes to 256 megabytes .
An example of how you can configure the client to connect through the MQTT broker :
Any data published or received by the MQTT broker will be encoded in binary format, since MQTT is a binary protocol. This means that the message must be interpreted in order to receive the original content. Here's what it looks like with Ably and JavaScript:
MQTT brokers can sometimes accumulate messages related to channels that do not have current subscribers. In this case, the messages will either be discarded or saved, depending on the instructions in the control message. This is useful in cases where new subscribers may need the most recently recorded data point, rather than waiting for the next send.
It is noteworthy that MQTT transmits security credentials in clear text, otherwise authentication or security functions are not supported. This is where the SSL framework comes into play , helping to protect transmitted information from being intercepted or otherwise tampered with.
In addition, in MQTT, you can use Ably authentication on tokens if you do not want to disclose your API key to the actual MQTT client at all (in the case of MQTT without SSL, tokens are required to prevent the transfer of API keys in clear text). Token authentication example:
According to IBM , MQTT has the following properties:
A member of the MQTT system can take on the role of publisher, consumer, or both roles at once.
One of the distinguishing characteristics of MQTT is its unique understanding of channels: each of them is treated as a file path, for example:
Channels ensure that every customer receives messages destined for him. By treating channels as file paths, MQTT performs all kinds of useful communication functions, including filtering messages based on where - at what level or in which branch - clients subscribe to the file path.
Look at the two components that make up each MQTT message:
The QoS flag in byte 1 deserves special attention, as it underlies the variable functionality that MQTT supports. QoS flags contain the following values based on the intent and urgency of the message:
Let's look at how to use different QoS levels in IoT devices and other applications.
As IoT applications are now being implemented on a massive scale, MQTT has come to the fore as an open, simple, and scalable way to deploy distributed computing and IoT functionality for a wider user base - both in the consumer and industrial markets.
As stated above, MQTT is a lightweight messaging protocol built for untrusted networks and devices with restrictions on power supply and CPU. However, this does not mean that the connection with potential packet loss is its only application. MQTT provides various levels of service for various types of IoT infrastructure, from repeating data sampling to managing industrial machines:
Developers have a wide selection of protocols for designing and deploying bidirectional IoT communication channels, including MQTT, HTTP, CoAP , WebSockets (if the CPU / battery allows) and others. Whether MQTT is the best choice depends on the hardware and the application task.
Designed for environments with extremely low bandwidth, the MQTT protocol can be quite inflexible in its quest to save every byte. For example, the specification defines only five error messages with which the server can reject the connection (for example, an invalid username / password or an unacceptable version of the protocol). If the server wants to point to some other error, it is out of luck. Even worse, if an error occurs after starting the connection, there is no mechanism for reporting an error at all. The server can only shrug its shoulders and abruptly interrupt the TCP connection, leaving the client without a clue why they dropped it (and without any way to distinguish intentional disconnection from a temporary network problem). For people accustomed to the more flexible and easy to debug (though less economical in terms of bandwidth) pub / sub protocols, such a Spartan approach may seem a little primitive.
MQTT is often referred to along with HTTP, so Google conducted a study comparing them in response time, traffic volume and other attributes important to developers. MQTT took first place in Google tests, but only in conditions when the connection can be reused to send several payloads.
HTTP and MQTT are a good choice for IoT applications due to the relatively small amount of traffic, low battery and memory requirements.
CoAP is another protocol that is often compared to MQTT for developing IoT systems. They are similar, but there are noticeable differences. MQTT is a many-to-many protocol, while CoAP is basically a one-to-one protocol for communication between a server and a client. At the same time, CoAP provides metadata, discovery, and content negotiation features that MQTT does not have .
In cases where clients should only receive data, Server-Sent Events is also a suitable option.
The MQTT repository on GitHub has an extensive list of open source MQTT libraries in various languages. The following are two examples of customization using the open source MQTT broker, JavaScript library, and .NET library.
Eclipse Mosquitto is an open source message broker (EPL / EDL licensed) that implements the MQTT protocols versions 5.0, 3.1.1 and 3.1. Mosquitto is lightweight and suitable for use on all devices: from low-power single-board computers to full-fledged servers.
MQTT.js is a client library for the MQTT protocol, written in JavaScript for Node.js and a browser. Here is an example of sending a message using MQTT.js:
MQTTnet is a high-performance .NET library that provides both the client and the MQTT server (broker).
MQTT Client Installation:
After you configure the MQTT client settings, you can establish a connection. The following code shows how to connect to the server:
Receive incoming messages:
Post Publication:
See the MQTTnet documentation and wiki for more examples .
For enterprise-class providers, there are ready-made MQTT servers for scalable messaging between mobile applications, industrial machines, and a wide range of other IoT uses. This guide tells you how to use MQTT through an enterprise level broker.
When it comes to scaling MQTT, there are two considerations to consider: 1) is this the correct protocol; 2) regardless of the protocol choice, what infrastructure and network capabilities are needed to handle the increased traffic between devices using MQTT.
Lightweight Machine-to-Machine (LWM2M) is another protocol that can be used with MQTT at the enterprise level. Compared to MQTT, it is sometimes better suited for long-term IoT systems . MQTT is ideally suited for a trial run of IoT effortlessly, while the LWM2M provides features for a long-term, versatile infrastructure. LWM2M also provides superior device management tools such as connection monitoring, firmware updates, and remote device actions. For enterprises with a large number of unmanaged devices that send large amounts of data to a central platform, LWM2M is the best choice. However, we are talking about large-scale deployments of IoT, so usually MQTT is more than an adequate option. In addition, MQTT is more common and has wider support.
Now about the possibilities of infrastructure. When it comes to server loading, the number of simultaneous connections is rarely the bottleneck. Most good MQTT servers / brokers support thousands of concurrent connections, but what is the workload required to process and reply to messages after the MQTT server has received the actual data? As a rule, there are all kinds of potential problems, such as reading and writing to and from the database, integration with the server, distribution and management of resources for each client, etc. As soon as one machine ceases to cope with the load, additional servers must be added. that is, think about load balancing, message synchronization between clients connected to different servers, generalized access to the client state regardless of the connection time or the specific server to which the client is connected - the list of products zhaetsya and continues.
Such problems deserve a separate article, and a lot of information can be found in the Engineering section of our blog . In particular, see the article on some of the complexities of servicing a large-scale real-time messaging infrastructure .
In April 2019, OASIS released MQTT v5.0 as an official standard. OASIS is a non-profit consortium of 600 member organizations and 5,000 individual members .
Version 5.0 introduces a number of new features that should be of interest to developers of real-time systems. These new features are backward compatible with current versions of MQTT. Among them:
For a complete list of the new MQTT 5.0 features, see Appendix C to the official standard.
In addition to the many consumer devices and services on the market, MQTT has found use in corporate infrastructure of all shapes and sizes . These are smartphones and tablets, energy monitoring systems, medical devices, oil rigs and rigs, the automotive and aerospace industries, as well as sensors and machine vision systems used in material handling, construction, supply chain, retail, and much more.
MQTT is a popular, widely supported, and relatively mature protocol. It is great for many real-time applications , and not just for deploying IoT. However, as real-time data production and consumption continue to grow exponentially , MQTT may not always be the right protocol to meet your streaming needs. Follow our Realtime Concepts section for information on other protocols and how they suit your situation.
Ably provides a broker and an MQTT protocol adapter with translation to Ably's own protocol in both directions, which allows you to integrate with any existing systems and connections. Supported WebSockets, HTTP, SSE, gRPC (in development), STOMP, AMQP and other protocols for organizing a distributed messaging infrastructure in real time. There are over 40 SDK client libraries and support for proprietary real-time protocols.
This means that lightweight, open, and affordable protocols will become even more important over time. This article provides a conceptual immersion in MQTT: how it works, how it is used now and how it will be used in the future.
Small introduction
MQTT is a publisher-subscriber-based messaging protocol (pub / sub). The initial version in 1999 was published by Andy Stanford-Clark of IBM and Arlene Nipper of Cirrus Link. They viewed MQTT as a way to maintain communication between machines in networks with limited bandwidth or unpredictable communications. One of the first options for its use was to ensure that fragments of the pipeline were in contact with each other and with the central links via satellites.
Given the harsh operating conditions, the protocol is made small and light. It is ideal for low power devices and with limited battery life. These now include the ubiquitous smartphones, and the ever-growing number of sensors and connected devices.
Thus, MQTT has become a protocol for streaming data between devices with limited CPU power and / or battery life, as well as for networks with expensive or low bandwidth, unpredictable stability or high latency. That is why MQTT is known as the ideal vehicle for IoT. It is built on the TCP / IP protocol, but there is an MQTT-SN branch for working via Bluetooth, UDP, ZigBee and other IoT networks other than TCP / IP.
MQTT is not the only real-time pub / sub messaging protocol of its kind, but it has already gained widespread acceptance in various environments that depend on machine-to-machine communications. His peers include Web Application Messaging Protocol , Streaming Text-Oriented Messaging Protocol, and Alternative Message Queueing Protocol .
MQTT is the logical choice for developers who want to create applications with reliable functionality and broad compatibility with devices and applications connected to the Internet, including browsers, smartphones and IoT devices.
How MQTT Works: The Basics
The communication system built on MQTT consists of a publisher server, a broker server, and one or more clients. The publisher does not require any settings for the number or location of subscribers who receive messages. In addition, subscribers do not need to tune to a specific publisher. There may be several message brokers in the system.
MQTT provides a way to create a hierarchy of communication channels - a kind of branch with leaves. Whenever the publisher has new data for distribution to customers, the message is accompanied by a note on delivery control. Higher-level clients can receive each message, while lower-level clients can receive messages related to only one or two basic channels, “branches” at the bottom of the hierarchy. This facilitates the exchange of information ranging in size from two bytes to 256 megabytes .
An example of how you can configure the client to connect through the MQTT broker :
var options = { keepalive: 60, username: 'FIRST_HALF_OF_API_KEY', password: 'SECOND_HALF_OF_API_KEY', port: 8883 }; var client = mqtt.connect('mqtts:mqtt.ably.io', options);
Any data published or received by the MQTT broker will be encoded in binary format, since MQTT is a binary protocol. This means that the message must be interpreted in order to receive the original content. Here's what it looks like with Ably and JavaScript:
var ably = new Ably.Realtime('REPLACE_WITH_YOUR_API_KEY'); var decoder = new TextDecoder(); var channel = ably.channels.get('input'); channel.subscribe(function(message) { var command = decoder.decode(message.data); });
MQTT brokers can sometimes accumulate messages related to channels that do not have current subscribers. In this case, the messages will either be discarded or saved, depending on the instructions in the control message. This is useful in cases where new subscribers may need the most recently recorded data point, rather than waiting for the next send.
It is noteworthy that MQTT transmits security credentials in clear text, otherwise authentication or security functions are not supported. This is where the SSL framework comes into play , helping to protect transmitted information from being intercepted or otherwise tampered with.
In addition, in MQTT, you can use Ably authentication on tokens if you do not want to disclose your API key to the actual MQTT client at all (in the case of MQTT without SSL, tokens are required to prevent the transfer of API keys in clear text). Token authentication example:
var options = { keepalive: 60, username: INSERT_TOKEN_HERE, password: '', port: 8883 }; var client = mqtt.connect('mqtts:mqtt.ably.io', options); client.subscribe("[mqtt]tokenevents", { /* Create a new token called 'NEW_TOKEN' */ client.end(); options.username = NEW_TOKEN; client = mqtt.connect('mqtts:mqtt.ably.io', options); });
MQTT functionality: deeper immersion
According to IBM , MQTT has the following properties:
- Neutral to message content
- Ideal for one-to-many distributed communications and disconnected applications
- Equipped with the LWT function (Last Will and Testament, "last will and testament") to notify the parties of an abnormal disconnection of the client
- Relies on TCP / IP for basic communication tasks
- Designed to deliver messages using the templates "at most once", "at least once" and "exactly once"
A member of the MQTT system can take on the role of publisher, consumer, or both roles at once.
One of the distinguishing characteristics of MQTT is its unique understanding of channels: each of them is treated as a file path, for example:
channel = "user/path/channel"
Channels ensure that every customer receives messages destined for him. By treating channels as file paths, MQTT performs all kinds of useful communication functions, including filtering messages based on where - at what level or in which branch - clients subscribe to the file path.
MQTT message format
Look at the two components that make up each MQTT message:
- Byte 1 : contains the type of message (client’s connection request, subscription confirmation, ping request, etc.), a duplication flag, instructions for saving messages, and information about the quality of service (QoS).
- Byte 2 : contains information about the remaining message length, including the payload and any data in the header of an optional variable.
The QoS flag in byte 1 deserves special attention, as it underlies the variable functionality that MQTT supports. QoS flags contain the following values based on the intent and urgency of the message:
- 0 = no more than once: the server is triggered and forgets. Messages may be lost or duplicated.
- 1 = at least once: the recipient confirms the delivery. Messages can be duplicated, but delivery is guaranteed
- 2 = exactly once: the server provides delivery. Messages arrive exactly once without loss or duplication
Let's look at how to use different QoS levels in IoT devices and other applications.
Where can I use MQTT?
As IoT applications are now being implemented on a massive scale, MQTT has come to the fore as an open, simple, and scalable way to deploy distributed computing and IoT functionality for a wider user base - both in the consumer and industrial markets.
As stated above, MQTT is a lightweight messaging protocol built for untrusted networks and devices with restrictions on power supply and CPU. However, this does not mean that the connection with potential packet loss is its only application. MQTT provides various levels of service for various types of IoT infrastructure, from repeating data sampling to managing industrial machines:
- Environmental Sensor Data : As already mentioned, MQTT supports the “no more than once” message delivery model. In networks with partial coverage or high latency, this means that information may be lost or duplicated . In areas where remote sensors record and transmit data at specified intervals, this is not a problem, as new readings are received on a regular basis. Sensors in remote environments are typically low-power devices, making MQTT the ideal solution for IoT sensors with a relatively low data transfer priority.
- Machine performance data : to quickly respond to problems and prevent downtime. For example, a wind power installation needs guaranteed delivery of current performance indicators to local teams even before this information reaches the data center. In such situations, the delivery of messages “at least once” ensures that the appropriate specialists are promptly noticed by the necessary specialists, even if they arrive as duplicates. This is important for higher priority machine communications.
- Billing systems : there are even more priority and accurate messages that need to be processed correctly. In business situations where duplication of records is unacceptable, including in billing systems, the “exactly once” QoS flag is useful. This eliminates the duplication or loss of packages in billing or billing systems, reduces the number of anomalies and unnecessary contradictions as agreed.
When not to use MQTT?
Developers have a wide selection of protocols for designing and deploying bidirectional IoT communication channels, including MQTT, HTTP, CoAP , WebSockets (if the CPU / battery allows) and others. Whether MQTT is the best choice depends on the hardware and the application task.
Designed for environments with extremely low bandwidth, the MQTT protocol can be quite inflexible in its quest to save every byte. For example, the specification defines only five error messages with which the server can reject the connection (for example, an invalid username / password or an unacceptable version of the protocol). If the server wants to point to some other error, it is out of luck. Even worse, if an error occurs after starting the connection, there is no mechanism for reporting an error at all. The server can only shrug its shoulders and abruptly interrupt the TCP connection, leaving the client without a clue why they dropped it (and without any way to distinguish intentional disconnection from a temporary network problem). For people accustomed to the more flexible and easy to debug (though less economical in terms of bandwidth) pub / sub protocols, such a Spartan approach may seem a little primitive.
MQTT is often referred to along with HTTP, so Google conducted a study comparing them in response time, traffic volume and other attributes important to developers. MQTT took first place in Google tests, but only in conditions when the connection can be reused to send several payloads.
HTTP and MQTT are a good choice for IoT applications due to the relatively small amount of traffic, low battery and memory requirements.
CoAP is another protocol that is often compared to MQTT for developing IoT systems. They are similar, but there are noticeable differences. MQTT is a many-to-many protocol, while CoAP is basically a one-to-one protocol for communication between a server and a client. At the same time, CoAP provides metadata, discovery, and content negotiation features that MQTT does not have .
In cases where clients should only receive data, Server-Sent Events is also a suitable option.
How to quickly set up MQTT
The MQTT repository on GitHub has an extensive list of open source MQTT libraries in various languages. The following are two examples of customization using the open source MQTT broker, JavaScript library, and .NET library.
Eclipse Mosquitto - open source MQTT broker
Eclipse Mosquitto is an open source message broker (EPL / EDL licensed) that implements the MQTT protocols versions 5.0, 3.1.1 and 3.1. Mosquitto is lightweight and suitable for use on all devices: from low-power single-board computers to full-fledged servers.
MQTT.js
MQTT.js is a client library for the MQTT protocol, written in JavaScript for Node.js and a browser. Here is an example of sending a message using MQTT.js:
var mqtt = require('mqtt') var client = mqtt.connect('mqtt://test.mosquitto.org') client.on('connect', function () { client.subscribe('presence', function (err) { if (!err) { client.publish('presence', 'Hello mqtt') } }) }) client.on('message', function (topic, message) { // message is Buffer console.log(message.toString()) client.end() })
MQTTnet
MQTTnet is a high-performance .NET library that provides both the client and the MQTT server (broker).
MQTT Client Installation:
// Create a new MQTT client. var factory = new MqttFactory(); var mqttClient = factory.CreateMqttClient();
After you configure the MQTT client settings, you can establish a connection. The following code shows how to connect to the server:
// Use WebSocket connection. var options = new MqttClientOptionsBuilder() .WithWebSocketServer("broker.hivemq.com:8000/mqtt") .Build(); await client.ConnectAsync(options);
Receive incoming messages:
client.UseApplicationMessageReceivedHandler(e => { Console.WriteLine("### RECEIVED APPLICATION MESSAGE ###"); Console.WriteLine($"+ Topic = {e.ApplicationMessage.Topic}"); Console.WriteLine($"+ Payload = {Encoding.UTF8.GetString(e.ApplicationMessage.Payload)}"); Console.WriteLine($"+ QoS = {e.ApplicationMessage.QualityOfServiceLevel}"); Console.WriteLine($"+ Retain = {e.ApplicationMessage.Retain}"); Console.WriteLine(); Task.Run(() => client.PublishAsync("hello/world")); });
Post Publication:
var message = new MqttApplicationMessageBuilder() .WithTopic("MyTopic") .WithPayload("Hello World") .WithExactlyOnceQoS() .WithRetainFlag() .Build(); await client.PublishAsync(message);
See the MQTTnet documentation and wiki for more examples .
For enterprise-class providers, there are ready-made MQTT servers for scalable messaging between mobile applications, industrial machines, and a wide range of other IoT uses. This guide tells you how to use MQTT through an enterprise level broker.
What about scaling?
When it comes to scaling MQTT, there are two considerations to consider: 1) is this the correct protocol; 2) regardless of the protocol choice, what infrastructure and network capabilities are needed to handle the increased traffic between devices using MQTT.
Lightweight Machine-to-Machine (LWM2M) is another protocol that can be used with MQTT at the enterprise level. Compared to MQTT, it is sometimes better suited for long-term IoT systems . MQTT is ideally suited for a trial run of IoT effortlessly, while the LWM2M provides features for a long-term, versatile infrastructure. LWM2M also provides superior device management tools such as connection monitoring, firmware updates, and remote device actions. For enterprises with a large number of unmanaged devices that send large amounts of data to a central platform, LWM2M is the best choice. However, we are talking about large-scale deployments of IoT, so usually MQTT is more than an adequate option. In addition, MQTT is more common and has wider support.
Now about the possibilities of infrastructure. When it comes to server loading, the number of simultaneous connections is rarely the bottleneck. Most good MQTT servers / brokers support thousands of concurrent connections, but what is the workload required to process and reply to messages after the MQTT server has received the actual data? As a rule, there are all kinds of potential problems, such as reading and writing to and from the database, integration with the server, distribution and management of resources for each client, etc. As soon as one machine ceases to cope with the load, additional servers must be added. that is, think about load balancing, message synchronization between clients connected to different servers, generalized access to the client state regardless of the connection time or the specific server to which the client is connected - the list of products zhaetsya and continues.
Such problems deserve a separate article, and a lot of information can be found in the Engineering section of our blog . In particular, see the article on some of the complexities of servicing a large-scale real-time messaging infrastructure .
What is the current situation with MQTT?
In April 2019, OASIS released MQTT v5.0 as an official standard. OASIS is a non-profit consortium of 600 member organizations and 5,000 individual members .
Version 5.0 introduces a number of new features that should be of interest to developers of real-time systems. These new features are backward compatible with current versions of MQTT. Among them:
- Error reporting has been improved : return codes can now inform that data is not being transmitted for some reason. Optional strings are supported to indicate the reason. They help improve troubleshooting diagnostics.
- Sharing Subscriptions : To help balance the load, subscriptions can be shared among multiple clients on the receiving side.
- Message Properties : Version 5.0 introduces metadata as part of the message header. It may convey additional information to the end user or facilitate some of the other functions listed below.
- Channel Alias : Publishers can replace channels with numeric identifiers to reduce the number of bytes to transmit.
- Message expiration date: messages can be marked for automatic deletion if the system is not able to deliver them within a specified period of time.
For a complete list of the new MQTT 5.0 features, see Appendix C to the official standard.
In addition to the many consumer devices and services on the market, MQTT has found use in corporate infrastructure of all shapes and sizes . These are smartphones and tablets, energy monitoring systems, medical devices, oil rigs and rigs, the automotive and aerospace industries, as well as sensors and machine vision systems used in material handling, construction, supply chain, retail, and much more.
MQTT and Ably
MQTT is a popular, widely supported, and relatively mature protocol. It is great for many real-time applications , and not just for deploying IoT. However, as real-time data production and consumption continue to grow exponentially , MQTT may not always be the right protocol to meet your streaming needs. Follow our Realtime Concepts section for information on other protocols and how they suit your situation.
Ably provides a broker and an MQTT protocol adapter with translation to Ably's own protocol in both directions, which allows you to integrate with any existing systems and connections. Supported WebSockets, HTTP, SSE, gRPC (in development), STOMP, AMQP and other protocols for organizing a distributed messaging infrastructure in real time. There are over 40 SDK client libraries and support for proprietary real-time protocols.
All Articles