MQTT: an open network protocol and its importance in the IoT

MQTT network protocol IoT

Remember the name MQTT, since it is a network communication protocol type M2M (Machine to Machine) that will sound quite a lot. It is becoming quite popular thanks to the new era of the Internet of Things (IoT). Also, it’s an open protocol, which gives a lot of advantages.

In fact, it has become one of the central pillars of the IoT, as it is quite good in devices with some transmission limitations like these. The acronym MQTT comes from Message Queuing Telemetry Transport, an open standard from OASIS and ISO (ISO/IEC 20922) for network communications and that is usually executed over the famous TCP/IP.

Network protocols

OSI model and its layers

The communication protocols are rules that make it possible for two or more devices or systems to communicate with each other. That is to say, it is a protocol to transmit information through diverse means and with a defined format, whether they are implemented by software or by hardware (or both).

The standard string of the protocol defines a multitude of communication features. It can range from synchronization rules, semantics, syntax, packet format, etc. And the truth is that they are not something negligible, since thanks to these protocols today we can use Internet and other communication networks …

And of course, there is not just one protocol, but many. For example, the famous ones DNS, FTP, MQTT, HTTP and HTTPS, IMAP, LDAP, NTP, DHCP, SSH, Telnet, SNMP, SMTP, etc., for the application layer. While in the transport layer there are some famous ones such as TCP, UDP, etc., as well as those of the Internet layer such as IPv4, or IPv6 (the one that has made possible the largest amount of available IPs and the arrival of the IoT), IPSec, etc., and others of the link layer such as DSL, Ethernet, WiFi, ARP, etc.

Over the IoT protocols

MQTT protocol

Of course, there are specific communication protocols or protocols that can be applied to the IoT. That is, given the previous section, they would be a series of rules defined so that two or more IoT devices can communicate and understand each other, and they are usually M2M, that is, machine-to-machine communication. Thanks to them, there can be many IoT devices connected and sharing information from sensors or other sources.

Due to the large number of IoT devices, these protocols must meet requirements beyond the limitations of bandwidth, speed, etc. (bear in mind that many devices are embedded and cheap), which are usually found in some devices. And I am referring to the fact that ‘string’ must be scalable, in order to be able to add more connected devices if necessary and without affecting the global system.

In addition, they must have a low dependency of coupling between devices, so that no problems are generated if a device is removed. And of course, at the same time, a high interoperability is sought to work with a large number of devices and very varied systems, as the world of IoT is quite heterogeneous.

Other useful features would be the ease of implementation, security, etc. Keep in mind that the IoT is generating great challenges in the safety aspect. Even more so when many of the connected devices are often critical in certain cases…for example, children’s toys.

Important concepts

Having said that, it must be said that the solutions for the IoT use a centralized server to receive the messages from all the connected devices that emit and distribute them to all the connected IoT devices that are listening in. That server is what is known as router or broker. Something that differs from the conventional client-server relationship in some ways.

On the other hand, the methodologies you can find in these communication protocols for IoT are:

  • PubSub: Publish/Subscribe is a messaging pattern where one device (Sub) informs the broker that it wants to receive a message, while another device (Pub) publishes messages for the broker to distribute to the other device/s waiting for them.
  • rRPC: Router Remoder Procedure Calls is another remote process execution pattern. In it, a device (Callee) informs the broker that it will perform this procedure and the broker distributes it to another device (Caller) where this process is executed.

However, to carry out these methodologies or patterns, a messaging infrastructure is needed. And in this sense two can be distinguished:

  • Message Queue: a messaging service where a single message queue is generated for all customers who initiate a subscription in the broker. The latter will keep the messages stored until they are delivered to the customer. If the customer or recipient is not connected, it is maintained until they are connected. These types of services are like those used in instant messaging apps such as Telegra, Whatsapp, Messenger, etc.
  • Message Service: This is another service in which the broker sends messages to the connected recipient client, filtering by the type of message. If the client or receiving device is disconnected, then the messages are lost (although it may have some logging system).

IoT Protocols

In view of the above, we are now going to look a little more closely at the most well-known IoT protocols. Among the most prominent of M2M are:

  • AMQP (Advanced Message Queuing Protocol): This is a PubSub type of Message Queue protocol. Designed to have a good interoperability and ensure reliability. Specially for corporate applications, high performance, high latency networks, critical, etc.
  • WAMP (Web Application Messaging Protocol): is another open protocol of type PubSub as rRPC, and runs over WebSockets.
  • CoAP (Constrained Application Protocol): is a protocol designed especially for low capacity applications.
  • TOMP (Streaming Text Oriented Messaging Protocol): very simple protocol to achieve maximum interoperability. HTTP is used to transmit text messages.
  • XMPP (eXtensible Messaging and Presence Protocol): another protocol used in IoT for instant messaging apps and based on XML. It is also open in this case.
  • WMQ (WebSphere Message Queue): protocol developed by IBM. It is Message Queue type, as its name indicates, and message oriented.
  • MQTT: (see next section)

All about MQTT

MQTT package

The MQTT protocol is a Message Queue communication protocol, which follows a PubSub pattern, and of the M2M type as I mentioned before. It is widely used in the IoT, and it is based on the TCP/IP stack used in the Internet.

In the case of MQTT, each connection is kept open and is reused in each necessary communication. Something different from what happens in other known protocols, that each communication needs a new connection.

Advantages

The advantages of the MQTT protocol are quite evident when it comes to M2M communications for IoT. In addition to all the above, it is a protocol that provides

  • Scalability, to connect more and more customers.
  • Decoupling between clients, for less dependence.
  • Asynchronism.
  • Simplicity.
  • Lightness to not consume too many resources (although with security TLS/SSL goes up).
  • Energy efficiency for devices that depend on battery or work 24/7, does not need a great bandwidth (ideal for slow connections, like some wireless).
  • Security and quality, for greater reliability and robustness in communications.

History

MQTT was created in the 90s, with a first version of the protocol in 1999. It was created by Dr. Andy Stanford-Clark of IBM and Arlen Nipper of Cirrus Link (formerly Eurotech).

The initial idea was to create a protocol to monitor a pipeline that travelled through the desert, with a communication protocol that was efficient (low bandwidth consumption), lightweight, and low power consumption. At that time it was very expensive, but now it has been transformed into a cheap and open protocol.

The initial protocol was improved with the appearance of new versions such as MQTT v3.1 (2013) under the OASIS specification (Organization for the Advancement of Structured Information Standards), etc. You should know that at the beginning it was a proprietary protocol of IBM, but that it would be released in 2010, and it ended up becoming standard in OASIS…

MQTT connection operation

The MQTT protocol uses a filter for the messages that are sent to each client, based on topics that are organized in a hierarchical way. That way, a client can post a message in a specific topic. In this way, all those clients or connected devices that subscribe to the topic will receive messages through the broker.

As it is MQ, the messages will be kept in the queue and are not lost until the client has received the message.

The connections, as I also indicated, are made through TCP/IP, and the server or broker will keep a record of the connected clients. By default, the devices will use communication port number 1883, although you can also find a port 8883 if you are using SSL/TLS for more security.

For the connection to be possible, not only clients, server and ports are needed. Also other packets or sent messages are needed for the communication to take place:

  • Establish connection: CONNECT message/package sent by the customer with all the necessary information. This information includes the customer’s ID, username, password, etc. The broker or server responds with a CONNACK package that will inform the customer that the connection was accepted, rejected, etc.
  • Sending and receiving messages: once the connection is established, PUBLISH packages or messages are used with the topic and payload of the message sent to the broker. On the other hand, the client or interested clients use SUBSCRIBE and UNSUBSCRIBE packages to subscribe or unsubscribe respectively. The broker will also respond with a SUBACK and UNSUBACK package respectively to report the success of the operation demanded by the customer.
  • Connection maintenance: to guarantee that the connection remains open, clients can periodically send a PINGREQ package that will be matched with a PINGRESP package from the server.
  • Ending the connection: when a client disconnects, it sends a DISCONNECT packet to report that event.

Those messages or packets I have talked about have a structure just like other packets of other network protocols:

  • Header: it is a fixed part that occupies between 2-5 bytes. It contains a control code, ID of the type of message sent, and its length. Between 1-4 bytes are used for length encoding, using the first 7 bits of each octet as length data and an additional continuity bit to determine that there is more than one byte making up the message length.
  • Variable header: it is not always mandatory, but optional. It is only contained in some packets in certain situations or specific messages.
  • Content or data: the packet data is what actually contains the message to be sent. It can be from a few kB to 256 MB in size.

If you are interested in knowing the corresponding code in hexadecimal for the types of messages sent are

Message Code
CONNECT 0x10
CONNACK 0x20
PUBLISH 0x30
PUBACK 0x40
PUBREC 0x50
PUBREL 0x60
PUBCOMP 0x70
SUSBSCRBE 0x80
SUBACK 0x90
UNSUSCRIBE 0xA0
UNSUBACK 0xB0
PINGREQ 0xC=
PINGRESP 0xD0
DISCONNECT 0xE0

Communication quality and security

Another important detail of the messages by MQTT is the quality of service or QoS and safety. The robustness of the communication system in case of failure and its security will depend on it.

As for their quality, you can determine 3 different levels:

  • QoS 0 (unacknowledge): the message is only sent once, and in case of failure it would not be delivered. It is used when it is not critical.
  • QoS 1 (acknowledge): the message will be sent as many times as needed until delivery to the customer is guaranteed. The bad thing is that the customer could receive the same message several times.
  • QoS 2 (assured): similar to the previous one, but it guarantees that it will only be delivered once. It is usually used for more critical systems where more reliability is needed.

On the other hand, as far as the safety of MQTT, several measures can be used to guarantee its strength in this sense. As I have commented before, the user and password authentication, like many other protocols, can be secured by SSL/TLS. Although many IoT devices with low capabilities, or low resources, might have problems with overload when using this kind of secure communication…

For that reason, many IoT devices that use MQTT use passwords and users in ‘plain text’, which could make it very easy for someone to sniff out network traffic. And if that is not enough, the broker can also be configured to accept anonymous connections, which would allow any user to establish communications, implying more risk.

Using MQTT with Arduino

Arduino UNO with MQTT

Of course, you can use the MQTT protocol with Arduino and other development boards, as well as with the Rapsberry Pi, etc. To do this, you must provide your Arduino board with connectivity, if it does not have it. In addition, the Arduino Client for MQTT library will help you with these tasks. This library is compatible with:

  • Arduino YUN
  • Arduino WiFi (shield)
  • Arduino Ethernet (shield)
  • ESP8266 Module
  • Intel Galileo/Edison
  • Rapsberry Pi

You know you can download and install the library in your Arduino IDE using the command: git clone https://github.com/knolleary/pubsubclient.git

As for the code to use MQTT in some application, the truth is that it is simple. In Fritzing’s image you can see an Arduino UNO board to which connectivity has been added by Arduino Ethernet and also connected a DHT22 temperature and humidity sensor, although it could have been anything else…

Well, that said, for the code you have to generate in Arduino IDE to work with the MQTT protocol in Arduino, it’s that simple:

  • To send MQTT messages

#{i1}includes{i0}
#{i1}include{i0} {i1}Ethernet-{i0}
#{i1}include <PubSubClient.{i0}
#{i1}include{i0} {i1}DHT-{i0}

#define DHTPIN 2
#define DHTTYPE DHT22

// Ethernet adapter MAC address
byte mac[] = { 0xCE, 0xAB, 0x0E, 0x3F, 0xFE, 0xD4 };

// Server IP (broker)
IPAddress mqtt_server(192, 168, 1, 4);

// Topic or theme you work with
const char* topicName = "test";

DHT dht(DHTPIN, DHTTYPE);
EthernetClient ethClient;
PubSubClient(ethClient);

void setup()
{
  Serial.begin(9600);
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Ethernet failure using DHCP");
  }
// Port 1883 of communication
  client.setServer(mqtt_server, 1883);
  dht.begin();
}

void loop()
{
  if (!client.connected()) {
    Serial.print("Connecting ...n");
    client.connect("Arduino client");
  }
  else {
    // Sending of temperature and humidity sensor information
    float temp = dht.readTemperature();
    char buffer [10];
    dtostrf(temp,0, 0, buffer);
    client.publish(topicName, buffer);
  }
  // Time between shipments in ms (every 10 seconds)
  delay(10000);
}

  • To receive messages by MQTT you only need the Arduino UNO board and connection, with Arduino Ethernet or any other element. As for the code, an example would be

#{i1}includes{i0}
#{i1}include{i0} {i1}Ethernet-{i0}
#{i1}include <PubSubClient.{i0}

// Ethernet adapter MAC address
byte mac[] = { 0xCE, 0xAB, 0x0E, 0x3F, 0xFE, 0xD4 };

// Server IP (broker)
IPAddress mqtt_server(192, 168, 1, 4);

// Topic to work with
const char* topicName = "test";

EthernetClient ethClient;
PubSubClient(ethClient);

void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("The message has arrived [");
  Serial.print(topic);
  Serial.print(") ");
  int i=0;
  for (i=0;i<length;i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}

void setup()
{
  Serial.begin(9600);
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Ethernet failure when using DHCP configuration");
  }
  client.setServer(mqtt_server, 1883);
  client.setCallback(callback)
}

void loop()
{
  if (!client.connected()) {
      Serial.print("Connecting ...");
      if (client.connect("rece_arduino")) {
        Serial.println("connected");
        client.subscribe(topicName);
      } else {
        delay(10000);
      }
  }
  // Listening customer
  client.loop();
}

Remember to change the IP to the appropriate server IP, and also alter the MAC address of your Ethernet network adapter or the one you are using, as well as the rest of the code if you want to adapt it to a different project. This is just an example! [/highlighted]

For more information, you can download our free PDF manual with the Arduino IDE course to start programming.

Leave a Comment

*

*