Wiren Board manufactures Linux controllers for industrial IoT. Devices collect data from oil wells and bank branches, monitor the microclimate in server and supermarkets. The controllers integrate with the top-level systems of the company's partners. Systems authenticate devices - they understand that they are talking with their sensor, and not with someone else's, and then they authorize. At this stage, the problem arises - there are thousands of controllers, hundreds of customers, but there is no single integration system. Simple traditional methods, such as login / password pairs, are vulnerable to attacks and inconvenient to deploy.
Therefore, the company has developed authentication in top-level systems using hardware keys - based on standard asymmetric cryptography using a hardware protected element for storing keys. Now a unified integration system is not needed — authentication and authorization are protected and work out of the box. Evgeny Boger will tell you how to do this: how they chose the “crypto chip”, how they screwed it to hardware and Linux, how common libraries and software were made to be friends with it. Particular emphasis on deployment: the introduction of device initialization in production, the introduction of support for various top-level software, including in someone else's and closed.
About the speaker: Eugene Boger ( evgeny_boger ) - CTO and co-founder of Wiren Board. He is engaged in embedded systems and, especially, embedded Linux.
Problems
To begin with, what we are doing and where did this problem come from. We at Wiren Board design and manufacture equipment in Russia. It used to be called M2M, but now it’s industrial IoT. This is the automation of building engineering systems, monitoring and scheduling. Briefly, all the work looks like this: sensors of various parameters, actuators, counters and controllers (edge-computing or IoT-gateway) collect different data from objects, process them, execute local logic, and then gather them in one large dispatch, monitoring or control system .
We do not have a whole ecosystem, unlike some competitors. We manufacture equipment that integrates with several top-level systems of our partners. There are many partner companies and they share responsibility. Without good technical means, integration will not work - it just cannot be negotiated.
There are two simple solutions to solving these problems. The first is to give the username / password to the client , as everyone does, and the second is to generate and sew up the "secret" in the workplace . Both options did not suit us - I will tell you why.
Simple solutions
The first solution is to issue a username and password to the client . All of us do so until recently.
To authenticate a device that sends data to some system, you can make a secret key - conditionally login / password ("secret"). It will be common on controllers and on a top-level system that collects data from several controllers.
A couple of username / password (common "secret") needs to be somehow given to the client - the company or the person. Someone must generate a secret pair, send by e-mail, authenticate the client by account number. This is standard procedure - low technology.
Problem . It is that we have many such systems. Our client, and he can send data to our partner’s system. This is a complex interaction between all involved parties.
In addition to the problem of many systems, there are others.
- Bad delivery and delivery to the customer .
- Logins and passwords are stored on the server . If we also store hashes, this will protect us a little from leaks. But even so, an unpleasant feeling arises when secret keys for all client controllers are stored on the server. Some of them can deal with critical tasks: outdoor lighting, monitoring oil rigs.
- Synchronization between services .
- Loss recovery . It is not clear what to do in case of loss, when the client erased the controller’s memory - in what memory should I write? You have to repeat it all over again.
- Protection against copying details . There are paid monitoring systems that provide the customer with a service and charge subscription fees. I don’t want the end client to be able to somehow bypass the system through us - pay once, but use two.
The second solution is to generate and suture the “secret” in production . This is an improvement on the previous solution.
The scheme is this: we, as a manufacturer of controllers, pre-generate usernames and passwords for everyone, sew them into our system and enter them into equipment. Logins and passwords from the equipment can neither be read nor changed. This is better than the previous option, because it does not require interaction between people.
The problems . All the problems remain, except the first, but the main one is the synchronization between services and the intranet . There are a lot of services and it is not clear how to synchronize them - because of this, we were not able to implement the second solution. We have clients who use equipment in their closed networks. We released a new controller, sold it to a client, and its system is closed. It is set up, it works once, and it’s difficult to convey the “secrets” further. Report in batches? Everything is complicated in organizations, although technically simple.
Both solutions did not suit us. Therefore, we decided to take a different path. But before that, they decided to outline common tasks and goals.
Tasks and Goals
First common tasks.
Authentication This is a way to understand who is talking to the top-level system, who exactly connects to the dispatch system.
Authentication is not granting or delimiting access rights, but a way of understanding who is talking to us.
The task of sending data . Our controllers are Linux computers designed for a special task. We need them to send data to top-level systems, connect via VPN. At the same time, we want the dispatch to work out of the box - without the settings and interaction of our customers and end users of the system with us and with customers.
Other tasks . This is a reliable connection, data channel encryption, but a separate issue is authorization . The authorization task is associated with external services and is divided into three parts.
- Free manufacturer service . Provide access by device serial number.
- White lists of serial numbers for the service of our partners - link purchases and accesses to the client’s account.
- Licenses Allow or deny access based on the options specified in the certificate.
Goals are what we want to achieve when we solve problems.
Issuance and delivery to the customer . Without the participation of people - information is sewn up by robots in production.
Loss recovery . We want there to be no loss of secret details at all.
Delivery from production to services . We want to do without it, so that you do not need to deliver anything to the services. When launching new equipment, we do not want to update the databases of all services that should authenticate these devices.
Storage on the server . It is advisable not to store anything there at all.
Synchronization between services and intranet . It is also advisable not to synchronize anything - because we will not store anything.
Protection against copying details . We want something secret, for which the money is taken, it was impossible to copy and receive for free.
Digital signature rushes to the rescue
Electronic digital signature (EDS) is a technology around which everything works for us.
It’s like an ordinary signature, only digital. EDS is easy to verify, but hard to fake. The familiar truths of cryptography, which are decades old.
An electronic signature is something that can be counted by a message if you know the secret private key (private key). If you know the public key, it is easy to verify that the electronic signature for the message is correct. The name is clear - it’s customary to tell the public to everyone, and the secret is only to the one who signs.
All signatures and keys are just numbers.
In our case, it is 32 bytes of data, which works on mathematical “magic”. Mathematics ensures that the signature is easy to verify, but hard to fake.
We use the signature ECDSA-256 + SHA-256:
-
e = HASH(m)
- the cryptographic hash function irreversibly converts the message m to the number e;
-
private key (dA)
- random number;
-
public key (QA)
- generated from a private key, but not vice versa;
-
signature (r,s) = sign(private key, e)
- signature;
-
verify(public key, signature, e)
- verification of the signature.
EDS authentication. First try
What can be done for our task using this tricky mechanism, which works in one direction simply and difficult in the other?
Issuance and delivery to the customer . We generate a random private key for each device in production. We don’t tell anyone, because we don’t even know him, and we write to the device.
Delivery from production to services . Next, we use only the public key of this device for authentication on services. On services, we only store a list of public keys instead of passwords.
Standard health check algorithm:
- the service sends a random message
m
to the controller;
- controller:
sign(private key, m)
;
- the controller sends a signature to the service;
- service:
verify(public key, signature, m)
.
The only thing that we decided in this way is that we no longer store common "secrets" on our services in open or cached form. This is not what we want.
EDS authentication. Second attempt
We do not want to store something on services. To achieve this, we can force our devices to send their public keys to the service.
At the last stage, we solved two problems. The first - we checked that they gave the key to the service . We have a public key, which means we also made a private key. The second - we made sure that the device owns a private key , which lies somewhere on the USB flash drive. If the device could sign something, then it has a private key.
Now the device will also send the public key to the service. How to check that no one intercepted him, did not fake him, and that everything works?
Checking the public key . Create another public key for ourselves. He will be our key, as a manufacturer. This is the root key “root private key + public key”. With this root secret key in production, we will sign the device public key and we will store this signature on the device. The device must send its public key and signature of its public key to the service. Now the service can check the public key of the device. If it is signed by the root key (root private key), then we issued this key.
Only the manufacturer - we can create and store a signature on the device, but check everything.We publish the public key on the site in the "Contact Us" section. Anyone can take it and check the public key of the device that sent the device to the service. Then you can check that the device itself has its own private key.
The general algorithm looks like this.
-
(once) random root private key
;
-
factory: random device private key
;
-
factory: sign(root private key, device public key) = signature_1
;
-
device->service: device public key + signature_1
;
-
service: verify(root public key, signature_1, device public key)?
Second attempt result
We solved the problem with delivery to the client - the information is sewn up at the production site and nothing needs to be restored .
It is important that we solved the problem of delivering “secrets” to top-level services , because all that needs to be stored on the service is the manufacturer’s public key. The whole key is 33 bytes. With their help and mathematical magic, you can continue to make a handshake connection and verify that the device has the corresponding private key.
On the server we only store the manufacturer key (root public key).
We do not have synchronization between services and the intranet , which we have already talked about. Also, we do not have protection against copying details .
The only thing we forgot about is authentication . The device sent a private key, and we checked that we did it and issued it, and checked that the device owns it. But we do not know what this device is, and we produce thousands of them.
Therefore, we applied a trick called “Certificate”.
Authentication and Certificates
At this step, in all the mathematical magic with signatures and their checks, we add additional information - a certificate . To do this, we sign at the factory not only the public key (device public key), but the key with additional information.
Additional information in our case.
- Date of manufacture and manufacturer.
- Model and hardware configuration.
- The serial number by which the device can be authenticated.
- Options: hardware and software. Different configurations may not physically differ from each other, but the certificate will contain information about what the client paid for.
- Customer name and account number.
We will sign all this information together with the public key with our producer key - root public key. After that, the information will go to the services and they will be able to make sure that it is correct. Since these are our and our partners' services, they trust us.
Goal Status
Information is also stitched at the factory, and delivery to services is not needed. On the server we store only the manufacturer’s key.
Loss recovery . We sew all the information from the certificates into the flash memory of the device. Theoretically, it can be accidentally or intentionally deleted, but there is nothing secret in this information in the certificate. Even the signature itself is not secret - there is a public key and a signature with our key. The only secret in the certificate is the sales volumes of devices with different options.
The certificate can be stored in the factory and sent to the client if he lost it. Clients rarely specifically erase the service area of memory. Usually we do this during the device recovery procedure: the device arrived from the client, it is completely passed through initialization, everything is erased, it is downloaded again, and the certificate is copied from the factory database.
We do not have loss recovery , copy protection and synchronization between services .
At the authentication stage, we receive and verify the certificate. We understand what kind of device it is - we know the manufacturer, model and serial number, what he can and cannot.
Login
The certificate allows you to store information for authorization.
Free manufacturer service . Knowing the serial number of the device, you can give access to everyone. In our services we give access to all our base customers.
White lists of serial numbers . For the service of our partners, you can create a table with a white list of serial numbers: “Client Vasily bought from us two controllers with such serial numbers that are tied to his account”
Licenses You can sell something in advance, and then allow or deny access based on the options specified in the certificate - a controller with a license for system X.
There is no common base between services, manufacturer or manufacturer of systems. Everything works exclusively on information from the controller that is signed by us, as a manufacturer, when it authenticates in the system.
Intermediate Certificate
Another technical problem that we solved on the road. In the scheme that I just talked about, there is a root certificate of the manufacturer - root private key. It is physically needed every time you create a device. But if there are many devices, then this key needs constant access for a limited circle of people. This is bad, because if you lose it, you will have to update the public keys on all services, and it should not get to the attackers. These are big organizational problems. But there is a solution.
We introduce intermediate keys to a batch of devices that are not so scary to lose.
We did the same, only the chain is longer.
With a manufacturer’s certificate, we sign the intermediate key. Physically, this is a "flash drive", which is given to the foreman at the factory for a day. Hardware limits the number of devices that a key can sign. In the middle of the diagram, we added an intermediate certificate, otherwise nothing changed.
Secure keystore
In all this, we do not have enough protection for the private key of the device - it is still a file that lies on a USB flash drive. An attacker can copy it, but most likely they will lose it or accidentally open access.
In the ideal case, it would be nice to protect the device’s private key from copying - put it in a black box.
The black box performs 4 operations:
- internally generates a key upon request, but does not give;
- gives the public key;
- Signs a message
- verifies the signature.
To verify the signature, you only need a public key, so three operations are enough.
In my understanding, this should be a hardware solution, preferably separate from the processor. There are several options, the best of which is a special crypto processor inside the SoC or as a separate chip.
The first black box option that we reviewed is the CAAM module in the NXP i.mx 6, 7, 8 processors that we use. The problem is that it is implemented programmatically in the processor’s Boot ROM.
It may contain bugs that can be found and even exploited through other processor functionality. A couple of years ago, a hole was found in this module that made it possible to bypass the signature verification when loading the firmware. This is not the functionality that we need, but the sediment remains. Another problem is that it is difficult to import processors with this module into Russia; they require filling out paperwork.
Therefore, we took a separate chip. Frankly, I counted on the fact that if we can not bring it to Russia, we’ll come up with something - the chip is small, it costs $ 1. But everything turned out well - they found the Microchip ATECC chip, which already has all the papers.
Microchip ATECC608A
This is a separate small chip that costs a penny. The chip is connected via I2C - two "legs" of the processor, which you can also share with other peripherals. The chip has a standard pinout. We used the chip in the first versions of the equipment and simply soldered it on top of another chip with the same protocol and pinout, because it is standard.
The chip can do what we need from such a chip: read signatures, store keys, and much more.
Characteristics:
- 16 key slots;
- can read ECSDSA signatures, hashes, MACs, and encrypt AES, can DH;
- has a random number generator and cryptographic counters;
- Enclosures: SOIC-8, DFN6;
- protocols: I2C, single wire;
- ~0.7$@1000pcs.
How to work with a microcircuit
There is decent documentation for it , but under the NDA. If you write immediately to gamma.spb.ru, then they will give it to you in 2 weeks. If in another company - after 3 months. We wrote to two companies, and when we had done everything, another Microchip dealer answered us.
There are few appnotes and they are worse than average. There is software on GitHub - a library with HAL. It's funny - the documentation is under the NDA, and the software that is written on it is on GitHub. The software does not support Linux, but supports the Raspberry Pi and Atmel MK - this is a bit different. The developers believe that on all equipment there is only one I2C bus, for example, the legs are called as on the Raspberry Pi.
There is integration with OpenSSL - it does not work well, but it does. There are no examples under Linux and no work with personalization .
Chip customization
Personalization is the biggest headache with the chip.
The problem is that the chip can do a lot of things. It has 16 slots in which 16 keys are stored: user data, or public keys, or temporary storage for other slots - there are many options.
You need to somehow restrict access to the slots, and there are also many configuration options: restrict by password, by authentication in another slot, by the time of access to the factories.
In the table, the type of key, read and write access, the relationship between the slots - SlotConfig, KeyConfig.
In the bit mask (16 bits) of each key that we use, there are different numbers everywhere.
The saddest thing is that the configuration zone is one-time, which sets the functions of the slots. We messed up 50 chips before doing everything right. The chip only works after locking the configuration . Separately, there is a lock for individual slots
There is no documentation in the examples or in the software. There is documentation for individual bits, but everything is complicated there. In all examples from Microchip it is written: “Download such a block, and it will somehow work for you as in the example of sending data to Amazon.”
It took a lot of time, but in the process they made a cool utility.
Atecc-util utility
This is a console utility that can perform most of the functions of the chip and allows you to work a little easier. It is on GitHub under a MIT license.
The utility uses CryptoAuthLib. She knows how to work more friendly with the config zone, she knows how to work with SHA, MAC, ECDSA, DH. The interface is batch friendly, because we created the utility for use in scripts in the first place. The fact that a person can cause it is a side feature. The utility can make a list - a plan of commands: "First, personalize this zone, then write down such a key."
An example of a utility call is quite human-readable.
atecc - b 10 - c 'serial' - c 'read-config /tmp/config.dump'
The utility is built under Linux, under AMD64 - it is in the Debian package.
Other personalization tools
We have an Excel plate to read the bits. If you show us an NDA scan with Microchip, we will give it to you.
We covered everything with tests, since there are many options when you can forget one bit and some service command will read your private key. Tests test the real device. They turn to the microcircuit and check the correct configuration on the device: can this slot be read, can such a signature be made?
In parallel with the bits, we created a list of guarantees that this device should satisfy, and checked how everything works. We use the bats framework - a very interesting thing. It looks like this.
List of tests for an example. The upper ones are passed, but the lower ones are not.
Settings in devices
For ourselves, we use only two slots for the task that I am talking about. In both we store the private key of the device. The difference is that the former is associated with a permanent certificate , which was issued in 1970 for 200 years.
This is due to the fact that in IoT time is not always synchronized. Certificate infrastructure involves verifying the validity period of a certificate. If the synchronization on the devices is broken, then some vital service may fail, for example, VPN.
Therefore, one slot is infinite - permanent . It is generated once and does not change throughout the life of the device. A certificate for 200 years is generated for this key - for closed networks.
Another slot is being updated. The maximum certificate lifetime is one year. This is done just in case something is compromised. A private renewable device key is generated as the validity period of the device certificate expires. Used for authentication in open networks, updated once a month or less, along with a certificate.
For users, we generated various combinations , including several slots for private ECDSA keys. Users can generate their key in a separate slot if they do not trust our private key. For this you need to trust only Microchip. Users can read signatures, do encryption - we gave everything that the chip can do.
So far, unfortunately, no one has used, but we hope so.
Infrastructure: Intermediate Keys
I already said that at some point we implemented intermediate certificates so as not to shine with a root certificate that should not be lost. He never appears in a factory.
Physically intermediate certificates are an ATECC508A chip. It differs slightly from 608, but in 508 there is functionality that came in handy for keys, but in 608 it is no longer there.
The chip is connected via a USB-I2C adapter. This is USBISP with tiny-usb-i2c firmware - a programmer that can be upgraded to a USB-I2C bridge. Intermediate certificates sign device certificates with their private key.
Two features of the microcircuit turned out to be useful to us.
Hardware password protection slot . The chip can be programmed to read the signature only if two conditions are met:
- when the device is stuck in a computer;
- password entered.
We give the production foreman an intermediate key and a password for a certain number of controllers. Accordingly, you need to steal both the key and the password in order to gain access. We got this opportunity for free, but it improves the security of the system.
Hardware limit on the number of uses . The cryptographic counter inside can only increase. When it reaches a predetermined limit once, the microcircuit does not sign anything else.
OpenSSL on the client
Let's consider how everything works on the client. We have OpenSSL on the controller. We did not invent anything - this is ordinary TLS, ordinary PKI. We additionally needed a client library. The vast majority of Linux software uses it for a secure connection.
We took the code from Microchip, added it a little bit, supported the fresh OpenSSL
1.1. As a result, he knows how to work with a hardware key - hardware supports passwords for private keys.
It looks something like this.
openssl req -new -engine ateccx08 -keyform engine -key ATECCx08:00:04:C0:00 -subj "/CN=wirenboard-AP6V5MDG" -out device AP6V5MDG.csr
This is a call to regular OpenSSL and an instruction to use the appropriate engine module. The key is set here: address, model, and the last two bytes is the number of the slot that is used. Everything is transmitted as if it is a key file, but it is not a file - you need to go into the device.
SSL on the server
Any SSL works on the server, including OpenSSL. No modifications and custom builds on the server side are needed. All that is needed on the server is to be able to check the chain of certificate bundle (device cert + intermediate cert), and store our public key , which we published on the site - Wiren Board ROOT CA.
Standard TLS says that both parties must authenticate each other. — — . — handshake.
: . , . letsencrypt SSL, , .
, — MQTT.
MQTT: mosquitto
IBM. .
Mosquitto — , , Linux. , OpenSSL engine ( ) «keyfile», . , 20 .
bundle.
.
mosquitto_sub -h mqtt.wirenboard.com -p 8884 -cert /etc/ssl/device/device_bundle.crt.pem --key 'engine:ateccx08:ATECCx08:00:04:C0:00' --capath /etc/ssl/certs/ -t /# -v
. —
-cert
. bundle- — .
--key
. .
,
--capath
, . SSL-, letsencrypt.
.
root@wirenboard-AXXVJI62:~# cat /etc/mosquitto/conf.d/bridge-hw.conf connection wb_devices_cloud.wirenboard-AXXVJI62 address contactless.ru:8884 bridge_capath /etc/ssl/certs/ bridge_certfile /etc/ssl/device/device_bundle.crt.pem bridge_keyfile engine:ateccx08:ATECCx08:00:04:C0:00 notifications true notification_topic /client/wirenboard-AXXVJI62/bridge_status topic/# both 1 ""/dient/wirenboard-AXXVJI62
Mosquito- .
Mosquitto — .
per _listener_settings true listener 8884 0.0.0.0 cafile/etc/mosquitto/certs/WirenBoard_Root_CA.crt certfile /etc/letsencrypt/live/contactless.ru/fullchain.pem keyfile/etc/letsencrypt/live/contactless.ru/privkey.pem require.certificate true use_identity_as_username true password_file /etc/mosquitto/passwd.conf allow_anonymous false acl_file /etc/mosquitto/ad.conf :~$ cat /etc/mosquitto/acl.conf pattern write /client/%u/# pattern read /client/%u/#
— .
- Root CA letsencrypt- — . .
- Mosquitto.
username
MQTT.
- , , , (CN) wirenboard-AXXVJI62, , .
-
per_listener_settings:
, / (>1.5.5).
MQTT- Wiren Board IoT Cloud Platform.
OpenVPN
OpenVPN , , . , .
OpenVPN , . , : bundle, , engine.
openvpn --capath /etc/ssl/certs/ --cert /etc/ssl/device/device_bundle.crt.pem --key engine:ateccx08:ATECCx08:00:04:C0:00
letsencrypt.
ca /etc/openvpn/WirenBoard_Root_CA.crt cert /etc/letsencrypt/live/vpn1.wirenboard.com/fullchain.pem key /etc/letsencrypt/live/vpn1.wirenboard.com/privkey.pem
— . - .
Nginx
. Nginx , , , SSL. nginx web-, reverse-proxy. — nginx.
nginx , HTTP-, . , : Common Name, , . , 400.
ssl_client_certificate WirenBoard_Root_CA.crt; ssl_verify_client on;
nginx . — , HTTP. Linux- nginx , SSL, , OpenSSL.
wget , bash , HTTP- TLS . 10 .
server { listen 8080; location / { proxy_pass https://example.com; proxy_ssl_name example.com; proxy_ssl_server_name on; proxy_ssl_certificate/etc/ssl/device/device_bundle.crt.pem; proxy_ssl_certificate_key engine:ateccx08;ATECCx08:00:04:C0:00; } }
Wiren Board 6 , . , .
web- cloud.wirenboard.com OpenVPN . Grafana InfluxDB, MQTT. saymon.info — (MQTT) .
, - , , Grafana, MQTT-, , , . — .
, , : — OpenSSL , — . !
InoThings Conf 2019 . YouTube- 2019 . Telegram. , , IoT.