How to use systemd-nspawn to restore a Linux system
A translation of the article was prepared specifically for students of the Linux Administrator course.
We deal with the ability of systemd to run containers to restore the root file system of a damaged system.
As long as GNU / Linux systems exist, system administrators will need to recover from damage to the root file system, accidental configuration changes, or other situations that prevent the system from loading into a “normal” state.
Typically, Linux distributions offer one or more menu options at boot time (for example, in the GRUB menu) that can be used to repair a damaged system; often they boot the system in single-user mode with the shutdown of most system services. In the worst case, the user can change the kernel command line in the bootloader to use the standard shell as the init process (PID 1). This method is the most complex and fraught with difficulties that can lead to loss of time and frustration, while the system needs to be restored.
Most importantly, all these methods assume that the damaged system has some kind of physical console, but in the era of cloud computing this can no longer be relied on. Without a physical console, there are only a few options (if they are still available) to influence the boot process in this way. Even physical machines may turn out to be small embedded devices that do not have an easy-to-use console, and finding the right cables and serial port adapters and setting up a terminal emulator, all for using the console on a serial port in an emergency, is often quite complicated.
When another system is available (of the same architecture and generally similar configuration), a common way to simplify the recovery process is to remove the storage devices from the damaged system and connect them to the working system as secondary devices. On physical systems, this is usually straightforward, and most cloud computing platforms can also support this, since they allow you to mount the root volume of the damaged instance in another instance.
After the root file system is connected to another system, the file system corruption problem is solved using fsck and other tools. Troubleshooting configuration errors, corrupted packages, or other problems can be more difficult because they require you to mount the file system and find and modify the correct configuration files or databases.
Before the advent of systemd, the way to fix configurations in practice was to edit configuration files using a text editor. Finding the necessary files and understanding their contents is a separate task that is beyond the scope of this article.
When the GNU / Linux system uses systemd , many configuration changes are best done using the tools it provides — for example, enabling or disabling services requires creating or deleting symbolic links in various places. The systemctl tool is used to make these changes, but its use requires the systemd instance to work and listen for requests (via D-Bus). When the root file system is mounted as an additional file system on another computer, a working instance of systemd cannot be used to make these changes.
Manual start of the systemd of the target system is also impractical, since it is designed as a PID 1 process to control all other processes, which may conflict with an already running instance in the system used for correction.
Fortunately, systemd has the ability to run containers - fully encapsulated GNU / Linux systems with their own PID 1 and environment, which uses various namespace functionality offered by the Linux kernel. Unlike tools like Docker and Rocket, systemd does not require a container image to run the container; he can run it with root privileges anywhere in the existing file system. This is done using the systemd-nspawn tool , which will create the necessary system namespaces and start the initial process in the container and then provide the console. Unlike chroot , which only changes the visible root of the file system, this type of container will have a separate file system namespace, suitable file systems mounted on / dev , / run and / proc , as well as a separate process namespace and IPC. Visit the main systemd-nspawn resource to learn more about its features.
In this example, a storage device containing the root file system of the damaged system is connected to a running system, where it appears as / dev / vdc . The name of the device will vary depending on the number of existing storage devices, the type of device and the method used to connect it to the system. The root file system can use the entire storage device or be located in a partition inside the device; since the most common (simple) configuration places the root file system in the first partition of the device, / dev / vdc1 will be used in this example. Be sure to replace the device name in the commands below with the correct device name of your system .
A damaged root file system can also be more complex than a separate file system on a device; it can be a volume in LVM or on a set of devices combined in a RAID array. In these cases, you must complete the necessary steps to create and activate a logical device containing a file system before it is available for mounting. Again, these steps are beyond the scope of this article.
First, make sure the systemd-nspawn tool is installed - most GNU / Linux distributions do not install it by default. It is provided by the systemd-container package on most distributions, so use the package manager of your distribution to install it. The instructions in this example were tested using Debian 9, but should work similarly with any modern GNU / Linux distribution.
Using the commands below will almost certainly require root privileges, so you will need to either log in as root, use sudo to get a shell with root privileges, or add the sudo prefix to each command.
First use fsck to check the structures and contents of the target file system:
If he finds any problems with the file system, answer the questions accordingly to fix them. If the file system is seriously damaged, it cannot be repaired, in which case you will have to look for other ways to extract its contents.
Now create a temporary directory and mount the target file system into it:
When the file system is mounted, run the container with it as the root file system:
Command line arguments to start the container:
If all goes well, you should see a output that looks something like this:
In this output, you can see that systemd starts as the init process in the container and determines that it runs inside the container so that it can adjust its behavior accordingly. To bring the container to working condition, various unit files are launched, then the root password of the target system is requested. You can enter the root password here if you want to request a shell with root privileges, or you can press Ctrl + D to continue the startup process, which will display the usual console login prompt.
When you make the necessary changes to the target system, press Ctrl +] three times in a row; this will shut down the container and return you to the original shell. From there, you can perform the cleanup by unmounting the file system of the target system and deleting the temporary directory:
That's all! Now you can remove the storage devices of the target system and return them back.
The idea of using systemd-nspawn in this way, especially the --boot option , came from a question posted on StackExchange. Thanks to Shibumi and kirbyfan64sos for the helpful answers to this question!
We deal with the ability of systemd to run containers to restore the root file system of a damaged system.
As long as GNU / Linux systems exist, system administrators will need to recover from damage to the root file system, accidental configuration changes, or other situations that prevent the system from loading into a “normal” state.
Typically, Linux distributions offer one or more menu options at boot time (for example, in the GRUB menu) that can be used to repair a damaged system; often they boot the system in single-user mode with the shutdown of most system services. In the worst case, the user can change the kernel command line in the bootloader to use the standard shell as the init process (PID 1). This method is the most complex and fraught with difficulties that can lead to loss of time and frustration, while the system needs to be restored.
Most importantly, all these methods assume that the damaged system has some kind of physical console, but in the era of cloud computing this can no longer be relied on. Without a physical console, there are only a few options (if they are still available) to influence the boot process in this way. Even physical machines may turn out to be small embedded devices that do not have an easy-to-use console, and finding the right cables and serial port adapters and setting up a terminal emulator, all for using the console on a serial port in an emergency, is often quite complicated.
When another system is available (of the same architecture and generally similar configuration), a common way to simplify the recovery process is to remove the storage devices from the damaged system and connect them to the working system as secondary devices. On physical systems, this is usually straightforward, and most cloud computing platforms can also support this, since they allow you to mount the root volume of the damaged instance in another instance.
After the root file system is connected to another system, the file system corruption problem is solved using fsck and other tools. Troubleshooting configuration errors, corrupted packages, or other problems can be more difficult because they require you to mount the file system and find and modify the correct configuration files or databases.
Using systemd
Before the advent of systemd, the way to fix configurations in practice was to edit configuration files using a text editor. Finding the necessary files and understanding their contents is a separate task that is beyond the scope of this article.
When the GNU / Linux system uses systemd , many configuration changes are best done using the tools it provides — for example, enabling or disabling services requires creating or deleting symbolic links in various places. The systemctl tool is used to make these changes, but its use requires the systemd instance to work and listen for requests (via D-Bus). When the root file system is mounted as an additional file system on another computer, a working instance of systemd cannot be used to make these changes.
Manual start of the systemd of the target system is also impractical, since it is designed as a PID 1 process to control all other processes, which may conflict with an already running instance in the system used for correction.
Fortunately, systemd has the ability to run containers - fully encapsulated GNU / Linux systems with their own PID 1 and environment, which uses various namespace functionality offered by the Linux kernel. Unlike tools like Docker and Rocket, systemd does not require a container image to run the container; he can run it with root privileges anywhere in the existing file system. This is done using the systemd-nspawn tool , which will create the necessary system namespaces and start the initial process in the container and then provide the console. Unlike chroot , which only changes the visible root of the file system, this type of container will have a separate file system namespace, suitable file systems mounted on / dev , / run and / proc , as well as a separate process namespace and IPC. Visit the main systemd-nspawn resource to learn more about its features.
An example to demonstrate how this works
In this example, a storage device containing the root file system of the damaged system is connected to a running system, where it appears as / dev / vdc . The name of the device will vary depending on the number of existing storage devices, the type of device and the method used to connect it to the system. The root file system can use the entire storage device or be located in a partition inside the device; since the most common (simple) configuration places the root file system in the first partition of the device, / dev / vdc1 will be used in this example. Be sure to replace the device name in the commands below with the correct device name of your system .
A damaged root file system can also be more complex than a separate file system on a device; it can be a volume in LVM or on a set of devices combined in a RAID array. In these cases, you must complete the necessary steps to create and activate a logical device containing a file system before it is available for mounting. Again, these steps are beyond the scope of this article.
Necessary preparations
First, make sure the systemd-nspawn tool is installed - most GNU / Linux distributions do not install it by default. It is provided by the systemd-container package on most distributions, so use the package manager of your distribution to install it. The instructions in this example were tested using Debian 9, but should work similarly with any modern GNU / Linux distribution.
Using the commands below will almost certainly require root privileges, so you will need to either log in as root, use sudo to get a shell with root privileges, or add the sudo prefix to each command.
Check and mount the file system
First use fsck to check the structures and contents of the target file system:
$ fsck /dev/vdc1
If he finds any problems with the file system, answer the questions accordingly to fix them. If the file system is seriously damaged, it cannot be repaired, in which case you will have to look for other ways to extract its contents.
Now create a temporary directory and mount the target file system into it:
$ mkdir /tmp/target-rescue $ mount /dev/vdc1 /tmp/target-rescue
When the file system is mounted, run the container with it as the root file system:
$ systemd-nspawn --directory /tmp/target-rescue --boot -- --unit rescue.target
Command line arguments to start the container:
- --directory / tmp / target-rescue provides the path to the container root file system.
- --boot looks for a suitable initialization program in the root file system of the container and starts it, passing parameters to it from the command line. In this example, the target system also uses systemd as the PID 1 of the process, so the rest of the parameters are for it. If the target system that you are restoring uses some other tool as the PID 1 of the process, you need to configure the settings accordingly.
- - Separates parameters for systemd-nspawn from those intended for PID 1 of the container process.
- --unit rescue.target tells systemd in the container the name of the target that it should try to achieve during the boot process. To simplify recovery operations on the target system, boot it in “recovery” mode, and not in normal multi-user mode.
If all goes well, you should see a output that looks something like this:
Spawning container target-rescue on /tmp/target-rescue. Press ^] three times within 1s to kill container. systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN) Detected virtualization systemd-nspawn. Detected architecture arm. Welcome to Debian GNU/Linux 9 (Stretch)! Set hostname to <test>. Failed to install release agent, ignoring: No such file or directory [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Encrypted Volumes. [ OK ] Created slice System Slice. Mounting POSIX Message Queue File System... [ OK ] Listening on Journal Socket. Starting Set the console keyboard layout... Starting Restore / save the current clock... Starting Journal Service... Starting Remount Root and Kernel File Systems... [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Started Remount Root and Kernel File Systems. Starting Flush Journal to Persistent Storage... [ OK ] Started Restore / save the current clock. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Set the console keyboard layout. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. [ OK ] Reached target System Time Synchronized. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Started Rescue Shell. [ OK ] Reached target Rescue Mode. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. You are in rescue mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to boot into default mode. Give root password for maintenance (or press Control-D to continue):
In this output, you can see that systemd starts as the init process in the container and determines that it runs inside the container so that it can adjust its behavior accordingly. To bring the container to working condition, various unit files are launched, then the root password of the target system is requested. You can enter the root password here if you want to request a shell with root privileges, or you can press Ctrl + D to continue the startup process, which will display the usual console login prompt.
When you make the necessary changes to the target system, press Ctrl +] three times in a row; this will shut down the container and return you to the original shell. From there, you can perform the cleanup by unmounting the file system of the target system and deleting the temporary directory:
$ umount /tmp/target-rescue $ rmdir /tmp/target-rescue
That's all! Now you can remove the storage devices of the target system and return them back.
The idea of using systemd-nspawn in this way, especially the --boot option , came from a question posted on StackExchange. Thanks to Shibumi and kirbyfan64sos for the helpful answers to this question!
More Linux Resources
All Articles