Linux descriptor file with examples
Once, in an interview, they asked me what you will do if you find a service that’s not working because the disk has run out of space?
Of course, I replied that I would see what this place was occupied and, if possible, I would clean the place.
Then the interviewer asked, what if there is no free space on the section, but you also don’t see the files that would occupy the whole place?
To this, I said that you can always look at open file descriptors, for example with the lsof command and understand which application took up all the available space, and then you can proceed according to the circumstances, depending on whether the data is needed.
The interviewer interrupted me at the last word, adding my question: “Suppose we don’t need the data, it’s just a debug log, but the application doesn’t work because it cannot record debug”?
“Okay,” I replied, “we can turn off debug in the application config and restart it.”
The interviewer objected: “No, we cannot restart the application, important data is still stored in our memory, and important clients are connected to the service itself, which we cannot force to reconnect”.
“Well,” I said, “if we cannot restart the application and the data is not important to us, then we can simply clear this open file through the file descriptor, even if we do not see it in the ls command on the file system.”
The interviewer was pleased, but I was not.
Then I thought, why does not the person testing my knowledge dig deeper? But what if the data is still important? What if we can not restart the process, and at the same time this process writes to the file system in a section on which there is no free space? What if we cannot lose not only already recorded data, but also the data that this process writes or tries to record?
At the beginning of my career, I tried to create a small application in which it was necessary to store information about users. And then I thought, how can I map the user to his data. For example, I have Ivan Ivanov Ivanich, and he has some data, but how to make friends with them? I can directly indicate that a dog named "Tuzik" belongs to this very Ivan. But what if he changes his name and instead of Ivan becomes, for example, Olya? Then it turns out that our Olya Ivanovna Ivanova will no longer have a dog, and our Tuzik will still belong to the non-existent Ivan. This database was helped by a database that gave each user a unique identifier (ID), and my Tuzik was attached to this ID, which, in fact, was just an ordinal number. Thus, the owner of the tuzik was with ID number 2, and at some point in time Ivan was under this ID, and then Olya became the same ID. The problem of humanity and animal husbandry has been practically solved.
The problem of the file and the program working with this file is about the same as that of our dog and person. Suppose I opened a file under the name ivan.txt and started writing the word tuzik into it, but I managed to write only the first letter “t” to the file, and this file was renamed by someone, for example, olya.txt. But the file remained the same, and I still want to write my ace in it. Every time I open a file with the open system call in any programming language, I get a unique ID that points me to a file, this ID is a file descriptor. And it doesn’t matter at all what and who does this file next, they can delete it, they can rename it, they can change the owner or take away the rights to read and write, I will still have access to it, because at the time of opening the file I had rights to read and / or write it and I managed to start working with him, which means I must continue to do this.
On Linux, the libc library opens for each running application (process) 3 descriptor file, with the numbers 0,1,2. You can find more information at the links man stdio and man stdout
If in your program you open a file for reading or writing, then most likely you will get the first free ID and this will be number 3.
A list of descriptor files can be viewed from any process if you know its PID.
For example, open a console with bash and see the PID of our process
In the second console, run
You can safely ignore the file descriptor number 255 within the framework of this article; it was opened for its needs by bash itself, and not by a linked library.
Now all 3 descriptor files are associated with the pseudo-terminal device / dev / pts , but we can still manipulate them, for example, run in the second console
And in the first console we will see
You can easily override these 3 descriptor files in any process, including in bash, for example, through a pipe connecting two processes, see
You can run this command yourself with strace -f and see what happens inside, but I will tell you briefly.
Our parent process bash with PID 15771 parses our command and understands exactly how many commands we want to run, in our case there are two of them: cat and sleep. Bash knows that he needs to create two child processes, and combine them with one pipe. Total bash will need 2 child processes and one pipe.
Before creating child processes, bash starts the pipe system call and receives new file descriptors for the temporary pipe buffer, but this buffer does not yet bind our two child processes.
For the parent process, it looks like pipe already exists, and there are no child processes yet:
Then, using a system call, clone bash creates two child processes, and our three processes will look like this:
Do not forget that clone clones the process along with all file descriptors, so they will be the same in the parent process and in the child ones. The task of the parent process with PID 15771 is to monitor child processes, so it just waits for a response from the child processes.
Therefore, he does not need pipe, and he closes the file descriptors with numbers 3 and 4.
In the first bash child process with PID 9004, the dup2 system call, changes our STDOUT file descriptor with number 1 to the file descriptor pointing to pipe, in our case it is number 3. Thus, everything that the first child process with PID 9004 will write to STDOUT will automatically fall into the pipe buffer.
In the second child process with PID 9005, bash changes the descriptor STDIN with number 0 using dup2. Now everything that our second bash with PID 9005 will read will read from pipe.
After that, the descriptors with numbers 3 and 4 are also closed in child processes, since they are no longer used.
I deliberately ignore the file descriptor 255, it uses bash for internal needs and will also be closed in child processes.
Further, in the first child process with PID 9004, bash launches the executable file that we specified on the command line with the system call exec , in our case it is / usr / bin / cat.
In the second child process with PID 9005, bash starts the second executable file that we specified, in our case it is / usr / bin / sleep.
The exec system call does not close the file descriptors unless they were opened with the O_CLOEXEC flag during the open call. In our case, after running the executable files, all current file descriptors will be saved.
Check in the console:
As you can see, the unique number of our pipe is the same in both processes. So we have a connection between two different processes with one parent.
For those who are not familiar with the system calls that bash uses, I highly recommend running the commands through strace and see what happens inside, for example, like this:
Let's get back to our problem with running out of disk space and trying to save data without restarting the process. Let's write a small program that will write to the disk about 1 megabyte per second. Moreover, if for some reason we could not write data to disk, we will simply ignore it and try to write data again after a second. In the example I use Python, you can use any other programming language.
Run the program and look at the file descriptors
As you can see, we have our 3 standard file descriptors and another one that we opened. Check the file size:
data is written, try changing the file permissions:
We see that the data is still being written, although our user does not have the right to write to the file. Let's try to remove it:
Where is the data written? And are they written at all? We check:
Yes, our descriptor file still exists, and we can work with this descriptor file as with our old file, we can read, clean and copy it.
We look at the file size:
File size 19923457. Trying to clear the file:
As you can see, the file size only increases and our trankate did not work. Refer to the open system call documentation. If we use the O_APPEND flag when opening a file, then each time the operating system checks the file size and writes data to the very end of the file, it does this atomically. This allows multiple threads or processes to write to the same file. But in our code we do not use this flag. We can see a different file size in lsof after trankate only if we open the file for overwriting, which means that instead of in our code
we have to put
Checking with the “w” flag
and with the “a” flag
Often, programmers use debuggers (for example GDB) or various levels of logging in the application when creating and testing programs. Linux provides the ability to actually write and change an already running program, for example, changing the values of variables, setting breakpoint, etc., etc.
Returning to the original question with not enough disk space to write the file, we will try to emulate the problem.
Create a file for our partition, which we will mount as a separate disk:
Create a file system:
Mount the file system:
Create a directory with our owner:
We open the file only for writing in our program:
We launch
Waiting a few seconds
So, we got the problem described at the beginning of this article. Free space 0 occupied 100%.
We remember that according to the conditions of the task, we are trying to record very important data that cannot be lost. And at the same time we need to repair the service without restarting the process.
Suppose we still have disk space, but in a different partition, for example, in / home.
Let's try to "reprogram on the fly" our code.
We look at the PID of our process, which ate all the disk space:
We connect to the process via gdb
We look at the open file descriptors:
We look at the information about the file descriptor with number 3, which interests us
Remembering what kind of system call Python makes (see above, where we ran strace and found the open call), while processing our code to open the file, we do the same on our own behalf, but we need the O_WRONLY | O_CREAT | O_TRUNC bits replace with a numerical value. To do this, open the kernel source, for example, here and see what flags are responsible for what
#define O_WRONLY 00000001
#define O_CREAT 00000100
#define O_TRUNC 00001000
We combine all the values into one, we get 00001101
Run our call from gdb
So we got a new descriptor file with number 4 and a new open file on another section, check:
We remember the pipe example - how bash changes file descriptors, and we already learned the dup2 system call.
We try to replace one file descriptor with another
We check:
We close the descriptor 4 file, since we do not need it:
And exit gdb
Check the new file:
As you can see, the data is written to a new file, check the old one:
Data is not lost, the application works, logs are written to a new place.
Imagine that the data is important to us, but we do not have disk space in any of the sections and we cannot connect the disk.
What we can do is redirect our data somewhere, for example to pipe, and the data from pipe, in turn, redirect to the network through some program, for example netcat.
We can create a named pipe with the mkfifo command. It will create a pseudo file on the file system, even if there is no free space on it.
We restart the application, and check:
There is no disk space, but we successfully create a named pipe there:
Now we need to somehow wrap all the data that gets into this pipe to another server through the network, for this all the same netcat will do.
On the remote-server.example.com server, run
On our problem server, run in a separate terminal
Now all the data that gets into the pipe will automatically go to stdin in netcat, which will send it to the network on port 7777.
All we have to do is start writing our data into this named pipe.
We already have a running application:
Of all the flags, we only need O_WRONLY since the file already exists and we don’t need to clear it
Checking the remote server remote-server.example.com
Data goes, we check a problem server
Data saved, problem resolved.
I take this opportunity to convey my regards to my colleagues at Degiro.
Listen to Radio T podcasts.
Good to all.
As a homework, I suggest thinking about what will be in the file descriptors of the cat and sleep process if you run this command:
Of course, I replied that I would see what this place was occupied and, if possible, I would clean the place.
Then the interviewer asked, what if there is no free space on the section, but you also don’t see the files that would occupy the whole place?
To this, I said that you can always look at open file descriptors, for example with the lsof command and understand which application took up all the available space, and then you can proceed according to the circumstances, depending on whether the data is needed.
The interviewer interrupted me at the last word, adding my question: “Suppose we don’t need the data, it’s just a debug log, but the application doesn’t work because it cannot record debug”?
“Okay,” I replied, “we can turn off debug in the application config and restart it.”
The interviewer objected: “No, we cannot restart the application, important data is still stored in our memory, and important clients are connected to the service itself, which we cannot force to reconnect”.
“Well,” I said, “if we cannot restart the application and the data is not important to us, then we can simply clear this open file through the file descriptor, even if we do not see it in the ls command on the file system.”
The interviewer was pleased, but I was not.
Then I thought, why does not the person testing my knowledge dig deeper? But what if the data is still important? What if we can not restart the process, and at the same time this process writes to the file system in a section on which there is no free space? What if we cannot lose not only already recorded data, but also the data that this process writes or tries to record?
Tuzik
At the beginning of my career, I tried to create a small application in which it was necessary to store information about users. And then I thought, how can I map the user to his data. For example, I have Ivan Ivanov Ivanich, and he has some data, but how to make friends with them? I can directly indicate that a dog named "Tuzik" belongs to this very Ivan. But what if he changes his name and instead of Ivan becomes, for example, Olya? Then it turns out that our Olya Ivanovna Ivanova will no longer have a dog, and our Tuzik will still belong to the non-existent Ivan. This database was helped by a database that gave each user a unique identifier (ID), and my Tuzik was attached to this ID, which, in fact, was just an ordinal number. Thus, the owner of the tuzik was with ID number 2, and at some point in time Ivan was under this ID, and then Olya became the same ID. The problem of humanity and animal husbandry has been practically solved.
File descriptor
The problem of the file and the program working with this file is about the same as that of our dog and person. Suppose I opened a file under the name ivan.txt and started writing the word tuzik into it, but I managed to write only the first letter “t” to the file, and this file was renamed by someone, for example, olya.txt. But the file remained the same, and I still want to write my ace in it. Every time I open a file with the open system call in any programming language, I get a unique ID that points me to a file, this ID is a file descriptor. And it doesn’t matter at all what and who does this file next, they can delete it, they can rename it, they can change the owner or take away the rights to read and write, I will still have access to it, because at the time of opening the file I had rights to read and / or write it and I managed to start working with him, which means I must continue to do this.
On Linux, the libc library opens for each running application (process) 3 descriptor file, with the numbers 0,1,2. You can find more information at the links man stdio and man stdout
- The file descriptor 0 is called STDIN and is associated with data input from the application
- The file descriptor 1 is called STDOUT and is used by applications to output data, for example, print commands
- The file descriptor 2 is called STDERR and is used by applications to output error reporting data.
If in your program you open a file for reading or writing, then most likely you will get the first free ID and this will be number 3.
A list of descriptor files can be viewed from any process if you know its PID.
For example, open a console with bash and see the PID of our process
[user@localhost ]$ echo $$ 15771
In the second console, run
[user@localhost ]$ ls -lah /proc/15771/fd/ total 0 dr-x------ 2 user user 0 Oct 7 15:42 . dr-xr-xr-x 9 user user 0 Oct 7 15:42 .. lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21
You can safely ignore the file descriptor number 255 within the framework of this article; it was opened for its needs by bash itself, and not by a linked library.
Now all 3 descriptor files are associated with the pseudo-terminal device / dev / pts , but we can still manipulate them, for example, run in the second console
[user@localhost ]$ echo "hello world" > /proc/15771/fd/0
And in the first console we will see
[user@localhost ]$ hello world
Redirect and Pipe
You can easily override these 3 descriptor files in any process, including in bash, for example, through a pipe connecting two processes, see
[user@localhost ]$ cat /dev/zero | sleep 10000
You can run this command yourself with strace -f and see what happens inside, but I will tell you briefly.
Our parent process bash with PID 15771 parses our command and understands exactly how many commands we want to run, in our case there are two of them: cat and sleep. Bash knows that he needs to create two child processes, and combine them with one pipe. Total bash will need 2 child processes and one pipe.
Before creating child processes, bash starts the pipe system call and receives new file descriptors for the temporary pipe buffer, but this buffer does not yet bind our two child processes.
For the parent process, it looks like pipe already exists, and there are no child processes yet:
PID command 15771 bash lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 3 -> pipe:[253543032] lrwx------ 1 user user 64 Oct 7 15:42 4 -> pipe:[253543032] lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21
Then, using a system call, clone bash creates two child processes, and our three processes will look like this:
PID command 15771 bash lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 3 -> pipe:[253543032] lrwx------ 1 user user 64 Oct 7 15:42 4 -> pipe:[253543032] lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21 PID command 9004 bash lrwx------ 1 user user 64 Oct 7 15:57 0 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:57 1 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:57 3 -> pipe:[253543032] lrwx------ 1 user user 64 Oct 7 15:57 4 -> pipe:[253543032] lrwx------ 1 user user 64 Oct 7 15:57 255 -> /dev/pts/21 PID command 9005 bash lrwx------ 1 user user 64 Oct 7 15:57 0 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:57 1 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:57 3 -> pipe:[253543032] lrwx------ 1 user user 64 Oct 7 15:57 4 -> pipe:[253543032] lrwx------ 1 user user 64 Oct 7 15:57 255 -> /dev/pts/21
Do not forget that clone clones the process along with all file descriptors, so they will be the same in the parent process and in the child ones. The task of the parent process with PID 15771 is to monitor child processes, so it just waits for a response from the child processes.
Therefore, he does not need pipe, and he closes the file descriptors with numbers 3 and 4.
In the first bash child process with PID 9004, the dup2 system call, changes our STDOUT file descriptor with number 1 to the file descriptor pointing to pipe, in our case it is number 3. Thus, everything that the first child process with PID 9004 will write to STDOUT will automatically fall into the pipe buffer.
In the second child process with PID 9005, bash changes the descriptor STDIN with number 0 using dup2. Now everything that our second bash with PID 9005 will read will read from pipe.
After that, the descriptors with numbers 3 and 4 are also closed in child processes, since they are no longer used.
I deliberately ignore the file descriptor 255, it uses bash for internal needs and will also be closed in child processes.
Further, in the first child process with PID 9004, bash launches the executable file that we specified on the command line with the system call exec , in our case it is / usr / bin / cat.
In the second child process with PID 9005, bash starts the second executable file that we specified, in our case it is / usr / bin / sleep.
The exec system call does not close the file descriptors unless they were opened with the O_CLOEXEC flag during the open call. In our case, after running the executable files, all current file descriptors will be saved.
Check in the console:
[user@localhost ]$ pgrep -P 15771 9004 9005 [user@localhost ]$ ls -lah /proc/15771/fd/ total 0 dr-x------ 2 user user 0 Oct 7 15:42 . dr-xr-xr-x 9 user user 0 Oct 7 15:42 .. lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21 [user@localhost ]$ ls -lah /proc/9004/fd total 0 dr-x------ 2 user user 0 Oct 7 15:57 . dr-xr-xr-x 9 user user 0 Oct 7 15:57 .. lrwx------ 1 user user 64 Oct 7 15:57 0 -> /dev/pts/21 l-wx------ 1 user user 64 Oct 7 15:57 1 -> pipe:[253543032] lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21 lr-x------ 1 user user 64 Oct 7 15:57 3 -> /dev/zero [user@localhost ]$ ls -lah /proc/9005/fd total 0 dr-x------ 2 user user 0 Oct 7 15:57 . dr-xr-xr-x 9 user user 0 Oct 7 15:57 .. lr-x------ 1 user user 64 Oct 7 15:57 0 -> pipe:[253543032] lrwx------ 1 user user 64 Oct 7 15:57 1 -> /dev/pts/21 lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21 [user@localhost ]$ ps -up 9004 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND user 9004 0.0 0.0 107972 620 pts/21 S+ 15:57 0:00 cat /dev/zero [user@localhost ]$ ps -up 9005 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND user 9005 0.0 0.0 107952 360 pts/21 S+ 15:57 0:00 sleep 10000
As you can see, the unique number of our pipe is the same in both processes. So we have a connection between two different processes with one parent.
For those who are not familiar with the system calls that bash uses, I highly recommend running the commands through strace and see what happens inside, for example, like this:
strace -s 1024 -f bash -c "ls | grep hello"
Let's get back to our problem with running out of disk space and trying to save data without restarting the process. Let's write a small program that will write to the disk about 1 megabyte per second. Moreover, if for some reason we could not write data to disk, we will simply ignore it and try to write data again after a second. In the example I use Python, you can use any other programming language.
[user@localhost ]$ cat openforwrite.py import datetime import time mystr="a"*1024*1024+"\n" with open("123.txt", "w") as f: while True: try: f.write(str(datetime.datetime.now())) f.write(mystr) f.flush() time.sleep(1) except: pass
Run the program and look at the file descriptors
[user@localhost ]$ python openforwrite.py & [1] 3762 [user@localhost ]$ ps axuf | grep [o]penforwrite user 3762 0.0 0.0 128600 5744 pts/22 S+ 16:28 0:00 | \_ python openforwrite.py [user@localhost ]$ ls -la /proc/3762/fd total 0 dr-x------ 2 user user 0 Oct 7 16:29 . dr-xr-xr-x 9 user user 0 Oct 7 16:29 .. lrwx------ 1 user user 64 Oct 7 16:29 0 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 7 16:29 1 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 7 16:29 2 -> /dev/pts/22 l-wx------ 1 user user 64 Oct 7 16:29 3 -> /home/user/123.txt
As you can see, we have our 3 standard file descriptors and another one that we opened. Check the file size:
[user@localhost ]$ ls -lah 123.txt -rw-rw-r-- 1 user user 117M Oct 7 16:30 123.txt
data is written, try changing the file permissions:
[user@localhost ]$ sudo chown root: 123.txt [user@localhost ]$ ls -lah 123.txt -rw-rw-r-- 1 root root 168M Oct 7 16:31 123.txt [user@localhost ]$ ls -lah 123.txt -rw-rw-r-- 1 root root 172M Oct 7 16:31 123.txt
We see that the data is still being written, although our user does not have the right to write to the file. Let's try to remove it:
[user@localhost ]$ sudo rm 123.txt [user@localhost ]$ ls 123.txt ls: cannot access 123.txt: No such file or directory
Where is the data written? And are they written at all? We check:
[user@localhost ]$ ls -la /proc/3762/fd total 0 dr-x------ 2 user user 0 Oct 7 16:29 . dr-xr-xr-x 9 user user 0 Oct 7 16:29 .. lrwx------ 1 user user 64 Oct 7 16:29 0 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 7 16:29 1 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 7 16:29 2 -> /dev/pts/22 l-wx------ 1 user user 64 Oct 7 16:29 3 -> /home/user/123.txt (deleted)
Yes, our descriptor file still exists, and we can work with this descriptor file as with our old file, we can read, clean and copy it.
We look at the file size:
[user@localhost ]$ lsof | grep 123.txt python 31083 user 3w REG 8,5 19923457 2621522 /home/user/123.txt
File size 19923457. Trying to clear the file:
[user@localhost ]$ truncate -s 0 /proc/31083/fd/3 [user@localhost ]$ lsof | grep 123.txt python 31083 user 3w REG 8,5 136318390 2621522 /home/user/123.txt
As you can see, the file size only increases and our trankate did not work. Refer to the open system call documentation. If we use the O_APPEND flag when opening a file, then each time the operating system checks the file size and writes data to the very end of the file, it does this atomically. This allows multiple threads or processes to write to the same file. But in our code we do not use this flag. We can see a different file size in lsof after trankate only if we open the file for overwriting, which means that instead of in our code
with open("123.txt", "w") as f:
we have to put
with open("123.txt", "a") as f:
Checking with the “w” flag
[user@localhost ]$ strace -e trace=open python openforwrite.py 2>&1| grep 123.txt open("123.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
and with the “a” flag
[user@localhost ]$ strace -e trace=open python openforwrite.py 2>&1| grep 123.txt open("123.txt", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
Program an already running process
Often, programmers use debuggers (for example GDB) or various levels of logging in the application when creating and testing programs. Linux provides the ability to actually write and change an already running program, for example, changing the values of variables, setting breakpoint, etc., etc.
Returning to the original question with not enough disk space to write the file, we will try to emulate the problem.
Create a file for our partition, which we will mount as a separate disk:
[user@localhost ~]$ dd if=/dev/zero of=~/tempfile_for_article.dd bs=1M count=10 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.00525929 s, 2.0 GB/s [user@localhost ~]$
Create a file system:
[user@localhost ~]$ mkfs.ext4 ~/tempfile_for_article.dd mke2fs 1.42.9 (28-Dec-2013) /home/user/tempfile_for_article.dd is not a block special device. Proceed anyway? (y,n) y ... Writing superblocks and filesystem accounting information: done [user@localhost ~]$
Mount the file system:
[user@localhost ~]$ sudo mount ~/tempfile_for_article.dd /mnt/ [sudo] password for user: [user@localhost ~]$ df -h | grep mnt /dev/loop0 8.7M 172K 7.9M 3% /mnt
Create a directory with our owner:
[user@localhost ~]$ sudo mkdir /mnt/logs [user@localhost ~]$ sudo chown user: /mnt/logs
We open the file only for writing in our program:
with open("/mnt/logs/123.txt", "w") as f:
We launch
[user@localhost ]$ python openforwrite.py
Waiting a few seconds
[user@localhost ~]$ df -h | grep mnt /dev/loop0 8.7M 8.0M 0 100% /mnt
So, we got the problem described at the beginning of this article. Free space 0 occupied 100%.
We remember that according to the conditions of the task, we are trying to record very important data that cannot be lost. And at the same time we need to repair the service without restarting the process.
Suppose we still have disk space, but in a different partition, for example, in / home.
Let's try to "reprogram on the fly" our code.
We look at the PID of our process, which ate all the disk space:
[user@localhost ~]$ ps axuf | grep [o]penfor user 10078 27.2 0.0 128600 5744 pts/22 R+ 11:06 0:02 | \_ python openforwrite.py
We connect to the process via gdb
[user@localhost ~]$ gdb -p 10078 ... (gdb)
We look at the open file descriptors:
(gdb) shell ls -lah /proc/10078/fd/ total 0 dr-x------ 2 user user 0 Oct 8 11:06 . dr-xr-xr-x 9 user user 0 Oct 8 11:06 .. lrwx------ 1 user user 64 Oct 8 11:09 0 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:09 1 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:06 2 -> /dev/pts/22 l-wx------ 1 user user 64 Oct 8 11:09 3 -> /mnt/logs/123.txt
We look at the information about the file descriptor with number 3, which interests us
(gdb) shell cat /proc/10078/fdinfo/3 pos: 8189952 flags: 0100001 mnt_id: 482
Remembering what kind of system call Python makes (see above, where we ran strace and found the open call), while processing our code to open the file, we do the same on our own behalf, but we need the O_WRONLY | O_CREAT | O_TRUNC bits replace with a numerical value. To do this, open the kernel source, for example, here and see what flags are responsible for what
#define O_WRONLY 00000001
#define O_CREAT 00000100
#define O_TRUNC 00001000
We combine all the values into one, we get 00001101
Run our call from gdb
(gdb) call open("/home/user/123.txt", 00001101,0666) $1 = 4
So we got a new descriptor file with number 4 and a new open file on another section, check:
(gdb) shell ls -lah /proc/10078/fd/ total 0 dr-x------ 2 user user 0 Oct 8 11:06 . dr-xr-xr-x 9 user user 0 Oct 8 11:06 .. lrwx------ 1 user user 64 Oct 8 11:09 0 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:09 1 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:06 2 -> /dev/pts/22 l-wx------ 1 user user 64 Oct 8 11:09 3 -> /mnt/logs/123.txt l-wx------ 1 user user 64 Oct 8 11:15 4 -> /home/user/123.txt
We remember the pipe example - how bash changes file descriptors, and we already learned the dup2 system call.
We try to replace one file descriptor with another
(gdb) call dup2(4,3) $2 = 3
We check:
(gdb) shell ls -lah /proc/10078/fd/ total 0 dr-x------ 2 user user 0 Oct 8 11:06 . dr-xr-xr-x 9 user user 0 Oct 8 11:06 .. lrwx------ 1 user user 64 Oct 8 11:09 0 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:09 1 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:06 2 -> /dev/pts/22 l-wx------ 1 user user 64 Oct 8 11:09 3 -> /home/user/123.txt l-wx------ 1 user user 64 Oct 8 11:15 4 -> /home/user/123.txt
We close the descriptor 4 file, since we do not need it:
(gdb) call close (4) $1 = 0
And exit gdb
(gdb) quit A debugging session is active. Inferior 1 [process 10078] will be detached. Quit anyway? (y or n) y Detaching from program: /usr/bin/python2.7, process 10078
Check the new file:
[user@localhost ~]$ ls -lah /home/user/123.txt -rw-rw-r-- 1 user user 5.1M Oct 8 11:18 /home/user/123.txt [user@localhost ~]$ ls -lah /home/user/123.txt -rw-rw-r-- 1 user user 7.1M Oct 8 11:18 /home/user/123.txt
As you can see, the data is written to a new file, check the old one:
[user@localhost ~]$ ls -lah /mnt/logs/123.txt -rw-rw-r-- 1 user user 7.9M Oct 8 11:08 /mnt/logs/123.txt
Data is not lost, the application works, logs are written to a new place.
Let's complicate the task a bit
Imagine that the data is important to us, but we do not have disk space in any of the sections and we cannot connect the disk.
What we can do is redirect our data somewhere, for example to pipe, and the data from pipe, in turn, redirect to the network through some program, for example netcat.
We can create a named pipe with the mkfifo command. It will create a pseudo file on the file system, even if there is no free space on it.
We restart the application, and check:
[user@localhost ]$ python openforwrite.py [user@localhost ~]$ ps axuf | grep [o]pen user 5946 72.9 0.0 128600 5744 pts/22 R+ 11:27 0:20 | \_ python openforwrite.py [user@localhost ~]$ ls -lah /proc/5946/fd total 0 dr-x------ 2 user user 0 Oct 8 11:27 . dr-xr-xr-x 9 user user 0 Oct 8 11:27 .. lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22 l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/123.txt [user@localhost ~]$ df -h | grep mnt /dev/loop0 8.7M 8.0M 0 100% /mnt
There is no disk space, but we successfully create a named pipe there:
[user@localhost ~]$ mkfifo /mnt/logs/megapipe [user@localhost ~]$ ls -lah /mnt/logs/megapipe prw-rw-r-- 1 user user 0 Oct 8 11:28 /mnt/logs/megapipe
Now we need to somehow wrap all the data that gets into this pipe to another server through the network, for this all the same netcat will do.
On the remote-server.example.com server, run
[user@localhost ~]$ nc -l 7777 > 123.txt
On our problem server, run in a separate terminal
[user@localhost ~]$ nc remote-server.example.com 7777 < /mnt/logs/megapipe
Now all the data that gets into the pipe will automatically go to stdin in netcat, which will send it to the network on port 7777.
All we have to do is start writing our data into this named pipe.
We already have a running application:
[user@localhost ~]$ ps axuf | grep [o]pen user 5946 99.8 0.0 128600 5744 pts/22 R+ 11:27 169:27 | \_ python openforwrite.py [user@localhost ~]$ ls -lah /proc/5946/fd total 0 dr-x------ 2 user user 0 Oct 8 11:27 . dr-xr-xr-x 9 user user 0 Oct 8 11:27 .. lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22 l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/123.txt
Of all the flags, we only need O_WRONLY since the file already exists and we don’t need to clear it
[user@localhost ~]$ gdb -p 5946 ... (gdb) call open("/mnt/logs/megapipe", 00000001,0666) $1 = 4 (gdb) shell ls -lah /proc/5946/fd total 0 dr-x------ 2 user user 0 Oct 8 11:27 . dr-xr-xr-x 9 user user 0 Oct 8 11:27 .. lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22 l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/123.txt l-wx------ 1 user user 64 Oct 8 14:20 4 -> /mnt/logs/megapipe (gdb) call dup2(4,3) $2 = 3 (gdb) shell ls -lah /proc/5946/fd total 0 dr-x------ 2 user user 0 Oct 8 11:27 . dr-xr-xr-x 9 user user 0 Oct 8 11:27 .. lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22 l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/megapipe l-wx------ 1 user user 64 Oct 8 14:20 4 -> /mnt/logs/megapipe (gdb) call close(4) $3 = 0 (gdb) shell ls -lah /proc/5946/fd total 0 dr-x------ 2 user user 0 Oct 8 11:27 . dr-xr-xr-x 9 user user 0 Oct 8 11:27 .. lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22 lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22 l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/megapipe (gdb) quit A debugging session is active. Inferior 1 [process 5946] will be detached. Quit anyway? (y or n) y Detaching from program: /usr/bin/python2.7, process 5946
Checking the remote server remote-server.example.com
[user@localhost ~]$ ls -lah 123.txt -rw-rw-r-- 1 user user 38M Oct 8 14:21 123.txt
Data goes, we check a problem server
[user@localhost ~]$ ls -lah /mnt/logs/ total 7.9M drwxr-xr-x 2 user user 1.0K Oct 8 11:28 . drwxr-xr-x 4 root root 1.0K Oct 8 10:55 .. -rw-rw-r-- 1 user user 7.9M Oct 8 14:17 123.txt prw-rw-r-- 1 user user 0 Oct 8 14:22 megapipe
Data saved, problem resolved.
I take this opportunity to convey my regards to my colleagues at Degiro.
Listen to Radio T podcasts.
Good to all.
As a homework, I suggest thinking about what will be in the file descriptors of the cat and sleep process if you run this command:
[user@localhost ~]$ cat /dev/zero 2>/dev/null| sleep 10000
All Articles