How to debug environment variables in Linux
It often happens that you come to the machine and find some kind of script running under the system user a week ago. Who launched it? Where to look for this run.php? Or you add an entry to / etc / crontab, and the script crashes there with the "command not found" error. Why? And what to do?
I have the answers to these questions.
In almost all modern operating systems, processes have environment variables. Technically, they are a collection of named strings. If a subprocess is started, then it automatically inherits a copy of the parent's environment.
Among others, there is the PATH variable, which indicates the paths to search for executable files, the HOME variable, which points to the user's home directory, variables that are responsible for the user's language preferences, and many others.
There are many reviews describing the meaning of these variables, but there are practically no articles on how to investigate problems. Fill this gap.
So, we found a script running under the system user a week ago. Who launched it? What for? Maybe they just forgot about him? Potentially 10-15 people could launch it, you won’t interview everyone. How to find who it was? And where does this run.php lie?
Process environment variables and the sudo feature come to the rescue. There is such a PWD variable in which the shell stores the current working directory; this value, in fact, saves information about the current directory at the time the command is run. Also, the sudo utility by default leaves information in the process environment variable about which user it was launched from.
Environment variables (and much more) for any running process can be found in / proc. Voila:
Ahem, I launched it myself. Well, who doesn’t happen to? ..
In general, using such a simple method in simple situations, you can find information about the process, which is generally not available.
One of the cases when you have to think about environment variables is when a script added to / etc / crontab crashes with an error. You go to the server via SSH, run the command, everything seems to work as it should. And when it starts automatically, it shows something like “hive: command not found”.
In general, it is good practice to write the full path to executable commands, but this is not always possible. In such cases, the developers get out as anyone can. Someone adds the desired path in PATH as part of the team in crontab. More experienced ones wrap their command in bash -l. And the crow-bombs taught by bitter experience still don’t forget to flock. Everything is so: made, added to monitoring and forgot.
After such manipulations, a sediment remains in the soul of a real engineer. Yes, the problem is solved. But I didn’t understand what was going on! How is one approach better than another? Where are all these settings stored and by whom are they changed?
Let's compare the environment variables that the process has when it is launched from the crown and the environment variables that we have on the command line. Log the output of the env command from the krona and our current environment:
Look what's in the PATH variable:
Mama Mia! So there under the crown only the very minimum! Of course, you need to load the normal environment variables.
Let's see what the environment will be if we add bash -l:
The difference is not so noticeable. All paths are presented. Some in a different order, some are repeated, but this is already much better than it was. The rest of the variables are also well tuned. There is, of course, a slight difference in the locale, in the variables from SSH, but this should no longer dramatically affect the operation of the script.
Now it’s clear why bash -l is needed in crontab entries. And, of course, do not forget about flock.
The problem seems to be solved, everything from the crown works. But how is it that some paths are duplicated in the PATH variable? So there is some kind of mess in the server setup. Let's try to figure it out.
We open some mans to initialize the environment, we read out which scripts and in what order are executed, with enthusiasm we start to run through their eyes - and after a few minutes a feeling of despair comes. Some endless stream of conditions about some special cases of architectures, terminals and incredibly important color settings for the ls command. Pain, despair, hate! We are interested in one damn variable PATH!
In fact, everything is somewhat simpler. Meet:
What does this team do? Creates a new bash process with a pristine environment, indicates that it is necessary to run initialization scripts and secure everything in detail in the login.log file. Now we have the opportunity not to execute all the scripts in our minds, but simply to read what, where and when it was executed and where this or that environment setting came from.
I will not analyze in detail how to read the resulting log. Everything is almost trivial there. I will only mention that one hit came from / etc / profile and two from /etc/bash.bashrc. Yes, somewhere they were too smart when setting up packages in a pappet. Well, nothing, it does not bother me to work.
But now I know and can!
PS In very difficult cases and in order to understand everything at all, you can wrap the command in strace:
I have the answers to these questions.
Environment variables
In almost all modern operating systems, processes have environment variables. Technically, they are a collection of named strings. If a subprocess is started, then it automatically inherits a copy of the parent's environment.
Among others, there is the PATH variable, which indicates the paths to search for executable files, the HOME variable, which points to the user's home directory, variables that are responsible for the user's language preferences, and many others.
There are many reviews describing the meaning of these variables, but there are practically no articles on how to investigate problems. Fill this gap.
Who started the process?
So, we found a script running under the system user a week ago. Who launched it? What for? Maybe they just forgot about him? Potentially 10-15 people could launch it, you won’t interview everyone. How to find who it was? And where does this run.php lie?
$ ps x | grep run.php 10684 ? Ss 472:25 /local/php/bin/php run.php
Process environment variables and the sudo feature come to the rescue. There is such a PWD variable in which the shell stores the current working directory; this value, in fact, saves information about the current directory at the time the command is run. Also, the sudo utility by default leaves information in the process environment variable about which user it was launched from.
Environment variables (and much more) for any running process can be found in / proc. Voila:
$ cat /proc/10684/environ | tr '\0' '\n' | grep SUDO_USER SUDO_USER=alexxz $ cat /proc/10684/environ | tr '\0' '\n' | grep PWD PWD=/home/etlmaster
Ahem, I launched it myself. Well, who doesn’t happen to? ..
In general, using such a simple method in simple situations, you can find information about the process, which is generally not available.
The script works from the command line, but does not work from cron
One of the cases when you have to think about environment variables is when a script added to / etc / crontab crashes with an error. You go to the server via SSH, run the command, everything seems to work as it should. And when it starts automatically, it shows something like “hive: command not found”.
In general, it is good practice to write the full path to executable commands, but this is not always possible. In such cases, the developers get out as anyone can. Someone adds the desired path in PATH as part of the team in crontab. More experienced ones wrap their command in bash -l. And the crow-bombs taught by bitter experience still don’t forget to flock. Everything is so: made, added to monitoring and forgot.
After such manipulations, a sediment remains in the soul of a real engineer. Yes, the problem is solved. But I didn’t understand what was going on! How is one approach better than another? Where are all these settings stored and by whom are they changed?
Let's compare the environment variables that the process has when it is launched from the crown and the environment variables that we have on the command line. Log the output of the env command from the krona and our current environment:
$ echo "* * * * * env > ~/crontab.env" | crontab; sleep 60; echo "" | crontab; $ env > my.env
Look what's in the PATH variable:
> grep ^PATH= crontab.env my.env Crontab.env: PATH=/usr/bin:/bin My.env: PATH=/local/hive/bin:/local/python/bin:/local/hadoop/bin:/local/hadoop/bin:/local/hive/bin:/local/hadoop/bin:/usr/local/bin:/usr/bin:/bin
Mama Mia! So there under the crown only the very minimum! Of course, you need to load the normal environment variables.
Let's see what the environment will be if we add bash -l:
$ echo "* * * * * bash -l env > ~/crontab.env" | crontab; sleep 60; echo "" | crontab; alexxz@bi1.mlan:~> grep ^PATH= crontab.env my.env Crontab.env: PATH=/local/hive/bin:/local/python/bin:/local/hadoop/bin:/local/hadoop/bin:/local/hadoop/bin:/local/hive/bin:/usr/local/bin:/usr/bin:/bin My.env: PATH=/local/hive/bin:/local/python/bin:/local/hadoop/bin:/local/hadoop/bin:/local/hive/bin:/local/hadoop/bin:/usr/local/bin:/usr/bin:/bin
The difference is not so noticeable. All paths are presented. Some in a different order, some are repeated, but this is already much better than it was. The rest of the variables are also well tuned. There is, of course, a slight difference in the locale, in the variables from SSH, but this should no longer dramatically affect the operation of the script.
Now it’s clear why bash -l is needed in crontab entries. And, of course, do not forget about flock.
Debug initialization of login scripts
The problem seems to be solved, everything from the crown works. But how is it that some paths are duplicated in the PATH variable? So there is some kind of mess in the server setup. Let's try to figure it out.
We open some mans to initialize the environment, we read out which scripts and in what order are executed, with enthusiasm we start to run through their eyes - and after a few minutes a feeling of despair comes. Some endless stream of conditions about some special cases of architectures, terminals and incredibly important color settings for the ls command. Pain, despair, hate! We are interested in one damn variable PATH!
In fact, everything is somewhat simpler. Meet:
env -i bash -x -l -c 'echo 123' > login.log 2>&1
What does this team do? Creates a new bash process with a pristine environment, indicates that it is necessary to run initialization scripts and secure everything in detail in the login.log file. Now we have the opportunity not to execute all the scripts in our minds, but simply to read what, where and when it was executed and where this or that environment setting came from.
I will not analyze in detail how to read the resulting log. Everything is almost trivial there. I will only mention that one hit came from / etc / profile and two from /etc/bash.bashrc. Yes, somewhere they were too smart when setting up packages in a pappet. Well, nothing, it does not bother me to work.
But now I know and can!
PS In very difficult cases and in order to understand everything at all, you can wrap the command in strace:
strace -f env -i bash -x -l -c 'echo 123' > login.log 2>&1
All Articles