I found a strange issue when I wanted to get the value of an env variable from Python script.
To reproduce, we can first set the value of
foo in the Bash shell:
Then execute the following python command:
$ python -c "import os; print(os.getenv('FOO', "None"))"
The output is
None. So the python process can not see the variable
However, if we run the two command in one line:
$ FOO=123 python -c "import os; print(os.getenv('FOO', "None"))"
The output will be
If we use
export FOO=123 instead of
FOO=123, we can also get the value
FOO in Python.
But why? What is the difference here? This is in fact a bit complex than what I think.
Bash builtin and external commands
First, we need to know that, when Bash executes commands, it treats the builtin and external commands differently. Builtin commands are those that are, literally, built into the Bash executable. External commands are those commands that are not part of the Bash shell.
When Bash executes builtin commands, it does not create a new process. When it executes external commands, it will create a new shell and then execute the external commands. The new shell will inherit the environment variables of its parent shell (see here on why Bash needs to create a new process to execute an external command).
How do we check if a command is a Bash builtin or an external command?
First, we can check the output of
help, it will list all Bash builtin
commands. We can also use the
type command to check if a command is a builtin
type -t some_command
For builtin commands, it will output
builtin. For external commands, it will
output different info, for example,
help type for more info).
- Check if a command is builtin.
- How does shell execute commands.
When will variable value be available in a process?
If we define variable without using
export, it is not part of the environment
variable. When Bash create a new process, this variable will not be available
One exception is that if you set the variable and run the command in the same line, it will be available only to that command. From the Bash manual:
The environment for any simple command or function may be augmented temporarily by prefixing it with parameter assignments, as described in Shell Parameters. These assignment statements affect only the environment seen by that command.
If a variable is not exported, it is only a shell variable. When it is exported, it is an environment variable that will be available in all subprocesses.
- Defining a variable with or without export.
- Bash: difference between “export k=1” vs. “k=1”.
- Difference between shell variables which are exported and those which are not in bash
- Difference between “a=b” and “export a=b” in bash
Back to the issue
Now with sufficient background knowledge on how Bash works, we are able to understand why certain ways work or fail.
# Fails $ FOO=123 $ python -c "import os; print(os.getenv('FOO', "None"))"
The above way to get the value of
FOO fails, because
python is an external
command for Bash. Bash will execute
python in a new process. However,
FOO=123 is only available the shell that invoking
command, not the sub-process (also a Bash shell) that actually run
# work $ FOO=123 python -c "import os; print(os.getenv('FOO', "None"))"
The above way works because the variable definition is in the same line with
python command. It will be available temporily for this command. If you
python -c "import os; print(os.getenv('FOO', "None")) in a second
line, it will not work any more.
$ export FOO=123 $ python -c "import os; print(os.getenv('FOO', "None"))"
This way also works because we set an environment variable
FOO will be available to all subprocesses created by this shell.
License CC BY-NC-ND 4.0