Bash Guidance
This document provides hits and overview for the use of the bash shell in general, namely a "bash 101 for DDCI developers".
Resources
There are lots of places on the web to get help with bash. For a particular installation you are best off with "info bash" or "man bash" from a command prompt to ensure you have the right version.
Execution Model
The bash shell is part of a family of programs that enable the execution and control of programs on Unix like systems. Some basic terminology:
- Executing a program causes a process to be created.
- Every process has (at least) these three files:
- 0 stdin
- 1 stdout
- 2 stderr
- Every process has the interface specified by this C prototype:
- int main(int argc, char **argv); // Ignoring envp.
Here is a nice simple C program "necho.c" that can be used to understand how it all works:
#include <stdio.h> // Program to print each argument on a separate line. int main(int argc, char **argv) { // If first argument starts with "e" send to stderr. FILE *f = ((argc > 1) && (argv[1][0] == 'e') ? stderr : stdout); fprintf(f, "ARGC = %d\n", argc); for (int a=0; a<argc; a++) fprintf(f, "ARGV[%d] = \"%s\"\n", a, argv[a]); if (f == stderr) return 1; // Failure return 0; // success }
Lets try some simple commands:
make necho ./necho ./necho a b c ./necho a b c > out # redirection of stdout cat out ./necho a b c 1> out # redirection of stdout long form. ./necho e f g 2> err # redirection of stderr. argv[1]="e" means necho has no stdout in this case! cat err
You can also append:
echo "hello" > msg cat msg echo "there" >> msg cat msg
The stdout of one process can be "piped" (used as stdin) to another process:
./necho a b c | sed -e "s/ARG/arg/g"
stdin, stdout, and stderr are inherited from the parent process. Parenthesis create a sub-shell, semicolon separates commands:
(./necho a b c; ./necho x y z) | sed -e "s/ARG/arg/g" (./necho a b c; ./necho e f g) | sed -e "s/ARG/arg/g" # stderr not piped!
Both stdout and stderr can be redirected:
(./necho a b c; ./necho e f g) >& both # the short way (./necho a b c; ./necho e f g) > both 2>&1 # Slightly longer way (./necho a b c; ./necho e f g) 2>&1 > both # A mistake. Order is important.
Both stdout and stderr can be piped:
(./necho a b c; ./necho e f g) |& sed -e "s/ARG/arg/g" # "|&" means stderr is piped!
The return status code can be checked using the "if" builtin:
if ./necho a; then echo "Success"; fi # output from conditional is ignored by "if". if ./necho e; then echo "Success"; fi
The programs "true" and "false" return success and failure respectively:
ls /bin/true /bin/false file /bin/true /bin/false
if true; then echo "Success"; fi if false; then echo "Success"; fi if ! false; then echo "Success"; fi
You can also use a "control operator":
true && echo "Success" true || necho a false || ./necho a (false || false) && echo "Success" (false || true) && echo "Success"
NOTE: you should almost never use ";" in make recipes. Use "&&" instead.
NOTE: You should almost never use the control operator syntax outside of the conditional in an "if" or "while" command because "set -e" considers failures as errors.
Globbing
Globs, also called patterns, are used to identify files. Globs are not the same as regular expressions.
"." means a literal period. "*" means any number of chars. "?" means exactly one character. "[]" is like in a regular expression. "**" any file in any subdir. In a directory position means any subdirectory (see below)
ls
./necho *
touch a b c # touch updates the date or creates the listed files.
make necho
touch necho.c
make necho
./necho ?
./necho a*
./necho ?? # if the pattern doesn't match, the pattern stays!
# see: shopt -s nullglob; and shopt -s failglob
touch aa bb cc
./necho ??
./necho a*
Brace Expansion
./necho a{1,2,3}b
mv output{,.orig} # Nice shortcut to rename things, especially if the path is long
echo mv output{,.orig} # prefixing with "echo" is a good way to see if your braces are right.
mv ~/scm/Deos/products/kernel/kernel/branches/experimental/output{,.orig}
./necho dir{1,2}/subdir{1,2}
mkdir -p dir{1,2}/subdir{1,2}
touch dir{1,2}/subdir{1,2}/{a,bb}
find dir?
echo */ # Show (non hidden) directories echo **/? # Show any single character file in any subdirectory echo **/a
Variables and Command Substitution
var=hello
./necho $var
./necho ${var}
var="hello there" ./necho $var ./necho "$var" ./necho '$var' # single quotes inhibit variable substitution ./necho ~ ./necho "~" # any quotes inhibit ~ expansion ./necho '~'
./necho ? single=$(echo ?) ./necho $single for f in $single; do echo file=$f; done for f in "$single"; do echo file=$f; done touch "x y" # Putting spaces in filenames will be punished. for f in *; do echo file=$f; done # is "x y" a file or not? echo * files=$(echo *) for f in $files; do echo file=$f; done # is "x y" a file or not?
Array Variables and Quoting
avar=(*)
for f in ${avar[@]}; do echo f=$f; done # unquoted array variable slice is still subject to reparsing
for f in "${avar[@]}"; do echo f=$f; done # double quoted works as expected.
for f in '${avar[@]}'; do echo f=$f; done # single quotes inhibit variable expansion
There are also associative arrays.
Subprocesses
sleep 5 & jobs ^Z and bg fg kill %1 ps ps -a # not just for sub-processes. ps -u
History
history ! ^R
Switch Conventions
--rm # two hypen for "long" options -i # one hypen for "short" options -it # short options can be combined
Of course not all programs adhere to the above, notably all the scripts that use toolCrib.py.
Python's argparse/argcomplete provide wonderful command completion.
Commonly Used Programs
- cat
- cat /etc/passwd
- wc
- wc -l /etc/passwd
- head
- head /etc/passwd
- tail
- tail /etc/passwd
- tail -f msg
- grep
- grep "^$USER" /etc/passwd
- sed
- sed -n -e "/^$USER/p" /etc/passwd
- cut
- grep "^$USER" /etc/passwd | cut -d: -f7
- grep "^$USER" /etc/passwd | cut -d: -f5,7
- awk
- awk -F: "/^$USER/"'{ printf "%s--%s\n",$5,$7; }' /etc/passwd
- find
- find /etc -iname '*passwd*'
- find /etc -iname '*passwd*' 2> /dev/null
- xargs
- find /etc -iname '*passwd*' 2> /dev/null | xargs grep "^$USER" # see also find's print0 and xarg's -0
- grep "^$USER" $(find /etc -iname '*passwd*' 2> /dev/null)
- grep "^$USER" $(find /etc -iname '*passwd*' 2> /dev/null) 2>/dev/null
General Usage Hints
- Incrementally build up "one line" scripts.
- Save the scripts for "tomorrow"
- "cd" using absolute paths: # this makes history more useful
- cd ~/scm/Deos/products/kernel/kernel/branches/experimental/code/HAL/aarch64/
- use command completion to list directories.
- Use sub-shells or pushd to work in multiple directories:
- (cd ~/scm/Deos/products/kernel/kernel/branches/experimental/ && build -Q -j install)
- Use "built in cd" option to various commands, notably make:
- make -C ~/scm/Deos/products/kernel/kernel/branches/experimental/output/arm-deos/debug/ install
- Add comments to commands to simplify history search
- make -C ~/scm/Deos/products/kernel/kernel/branches/experimental/output/arm-deos/debug/ install # kbuild
