Advanced Bash-Scripting Guide

Learning the bash Shell,
3rd Edition
Cameron Newham

Chapter 1. bash Basics

Chapter 2. Command-Line Editing

2.2 The History List

Whenever you log in or start another interactive shell, bash reads an initial history list from the file .bash_history in your home directory. From that point on, every bash interactive session maintains its own list of commands. When you exit from a shell, it saves the list in .bash_history.

Chapter 3. Customizing Your Environment

The files .bash_profile, .bash_logout, and .bashrc that are read by bash when you log in and out or start a new shell.
Your account is using only the default system file /etc/profile initially.

• .bash_profile
Every time you log in to the system, .bash_profile is read. This file defines the basic environment for your login account.
bash allows two synonyms for .bash_profile:
• .bash_login
• .profile
If .bash_profile doesn't exist in your home directory, then bash will look for .bash_login. If that doesn't exist it will look for .profile.
• .bashrc
.bash_profile is read and executed only by the login shell.
If you start up a new shell (a subshell), the file .bashrc is read.
This scheme allows you the flexibility to separate startup commands needed at login time from those you might need when you run a subshell.
• .bash_logout
The file .bash_logout is read and executed every time a login shell exits.

3.2 Aliases

Aliases can be defined using this form:

alias name=command

3.3 Options

Options are one way to change the shell's behavior.
A shell option is a setting that is either "on" or "off".
The basic commands that relate to options are:

set -o optionname
set +o optionname

the -o turns the named option on, while the +o turns it off. (becaus - is the conventional UNIX way of specifying options to a command )
To check the status of an option, just type set -o.

allexport      	off
braceexpand    	on
emacs          	on
errexit        	off
errtrace       	off
functrace      	off
hashall        	on
histexpand     	on
history        	on
ignoreeof      	off
interactive-comments	on
keyword        	off
monitor        	on
noclobber      	off
noexec         	off
noglob         	off
nolog          	off
notify         	off
nounset        	off
onecmd         	off
physical       	off
pipefail       	off
posix          	off
privileged     	off
verbose        	off
vi             	off
xtrace         	off

To returns a non-zero exit code, turn errexit ON: (the shorthand version)

$ set -e

3.4 Shell Variables

The syntax for defining variables

varname=value

To use the value of a variable in a command, precede its name by a dollar sign ($).
The shell doesn't try to interpret character which is preceded with backslash (\).

$ echo "The value of \$varname is \"$varname\"."
The value of $varname is "".

Built-In Variables:

• Editing mode variables
• HISTCMD
• HISTCONTROL
• HISTIGNORE
• HISTFILE
• HISTFILESIZE
• HISTSIZE
• HISTTIMEFORMAT
• FCEDIT
• Time stamp formats
• Mail variables
The shell can't actually check for incoming mail, but it can look at your mail file periodically and determine whether the file has been modified since the last check.

• MAIL
Name of file to check for incoming mail
• MAILCHECK
How often, in seconds, to check for new mail (default 60 seconds)
• MAILPATH
List of filenames, separated by colons(:), to check for incoming mail
• Prompting variables
• Command search path
• Command hashing
• Directory search path and variables
• Miscellaneous variables

3.5 Customization and Subprocesses

A special class of shell variables called environment variables is known to all kinds of subprocesses.
Any variable can become an environment variable after it is exported.
You can find out which variables are environment variables and what their values are by typing export.
bash's default environment file is the .bashrc file.

3.6 Customization Hints

Chapter 4. Basic Shell Programming

4.1 Shell Scripts and Functions

A script (a file that contains shell commands) is a shell program.
To define a function, you can use either one of two forms:

function functname
{
    shell commands
}    
    

functname( )
{
    shell commands
}    
    

You can also delete a function definition with the command :

unset -f functname.

You can find out what functions are defined in your login session by typing:

declare -f

4.2 Shell Variables

4.2.1 Positional Parameters

Positional parameters hold the command-line arguments to scripts when they are invoked. Positional parameters have the names 1, 2, 3, etc., meaning that their values are denoted by $1, $2, $3, etc.
The positional parameter 0 is the name of the script.
Two special variables contain all of the positional parameters:

• *
$* is a single string that consists of all of the positional parameters
• @
$@ is a list of double-quoted strings separated by space.

The variable # holds the number of positional parameters.
Shell functions use positional parameters and special variables like * and # in exactly the same way as shell scripts do.

A double dash -- is used to signify the end of the command's option-argument. Any following arguments should be treated as operands.

  
$ touch --test
touch: unrecognized option '--test'
Try 'touch --help' for more information.
$ touch -- --test

4.2.2 Local Variables in Functions

A local statement inside a function definition makes the variables involved all become local to that function.

test1=main

func(){
        local test1="func"
        echo "func:$test1"
}

func
echo $test1

$ ./test.sh
func:func
main

4.2.4 More on Variable Syntax

The syntax of $varname for taking the value of a variable is actually the simple form of the more general syntax, ${varname}.

4.3 String Operators

The curly-bracket syntax allows for the shell's string operators.
String operators allow you to manipulate values of variables.

4.3.1 Syntax of String Operators

The string-handling operators:

1. tests for the existence of variables
2. allows substitutions of default values under certain conditions

• ${ varname :- word }
If varname exists and isn't null, return its value; otherwise return word. (Returning a default value if the variable is undefined.)
• ${ varname := word}
If varname exists and isn't null, return its value; otherwise set it to word and then return its value.(Setting a variable to a default value if it is undefined.)
• ${varname:?message }
If varname exists and isn't null, return its value; otherwise print varname: followed by message, and abort the current command or script.(Catching errors that result from variables being undefined.)
• ${ varname:+word }
If varname exists and isn't null, return word; otherwise return null.(Testing for existence of a variable.)
• ${varname:offset:length}
It returns the substring of $varname which is started at the offset and up to length characters. (Returning parts of a string)
The first character in $varname is position 0.

4.3.2 Patterns and Matching

The next type of string operator is used to match portions of a variable's string value against patterns.
Pattern-matching operators:

• ${variable#pattern}
If the pattern matches the beginning of the variable's value, delete the shortest part that matches and return the rest.
• ${variable##pattern}
If the pattern matches the beginning of the variable's value, delete the longest part that matches and return the rest.
• ${variable%pattern}
If the pattern matches the end of the variable's value, delete the shortest part that matches and return the rest.
• ${variable%%pattern}
If the pattern matches the end of the variable's value, delete the longest part that matches and return the rest.
• ${variable/pattern/string} or ${variable//pattern/ string}
The longest match to pattern in variable is replaced by string.
In the first form, only the first match is replaced.
In the second form, all matches are replaced.

4.3.3 Length Operator

${#varname }, which returns the length of the value of the variable as a character string.

4.3.4 Extended Pattern Matching

Bash provides a further set of pattern matching operators if the shopt option extglob is switched on.

4.4 Command Substitution

The "command substitution" allows you to use the standard output of a command as if it were the value of a variable named with that command.

$(UNIX command)

Here are some simple examples:

• $(pwd)
the current directory
• $(> alice)
the contents of the file alice with any trailing newlines removed.

4.5 Advanced Examples: pushd and popd

Chapter 5. Flow Control

5.1 if/else

if condition
then
    statements
elif condition
then
    statements
else
    statements
fi

For ex.,

val=$1

if [ $val = 1 ]
then
        echo "1"
elif [ $val = 2 ]
then
        echo "2"
else
        echo "out,$val"
fi

5.1.1 Exit Status

Every UNIX command returns an integer code to its calling process. This is called the exit status.
0 is usually the OK exit status.
if checks the exit status of the last statement in the list following the if keyword.
If the status is 0, the condition evaluates to true; if it is anything else, the condition is considered false.

5.1.2 Return

return N

which causes the surrounding function to exit with exit status N.
N is actually optional; it defaults to the exit status of the last command.

5.1.3 Combinations of exit Statuses

exit statuses can be combined logically.

• statement1 && statement2
"execute statement1, and if its exit status is 0, execute statement2."
• statement1 || statement2
"execute statement1, and if its exit status is not 0, execute statement2."
• Preceding a statement with ! will cause it to return 0 if it fails and 1 if it succeeds.

5.1.4 Condition Tests

Except for the exit statuses that an if construct can test, the shell provides two ways for testing a variety of conditions.

1. [ condition ]
[ condition ] is actually a statement just like any other, except that the only thing it does is return an exit status that tells whether condition is true.(The spaces after the opening bracket "[" and before the closing bracket "]" are required.)

2. [[ condition ]]

The second version is identical to the first except that word splitting and pathname expansion are not performed on the words within the brackets.
The square brackets ([ ]) surround expressions that include various types of operators.

• String comparison operators
• str1 = str2
True if str1 matches str2
• str1 != str2
True if str1 does not match str2
• str1 < str2
True if str1 is less than str2
• str1 > str2
True if str1 is greater than str2
• -n str1
True if str1 is not null (has length greater than 0)
• -z str1
True if str1 is null (has length 0)
• File attribute checking operator
• -a file
True if file exists
• -d file
True if file exists and is a directory
• -e file
True if file exists; same as - a
• -f file
True if file exists and is a regular file (i.e., not a directory or other special type of file)
• -r file
True if You have read permission on file
• -s file
True if file exists and is not empty
• -w file
True if You have write permission on file
• -x file
True if You have execute permission
• -N file
True if file was modified since it was last read
• -O file
True if You own file
• -G file
True if file 's group ID matches yours
• file1 -nt file2
True if file1 is newer than file2
• file1 -ot file2
True if file1 is older than file2

-a (AND) and -o (OR) are ogical operators and only available inside a test conditional expression.

5.1.5 Integer Conditionals

The integer comparison operators:

• -lt
Less than
• -le
Less than or equal
• -eq
Equal
• -ge
Greater than or equal
• -gt
Greater than
• -ne
Not equal

5.2 for

The for loop is ideal for working with arguments:

for name in list
do
  statements that can use $name ...
done

The list is a list of strings.
If in list is omitted, the list defaults to "$@", i.e., the quoted list of command-line arguments.
The environment variable IFS (internal field separator) indicates the field separator which is a space, TAB, and NEWLINE by default.
To print out information about each directory:

IFS=:
for dir in $PATH
do
    ls -l $dir
done

5.3 case

bash's case construct lets you test strings against patterns that can contain wildcard characters.

case expression in
    pattern1 ) statements ;;
    pattern2 ) statements ;;
    ...
esac

If expression matches one of the patterns, its corresponding statements are executed.

5.4 select

select conctruct allows you to generate simple menus easily.

select brand in Samsung Sony iphone symphony Walton
do
    echo "You have chosen $brand"
done

$ ./test.sh
1) Samsung
2) Sony
3) iphone
4) symphony
5) Walton
#? 

5.5 while and until

while condition
do
    statements...
done

until command
do
    statements...
done

The only difference between while and until is the way the condition is handled.

• while
the loop executes as long as the condition is true
• until
it runs as long as the condition is false

Chapter 6. Command-Line Options and Typed Variables

Chapter 7. Input/Output and Command-Line Processing

eval

The eval statement tells the shell to take eval's arguments and run them through the command-line processing steps.
Cases study:

$ eval ls
build.sh  
$ listpage="ls | more"
$ $listpage
ls: cannot access '|': No such file or directory
ls: cannot access 'more': No such file or directory
 

Thi is due to that only the 1st argument is considered as a command.
Now consider eval :

  
$ eval $listpage
build.sh

It will run the command eval with arguments ls, |, and more.
It enables you to write programs that can "write" and execute other programs.

Chapter 8. Process Handling

Chapter 9. Debugging Shell Programs

Chapter 10. bash Administration

Chapter 11. Shell Scripting

Chapter 12. bash for Your System

Advanced Bash-Scripting Guide

Part 1. Introduction

1. Shell Programming!

2. Starting Off With a Sha-Bang

2.1. Invoking the script

Having made the script executable, you may now test it by ./scriptname.

2.2. Preliminary Exercises

In the simplest case, a script is nothing more than a list of system commands stored in a file.

#!/bin/bash
# Cleanup
# Run as root, of course.

cd /var/log
cat /dev/null > messages
cat /dev/null > wtmp
echo "Log files cleaned up."

The sha-bang ( #! ) at the head of a script tells your system that this file is a set of commands to be fed to the command interpreter indicated.

Part 2. Basics

3. Special Characters

A special character beyonds its meaning.
Special Characters Found In Scripts and Elsewhere:

• #
Comments
• ;
Command separator [semicolon].
• ;;
Terminator in a case option [double semicolon].
• .
"dot" command [period]. This is a bash builtin. Equivalent to source.
Note,
• When working with filenames, a leading dot is the prefix of a "hidden" file.
• When considering directory names, a single dot represents the current working directory, and two dots denote the parent directory.
• "
partial quoting [double quote]. "STRING" preserves (from interpretation) most of the special characters within STRING.
• '
full quoting [single quote]. 'STRING' preserves all special characters within STRING. This is a stronger form of quoting than "STRING".
• ,
comma operator. The comma operator is a list operator. A list of arithmetic expressions will be evaluated from left to right, the last expression result is the return value.


$ echo "$(( a=1, ++a, ++a ))"
3
  

• ,,
Lowercase conversion in parameter substitution

$ var=veryMixedUpVariable
$ echo ${var}
veryMixedUpVariable
$ echo ${var,,}          # verymixedupvariable
verymixedupvariable
  

• \
escape [backslash]. A quoting mechanism for single characters.
• `
command substitution. This is also known as backquotes. The `command` construct makes available the output of command for assignment to a variable.

4. Introduction to Variables and Parameters

4.1. Variable Substitution

If variable1 is the name of a variable, then $variable1 is a reference to its value.
Enclosing a referenced value in double quotes ("$variable1") does not interfere with variable substitution.
Using single quotes ('$variable1') causes the variable name to be used literally.
Note that $variable is actually a simplified form of ${variable}.

$ a=375
$ hello=$a
$ echo hello
hello
$ echo $hello
375
$ echo ${hello}
375
$ echo "$hello"
375
$ echo "${hello}"
375
$ echo '$hello'
$hello
$ hello=
$ echo "\$hello (null value) = $hello"
$hello (null value) = 

4.2. Variable Assignment

= is the assignment operator (no space before and after).
Note that = can be either an assignment or a test operator, depending on context.

$ echo "The value of \"a\" is $a."
The value of "a" is 879.

Variable assignment using the $(...) mechanism

R=$(cat /etc/redhat-release)
arch=$(uname -m)

4.3. Bash Variables Are Untyped

Essentially, Bash variables are character strings.
But, depending on context, Bash permits arithmetic operations and comparisons on variables. The determining factor is whether the value of a variable contains only digits.

$ a=2334 
$ let "a += 1"
$ echo "a = $a "
a = 2335 
  

4.4. Special Variable Types

• Local variables
Variables visible only within a code block or function
• Environmental variables
Variables that affect the behavior of the shell and user interface.
Every time a shell starts, it creates shell variables that correspond to its own environmental variables.
• Positional parameters
Arguments passed to the script from the command line : $0, $1,$2, $3 . . .
$0 is the name of the script itself, $1 is the first argument,...
The special variables $* and $@ denote all the positional parameters.
The shift command reassigns the positional parameters, in effect shifting them to the left one notch.
The old $1 disappears, but $0 (the script name) does not change.


#!/bin/bash
# shift-past.sh

shift 3    # Shift 3 positions.
echo "$1"

exit 0

$ sh shift-past.sh 1 2 3 4 5
4
  

5. Quoting

A character is "special" if it has an interpretation other than its literal meaning.
Quoting is for protecting special characters in the string from reinterpretation or expansion by the shell or shell script.

5.1. Quoting Variables

When referencing a variable, it is generally advisable to enclose its name in double quotes. This prevents reinterpretation of all special characters within the quoted string.
Single quotes (' ') operate similarly to double quotes, but do not permit referencing variables, since the special meaning of $ is turned off.

$ echo "Why can't I write 's between single quotes"
Why can't I write 's between single quotes
$ echo 'Why can't I write 's between single quotes'
Why cant I write s between single quotes

  

5.2. Escaping

Escaping is a method of quoting single characters.
The escape (\) preceding a character tells the shell to interpret that character literally.
Special meanings of certain escaped characters:

• \n
means newline
• \r
means return
• \t
means tab
• \v
means vertical tab
• \a
means alert (beep or flash)
• \0xx
translates to the octal ASCII equivalent of 0nn, where nn is a string of digits.
", 34, 0x22

$ quote=$'\042'        # " assigned to a variable: 34 = 0x22 = 42 in octal decimal.
$ echo "$quote Quoted string $quote and this lies outside the quotes."
" Quoted string " and this lies outside the quotes.
  
  

6. Exit and Exit Status

The exit command terminates a script, just as in a C program. It can also return a value, which is available to the script's parent process.
A successful command returns a 0, while an unsuccessful one returns a non-zero value that usually can be interpreted as an error code. Well-behaved UNIX commands, programs, and utilities return a 0 exit code upon successful completion, though there are some exceptions.
Likewise, functions within a script and the script itself return an exit status.

• When a script ends with an exit that has no parameter, the exit status of the script is the exit status of the last command executed in the script
• $? reads the exit status of the last command executed.

7. Tests

7.1. Test Constructs

An if/then construct tests the condition.
The condition is a list of statements and the condition is the exit status of commands.

if [ 0 ] 
then
  echo "0 is true."
else 
  echo "0 is false."
fi 

When if and then are on same line in a condition test, a semicolon must terminate the if statement.
Not only the eit status of commands can be checked, you can ues the condition test command [ condition ] to check attributes of a file , comparsions of files o strings.:

• string comparisions
• str1 = str2
• str1 != str2
• str1 < str2
• str1 > str2
• -n str1
return True if str1 is not null
• -z str1
return True if str1 is null(length 0)
• file attribute checking
• -e
file exists
• -f
file is a regular file (not a directory or device file)
• -d
file is a directory
• -s
file exists and not zero size
• -b
file is a block device
• -c
file is a character device
• f1 -nt f2
file f1 is newer than f2
• f1 -ot f2
file f1 is older than f2
• integer comparison operators
• $a -eq $b
$a is equal to $b
• $a -ne $b
$a is not equal to $b
• $a -gt $b
$a is greater than $b
• $a -ge
$a is greater than or equal to $b
• $a -lt $b
$a is less than $b
• $a -le
$a is less than or equal to $b

  	a=1
    b=2
    if [ $a -gt $b ];then
    	echo "Big a"
    else
    	echo "Big b"
	fi
  

7.4. Nested if/then Condition Tests

7.5. Testing Your Knowledge of Tests

8. Operations and Related Topics

8.1. Operators

• variable assignment
=

• arithmetic operators
The shell arithmetic expansion can be performed using the double parentheses (( )) and $(( )) or with the let built-in command. The shell interprets words surrounded by (( )) as an arithmatic expressions.
Variables in arithmatic expressions do not need to be proceeded by $ sign.


i=12
echo $i
12
i=$((i+1))
echo $i  
13
((i=i+1))
echo $i  
14
	

• bitwise operators
• logical (boolean) operators

8.2. Numerical Constants

8.3. The Double-Parentheses Construct

8.4. Operator Precedence

Part 3. Beyond the Basics

9. Another Look at Variables 10. Manipulating Variables 11. Loops and Branches 12. Command Substitution 13. Arithmetic Expansion 14. Recess Time

Part 4. Commands

15. Internal Commands and Builtins 16. External Filters, Programs and Commands 17. System and Administrative Commands

Part 5. Advanced Topics

18. Regular Expressions 19. Here Documents 20. I/O Redirection 21. Subshells 22. Restricted Shells 23. Process Substitution 24. Functions 25. Aliases 26. List Constructs 27. Arrays 28. Indirect References 29. /dev and /proc 30. Network Programming 31. Of Zeros and Nulls 32. Debugging 33. Options 34. Gotchas 35. Scripting With Style 36. Miscellany 37. Bash, versions 2, 3, and 4

Sed

Sed is a non-interactive line editor. It

• receives text input, whether from stdin or from a file
• performs certain operations on specified lines of the input, one line at a time
• outputs the result to stdout or to a file

Sed determines which lines of its input that it will operate on from the address range passed to it. If no address range is specified, the default is all lines.
Specify this address range either by line number or by a pattern to match.
we will focus primarily on the three most commonly used ones.

Operator	Name	Effect
[address-range]/p	print	Print [specified address range]
[address-range]/d	delete	Delete [specified address range]
s/pattern1/pattern2/	substitute	Substitute pattern2 for first instance of pattern1 in a line
[address-range]/s/pattern1/pattern2/	substitute	Substitute pattern2 for first instance of pattern1 in a line, over address-range
[address-range]/y/pattern1/pattern2/	transform	replace any character in pattern1 with the corresponding character in pattern2, over address-range (equivalent of tr)
g	global	Operate on every pattern match within each matched line of input