------------------------------------------------------------------------------- General Shell Script Programming I have been working with shell scripts for more than 30 year, and started saving: hints, tips, and notes; about various aspects of shell programming in 1998, in this very file. Of course it has evolved well beyond what it looked like back then, but it was one of my most important resources. Many more complex aspects has since moved to a more dedicated file, starting with the very advance shell construct of File Handles, to open a file across multiple commands, and to redirect output. More specific things Option and Argument Handling https://antofthy.gitlab.io/info/shell/options.txt Variables and Array Processing https://antofthy.gitlab.io/info/shell/variables_n_arrays.txt Files and Filenames https://antofthy.gitlab.io/info/shell/file.txt File Handles (open files for longer periods) https://antofthy.gitlab.io/info/shell/file_handles.txt Other Resources 10 Resources for Learning Shell https://itsfoss.com/shell-scripting-resources/ Shell Tutorial Video for beginners... http://ontwik.com/linux/shell-programming-for-beginners/ Fast cheat sheet curl cheat.sh/bash/:learn | less ------------------------------------------------------------------------------- GLOB Glob glob matching x* will match files and output those starting with 'x', sorted [zyx]* will match files and output in sorted x,y,z order! {z,y,x}* will output strings in that specified order (unsorted) That is it is just iterated though the given sequence Though each sub-set is sorted In all cases if the glob does not match, the original string is returned echo no_file_* no_file_* Curly Braces Sequences (forward or backwards) echo {C..H} C D E F G H echo {H..C} H G F E D C Multiple Curly Braces (process order) echo {A,Z,B}{4..2} A4 A3 A2 Z4 Z3 Z2 B4 B3 B2 Repeated comma values can be used! echo {H,5,H} H 5 H WARNING: Sequences cannot mix numbers and letters echo {0..F} {0..F} WARNING: Nor can you use commas with a sequence echo {0..9,A} 0..9 A echo {0..9,A..F} # attempt to do a hexadecimal sequence 0..9 A..F seperating them works though :-( echo {0..9} {A..F} 0 1 2 3 4 5 6 7 8 9 A B C D E F Bash... Globs in conditional testing. Do not use quotes for the glob! name=abcd # glob match test [[ "$name" = a* ]] && echo true || echo false true [[ "$name" != a* ]] && echo true || echo false false name=xyz # glob mis-match test [[ "$name" = a* ]] && echo true || echo false false [[ "$name" != a* ]] && echo true || echo false true ------------------------------------------------------------------------------- Glob/Argument Sorting By Shells Essentually {..} preserves expansion order. While other Glob meta-chars sort the results. That is to say... {b,a}* becomes b* a* And thus generate two separated sets of arguments. WARNING: ls will sort its arguments a second time internally (unless GNU '-U' option is used). [...] will be sorted by the glob expandsion {..,..} will preserve the ordering of the expandsion EG compare the output of these commands echo b* a* # sorted b before a - word based echo [ba]* # sorted a and b together - word based echo {b,a}* # sorted b before a - word based ls -d1 b* a* # ls sorts its output - line based ls -d1 {b,a}* # ls sorts its output - line based ls -d1U {b,a}* # sorted b before a - line based ls -d1U * # the '*' sorted the arguments not 'ls' echo [aa]* # the a is only output once echo {a,a}* # but now a is output twice! This has been tested and works as described in csh, tcsh, bash, and zsh NOTE: bourne-sh, and 'dash' does not have '{}' functionality. ------------------------------------------------------------------------------- Sanitising Input.. Numerical value Bourne Shell built-ins only... case "$var" in '' | *[!0-9]*) echo "non-numeric" ;; *) echo "numeric" ;; esac Using BASH '[[' globs [[ $var != *[^0-9]* ]] || echo "non-integer or Empty" Using BASH '[[' regular expression... [[ $var =~ ^[^1-9][0-9]*$ ]] || echo "non-integer" Using egrep RegExp... echo "$arg" | egrep '^[1-9][0-9]*$' >/dev/null || echo "non-integer" Using expr. (fails for 'zero' input, or some strings). -- AVOID expr match "$var" '\([0-9]*\)$' || echo "not a non-zero-numeric value" IP Simplistic case "$var" in '' | *[!0-9.]*) echo "non-numeric" ;; *) echo "numeric" ;; esac Complex but complete... if [[ $ip =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then local ip IFS='.' ip=($ip) [[ 10#${ip[0]} -le 255 \ && 10#${ip[1]} -le 255 \ && 10#${ip[2]} -le 255 \ && 10#${ip[3]} -le 255 ]] stat=$? fi return $stat Third party solution... ipcalc -cs 10.10.10.257 && echo vaild || echo invalid ------------------------------------------------------------------------------- Integer Shell Mathematics Also see "apps/printf.txt" especially with regards to handling numbers that may have leading zeros (which BASH treats as octal). EG: printf "%d\n" 013 # => 11 printf "%d\n" $((10#013)) # => 13 --- EXPR.... This is the traditional (bourne shell) way to do integer maths all elements must be separate arguments, and special characters like '*' (for multiply) must be escaped. However it is not very secure. expr + 5 / 2 2 --- BASH... In BASH you can use some special constructs (no $ variable prefix needed) Specifically ((...)) for conditionals (EG: exit code testing) $((...)) for variable substitution. let var= for assignment NOTE: Do not confuse this with $(...) for command substitution WARNING: $[...] is an older math expression construct and is depreciated due to its confusion with [...] and [[...]] string/file handling conditionals. a=1 if (( a == 0 )); then echo true; else echo false; fi a=5 b=2 echo $(( a/b )) 2 Using a built-in integer maths is many orders of magnitude faster. BUT as BASH and not compatible with older Bourne shells (on older machines) The "expr" command is not a built-in, and has some problems (see below). time for I in $(seq 1 1000); do A=$((1+1)); done real 0m0.009s time for I in $(seq 1 1000); do expr 1 + 1 > /dev/null; done real 0m1.330s --- KSH... The [[...]] was introduced by ksh88 as a [...] replacement and should be used for string comparisons, while ((...)) are for numerical comparisons. You can still use [...] for the older traditional test command in ksh88 BASH inherited this convention form KSH. ------------------------------------------------------------------------------- Floating Point Math.... Shell can't do floating point, but many other commands can... Also see "apps/printf.txt" especially with regards to handling numbers that may have leading zeros (which BASH and other languages treat as octal). See also my "math" script. BC echo 5/2 | bc # => 2 BC rounds to intergers by default... BC Scaling will add decimal places (fixed point integer math) echo 'scale=5; 5/2' | bc # => 2.50000 # math function using bc math() { echo "scale=5; $1" | bc; } math '5/2' # => 2.50000 BC with a -l flag (true floating point and math functions) This also enables scaling precision (scale=20) echo 5/2 | bc -l # => 2.50000000000000000000 # a() is arctan so a(1)*4 => PI echo 'a(1)*4' | bc -l # => 3.14159265358979323844 --- ImageMagick... It is a rather long winded, but if you are already using it! It also syntax checks the arguments so only a quote taint check is needed. This makes it safer than some of the more general ones later. convert null: -precision 10 -format '%[fx:atan(1,1)*4]' info: # => 3.141592654 # math function using imagemagick math() { convert xc: -print "%[fx: $1 ]\n" null:; } math '5/2' # => 2.5 --- Awk / Perl WARNING: argument should be taint checked for safety, as the string substitution may allow it to be easily 'hacked'. math() { awk 'END{ printf "%.5g\n", '"$1"' }' /dev/null; } math "5/2" # => 2.5 math() { perl -e 'printf "%.5g\n", '"$1"; } math "5/2" # => 2.5 ------------------------------------------------------------------------------- Verbose Log Handling # Set a flag (see variables above) verbose=verbose_true # direct test of variable then echo [ "$verbose" ] && echo "I am being verbose" # log() functional doing the verbose test log() { [ "$verbose" ] && echo "$@"; } log "I am being verbose" # Remove repeated 'if' testing, at function definition # But verbose must be set first, and can not be changed later if [ "$verbose" ]; then log() { echo "$@"; } else log() { :; } fi log "I am being verbose" # Colapsing Function (function redefines itself on first call) # The "$verbose" variable can now be defined LATER, # as long as it is done before its first use. log() { if [ "$verbose" ]; then log() { echo "$@"; } else log() { :; } fi log "$@" } log "I am being verbose" ------------------------------------------------------------------------------- Inverted TR compatibility problems "tr" is used to translate, or delete characters from piped strings WARNING: The command acts very differently with regards [...] and character ranges between older solaris and linux. On Solaris 5.8, tr -cd '[a-z]' deletes all characters except abcdefghijklmnopqrstuvwxyz tr -cd 'a-z' deletes all characters except a, z, and hyphen. On Linux (gnu), tr -cd '[a-z]' deletes all characters except a to z and brackets tr -cd 'a-z' deletes all characters except abcdefghijklmnopqrstuvwxyz ------------------------------------------------------------------------------- Command search (bash) 1/ REDIRECTION is done (before anything else!) 2/ simple alias expandsion 3/ Parameter expansion, command substitution, arithmetic expansion, and quote removal, variable assignment performed if '=' found. 4/ functions exectuted 5/ shell built-in commands 6/ lookup command in hash table (error hash lookup does not exists) 7/ search the command PATH variable 8/ else error "Command not found" ------------------------------------------------------------------------------- One line if-then-else shell command cmd1 && cmd2 || cmd3 This however will execute cmd3 if EITHER cmd 1 or cmd2 fails! If cmd2 never fails (IE "echo" ) - no problems This means quick status check can be written as command && echo TRUE || echo FALSE ------------------------------------------------------------------------------- Results from commands a=`command` a=$(command) Quotes are not needed, unless appending other things, but does not hurt Split result into words... read -r a b c < <( echo one two three ) echo "a=\"$a\" b=\"$b\" c=\"$c\"" This is Equivelent to the use of a pipline using a named pipe (bourne sh) mknod -p cmd_pipe echo one two three > cmd_pipe & read -r a b c < cmd_pipe rm -f cmd_pipe echo "a=\"$a\" b=\"$b\" c=\"$c\"" Spilt into an array... You can also directly read into a bash array (sequential)... array=( $( echo one two three ) ) echo "${array[@]}" Using Read... NOTE: This only reads first line, and junks the rest of the output... That itself can be useful, for sorted output... # get count and directory with highest inode count (number of files) read count directory < <(du -xs --inodes /app/* | sort -nr ) echo $count $directory Split by column (read comma separated CSV file) 4th and later column will be ignored in 'd' This will fail with regard to quoted CSV elements! OIFS=$IFS IFS=',' while read a b c d do echo "LOGIN:$a LASTNAME:$b FIRSTNAME:$c" done < fileA.csv Read results of multiple lines { read -r line1 read -r line2 read -r line3 } < <( command ) For array of lines (preserving leading space)... # Using mapfile (-t to remove newlines) mapfile -t arr <(your command) # unset arr i while read -r; do arr[i++]=$REPLY done < <(your command) # Now get the final line (if it has no final newline, though it may be empty) [[ $REPLY ]] && arr[i++]=$REPLY NOTE: "read" will return FALSE, if the last line did not end in a newline See "info/shell/input_reading.txt" ------------------------------------------------------------------------------- Break string into words... Bourne shell "set" method with IFS save. (Only shell built-ins!) WARNING: this replaces script/function args. (Dangerious) (NOTE: no quote in the 'set' - making it very dangerious security-wise) line="one two three" set -- $line a="$1" b="$2" c="$3" echo "a=\"$a\" b=\"$b\" c=\"$c\"" # -> a="one" b="two" c="three" Using IFS to change word separator character line="one:two:three" OIFS=$IFS; IFS=":"; set -- $line; IFS="$OIFS" a="$1" b="$2" c="$3" echo "a=\"$a\" b=\"$b\" c=\"$c\"" # -> a="one" b="two" c="three" sed/awk/grep -- to separate variable assignments Very slow - but can extract values from very complex lines in any order. Basically you read ALL the input into a variable then extract what you want. line="one:two:three" a=`echo "$line" | awk -F: '{print $1}'` b=`echo "$line" | awk -F: '{print $2}'` c=`echo "$line" | awk -F: '{print $3}'` echo "a=\"$a\" b=\"$b\" c=\"$c\"" # -> a="one" b="two" c="three" OR line="one:two:three" a=`echo "$line" | sed 's/:.*//'` b=`echo "$line" | sed 's/.*:\(.*\):.*/\1/'` c=`echo "$line" | sed 's/.*://'` echo "a=\"$a\" b=\"$b\" c=\"$c\"" # -> a="one" b="two" c="three" Bash 'here string' reading (Safer - Bash built-ins only) NOTE: The "$var" is passed as stdin, with newline appended. The "read" command will then remove that newline as a delimiter. var="one two three" read -r a b c <<< "$var" echo "a=\"$a\" b=\"$b\" c=\"$c\"" # -> a="one" b="two" c="three" --- With field separators, including blank fields... IFS=":" read -r a b c <<< "hello::world"; echo "a=\"$a\" b=\"$b\" c=\"$c\"" # -> a="hello" b="" c="world" or you can pre-process the line so you can skip fields at start, middle, or end read -r a b c <<< $(echo "::hello::world::" | tr ':' ' ') echo "a=\"$a\" b=\"$b\" c=\"$c\"" # a="hello" b="world" c="" The last can be simplified to read command output directly instead of as a string... read -r a b c < <( echo "::hello::world::" | tr ':' ' ' ) echo "a=\"$a\" b=\"$b\" c=\"$c\"" # a="hello" b="world" c="" WARNING: IFS normally allows multiple spaces between fields (whitespace). But if it is not the whitespace default setting, then the fields are assumed to be separated by exactly one character. ------------------------------------------------------------------------------- builtin cat command This cat only uses the shell builtins! As such can be used on a machine which has no access to shared libraries and nothing but the statically linked bourne sh can run. shcat() { while test $# -ge 1; do while read -r i; do echo "$i" done < "$1" shift done } Of course the real cat command is only as big as it is, to protect the user from himself and to provide a huge number of options. PS: If the "ls" command is also not available then you can use echo * For directory listings, using only builtins. Though this will not tell you if files are executables or sub-directories, or handle problems like spaces in filenames, or unprintable characters. # bulti-in ls example? ------------------------------------------------------------------------------- The useless use of 'cat'! You often see in shell scripts... cat input_file | some_command The cat is useless as it is exactly the same as some_command < input_file Or even < input_file some_command Without needing to fork the extra "cat" process or creating a pipeline. However it is sometimes useful to use cat anyway to make the code more readable. Particularly in long pipelines, or where "some_command" is part of large while loops or if statements. WARNING: "some_command" will be executed as a sub-shell in the first instance, but is part of the main-shell in the second instance. This is important for shell built-ins that set variables. ------------------------------------------------------------------------------- Non-POSIX shell bugs... foo=non-postix while read line; do foo=postix done < /etc/passwd echo "foo = $foo" You should get the output "foo = postix". But some bad (old) shell implementations redirecting a file into a control structure will use a sub-shell for that structure. As such in many old shells you will get a output of "foo = non-postix" The POSTIX standard forbids this behaviour. ---- Piped Loop Shell Bug HOWEVER: Pipelines into sub-shells IS still a problem!!! foo=nothing echo "testing 1 2 3" | while read line; do foo=something done echo "foo = $foo" # -> foo = nothing Note $foo was not assigned "something" as the 'while' was in a sub-shell. One solution is to use 'named pipes' technique such as A POSIX shell solution is to use a named pipe mkfifo mypipe foo=nothing echo "testing 1 2 3" > mypipe & while read line; do foo=something done < mypipe echo "foo = $foo" rm mypipe Bash can also use builtin named pipes NOTE: <(...) is a file name and still needs a extra < to pipe it in foo=nothing while read line; do foo=something done < <(echo "testing 1 2 3") echo "foo = $foo" # -> foo = something See http://mywiki.wooledge.org/BashFAQ/024 See also co-processes.hints http://www.imagemagick.org/Usage/info/shell/co-processes.hints on bi-directional I/O to a background process, daemon, or internet service. ------------------------------------------------------------------------------- Inserting a complex command line inside a shell script... For example embedding, sed, awk, or perl scripts, into a shell script. The normal way is to use Single Quotes around the command line script. For Example... awk '# add prefix to lines { print "prefix> " $0 } ' list.txt Note the whole script is inside single quotes on the awk command line! ASIDE: Old versions of awk must have something on the first line thus the addition of the # comment to keep it happy! Perl needs no such comment but does require a -e option to execute a command line argument. CAUTION: Watch for single quotes inside COMMENTS in the script code block! The comments are within the single quotes so are scanned for those quotes. Environment Variables Both "awk" and "perl" can read environment variables for data passing. export column=2 awk 'BEGIN { column = ENVIRON["column"] } /^[^#]/ { print $column } ' file Insert a Shell variable in quoted code... To include an shell variable, you need to switch quote modes just for that variable... file="list.txt" awk '# add prefix to lines { print "'"$file"'> " $0 } ' "$file" For awk you can declare the variable on the command line, file="list.txt" awk -v file="$file" '# awk code { print file "> " $0 } ' "$file" OR put assign variable after the code, before reading the file file="list.txt" awk '# awk code { print file "> " $0 } ' file="$file" "$file" Note the latter may or may not have the 'file' variable available inside a "BEGIN {}" block! ---- Single Quote Insertions... There are a number of ways... awk '# print single quotes BEGIN { print "'\''" print "\047" q="\047"; print "Can or can" q "t?" exit }' The first print in the above works for any command line script. The rest are "awk" (or whatever) dependant. You can also use the special $'...' form of single quote delimited strings which is specific BASH and KSH, (ansi-c style quoting). That is force all backslashed escapes to be performed. EG: All backslashes are handled as be the ANSI-C compilion of strings. EG: \n \t \' \" all handled. echo $' single_quote=\' double_quote=\" ' Alternative Method for code blocks in shell scripts. Perl has the added advantage that you can pipe the command to execute into it. For example using a 'herefile' perl <<'EOF' print "Hello world from perl in a here document.\n"; EOF Of course you have less control over the escaping and external variables and values in the script. Also see HERE files in "info/shell/file.hints" CSH SCRIPTS: If you must do this in a csh script (don't do it!), you will need to escape (backslash) the new line at the end of every line, even though single quotes are being used. Also watch out for history escapes '!' which in csh work inside single quotes! ----------- Warning: Some precautions are necessary with the contents of the variable, as the inserted value will be parsed by the command (awk/sed/perl/etc..). In perl characters than may need special handling include $ @ " ' \ and so on. A better method may be to 'pipe' the variable into the command For example this works... encode() { # WWW url encoding (typically needed for passwords) echo -n "$1" | perl -e '$_ = <>; s/[^\w.]/sprintf "%%%02x",ord $&/eg; print;' } The more direct method (below) will fail on characters such as '@' or '$' or '"' in the inserted variable (perl quoting rules) encode() { # WWW url encoding (typically needed for passwords) perl -e '$_ = "'"$1"'"; s/[^\w.]/sprintf "%%%02x",ord $&/eg; print;' } This is especially important for 'tainted' input data. ------------------------------------------------------------------------------- Is COMMAND available There is two techniques available to test if a command is available. The first is to use the `status' return of the "type" or "which" command of the shell you are using. The Second is to examine the output of that command itself. Using status return. This is a simple method of testing the existence of a command. But DOES NOT WORK ON ALL SYSTEMS! The problem is that old shells (For Example: SunOS Bourne sh) always returns a true status weather the command is present or not! Bourne Shell if type COMMAND >/dev/null 2>&1; then # COMMAND is available fi C-Shell Warning: The "which" command in C shell is not a builtin but a external script which sources your .cshrc (YUCK). As such the Bourne shell alias is preferred which will only search your current command PATH. if ( ! $?tcsh ) then alias which 'sh -c "type \!:1 2>&1"' endif if ( -x "`which COMMAND >/dev/null`" ) then # COMMAND Available endif TC-Shell Tcsh 6.06 also does not return the correct status in its which command. Use it like the csh which above. WARNING: this method will also test positive for :- subroutines, bash aliases, and probably other non-command definitions. Examine output The other am more reliable method is to examine the output of the "type" or "which" command to looks for the string "not found" (See below). This is more complex than the above status check but should work on ALL unix machines regardless of age. Bourne Shell cmd_found() { case "`type "$1" 2>&1`" in *'not found'*) return 1 ;; esac; return 0 } # ... if cmd_found COMMAND; then # COMMAND is available fi # WARNING: Do not use... cmd_found COMMAND && COMMAND args & # As it leaves behind a waiting sub-shell BASH # As Bourne shell, OR cmd_found() { type -t "$1" >/dev/null; } # OR cmd_found() { command -v "$1" >/dev/null; } C-Shell & Tcsh (See notes below) if ( ! $?tcsh ) then alias which 'sh -c "type \!:1 2>&1"' endif ... if ( "`which less`" !~ *'not found'* ) then # COMMAND Available endif NOTES for "which/type" commands :- The above methods look for the specific string "not found" This is important as the sh type command and tcsh which command produce different output, and this may also vary from bourne shell to bourne shell or other shell types. Csh -- "which" is an unreliable shell script! fudge it into a shell script "type" command. See the "Which Problem" below. Tcsh > which less /opt/bin/less > which junk junk: Command not found. > which which which: shell built-in command. > alias a alias > which a a: aliased to alias Solaris Bourne shell $ type less less is /opt/bin/less $ type junk junk not found $ type type type is a shell builtin $ func(){ echo ha; } $ type func func is a function func(){ echo ha } Solaris Ksh As per Sh, but the actual function definition is NOT listed BASH $ type less less is /opt/bin/less $ type junk bash: type: junk: not found $ type type type is a shell builtin $ func(){ echo ha; } $ type func func is a function func () { echo ha } NOTE: the type returns a status on if command is found (even as a function) BASH also has a type -t which responds with a single word "file", "alias", "function", "builtin", "keyword", or nothing if command does not exist. A -p will print the disk file name, or nothing. A -a prints all the places that have that name. BASH also has a different builtin that is simular... $ command -v less /usr/bin/less From the results above, only the appearence of "not found" for a false result is consistant across all shells. Anything else is true. This is especially important if you want shell functions to report true. The expanded Bourne shell form, without using the "cmd_found" function is as follows, But is is a low simpler and easier to read if you use the funtion. If command present if expr + "`type COMMAND 2>&1`" : '.*not found' == 0 >/dev/null; then # COMMAND is available fi and its inverse (not present) if expr + "`type COMMAND 2>&1`" : '.*not found' >/dev/null; then # Comand is NOT present fi Finally only using built-in commands... case "`type COMMAND`" in *'not found'*) # Command not found ;; *) # Command found ;; esac Functional forms cmd_found() { expr + "`type $1 2>&1`" : '.*not found' == 0 >/dev/null } OR cmd_found() { case "`type $1 2>&1`" in *'not found'*) return 1 ;; esac; return 0 } Final form (for all bourne-like shells) =======8<-------- # Is the command available? case `type cmd_found 2>&1` in *'not found'*) cmd_found() { case "`type "$1" 2>&1`" in *'not found'*) return 1 ;; esac; return 0 } ;; esac =======8<-------- Bash equivelent cmd_found() { type -t "$1" &>/dev/null; } declare -fx cmd_found # export this function! The latter lets you export the function to sub-shells and scripts ------------------------------------------------------------------------------- CSH Which problem! On SunOS: The "which" command is a csh script that specifically reads the ".cshrc" file to find out about aliases. To avoid having ".cshrc" do lots of odd things to your script, run it with a HOME set... set program = `/bin/env HOME= /usr/ucb/which $program` This is NOT a problem in Tcsh, where "which" is a built-in. The best solution it to replace "which" with the bourne shell "type" command. For example to do this to Csh but not Tcsh... if ( ! $?tcsh ) then alias which 'sh -c "type \!:1 2>&1" | sed "s/.* is //"' endif set program = `which $program` ------------------------------------------------------------------------------- echo without a return # Create no-return echo function... if [ "x`echo -n`" = "x-n" ]; then echo_n() { echo ${1+"$@"}"\c"; } else echo_n() { echo -n ${1+"$@"}; } fi For Csh or Tcsh (which you shouldn't use for scripts anyway) # echo without a return (bsd & sysV) if ( "x`echo -n`" == 'x-n' ) then alias echo_n 'echo \!:* "\c"' else alias echo_n 'echo -n \!:* ' endif For all shells, but uses non-builtin commands echo '----- Press return to continue. ' | tr -d '\012' ALTERNATIVE.. Don't use echo... Use "printf" which is either a builtin (bash) or in /bin/printf printf 'press return to continue: ' WARNING: if a username or password is being printed, you must ensure you are using a built-in. The builtin also lets you assign the output to a variable! ------------------------------------------------------------------------------- Looping and Generation of a list of numbers Bash allows a C-like for loop... for (( i=10; i; i-- )); do echo $i; done Bash Brace Expandsion (versions > 3.2) echo {5..15} # forward can be negative numebrs # => 5 6 7 8 9 10 11 12 13 14 15 echo {15..5} # reverse order # => 15 14 13 12 11 10 9 8 7 6 5 echo {5..15..2} # increment supported in BASH v4.0+ # => 5 7 9 11 13 15 echo {x..z} # single letters (only single letters) # => x y z echo {c..a}{A..C} # or letter sequence (multiple expand) # => cA cB cC bA bB bC aA aB aC echo {1..3}{,} # duplicate the output # => 1 1 2 2 3 3 echo "_oOo_"{,,,} # a string four times! # => _oOo_ _oOo_ _oOo_ _oOo_ echo {{1..5},{23..27}} # nested brace expandsions # => 1 2 3 4 5 23 24 25 26 27 Formatted. echo {0007..11} # => 0007 0008 0009 0010 0011 printf "__%d__" {1..5}; echo "" # printf auto-loops multiple args # => __1____2____3____4____5__ WARNING: you cannot use variables in this syntax! This for example it will NOT work ... {1..$limit} limit=5 echo {1..$limit} Using eval substitution will work (painful though it is)... echo $(eval echo {1..${MAX_SEGMENTS}}) You better of using a "for( ; ; )" loop! seq The Gnu "seq" command is the simplest and most wide spread... Args: {start} [[{incr}] {end}] Note that {incr} is in middle which is logical but can be awkward. seq 10 seq 5 10 seq 10 -1 1 seq 1 .1 2 seq -f %03g 5 5 100 To remove newlines use... echo $(seq 10) # => 1 2 3 4 5 6 7 8 9 10 As you can see it can go forwards, from any range, with any increment, even backwards, or using factions (floats), and with format control. WARNING: "seq" may not be available under MacOSX :-( See 'seq' replacement functions below multi_seq (personal script) handles multiple ranges of numbers ISO dates multi_seq "%d-%02d-%02d" 2010,2011 12 31 Human dates multi_seq -f "%02d/%02d/%d" 31 12 2010,2011 DIY seq() replacements # seq_count {end} # seq_count() { i=1; while [ $i -le $1 ]; do echo $i; i=`expr $i + 1`; done } # simple_seq {start} {incr} {end} simple_seq() { i=$1; while [ $i -le $3 ]; do echo $i; i=`expr $i + $2`; done } # seq_integer [-f {format}] [{start} [{incr}]] {end} # WARNING: Only positive increments at this time, no error checking seq_integer() { if [ "x$1" = "x-f" ] then format="$2"; shift; shift else format="%d" fi case $# in 1) i=1 inc=1 end=$1 ;; 2) i=$1 inc=1 end=$2 ;; *) i=$1 inc=$2 end=$3 ;; esac while [ $i -le $end ]; do printf "$format\n" $i; i=`expr $i + $inc`; done } See also "Columns of Data" (below) for more examples... ---- Other techniques... Awk awk "BEGIN {for (i=1;i<$COUNT;i++) {printf(\"%03d\n\",i)} }" Perl perl -le 'for(1..9){print}' Dev Zero -into-> newlines -into-> cat line counting: dd | tr | cat This reads the right number of 'NUL' chars from /dev/zero, then converts that return characters, which cat numbers. Finally any spaces are removed. use dd and cat to generate a list of numbers dd 2>/dev/null if=/dev/zero bs=10 count=1 | tr \\0 \\012 |\ cat -n | tr -d '\40\11' shuf Will randomly shuffle numbers (and much more) seq 9 | shuf sort sort will do this too... seq 9 | sort -R jot It seem "jot" can also be used, but it is not a standard install. The first number by default is the total number of numbers wanted spread out equally accross the range given (even floating point) jot 3 # 1 2 3 jot 2 12 # 12 13 jot 3 12 16 # 12 14 16 jot 4 12 17.0 # 12.0 13.7 15.3 17.0 jot -p4 4 12 17.0 # 12.0000 13.6667 15.3333 17.0000 Using '-' will let it work more like "seq" using args: {start} {end} {incr} jot - 2 10 2 # 2 4 6 8 10 It can create random numbers in range jot -r 5 1000 9999 # 3435 1811 2641 7472 2528 or repeat a specific word or string jot -b word 3 # word word word See also random number generation... "info/shell/random.txt" ------------------------------------------------------------------------------ Arrays of Numbers You can make an array say A-1 to C-3 easilly in bash using echo {A..C}-{1..3} A-1 A-2 A-3 B-1 B-2 B-3 C-1 C-2 C-3 Or using "multi-seq" multi_seq "%X_%d" 0xa,0xc 3 | tr '\n' ' '; echo A_1 A_2 A_3 B_1 B_2 B_3 C_1 C_2 C_3 You can even add prefix or postfix strings echo s {A..C}-{1..3} e s A-1 A-2 A-3 B-1 B-2 B-3 C-1 C-2 C-3 e But what if you want a prefix/postfix string around each row that gets a lot more complicated... Lets make this a bit clearer, so as to make working it out easier echo Array \ ' s' A-{1-3} 'E ' \ ' s' B-{1-3} 'E ' \ ' s' C-{1-3} 'E ' \ End Array s A-{1-3} e s B-{1-3} e s C-{1-3} e End So we need to reproduce that by merging the lines echo Array \ '" s" '{A..C}'_{1..3} "e "' \ End Array " s" A_{1..3} "e " " s" B_{1..3} "e " " s" C_{1..3} "e " End And evaluate it to remove one layer of quotes. eval echo Array \ '" s" '{A..C}'_{1..3} "e "' \ End Array s A_1 A_2 A_3 e s B_1 B_2 B_3 e s C_1 C_2 C_3 e End Basically it is messy... Better to use a verbose looped solution. ------------------------------------------------------------------------------- Columns of Data No ideal solution has been found... The "ls" command is close to an ideal columnised data. Vertically ordered with variable column size. But it does not provide any control (other than disable) NOTE: if all elements are of equal length you can also use paragraph word wrapping and formatting techniques (see below). column Vertical ordered (TAB seperated) to fit a screen width. By default width is from 'COLUMNS' environment, tty width. No easy way control the final number of columns WARNING: Any meta-characters found may cause column silently ABORT, truncating the file being processed! Normal vertical columns... seq 13 | column -c40 1 4 7 10 13 2 5 8 11 3 6 9 12 Or across the page... seq 13 | column -x -c40 1 2 3 4 5 6 7 8 9 10 11 12 13 Using tablulat mode will get rid of the tabs seq 13 | column -c40 | column -t 1 4 7 10 13 2 5 8 11 3 6 9 12 Limitations... You can NOT specify an output seperator (only TAB) Unless you use -t table mode, and adjust -c for the data. (see last example) However this can be awkward to control the number of columns! due to the number of rows needed... seq 30 | column -c63 | column -o' ' -t # 6 columns 1 6 11 16 21 26 2 7 12 17 22 27 3 8 13 18 23 28 4 9 14 19 24 29 5 10 15 20 25 30 seq 30 | column -c64 | column -o' ' -t # 8 columns 1 5 9 13 17 21 25 29 2 6 10 14 18 22 26 30 3 7 11 15 19 23 27 4 8 12 16 20 24 28 Table Usage (what is it made for)... This can also format text tables using "-t" function of column to preserve the line by line column structure of the input file. In this mode you can specify both input and output field separators. column -s: -t -o ' ' /etc/group OR a more complex example... ( echo "PERMS LINKS OWNER GROUP SIZE MON DAY HH:MM FILENAME"; \ ls -l | tail -n+2; \ ) | column -t Columns are equal width, unless in table mode. Compare outputs of... ls ls | column ls | column | column -t paste simple, across ordered, TAB separated, Number of - arguments given determines number of columns (tab separated). But result is NOT vertically aligned if elements are longer than a TAB. seq 14 | paste - - - - 1 2 3 4 5 6 7 8 9 10 11 12 13 14 You can use this with table mode of "column" to fix the columns and allow adjustment of column seperators. seq 15 | paste - - - - | column -t 1 2 3 4 5 6 7 8 9 10 11 12 13 14 datamash (non-standard package) Can be used to reorganise columns of "paste" Reverse columns seq 7 | paste - - | datamash reverse 2 1 4 3 6 5 7 Transpose Columns seq 7 | paste - - | datamash transpose 1 3 5 7 2 4 6 pr To convert a file (or part of a file) to VERTICAL ordered columns. Warning: This program will truncate data to fit into space available! UPDATE: number of lines now auto-adjusts! A -t is needed to remove header and 66 line paging seq 14 | pr -t -3 -w30 1 5 9 12 2 6 10 13 3 7 11 14 4 8 Reduced line count... NOTE the way the column breaks and page reset seq 14 | pr -t -3 -l3 -w30 1 4 7 2 5 8 3 6 9 10 12 14 11 13 For across order, use a line count of 1 seq 14 | pr -t -4 -l1 -w30 1 2 3 4 5 6 7 8 9 10 11 12 13 14 xargs simple, across ordered, unformatted (ancient) seq 13 | xargs -n 3 echo 1 2 3 4 5 6 7 8 9 10 11 12 13 Feed to column -t to re-align columns seq 13 | xargs -n 3 echo | column -t 1 2 3 4 5 6 7 8 9 10 11 12 13 You can also wrapped the output with start and end wrappings seq 13 | xargs -n 3 sh -c 'echo "(" "$@" ")";' sh ( 1 2 3 ) ( 4 5 6 ) ( 7 8 9 ) ( 10 11 12 ) ( 13 ) perl Array::PrintCols NON-STANDARD perl module (from CPAN) seq -f %03g 100 |\ perl -MArray::PrintCols -e '@a=<>; chomp @a; print_cols \@a' perl DIY Otherwise manually handle columns in perl This reads all the data in first, but does not examine data widths. Outputs in row order, and is very manual Example: seq 13 |\ perl -e ' @a=<>; chomp @a; my $columns = 4; my $i = $columns; foreach ( @a ) { print("\n"),$i=$columns unless $i; printf " %4s", $_; $i--; } print "\n"; ' 1 2 3 4 5 6 7 8 9 10 11 12 13 FUTURE: example of vertical columns ------------------------------------------------------------------------------- Column Extraction cut Extract columns from a file in terms of character separated fields or by character position. Does not handle 'white space separated' cut -f3 - colrm Remove unwanted columns of text from the file, in terms of character positions. "cut" and "colrm" are the inverse of each other. join Extract white space seperated fields from one file For example 3rd field (from stdin, file2 is /dev/null) join -a1 -o1.3 - /dev/null awk Extract columns either by whitespace or character separators. can also process the feilds in other ways. awk '{print $3}' - perl Can basically handle anything. (just need to work it out) perl -peF 'print $F[2],"\n";' You can also use "vim"s visual blocks (Ctrl-V) ------------------------------------------------------------------------------- Merge data using a column of keys input_1 input_2 X A M A Y B N B join -j 2 input_1 input_2 ------------------------------------------------------------------------------- Remove all blank lines This is simple... But do you want to remove: empty lines, or lines with only white space Vim :%g/^\s*$/d perl -ne 'print unless /^$/' perl -lne 'print if length' perl -ne 'print unless /^\s*$/' perl -ne 'print if /\S/' # keep any line with a non-space character ------------------------------------------------------------------------------- Compress multiple blank lines into one line. NOTE: You may need to first remove white space on otherwise blank lines. Typically removing all trailing whitespace fromi all lines will do this. This is not easy, as we want to preserve a blank line, but remove extras. Two basic techniques: print paragraphs, or delete extra blanks. cat -s # GNU version, will do this (if available) perl -00pe0 perl -ne 'if (/\S/) { print; $i=0 } else {print unless $i; $i=1; }' perl -ne 'print if /\S/../^\s*$/' awk '{ printf "%s ", $0 } NF == 0 { print "\n" }' filename sed '/./,/^$/!d' # no blank lines at top, one at bottom sed '/^$/N;/\n$/D' # one blank at top, no blank lines at bottom sed '/[^ \t]/,/^[ \t]*$/!d' # For blank lines with spaces and tabs # This one keeps one blank line at top and bottom if they are present sed ':x;/^\n*$/{N;bx;};s/^\(\n\)*/\1/' # Line joining in the vim editor! as a macro :map QE :$s/$/\\rZ/:g/^[ ]*$/,/[^ ]/-jGdd or :%s/\n\s*\n\(\s*\n\)*/\r\r/ See also https://www.gnu.org/software/sed/manual/html_node/cat-_002ds.html ------------------------------------------------------------------------------- Multi-line Paragraph to Single-line Paragraph Most text files including this one consists of paragraphs of multiple lines seperated by a blank line. This converts to single-line paragraphs Convert all paragraphs into a single line (blank line seperated) sed '/^$/d; :loop y/\n/ /; N; /\n$/! b loop; s/ */ /g; s/^ //; s/ $//' Also remove blank lines between paragraph lines sed '/^$/d; :loop N; s/\n$//; T loop; y/\n/ /; s/ */ /g; s/^ //; s/ $//' WARNING: Better space handling is probably needed, especially for a last paragraph that has no final blank line after it. ------------------------------------------------------------------------------- Limit Line Length (replace with ...) sed -u 's/\(.\{77\}\).*/\1.../' ------------------------------------------------------------------------------- One word per line This is typically used for pre-processing for text formatting (see next) or as part of a word-diff type utility (see: "info/apps/word-diff"). cat file | sed 's/[^[:alnum:][:space:]]/\\&/g' | xargs -rn1 echo The 'sed' in the above is to quote non-alphabetrical characters. To remove punctuation too, delete the '\\&' form the above Note paragraph blank lines are also removed. ASIDE: Removing '[:space:]' is the accepted way of handling return delimited line input into "xargs" safely. The better way is to use -0 filename listing format. ------------------------------------------------------------------------------- Word-Wrapping or Text Formatting Single-line Paragraphs to Multi-line Paragraphs WARNING: There are two parts to doing this, spliting long lines and join short lines. vim has built in formating... gg}gqGgg} Usually you would use "fmt" or "fold" which is a standard addition for LINUX systems. fmt -u -w 72 NOTE: fmt preserves the indentation of the lines... fold -s -w 72 [file...] NOTE: fold does not do the join operation on multi-line paragraphs! The solution is to convert paragraphs to single-lines first. expand < file | sed '/^$/d; :loop y/\n/ /; N; /\n$/! b loop; s/ */ /g; s/^ //' | fold -s -w 72 This replaces fold with a poor mans word-wrap... It first joins all lines, then adds markers at line breaks, and finally used "tr" to split lines AFTER 55 characters. expand < file | sed '/^$/d; :loop y/\n/ /; N; /\n$/! b loop; s/ */ /g; s/^ //' | sed 's/ $/@/; s/[^@]\{55\} /&@/g' | tr '@' '\012' | sed 's/^/ /' Note the second sed command wraps on a space AFTER '55' characters. As such it can be fooled by very very long words. True word wrap, splits on the space before the given column limit. In "gnu-sed" you can replace '@' with '\x01' and in "tr" use '\001' so as to avoid matching existing substitution characters in the text. A variation to prevent matching existing characters, is to first remove all extra spaces including those at the start and end of line, then treat double-spaces as the paragraph break. expand < file | sed 's/ */ /g; s/^ //; s/ $//;' | tr '\012' ' ' | sed 's/ $/@/; s/ */@@/g; s/[^@]\{55\} /&@/g' | tr '@' '\012' Perl Text Format (using the standard Text::Wrap module) #!/bin/perl use Text::Wrap qw(&wrap $columns, $huge); $columns=72; $huge='overflow'; ($/, $\) = ( '', "\n\n"); # read and output mutli-line paragraphs while(<>) { s/\s*\n\s*/ /g; # remove newlines and indents print wrap('', '', $_); # format paragraph } Using xargs! The number is column limit + 5 for the "echo " NOTE: This only handles a single paragraph, but does BOTH the joining and spliting of the lines. expand < file | sed 's/[^[:alnum:][:space:]]/\\&/g' | xargs -r -s 85 echo It can also be used in perl, and indentation is easilly added. NB: -r (no run if empty) is not available under solaris "xargs". system("echo '@list' | xargs -s 70 echo ' ' "); ------------------------------------------------------------------------------- Interleave Lines a b c 1 2 3 -> a 1 b 2 c 3 paste (file A with file B) paste -d\\n a b vim mark the blocks in some way.. For example prefix with 'A' and 'B' :'a,'bg/^A/ /^B/m . More complex.. https://vi.stackexchange.com/questions/4575/merge-blocks-by-interleaving-lines function! Interleave() " retrieve last selected area position and size let start = line(".") execute "normal! gvo\" let end = line(".") let [start, end] = sort([start, end], "n") let size = (end - start + 1) / 2 " and interleave! for i in range(size - 1) execute (start + size + i). 'm' .(start + 2 * i) endfor endfunction " now select your two continous same lenght blocks and call function :call Interleave() Or with line ranges function! Interleave(start, end, where) if a:start < a:where for i in range(0, a:end - a:start) execute a:start . 'm' . (a:where + i) endfor else for i in range(a:end - a:start, 0, -1) execute a:end . 'm' . (a:where + i) endfor endif endfunction :call Interleave(5, 8, 1) " 5 is first line to move " 8 is last one " 1 where to move ------------------------------------------------------------------------------- Capitalize the first word in string The initial solutions in the news group came out to more than 10 lines of code! # Ken Manheimer (expr-tr-expr) -- see Expr Warning Word=`expr + "$word" : "\(.\).*" | tr a-z A-Z``expr + "$word" : ".\(.*\)"` # Paul Falstad (cut-tr-sed) Word=`echo $word|tr a-z A-Z|cut -c1``echo $word|sed s/.//` ==> # Logan Shaw (cut-tr-cut) Word=`echo "$word" | cut -c1 | tr [a-z] [A-Z]``echo "$word" | cut -c2-` # Harald Eikrem (sed only) Word=`echo $word | sed -e ' h; 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/; G; 's/\(.\).*\n./\1/; ' ` # Tom Christiansen (perl) Word=`echo $word | perl -pe 's/.*/\u$&/'` # Jean-Michel Chenais # (korn shell built-ins only) typeset -u W;typeset -l w;W=${word%${word#?}};w=$W;Word=$W${word#${w}} # perl of course makes this easy perl -e 'print "\u'"$word"'"' ------------------------------------------------------------------------------- Getting environment from a CSH startup scripts into Bourne Shell... The trick is to process the startup scripts in a sub-shell and then extract the resulting environment. env - DISPLAY=$DISPLAY HOME=$HOME TERM=dumb \ csh -cf 'set prompt="> "; source .cshrc; source .login; echo "#------ENVIRONMENT------" env' | sed -n '/#------ENVIRONMENT------/,$p' WARNING: Scripts may have other 'side-effects', for example set permissions, create directories... This technique has been used by other scripts, but that were then found to cause problems due to such side-effects. ------------------------------------------------------------------------------- Am I a Non-Interactive Shell bourne sh: if [ -z "$PS1" ]; then # script/remote execution (non-interactive) csh: if ( ! $?prompt ) then # script/remote execution (non-interactive) In the bourne shell a better way is to test the shell options ``$-'' for the interactive flag directly. case $- in *i*) ;; # do things for interactive shell *) ;; # do things for non-interactive shell esac ------------------------------------------------------------------------------- Auto Background a shell script #!/bin/csh -f if ( $1:q != '...' ) then ( $0 '...' $*:q & ) exit 0 endif shift ...rest of script to run in background... OR #!/bin/sh foreground stuff ( ( background stuff ) & ) exit ------------------------------------------------------------------------------- Suppressing the shell background fork message The trick is to redirect the standard error output of the _shell_ itself Korn (with loss of standard error) ( command & 2>&3 ) 3>&2 2>/dev/null csh/tcsh (this appears to work) ( command & ) bourne shell does not give a message about background forks ever. The real problem is doing this without loosing the stderr of the command BASH This works. ( command & ) But on older BASH you can use this... ( command 2>&5 & ) 5>&2 2>/dev/null For example... This produces extra forking JUNK.. ( echo "stdout"; echo "stderr" >&2; ) & [1] 29886 stdout stderr [1]+ Done ( echo "stdout"; echo "stderr" 1>&2 ) This does not ( ( echo "stdout"; echo "stderr" >&2; ) & ) stdout stderr ------------------------------------------------------------------------------- Background Remote Processes A remote command will leave lots of `extra' processes, and hold a communications channel open for IO. The following will background the remote shell and completely close the IO channel of the rsh command. Csh on remote machine ssh machine -n 'command >&/dev/null /dev/null 2>&1 <&1 &' Either Shell or other shell ssh machine -n '/bin/sh -c "exec command >/dev/null 2>&1 <&1" &' ------------------------------------------------------------------------------- Daemonizing a background process. That is completely seperating it from a terminal and user session so it does not stop when the ssh terminal closes. If a process is not running yet... nohup command & /path/to/command_log If it is already running in forground... CTRL + Z # to unschedule the running process bg # to resume the process in background disown -h %1 # to move the PID out of your tree # so it is not signaled when temrinal is disconnected ------------------------------------------------------------------------------- Loop until parent process dies Background tasks have an annoying habit of continuing to run AFTER you have logged out. This following example program looks up the launching parent process (typically your login shell) and only loops if the parent process is still alive. WARNING: The PS command varies from UNIX system to UNIX system so you will have to tweek the arguments to the `ps' command to make this script work on your UNIX system =======8<-------- #!/bin/sh # # Loop until parent dies # sleep_time=300 # time between background checks # Pick the appropriate ps options for your UNIX system # Uncomment ONE of the following lines #job_opt=xj; sep=""; ppid=1; # SunOS job_opt=xl; sep=" "; ppid=4; # Solaris #job_opt=xl; sep=" "; ppid=4; # IBM UNIX (aix) #job_opt=xl; sep=" "; ppid=4; # SGI UNIX (irix) # Discover the parents process ID set - `ps $job_opt$sep$$ | tail -n+2` "1" eval parent=\$$ppid # While parent is still alive # The kill command "-0" option checks to see if process is alive! # It does NOT actually kill the process (EG: test process) while kill -0 $parent 2>/dev/null; do # ... # Do the background job here # ... sleep $sleep_time done # Parent process has died so we also better die. exit 0 =======8<-------- Also see the script https://antofthy.gitlab.io/software/#homespace as an example of a shell script that is designed to run on many different hosts. ------------------------------------------------------------------------------- Setting a timed alarm in shell This can be done (as a background task) exactly how is an another matter As an educated guess probably something like.. # Trap the USR1 signal trap "do timeout commands" 16 ( sleep $timeout; kill -16 $$; ) & See also "Command Timeout" (next). ------------------------------------------------------------------------------- Command Timeout Run a command but kill it if it runs for too long. This prevents scripts hanging on a command, especially network related commands, like nslookup. Another good summery of this is http://mywiki.wooledge.org/BashFAQ/068 though this was developed independantally. Simple C solution... Look at ~/store/c/programs/timeout.c which will timeout a command simply and easilly. A lot better than the complex solutions below. But of course it is binary, so does not work on all systems. A advanced version is on linux systems in /usr/bin/timeout There is a "timeout" program written in shell. ~/store/scripts/command_run/timeout_bad It does a looped sleep to do interval testing, as such may not instantly return when a command finishes. Also while it works fine from the command line it fails when used within a shell script! The following are my attempts to build it in shell! And shows just what a pain it can be. ---- Attempt 1 ---- Simply runs the command in the background and waits for it to complete. A sleep command provides a timeout. Works but full timeout period is always waited before exiting TIMEOUT=60 # timelimit for command ( command_which_can_hang & sleep $TIMEOUT; kill $! 2>/dev/null ) The above will wait for the sleep for the full TIMEOUT period, regardless of how fast the command completes. If the command is known to never finish then the above will be fine! For example.. at 3 in the afternoon, play random noise for 10 seconds The 'cat' normally would never end, so will never abort early. The sleep will time out the never ending command, just fine! -- EASY > at 3pm cat /dev/urandom > /dev/dsp & sleep 10; kill $! As a test try this... sleep 1 cat /dev/urandom > /dev/dsp & sleep 1; kill $! ---- Attempt 2 ----- Works for modern day shells... Example: NFS quota can hang, (so can any network command!) =======8<-------- QUOTA_TIMEOUT=5 do_quota() { # lookup the users disk quota but with a timeout quota -v "$@" & # output to stdout command_pid=$! # get its pid (to terminate) ( trap "exit 1" 1 2 3 15 # be quiet if terminated sleep $QUOTA_TIMEOUT echo >&2 "Quota Timeout" kill -9 $command_pid exit 0 ) & timeout_pid=$! # pid of the timeout process wait $command_pid status=$? kill $timeout_pid exit $status } quota=`do_quota $USER` =======8<-------- The problem here is that this works fine but the sleep will still continue for the full timeout period. Basically the parent has no simple way of killing the sleep that was lanched in the sub-shell, and does not die naturally. This is not a problem if the sleep is not too long, it does not use any resources other than a entry in the process table. ---- Attempt 3 ---- Kill the timer sleep too! This is amost exactly the same, but also ensures the timer sub-process cleans up the sleep if it is still running. =======8<-------- DNS_TIMEOUT=5 do_nslookup() { # lookup the users disk quota but with a timeout nslookup "$@" | sed -n 's/Address: *\(.*\)/\1/p' & command_pid=$! # get its pid (to terminate) # run a timer to abort the above command ( trap 'kill -ABRT $sleep_pid; exit 1' 1 2 3 15 sleep $DNS_TIMEOUT & sleep_pid=$! wait $sleep_pid kill -9 $command_pid exit 1 ) & timeout_pid=$! # pid of the timeout process # Wait for command, or timeout wait $command_pid status=$? # cleanup timer (if still running) kill $timeout_pid 2>/dev/null wait $timeout_pid 2>/dev/null exit $status } IP=`do_nslookup google.com` echo "Google is at IP address: $IP" =======8<-------- The last method has now been built into a program https://antofthy.gitlab.io/software/#timeout Which includes linkes to a simple test progrom called "countdown" It can be tested with timeout 5 countdown timeout 12 countdown --- Attempt 4 --- This runs the check program as a sub-shell monitoring the original process ID, which exec'ed to the given command. As such when the command exits the main process exists immediatally. However the 'sleep' will still what for a maximum 'interval' period before the sub-process realises the command has finished, and it no longer needs to monitor it. As such it is not very responsive, as it does not return until at least interval seconds has passed between checks. It could be used to provide a looped working indicator (spinner) But the pid of the command is preserved so parents can still kill its child if they want to. Program by Geoff Clare , 13 Feb 1998 Updated by Chet Ramsy (bash maintainer) timeout=60 # timeout interval=15 # intervial beteen 'still runing' checks SIG=-TERM # signal to send delay=2 # delay from signal and doing a KILL ( for t in $timeout $delay do while (( $t > $interval )) do sleep $interval kill -0 $$ || exit t=$(( $t - $interval )) done sleep $t kill $SIG $$ && kill -0 $$ || exit SIG=-KILL done ) 2> /dev/null & exec COMMAND TO RUN ------------------------------------------------------------------------------- Protecting shell scripts from ^Z DANGEROUS This signal can't normally be stopped in a shell, the trick is to change key generating the signal (Don't forget to return it to normal). stty susp undef -- if available stty susp '^-' -- maybe system dependant ------------------------------------------------------------------------------- Pre and Post Commands NOTE: These should be small and quick, and NEVER lock up. Generally used for history updates, status reports (you have mail), prompt changes, and debugging. Post: After a command completed, before prompting for a new one PROMPT_COMMAND='...' $? Reports the last exit result of the previous command line. NOTE: This may have been multiple commands! Pre: Before running a command trap '...' DEBUG This runs before every 'simple' command about to be executed. This may happen multiple times on a single CLI line entered! especially if a loop is involved. BASH_COMMAND will be set to the simple command about to be executed. history 1 | sed -e "s/^[ ]*[0-9]*[ ]*//g" Will get you the full command line that was just entered. Before running a CLI command (sequence) Running commands entering CLI commands, before execution http://hints.macworld.com/dlfiles/preexec.bash.txt this uses both PROMPT_COMMAND, and "trap DEBUG" to exeust on user return. pre_command() { [[ "$enable_cli" ]] || return; enable_cli="" local this_command=$(history 1 | sed -e "s/^[ ]*[0-9]*[ ]*//g") echo " \"$this_command\"" echo "About to execute \"$this_command\"" } PROMPT_COMMAND="enable_cli=yes" trap 'pre_command' DEBUG The above is not complete. See the link for a complete solution. ------------------------------------------------------------------------------- Progress Reporting, and monitoring --- waiting for condition # Wait for a web server to appear (with rotating 'spinner') # Initialization printf -v start '%(%s)T' -1 # start time heartbeat='-\|/'; beatlen=4 timeout=30 # Wait loop while true; do # wait for a server to appear -- adjust to suit # http_code=$(curl -s -o/dev/null -w'%{http_code}' https://localhost:8443/) # case "$http_code" in # 501|504) : ;; # not started yet # 302) break ;; # SUCCESS - break loop # *) echo "Unknown HTTP response $http_code"; break ;; # esac printf -v elapsed '%(%s)T' -1 # elapsed time if (( elapsed - start > timeout )); then printf "FAILED" break #exit 10 fi beat=$(( (beat += 1)%beatlen )) printf "%c Waiting... %d\r" "${heartbeat:$beat:1}" $(( elapsed - start )) sleep .1 # max speed of rotating line done printf " \n" --- File Processing file_progress Extract read offset of a file using "lsof" This can be from ANY process with same UID or root https://antofthy.gitlab.io/software/#file_progress Pipeline Progress "pv" or "pipeview" When reading from a fixed file the 'pv' or "pipe viewer" command tells you how much progress has been made. NOTE: there is a buffer at the start! pv somefile | gzip > rt94-171-06.gz 128MB 0:00:15 [ 9.1MB/s] [=====>.....................] 18% ETA 0:01:07 When taring up a directory getting disk size first is needed Does not work is compressing as well. tar -cf - . | pv -s $(du -sb | grep -o '[0-9]*') > out.tar 44.3MB 0:00:27 [1.73MB/s] [>..........................] 0% ETA 13:36:22 If size is not known, it just shows 'activity' indicator, and how much has been read, for how long, and average speed of data transfer pv /dev/urandom > /dev/null 38.2MB 0:00:03 [12.8MB/s] [ <=> ] See 'perl' notes, for a method showing time : spent, left, and total; This is better as it shows how long it has run and how long to go. The predicted total is useful for seeing if the time calcualtion is varying. However this fails if there are large scale changes in progress rate. ------------------------------------------------------------------------------- Bar Graph... percent Easy barchart display script https://antofthy.gitlab.io/software/#percent This is a all bash method (replace RANDOM line and 'sleep' with action) Of course 'i' and '100' should be set from somewhere meaningful. bar="==================================================" barlength=${#bar} i=0 while ((i < 100)); do ((i += RANDOM%10+2)); sleep 1; # simulate progress i = percentage n=$((i*barlength / 100)) # Number of bar segments to draw $((i/2)) printf "\r[%-${barlength}s]" "${bar:0:n}" done echo Terminal background coloring Prints reverse video spaces! width=60 i=0 n=100 rev=$(tput rev); norm=$(tput sgr0) # terminal coloring while ((i < n)); do ((i += RANDOM%10+2)); sleep 1; # simulate progress i = percentage printf "$rev%$((width*++i/n))s$norm\r" " " done; \ echo "$norm" The various "dialog" apps has a progress bar, but they either totally take over the terminal window or are X window popups. See also http://mywiki.wooledge.org/BashFAQ/044 ------------------------------------------------------------------------------- Line of repeated characters Instead of bar="==================================================" We want to generate it bar=$( ...some command... ) Logical Bash (see below for alternatives) for i in {1..60}; do echo -n "$1"; done External Commands (high startup cost) perl -e 'print "=" x 60' ruby -e 'puts "=" * 60' python -c 'print "=" * 60' Smaller External Commands # awk awk -v OFS="=" 'BEGIN{NF=60+1;print }' awk 'BEGIN{while(c++<60)printf"="}' awk NF=61 OFS== <<<'' # awk with bash input string # unusual use of "awk" awk 'BEGIN{$(60)=OFS="*";print}' # sed loop (print and quite when it is long enough) sed -r ':a s/^(.*)$/=\1/; /^={60}$/q; ba' <<<'' # seq seq 60 | xargs -n1 -I{} echo -n = seq -s= 0 60 | tr -d '[:digit:]' # head-tr head -c 60 /dev/zero | tr '\0' = # "head' is faster and neater than "dd" yes '' | head -60 | tr '\n' = # no newline yes = | head -60 | tr -d "\n" # no newline yes = | head -60 | paste -s -d '' - # newline preserved One External Command printf %60s | tr ' ' = # POSIX *** printf '%*s' 60 | tr ' ' = # alternative printf "%*s" 60 | sed 's/ /═/g' # "sed" can handle unicode # How does ths following work? echo -e $_{0..60}'\b=' # if you don't mind space-backspaces in it echo -e $_{1..60}'\b=' | col # external cleanup of space-backspace Pure BASH (small counts) ****** for ((i=0; i<60; ++i)); do echo -n =; done # very verbose loop printf -v line '%*s' 60; echo ${line// /=} # create and substute a variable printf -- '-%.s' {1..60} # The %.s allows it to repeat for each arg # the -- needed to stop it being an option printf '%.s=' {1..60} # The '60' must be hardcoded printf '%.s=' $(seq 60) # % is first in format so char can be '-' See summery in... https://stackoverflow.com/a/30288267/701532 Repeat strings printf %6s | sed 's/ /_\/\\_/g' # => _/\__/\__/\__/\__/\__/\_ printf -v t "%6s" '' ; echo ${t// /'_/\_'} printf '%.s_/\_' {1..6} # you can have seperate lines too Unusual techniques yes '/' | head -12 | paste -s -d '\\' # => /\/\/\/\/\/\/\/\/\/\/\/ yes '-' | head -12 | paste -s -d '-^-v' # => ---^---v---^---v---^--- ------------------------------------------------------------------------------- Center a line right_justify(){ COLS=$(tput cols) # use the current width of the terminal. printf "%*s\n" "$(( COLS-1 ))" "$1" } # Now calculate to right column for center justified center_justify(){ COLS=$(tput cols) # use the current width of the terminal. printf "%*s\n" "$(( ( COLS+${#1} )/2 ))" "$1" } ------------------------------------------------------------------------------- Increment a character in shell Using 'tr' char=`echo $char | tr ' -~' '\!-~'` or (sutable for hex chars) str1="ABCDEFGHIJKLMNOPQRSTUVWXYZ" str2="BCDEFGHIJKLMNOPQRSTUVWXYZA" pos=`expr index $str1 $char` char=`expr substr $str2 $pos 1` echo $char In perl the "increment alpha string" makes this easy char=abzzz increment=`perl -e '$n="'"$char"'"; print ++$n'` echo $increment # => acaaa ------------------------------------------------------------------------------- Argument Replacement Wrapper (BASH) This function replaces the "convert" command options with 'newer' versions of the same option. convert() { # go through arguments and replace "-rotate" with "+distort" "SRT" args=() # empty array for arg in "$@"; do case "$arg" in "-rotate") args=( "${args[@]}" "+distort" "SRT" ) ;; "-matte") args=( "${args[@]}" "-alpha" "set" ) ;; "+matte") args=( "${args[@]}" "-alpha" "off" ) ;; *) args=( "${args[@]}" "$arg" ) ;; # just copy argument as is esac done # call the the REAL command (with debugging output) echo "DEBUG: convert ${args[*]}" command convert "${args[@]}" } convert rose: -matte -background none -rotate 40 show: The above also allows you to add things like * set up environment (LD_LIBRARY_PATH, TMPDIR) for the command * path to the command to execute, or call a different command * prefix extra setup options or other hidden options * create 'fake' options that expand (macro-like) to more complex options * call other programs to pre-determine the options to use (for the version) WARNING: Watch out that for arguments that look like options... For example convert rose: -gravity center -annotate 0 "-matte" show: Here "-matte" is a string argument, and not a option! Used with the above example function will results in... convert rose: -gravity center -annotate 0 -alpha set show: which will annotate the string "-alpha" instead of "-matte", and generating the warning: "unable to open image 'set'" The only real fix is to give the wrapper a understanding of how many arguments each and every option needs, which is getting a little ridiculous (see next). --- The following is a more complex version that will take care options that could take multiple arguments, and keep such arguments separate. However this is really getting too difficult as you need to start looking at very option that has arguments! convert() { # go through arguments and replace "-rotate" with "+distort" "SRT" args=() # the arguments for the real command option=() # option and arguments (may have multiple arguments) count=0 # number of arguments to still add to option for arg in "$@"; do # collect arguments into multi-word options if [ $count -eq 0 ]; then option=() option="$arg" # note that this is an option case "$option" in "-annotate") count=2 ;; # annotate has two arguments "-rotate") count=1 ;; # rotate has one argument esac else # read in any more option arguments needed option=( "${option[@]}" "$arg" ) count=$( expr $count - 1 ) fi [ $count -gt 0 ] && continue # Substitute options with replacements case "$option" in "-matte") option=( "-alpha" "set" ) ;; "+matte") option=( "-alpha" "off" ) ;; "-rotate") option=( "-virtual-pixel" "background" "+distort" "SRT" "${option[1]}" ) ;; esac args=( "${args[@]}" "${option[@]}" ) # just copy option as is done # call the command - printing what will actually be executed echo "DEBUG: magick ${args[*]}" command magick "${args[@]}" } For example this will now work... convert rose: -gravity center -annotate 0 "-matte" \ -matte -background none -rotate 40 show: ------------------------------------------------------------------------------- Find processes of a particular name Using grep (but ignore all grep processes) ps auxgww | grep "$NAME" | grep -v grep | cut -c1-15,36-99 Merging the two greps.. ps uxw | grep "[s]sh-agent" | awk '{print $2}' The regular expression "[s]sh-agent" will not match the grep process itself! EG: It will not match the "[s]sh-agent" string in the grep process Using awk (but ignoring all awk processes)... ps auxgww | awk "/$NAME/ && \! /(awk)/" | cut -c1-15,36-99 or for a exact process name ps auxgww | awk '/(^| |\(|\/)$NAME( |\)|$)/' | cut -c1-15,36-99 or alturnativeally which matches under a lot of conditions... EG: matches :01 NAME arg :01 some/path/NAME :01 NAME: ps auxgww | \ awk '/:[0-9][0-9] (|[^ ]*\/)$NAME($| |:)/' | cut -c1-15,36-99 Perl version (also matches on username) ps auxgww | \ perl -nle 'print if $. == 1 \ || /^\s*\!:1\s/o \ || /:\d\d (|\[ *|[^ ]*\/)\!:1($|[]: ])/o; ' As you can see things can get complex very quickly ------------------------------------------------------------------------------- Context Grep Or how to display lines before/after search pattern. GNU grep, has context built in to it, check out the -A, -B, and -C options. No gnu-grep grep -v pattern file | diff -c3 - file | grep '^. ' | colrm 1 2 or grep -n $1 $2 | awk -F: '{ print $1 }' | while read linenum do awk 'NR>target-5 && NR (2**10) => (1<<10) case "$1" in *???????????) echo "$(( swap / (1<<30) ))Tb" ;; *????????) echo "$(( swap / (1<<20) ))Gb" ;; *?????) echo "$(( swap / (1<<10) ))Mb" ;; *) echo "${swap}Kb" ;; esac } humanize() { awk 'BEGIN { size='"$1"'; if ( size < 800 ) { printf("%.1f Kbytes", size); exit; } size /= 1024; if ( size < 800 ) { printf("%.1f Mbytes", size); exit; } size /= 1024; if ( size < 800 ) { printf("%.1f Gbytes", size); exit; } size /= 1024; if ( size < 800 ) { printf("%.1f Tbytes", size); exit; } size /= 1024; printf("%.1f Pbytes", size); }' } For humanized time handling see 'info/apps/date.txt' ------------------------------------------------------------------------------- Parellelized Workers Also See "info/apps/find.txt" for recursive compression of files that are not already compressed Download Multiple files from web... cat url_list | parallel "wget -q {} 2>/dev/null || echo {} >> url_failed " #or cat url_list | xargs -r -i -n1 -P8 "wget -q {} 2>/dev/null || echo {} >> url_failed " Process in batches of N (bzip2) # Run N jobs, and wait for all of them, before starting next 8 while :; do parallel=8 files_todo=( `ls * | grep -v '.bz2$' | head -$parallel` ) [ "x$files_todo" = "x" ] && break for file in "${files_todo[@]}"; do bzip2 -9 "$base" & done wait done Using Bash job control #!/bin/bash # # Check how many jobs are running, and add more if not enough. # Runstuff is the worker job (whereever that may be) # # WARNING: all jobs remain 'zombied' until all jobs finished! # This needs some major improvement. # runstuff() { : do the task given with arguments provided } maxjobs=8 parallelize () { while [ $# -gt 0 ]; do count=(`jobs -p`) # any free job slots? if [ ${#count[@]} -lt $maxjobs ]; then runstuff $1 & shift else sleep .0001 # wait a bit before trying again fi done wait # wait for last jobs to finish } parallelize argv1 "argv2-1 argv2-2b" argv3 ... argvn ------------------------------------------------------------------------------- About Functions... Functions exported and imported from the environment. Basically so they can be inherited by sub-shells... BUT they have a very specific setup thanks to "Shell Shock" Export Function to environment... bash -c 'xyzzy() { printf "Xyzzy Run\n"; }; declare -fx xyzzy; echo $( env | grep -A1 xyzzy); ' # => BASH_FUNC_xyzzy%%=() { printf "Xyzzy Run\n"; } Import Function from environment... env - 'BASH_FUNC_xyzzy%%=() { printf "Xyzzy Run\n"; }' \ bash -c 'xyzzy' # Outputs => Xyzzy Run Or get bash to export it so you can see the current requirements Read all about the problems with this in "Shell Shock Bug" https://antofthy.gitlab.io/info/shell/shell_shock.txt --- Shell functions can have '/' or :: in there names! But you can no longer import such functions - post shell shock bash -c '/bin/echo() { builtin echo Whoops; }; /bin/echo' # => Whoops bash -c 'foo::bar() { builtin echo Yes; }; foo::bar' # => Yes However shell shock patches fixed the ability to import such unusual functions --- You can replace a builtin with a function bash -c 'echo() { printf "Subverted\n"; }; echo Normal' # Outputs => Subverted You can use "builtin" to still callc the builtin command bash -c 'echo() { printf "Subverted\n"; }; builtin echo Normal' # => Normal Unless the builtin itself was replaced with a function! The same goes for unset! bash -c 'echo(){ printf "Subverted\n"; } unset() { : do nothing; } builtin() { "$@"; } unset builtin echo builtin echo Normal' builtin unset echo # => Subverted --- MAJOR WARNING! HOWEVER "Shell Shock" patches do NOT prevent the replacement of builtins! env - 'BASH_FUNC_echo%%=() { printf "Subverted\n"; }' bash -c 'echo Normal' # => Subverted Using option '-p' (privileged mode) will prevent shell functions and other environonment variables from being imported. env - 'BASH_FUNC_echo%%=() { printf "Subverted\n"; }' bash -cp 'echo Normal' # => Normal SUID shell scripts also do not allow you to inherit functions (but still risky). Using "sudo" does clean the environment, so that is safe. -------------------------------------------------------------------------------