------------------------------------------------------------------------------- Shell File Handles and Descripters Shells can 'open' files, and keep them open. It is just rarely used. Complex file descriptor handling are often just not needed, simpler use of named pipes and background commands can be used to simplify things. For a practical example see "co-processing/shell_example.txt" where a shell script does "expect" like interaction with a telnet session. WARNING: While POSIX allows the use of file descriptors 10 an larger Not all shells do, leaving only fd 3 to 9 usable by scripts. With 'echo 10>a' shells should use fd 10 or give nan error, But few shells actually do, instead parsing this as 'echo 10 >a' That is write 10 and newline into file "a". Example dash -c 'echo 10>a; wc -c a' # => 3 a 10 written into file "a" bash -c 'echo 10>a; wc -c a' # => 0 a A file descriptor 10 created pointing to a, an nothing written. Be careful of what type of shell "sh" points to. WORKS: bash FAILS: dash, ksh, posh The rest of this document is mostly bash. See "Syntax error: word unexpected on io redirection on file descriptor ≥10" https://unix.stackexchange.com/questions/166805/ ------------------------------------------------------------------------------- Bash-isms <( command ) read from command (a pipe is created) <<< "$var" read from a variable (bash 'here' string) WARNING: return is appended! try: od -xc <<< "hi" Don't bother with <<[-]EOF type 'here files' See "file.txt" and look for "HERE, Here, here files" for better alternatives. ------------------------------------------------------------------------------- Basic Bourne shell file descriptor handling... Opening Files (Read) In the csh, all you've got is $<, which reads a line from your tty. What if you've redirected? Tough noogies, you still get your tty. Read in the Bourne shell allows you to read from stdin, which catches redirection. It also means that you can do things like this: exec 3file.log echo >&4 "an error has happened" Closing FDs In the Bourne shell, you can close file descriptors you don't want open, like 2>&-, which isn't the same as redirecting it to /dev/null. exec 3<&- exec 4>&- WARNING: both mean the same thing in BASH, a full close() of the FD. Bash has no concept of a socket 'shutdown()' of just one side of the connection. Clone a File descriptor exec 5>&4 Move a File descriptor (essentuall clone and close) exec 5>&4- # equivelent to exec 5>&4- 4>&- Open File/Network (Read and Write full example) exec 4<>/dev/tcp/www.some.comain/80 echo >&4 GET http://www.some.comain:80/file HTTP/1.0 echo >&4 while read line; do echo $line done <&4 # not you need to close each side separately exec 4>&- 4<&- Swap stdout and stderr command 3>&1 1>&2 2>&3 3>&- Select for 'ready' file handles File handles are inherited by sub-processes, so while shell can't use the "select()" system call, other programs (C, perl) can. See "shell_select" perl script in "info/co-processing" and specifically the "info/co-processing/shell_select_example" examples. https://antofthy.gitlab.io/software/#shell_select Seek on filehandles As file handles are inherited by sub-processes a program can be created to do this. For example, this will rewind a shell file handle F=$(mktemp tmp.XXXXXX) exec 3<>$FD # open rm $F # delete open file echo "Hello world" >&3 # write perl -e 'open(FD,">&3"); seek(FD,0,0);' # rewind echo "Goodbye Creal world" >&3 # overwrite file cat /dev/fd/3 # read file (fd is uneffected) See stack overflow "Bash read-write file descriptors seek to start of file" https://stackoverflow.com/questions/3838322 A perl script 'shell_seek' has been created to make this easier, and to also allow you to 'tell()' to report location of the file descriptor, and and even 'truncate()' the file the descriptor points to. https://antofthy.gitlab.io/software/#shell_seek F=$(mktemp tmp.XXXXXX) exec 3<> $F # open rm $F # delete open file echo "Hello world" >&3 # write shell_seek start # rewind echo "Goodbye Creal world" >&3 # overwrite file cat /dev/fd/3 # read file (fd is uneffected) Tee intermediate data to programs STDOUT This is NOT just a simple matter of doing ... | tee /dev/tee | ... Though often that will do the job desired. As the programs stdout may not be a terminal, you need to save stdout so the 'tee' can use it. exec {so}>&1 # save the programs stdout to variable (or use '3') some_command | tee /dev/fd/$so | # get and output data to progs stdout awk 'NR==2 {g=$1} END {print g}' | # get the second line - read all input more_processing # process 2rd line What file descriptors are open by a shell ls /proc/$$/fd A long listing will show symbolic links to the actual file that is open! If file was deleted, the symblic will show the original filename and append the string ' (deleted' to it. (Nice!) This lets you find an unused file descriptor number, for older shells. But newer BASH can open a unused handle, an put into a variable (see next) Closing open files of some running process... First get the processes PID, and run gdb on it gdb -p 1598 Then, call the close system call on the fd you want to close: (gdb) call (int)close(999) $1 = 0 if it is a 'leaked' fd no problem, and program has a bug But program will get a system error if it later tries to use it. The '(int)' cast is just in case close() debug info is missing. --- Connect open descriptor to /dev/null using gdb sudo gdb -p 234532 (gdb) set $dummy_fd = open("/dev/null", 0x200000) // O_PATH (gdb) p dup2($dummy_fd, offending_fd) (gdb) p close($dummy_fd) (gdb) detach (gdb) quit --- Alturnatve (for network connections) is to terminate the connection by adding then removing an iptable rule Find your destination address and port: netstat --program [ --numeric-host --numeric-ports ] | grep []/[] Add a rule to cut iptables -A OUTPUT --destination 10.56.4.79 --dport 57000 --jump DROP and remove it again iptables -D OUTPUT --destination 10.56.4.79 --dport 57000 --jump DROP Not only works for 'keepalive' connections. See https://stackoverflow.com/questions/5987820/ and https://unix.stackexchange.com/questions/491823/ ------------------------------------------------------------------------------- Using variables to hold file descriptor numbers (Bash) Basic usage exec {fd}<~/lib/line-page # File Descriptor Assignment added Bash 4.1 echo "file descriptor opened : $fd" read line <&$fd # read first line (old way) echo "$line" read -u$fd line # read second line (bash way) echo "$line" read -d'\n' -u$fd -a lines # read rest of lines into an array printf '%s\n' "${lines[@]}" # print the array of lines exec {fd}<&- # close Map file also reads into an array but also lets you use a callback function exec {fd}<~/lib/line-page # open file print_line() { echo "$2"; } # action for a mapfile callback mapfile -C print_line -u$fd -O1 -c1 -n10 -t # print 10 lines unset MAPFILE # Junk the lines read into the default array exec {fd}<&- # close Using a loop for greater control Note you can use other things other than read and echo. For example head can be use to read or skip N lines. exec {fd}/dev/null # read and junk first 50 lines cnt=0 while read -r -u$fd line; do # read line in a loop echo "$line" (( cnt++ >= 20 )) && break # count lines done exec {fd}<&- # close descriptor NOTE: it is important to close a descriptor before re-using variable or the file will remain open. Re-open reassigns a new file descriptor, and does not close the old one. ------------------------------------------------------------------------------- Handle STDERR different to STDOUT Filter STDERR stuff () { echo standard output echo more output echo standard error 1>&2 echo more error 1>&2 } filter () { grep a } { stuff 2>&1 1>&3 | filter 1>&2; } 3>&1 standard output more output standard error # Note that 'more error' was filtered out as it has no 'a' Practical example... # Find the directory with largest inode count, # Ignoring directory errors for dirs that dissappear during the search. # raw=$( { ( du -xs --inodes /app/docker/overlay2/* | sort -nr | head -1 ) 2>&1 1>&3 | grep -v 'du: cannot access' 1>&2; } 3>&1 ) Using built-in named pipes... command > >(stdout_pipe) 2> >(stderr_pipe) OR { command | stdout_pipe; } 2> >(stderr_pipe) Note stdout_pipe can also produce errors, which due to order will not go though stderr_pipe. Direct both output and errors to a log, but errors also to screen... (./doit >> log) 2>&1 | tee -a log WARNING: any seperation like this may cause order of stdout and stderr to become out of sync! Caution is needed. ------------------------------------------------------------------------------- Using named pipes This can be simplier, especially if you need an actual filename mknod /tmp/pipe$$ p echo "A plumbers job is never done" >/tmp/pipe$$ & cat /tmp/pipe$$ rm -f /tmp/pipe$$ Bash allows this to be a little more direct cat <( echo "A plumbers job is never done" ) In this case the cat was given an actual filename. ------------------------------------------------------------------------------- More Elaborate Combinations Maybe you want to pipe stderr to a command and leave stdout alone. Not too hard an idea, right? You can't do this in the csh. In a Bourne shell, you can do things like this: exec 3>&1; grep yyy xxx 2>&1 1>&3 3>&- | sed s/file/foobar/ 1>&2 3>&- grep: xxx: No such foobar or directory Normal output would be unaffected. The closes there were in case something really cared about all it's FDs. We send stderr to the sed, and then put it back out 2. --------------------------------------------------------------------------- Anthony's General Example Of complex File descriptor use (bourne shell)... Extracting the output of 3 (or more) channels of data is a difficult task. Here the third channel is used to pass the status of the command, ($?) for error checking. This is vital as the status of the first command in a pipe line is normally unavailable (status of a pipe line is that of the last command!) ASIDE: Bash lets you get the status of ALL pipeline commands with $PIPESTATUS array. Note the use of sub-shells so that the output stream (example fd 3) is taken from the sub-shell where it is defined and NOT from the previous command. exec 9>&1 # set fd 9 to be the normal (at this time) stdout of program cmd() { echo OUTPUT; echo >&2 ERROR; } # fake command for testing. (( ( cmd; echo STATUS >&3 ) \ | sed 's/^/out:/' >&9 ) 2>&1 | sed 's/^/err:/' >&9 ) 3>&1 | sed 's/^/stat:/' >&9 unset cmd exec 9>&- # close fd 9 results in the following output to fd 9 (order of lines may vary) out:OUTPUT err:ERROR stat:STATUS Note however the point is not to output the status but to somehow save it (for parent shell) while doing further processing of the other channels. See also my "cmdout" script. https://antofthy.gitlab.io/software/#cmdout Dan Bernstein --- brnstnd@nyu.edu No exit status report... exec 9>&1; ( exec 2>&1; ls /tmp /foo | sed 's/^/out: /' >&9 ) \ | sed 's/^/err: /' Simplified to just tag error output exec 9>&1; ls /tmp /foo 2>&1 >&9 | sed --unbuffered 's/^/ERROR: /' ------------------------------------------------------------------------------- Anthony's Pratical example I had a pipe in which the first command may or may-not exist on the system, and if it did exist its location was unknown (non-standard unix command). If it was not present I wanted to do something else. One solution was to just run it, get its status, and redirect stderr to device null. The command before this conversion was zoo xp $Zoo $mesgname | tail -n+6 and afterward status=`( ( zoo xp $Zoo $name; echo $? >&3 ) | tail -n+6 >&9 ) 9>&1 2>/dev/null` if [ $status -ne 0 ]; then .... fi Of course it is simplier to just do a "type" test on the command to see if it is present or not (See 'general_hints.txt", "Is COMMAND available" ) But this works for unexpected errors to. Also note that saving output to a temporary file may also have been simpler, unless the command takes a very very long time, and you want to display what you have recieved so far. Complex stdout and stderr example (original): Here's something I had to do where I ran dd's stderr into a grep -v pipe to get rid of the statistics dd produces, but retain any erros, and to return the dd's exit status, not the grep's: device=/dev/rmt8 dd_noise='^[0-9]+\+[0-9]+ records (in|out)$' exec 3>&1 status=`((dd if=$device ibs=64k 2>&1 1>&3 3>&- 4>&-; echo $? >&4) | egrep -v "$dd_noise" 1>&2 3>&- 4>&-) 4>&1` exit $status; This is one example where a named pipe and a backgrounded egrep may have been more practical, and simpler to understand. Of course BASH has things that can make this much easier. ------------------------------------------------------------------------------- Linux special filenames /dev/stdin read from standard input as a pipeline file /dev/fd/0 the file descriptors for the current process ls /proc/$pid/fd/ list of open descriptors and posible the files attached ------------------------------------------------------------------------------- BASH special file handles 'tcp' reading network ports, using <> for a bi-directional stream exec {w}<>/dev/tcp/www.google.com/80 printf "%s\n\n" "GET http://www.google.com/ HTTP/1.0">&$w while read -u$w line; do echo "$line" done exec {w}>&- ------------------------------------------------------------------------------- Rewind a Read/Write File Handle Write to then re-read from a RW file handle is tricky. Here we pre-delete a temporary file after opening it. F=$(mktemp) exec {tmp}<> "$F" rm -f "$F" echo "Hello world" >&$tmp { exec < /dev/stdin; cat; } <&$tmp exec {tmp}>&- {tmp}<&- When you open a file descriptor in bash it is also visible in /dev/fd/ As such cat /dev/fd/3 or cat /proc/self/fd/3 will rewind the file descriptor for you. While echo "something" >> /dev/fd/3 or echo "something" >> /proc/self/fd/3 will append to the end. Both are repeatable as many times as you like. -------------------------------------------------------------------------------