-------------------------------------------------------------------------------
Using Bash Co-Processing

From Technetra
  http://www.technetra.com/2009/04/26/discovering-web-access-latencies-using-bash-coprocessing/

This is an example of using co-processing in BASH using FIFO named pipes,
which are linked to an array of file descriptors.

However it suffers from assumptions about the file descriptor numbers
being used, and does not have good cleanup coding.  It is also devoid of
comments on exactly what it is doing in many places.

Basically it is a start of an all bash version of the "empty" coprocessing
methodology, but needs more work.

-------------------------------------------------------------------------------
Discovering Web Access Latencies Using Bash Co-Processing
Robert Adkins,  April 26th, 2009 at 10:03 am

This tutorial describes a Bash co-processing script latency.sh (Listing 1)
that generates and sorts web access latencies for a user supplied list of
hosts. It demonstrates a co-processing scheme that supports multiple,
simultaneous co-processes and works in all versions of Bash.

It is an updated and parallelized version of an earlier script with a similar
purpose, ffmirror.sh (see http://grulos.blogspot.com). This new script uses
customized Bash co-processing support for parallelization. In addition, unlike
the grulos script, this script does not require a Bash executable that has
been compiled for special network redirection (that is, with compiled-in
support for /dev/tcp, etc.). You can use either tcp redirection or wget by
specifying the desired mode as an argument on the command line. This is good
news for Debian and Ubuntu users because these distributions normally do not
ship with net redirections enabled. Note that while the Bash net redirection
mode is a bit faster than launching and using wget, the latter may be more
robust across many iterations of running the script against large lists of
both existing and non-existent hosts.

Co-processing can reduce the performance profile of the principal work path of
the script from linear, or O(n), to nearly O(1). The principal work path in
our case is setting up host connections. In our implementation, collecting the
latency results is still linear, but with a negligible run-time coefficient.

latency.sh can be run with the default wget connection strategy:

$ ./latency.sh < myhostlist

or with the Bash network redirection strategy where available:

$ ./latency.sh tcp < myhostlist

Using the 20-entry sample host list at grulos.blogspot.com, the time to run
this script on a normal desktop can be under 2 seconds, while the original
code requires over 18 seconds. This makes it feasible to run such a command
under programs like /usr/bin/watch to create dynamically updating displays.
Figure 1: Trial run (using the original sample host data from
grulos.blogspot.com)

# time ./latency.sh tcp < new_hostlist.txt
04/26/2009 19:40:49 running tcp version
270ms www.google.com
540ms www.redhat.com
900ms www.kernel.org
940ms www.microsoft.com
940ms www.slackware.org
940ms www.w3.org
960ms www.ibm.com
990ms www.debian.org
1000ms www.wikipedia.org
1000ms www.gentoo.org
1050ms www.altavista.com
1070ms www.yahoo.com
1070ms www.dell.com
1090ms www.netbsd.org
1100ms www.freebsd.org
1220ms www.openbsd.org
1860ms www.linux.org
(not set) www.shakakalasfdksasdfasdf.com
(not set) www.oooooooos.com
(not set) www.tttttttttt.org

real    0m1.982s
user    0m0.156s
sys     0m0.160s

Note that hosts whose latencies cannot be computed, for whatever reason, are
labelled as ‘(not set)’.

The Bash co-processing function presented in Listing 2 below for
coprocess.bash is based on a similar example from the GNU Bash distribution.
The difference is that this co-processing function implements a simple queuing
facility to handle multiple simultaneous coprocesses. This is essential for us
to be able to parallelize our web access application. It also uses
parameterized named pipes (fifos) for communicating between coprocesses.

The script latency.sh works under both Bash 3.x and Bash 4.0. It should be
noted that the recent native support for co-processes in Bash 4.0 is limited
to a single co-process at a time and so could not meet our need for multiple
simultaneous web connections.

Listing 1: latency.sh

=======8<--------CUT HERE----------axes/crowbars permitted---------------
#!/bin/bash
#
# latency.sh - Generate and sort web access latencies
#              for a list of hosts
#
# Copyright (c) 2009 Technetra Corp
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
# This is an updated and parallelized version
# of ffmirror.sh (http://grulos.blogspot.com)
#
# To run the default wget based version:
#    ./latency.sh < hostlist.txt
# or for the tcp (Bash net redirection) version:
#    ./latency.sh tcp < hostlist.txt
#
###########################################################################

. ./coprocess.bash

WHICH_BACKGROUNDER=${1:-wget}
echo "$(date '+%m/%d/%Y %H:%M:%S') running $WHICH_BACKGROUNDER version"

chk_time () {
  # get first part of uptime - time in seconds since last reboot
  local t=$(</proc/uptime)
  t=${t%% *}
  t=${t/./}
  eval $1=$t
}

wget_backgrounder () {
  local host=$1 a b
  [[ -z "$host" ]] && read host
  chk_time a
  wget --spider -T 10 -q -O /dev/null $host && chk_time b
  echo "$((b-a))|$host"
}

tcp_backgrounder () {
  local host=$1 a b
  [[ -z "$host" ]] && read host
  chk_time a
  exec 2>/dev/null 3<>/dev/tcp/$host/80 &&
    # send a GET request and read 1st line of response
    printf "GET / HTTP/1.1\r\nHost: $host\r\n\r\n">&3 &&
    read -u 3 -t 10 -a response &&  exec 3>&- && chk_time b
  echo "$((b-a))|$host"
}

# spin off parallel connections (the 'map' in 'MapReduce')
while read hostname; do
  # provide hostname to wget or /dev/tcp as command-line argument
  coprocess open qindex ${WHICH_BACKGROUNDER}_backgrounder "$hostname"
  # alternatively, send hostname to wget or /dev/tcp via pipe:
  #   coprocess open qindex wget_backgrounder
  #   coprocess put $qindex "$hostname"
done

# collect and process results (the 'reduce' in 'MapReduce')
coprocess size len
for ((i=0;i<$len;i++)); do
  # read result from pipe
  coprocess get $i c
  coprocess close $i
  h=${c##*|}
  d=${c%%|*}
  if [[ d -ge 0 ]]; then
    # bucket sort variant
    e=$((d*100))
    until [ "${slist[e]}" == "" ]; do
      ((e++))
    done
    slist[e]="${d}0ms $h"
  else
    elist[not_set++]="(not set) $h"
  fi
done

printf "%sn" "${slist[@]}"
[[ ${#elist[*]} -ge 0 ]] &&
  printf "%sn" "${elist[@]}"

=======8<--------CUT HERE----------axes/crowbars permitted---------------


Listing 2: coprocess.bash

This is source by the previous program and provides coprocessing
routines.

No Manual Provided!

=======8<--------CUT HERE----------axes/crowbars permitted---------------
# coprocess.bash
#
#    Copyright (c) 2009 Technetra Corp
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
##########################################################################

declare -i FIFO_IN FIFO_OUT FIFO_START=60
declare -a COPROCESSQ

coprocess () {
  local CMD="$1" N="$2"
  shift 2
  [[ "$N" == [[:digit:]]* ]] &&
      FIFO_IN=$((N*2+FIFO_START)); FIFO_OUT=$((FIFO_IN+1))

  case "$CMD" in

    open)
      local M=${#COPROCESSQ[*]}    # get next co-process index
      FIFO_IN=$((M*2+FIFO_START)); FIFO_OUT=$((FIFO_IN+1))
      local fifo="/var/tmp/coprocess.$FIFO_IN.$FIFO_OUT.$$.$RANDOM"

      mkfifo "$fifo.in" || return $?
      mkfifo "$fifo.out" || {
        ret=$?
        rm -f "$fifo.in"
        return $ret
      }

      # Launch the background process
      ( trap "rm -f $fifo.in $fifo.out" 0
        "$@" <$fifo.in >$fifo.out
      ) &

      COPROCESSQ[M]=$fifo
      eval "$N=$M"
      eval "exec $FIFO_IN>$fifo.in $FIFO_OUT<$fifo.out"
      return 0
      ;;

    close)
      eval "exec $FIFO_IN>&- $FIFO_OUT<&-"
      [ "$1" = "-SIGPIPE" ] && return 1
      return 0
      ;;

    get)
      local old_trap=$(trap -p SIGPIPE)
      trap "coprocess close $N -SIGPIPE" SIGPIPE      # is this needed?
      builtin read "$@" <&$FIFO_OUT
      local ret=$?
      eval "$old_trap"
      return $ret
      ;;

    put)
      local old_trap=$(trap -p SIGPIPE)
      trap "coprocess close $N -SIGPIPE" SIGPIPE
      builtin echo "$@" >&$FIFO_IN
      local ret=$?
      eval "$old_trap"
      return $ret
      ;;

    size)
      local M=${#COPROCESSQ[*]}
      eval "$N=$M"
      ;;

  esac
}

=======8<--------CUT HERE----------axes/crowbars permitted---------------