------------------------------------------------------------------------------- docker Run an application in a user level micro-virtal environment NOTE: applications are now generally using "podman" instead. Other than a lack of 'swarm' mode, it is essentually the same. docker-compose Simpler way to set up, shutdown, and monitor logs of a docker application that uses a docker-compose.yml to define the docker application docker-compose up # just run it in foreground docker-compose up -d # daemonized run docker-compose down # shut it down docker-compose logs # the log output of the docker application Swarm Mode docker swarm mode # run multiple containers, local network docker swarm leave --force # remove node from the swarm ------------------------------------------------------------------------------- Installation Set up a docker repository echo ' | [docker] | name=Docker Repository | baseurl=https://yum.dockerproject.org/repo/main/fedora/$releasever/ | enabled=1 | gpgcheck=1 | gpgkey=https://yum.dockerproject.org/gpg ' | perl -ne 's/^\s*\| ?// && print' > /etc/yum.repos.d/docker.repo dnf install docker-engine Start the docker deamon systemctl enable docker systemctl start docker Fix permissions to allow user running chgrp docker /var/run/docker.sock chmod 660 /var/run/docker.sock usermod anthony -G docker Images are located in /var/lib/docker/image/overlay2/imagedb/content/sha256/ first part of image name is the ID in "docker images" list The specific (extra) files are in /var/lib/docker/containers/.../ When running the container consists of Hostname is the container ID container / is main system / in size The /etc/hosts file is mounted using /dev/{disk} !!!! /etc/hostname and /etc/resolve.conf are copies from 'containers' a "ps aux" only shows two processes (bash and ps) and bash is processor 1 As soon as the command finished the container is stopped ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- General Commands General example docker run hello-world # run docker hello world (pull it if needed) docker --version #\ docker version # > increasing levels of information docker info #/ docker pull ubuntu:xenial # install ubuntu with the xenial tag (don't run) # run it interactively, connecting my tty, running a bash (as root) # Add -d to just run it without attaching (disconnect) docker run -i -t ubuntu:xenial /bin/bash --- Base Container handling docker images # what docker images are installed docker images | perl -pe 's/(? | RUN apt-get update | RUN apt-get install -y telnet openssh-server ' perl -ne 's/^\s*\| ?// && print' > Dockerfile docker build -f Dockerfile -t "anthony/unbuntu-ssh:v2" . If the image has already been build a specific TASK (step in "Dockerfile") may just be retrieved from the build cache. To prevent this use the option --no-cache NOTE: During the 'build' the new 'buildkit' process will not should the output of the commands that are run unless something goes wrong. You get it to show the output of command run (not already successfully cached) either add the option --progress plain or set ten environment variable... export BUILDKIT_PROGRESS=plain To not use 'buildkit' but use the older docker build engine, use... (This is legacy and now depreciated). export DOCKER_BUILDKIT=0 --- Run a container # start a new container, and run command, container stops on command exit # output is logged and avaiable from "docker logs" docker run -rm {image} command... # run container as a daemon docker run -d -rm {image} \ /bin/bash -c 'white true; do echo HELLO; sleep 1; done' docker logs {container} docker stop {container} # Run the default command built into the image in background # This is visible using "docker inspect {image}" # The -rm has it remove run time when it stops. docker run -d -rm {image} --- Run commands or connect to a runing container # Run a command in a running container (running a bash shell) # -- it is not logged and container does not stop docker exec {container} command... # connect to a container as root user! # if not the main user - exiting does not stop the container docker exec -u 0 -it {container} /bin/bash ---- Cleanup to stop gracefully use docker stop {id} to just kill it use docker kill {id} # to clean up and remove the instantiated container docker rm -v {id} #Without the -v it leaves container files behind, so always use it. docker rmi {image} # remove image docker stop $(docker ps -a -q) # Stop all containers docker rm $(docker ps -a -q) # Remove all containers docker rmi -f $(docker images -q) # Remove all images --- Auto Clean Up (Cron) # Clean up unused containers docker rm -v $(docker ps -aq -f status=exited -f status=created) # Clean up Old Unused Images docker rmi $(docker images -qf dangling=true) # Clean up unused volumes docker volume rm $(docker volume ls -qf dangling=true) --- Major Clean Up # clean up -- images, networks, volumes, etc that are not needed docker system prune --volumes --force # Overly clean - remove all images NOT CURRENTLY in use docker system prune -a --force # Clean up just the build cached images docker builder prune -af --- Exposing Ports # any services will normally only be visible on the IP address # assigned to a container # redirect a local port to a container port .... run -d -p 8080:80 ... # docker ps lists the redirected ports. # but you can see them in lsof -Pi > lsof -Pi :8080 docker-proxy 26893 root 4u IPv6 3376063 0t0 TCP *:8080 (LISTEN) # And in iptable rules > iptables -t nat -n -L | grep 172.18.0.2 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 to:172.17.0.2:80 --- Copy files directly to/from a running container (ephemeral) Files... docker cp foo.txt mycontainer:/foo.txt Directory docker cp src/. mycontainer:/target docker cp mycontainer:/src/. target NOTE: Though this is NOT a permanent part of the image, just that running instance (container). --- Migrating images to a different server (no repo) docker save {image} | bzip2 > {image.tar.bz2} bzip2 -d < {image.tar.bz2} | docker load Note use the 'repo:tag' so the save will retain the tag. EG: docker save -o ~/centos_16.tar centos:16 You can re-tag an image using docker tag {image_id} {image_name} ------------------------------------------------------------------------------- Docker Swarm (using Services) https://docs.docker.com/engine/swarm/swarm-tutorial/ Start a Docker Swarm docker swarm init Stop a Swarm docker swarm leave --force ON a manager node -- Get a token to start a Worker Node (copy output to the worker node to connect it to swarm) docker swarm join-token worker # for a new worker node docker swarm join-token manager # for a new manager node # make another node a manager docker node promote {node_id|hostname} # stop a node being a manger docker node promote {node_id|hostname|self} Swarm information docker info # Check the swarm # To find the managers, look in the output for "Manager Addresses" # and it will list the IP's with port:2377 docker info 2>/dev/null | sed '1,/Manager Addresses:/d; /[A-Za-z]/Q; s/:.*//' docker node ls # what nodes are in swarm (on manager only) # Is this node the swarm leader? (true, false) # NOTE this can change and migrate between managers! docker node inspect --format '{{.ManagerStatus.Leader}}' self # Is this node a swarm manager? (This is usually more important) docker node inspect self >/dev/null 2>&1 && echo "true" || echo "false" # what is running on a node (including stopped containers) # though a node is clean, this shows the node still had stopps processes! docker node ps {node_id|hostname|self} Drain a Swarm Node # from manager node docker node update --availability drain {hostname} # After it starts draining (30 seconds) you can make it available again docker node update --availability active na-prd-swarm-5.itc.griffith.edu.au The old containers will continue to be be moved, even as new containers will re-populate the node Hopefully clearing swap/memory problems. --- Deploy a image as a service on the swarm docker service create --replicas 1 --name {name} {docker_image} # for options on how long updates roll out at (see below) Service Stats (on manager node only) docker service ls docker service ls --format "{{.Name}} {{.Image}}" # compact service stats docker stats --no-stream | sed 's/[^ ]* *//; s/\.1\..\{26\}//;s/ \/ [^ ]*//g; 1s/.*/Name CPU% MEM MEM% NET_IO BLK_IO PIDS/' | sort | column -t # sorted by memory usage -- With overall stats ("contained_status" script) mem=$(free -h | sed -n 's/Mem: *\([^ ]*\).* \(.*\)/\2\/\1/p'); load=$(uptime | sed 's/.*://'); docker stats --no-stream | sed 's/[^ ]* *//; s/\.1\..\{26\}//;s/ \/ [^ ]*//g; 1s/.*/Name CPU% MEM MEM% NET_IO BLK_IO PIDS/' | sort -ts -k2n | column -t | awk '{print; lines++;} # count up the number of lines END {printf "Containers: %d Free Mem: %s Load Avg: %s\n", lines-1, "'"$mem"'", "'"$load"'" }' Inspect a service # everything formatted nicely (without --pretty, json is returned) docker service inspect --pretty {service_name} # get its ID docker service inspect --format '{{.ID}}' {service_name} # everything about service... docker service inspect --format '{{json .}}' {service_name} | json_reformat # get the containers default environment variables docker service inspect {service_name} \ --format '{{json .Spec.TaskTemplate.ContainerSpec.Env}}' | json_reformat # OR docker service inspect {service_name} | jq '.[] | .Spec.TaskTemplate.ContainerSpec.Env' # OR.. docker service inspect -f '{{range $index, $value := .Spec.TaskTemplate.ContainerSpec.Env}}{{println $value}}{{end}}' {service_name} # service processes that have shutdown docker service ps {service_name} -f desired-state=Shutdown # Question: how to rm a shutdown container, when it is not running # on the node indicated --- docker service logs {service_name} docker ps # on the worker node to look at the 'task' docker service ps {service} # all the worker 'tasks' running # Add --no-trunc to see the FULL error of a 'failed service' docker service ps {service_name} --format '{{json .}}' --no-trunc | \ head -1 | json_reformat docker service ps {service_name} --format='{{.Node}}' # node service is running on docker service ps {service_name} --format='{{.DesiredState}}' # and if it is running! Exec a service service=control_redis container=$(docker ps --filter name="^$service.1" --quiet) docker exec -it $container sh docker exec -it $container redis-cli keys \* Scale Service - Number of containers docker service scale {service}={number} Remove service docker service rm {name|service} # Note it may take a few seconds for 'tasks' (running service containers) # to shutdown Update a service (rolling update) docker service update --image {new-image} {name} # new-image can be same as old name # --update-failure-action what to do on failure to stop-restart service # --update-delay {time} roll out time format: 10s 3m 1h 10m30s # --update-parallelism number simulatious updates (default 1) if update failed or pause - restart it using docker service update {service} Drain (stop worker node recieving new tasks (running containers) docker node ls # list nodes docker node update --availability drain {worker_name} docker node inspect --pretty {FQDN} # check availability docker service ps {service|name} # monitor the movement of tasks docker node update --availability active {worker_name} # node back online Publish Ports (route port on any node to active task port) # on creation docker service create --name {name} \ --publish published={external_port},target={container_port} \ {image} # add using update docker service update --publish-add \ published={external_port},target={container_port}:mode=host \ {service} # or (not recommended) docker service update --publish-add {port} {service} # check active ports docker service inspect --format="{{json .Endpoint.Spec.Ports}}" \ {name} | json_reformat Services on specific nodes.... # nodes available (names and roles) docker node ls docker node inspect {FQDN} # create (or update) service on manger or workers docker service create --constraint node.role==manager ... docker service create --constraint node.role==worker ... # create (or update) service to specific node docker service create --constraint node.hostname=={FQDN} ... docker service create --constraint node.hostname!={FQDN} ... # label a node with a particular nam docker node update --label-add {label}={name} {FQDN} # create (or update) services to a specific node by label docker service create --constraint node.labels.{label}=={name} ... docker service create --constraint node.labels.{label}!={name} ... ------------------------------------------------------------------------------ Docker Stacks (grouped services) Example: https://docs.docker.com/engine/swarm/stack-deploy/ Create stack non-swarm (yaml file "docker-compose.yaml") docker-compose up -d # start docker-compose ps # check ... do whatever ... docker-compose down --volumes # stop Deploy to a swarm docker stack deploy -c stack.yml stackdemo docker stack services stackdemo # see just services on this stack docker stack ps control # status of all process for stack docker service update stackdemo_worker # update (restart, services) # go to a newer image! docker service ps stackdemo_worker # processes of a specific stack service # Add --no-trunc to above for the FULL ERROR on failed services docker service rm stackdemo_worker # shutdown a specific stack service Deploy but with 'development' additions, or changes See https://github.com/docker/cli/issues/939#issuecomment-390931468 "What if I want additional environment variables to be set" docker stack deploy -c stack.yml stack_devel.yml stackdemo Store/Deploy Image in a registry... echo "password" | docker login -u loginname --passwd-stdin dir_to_use docker-compose push # push image to a registry docker stack deploy -c stack.yml {name} --with-registry-auth # URL={hostname docker {options} stack ... # --host=swarm-hostname.domain # --tlsverify --tlscacert={cert}... ---tlskey={key} Remove a deployed stack docker stack rm Look at a stack docker stack ls docker stack services ingress docker service ls docker network ls docker config ls docker secret ls docker stack ps ingress To remove manually replace "ls" with "rm {..}" --- Update a docker 'config' file (used by a ingress stack) # list the configs docker config ls # details (NOTE that "Data" is a LONG base64 string, thsi contents) docker config inspect ingress_nginxconf # print/extract the current config docker config inspect | sed -n 's/"$//;s/ *"Data": "//p' | openssl enc -base64 -A -d # OR docker config inspect -f '{{json .Spec.Data }}' ingress_nginxconf | \ cut -d'"' -f2 | base64 -d # # Update Running Service # (using a temporary, or you can use a incrementing number) # # update the service service="ingress_nginx" conf="ingress_nginxconf" file="nginx.conf" target="/etc/nginx/nginx.conf" OLD="" # or use versions ".1" NEW=".TEMP" # ".2" # # Create temporary config entry docker config create "$config$NEW" "$file" # docker service update \ --config-rm "$config$OLD" \ --config-add "src=$config$NEW,target=$target" $service # # remove the old one docker config rm "$config$OLD" # Optionally... # You can do it again to remove the $NEW suffix # # create a config without the suffix docker config create "$config" "$file" # # and install it... docker service update \ --config-rm "$config$NEW" \ --config-add "src=$config,target=$target" \ $service # # remove temporary config docker config rm "$config$NEW" # NOTE: many services only read the file on startup # so you may need to force a restart of the service. docker service update $service --force --- Add or Update a docker 'secret'. NOTE: A 'secret' is as per a 'config' (putting a file on a service) But these are generally set by hand so as not to put into 'source control'. they are handled in much the same way as a 'config'. See below. The secret file within the container is typically placed in... "/run/secrets/{secret_name}" unless you specifically set a different target location for that data, this may be done when updating the data (as per a 'config" above) # Create a secret... printf 'some password' | docker secret create auth_password - docker secret inspect docker secret ls docker secret rm docker service create ... --secret ... docker service update ... --secret-add ... docker service update and --secret-rm ... # show secrets in a service container (must be on local machine) service=control_worker container=$(docker ps --filter name=$service -q | tail -1) docker container exec $container bash -c 'head /run/secrets/*' # show secrets using a personal script... docker_user control_worker -c 'head /run/secrets/* /dev/null' --- Migrate all services to a new network! https://git.griffith.edu.au/snippets/52 # create the LARGER (B-class) network docker network create --driver overlay --subnet 10.1.0.0/16 ingress_net_temp # get the ID's of network and service INGRESS_NET=$(docker network inspect ingress_net --format '{{.Id}}') INGRESS_NET_TEMP=$(docker network inspect ingress_net_temp --format '{{.Id}}') INGRESS_ID=$(docker service inspect ingress_nginx --format '{{.ID}}') # Attach the new network to services, so it's on both networks docker service update $INGRESS_ID --network-add $INGRESS_NET_TEMP # Migrate services to temp network for service in $(docker service ls -q); do if docker service inspect $service --format '{{range .Spec.TaskTemplate.Networks}}{{println .Target}}{{end}}' | grep -qw $INGRESS_NET; then # skip the main service if [ "${INGRESS_ID:0:12}" = "$service" ]; then continue fi # show the human name of the service docker service inspect $service --format '{{.Spec.Name}}' # Remove old network and attach the new temp one docker service update $service \ --network-add $INGRESS_NET_TEMP \ --network-rm $INGRESS_NET \ --update-order "start-first" \ --update-parallelism 1 \ --update-monitor "30s" \ --update-delay "30s" echo -e "\n\n\n" fi done # Detach the service from the public_ingress docker service update $INGRESS_ID --network-rm $INGRESS_NET ------------------------------------------------------------------------------- Publish a Port on a RUNNING service... The following manually sets up a route from external port to the service However as docker does not know about it, it can be reset and lost. The BEST solution is a full restart of the service, if posible. Opening port 3001 without stopping ingress service on 172.18.0.2... iptables -t nat -A DOCKER ! -i docker_gwbridge -p tcp -m tcp \ --dport 3001 -j DNAT --to-destination 172.19.0.2:3001 iptables -t nat -A POSTROUTING -j MASQUERADE -p tcp \ --source 172.18.0.2 --destination 172.19.0.2 --dport 3001 iptables -A DOCKER -j ACCEPT -p tcp --destination 172.19.0.2 \ \! -i docker_gwbridge -o docker_gwbridge --dport 3001 WARNING: When the service is properly installed, so that docker does the routing, the above rules MUST be removed. Duplicate rules will NOT work. ASIDE: probably a duplicate POSTROUTING is the cause of failure. To delete a specific rules from iptables... iptables -t nat -S DOCKER | cat -n # now get the line number and subtract 1 (starts at 0) # list that rule to double check iptables -t nat -S DOCKER 8 # delete that rule... iptables -t nat -D DOCKER 8 Using -S will output the exact 'iptable' command options needed to re-add that rule. Repeat the removed for each of the three additions, to ensure there are no exact duplicates. Discussion of problem (no good solution)... https://github.com/moby/moby/issues/33420 Solution https://stackoverflow.com/questions/19897743/ https://stackoverflow.com/questions/19335444/ ---- A 'socat' port forwarder 8080 to 80... socat TCP-LISTEN:8080,fork,reuseaddr,user=node,group=node,mode=777 \ TCP:172.19.0.2:80 & You could do this is from a docker container too.. docker run --rm -p 8080:1234 \ verb/socat TCP-LISTEN:1234,fork TCP-CONNECT:172.19.0.2:80 From https://stackoverflow.com/questions/41555683/ https://stackoverflow.com/questions/19897743/ --- Using docker networks ??? https://stackoverflow.com/questions/19335444/ --- To get the containers IP... docker ps # get the containers name docker inspect container_name | grep IPAddress But this does not get the contains 'ingress network ip' (a 172.0.0.0 subnet) docker exec container_name hostname -I will list ALL the containers network IP's ------------------------------------------------------------------------------- Docker Image Transfers without using repositories Individaully: https://stackoverflow.com/questions/23935141/ All of them: https://stackoverflow.com/questions/35575674/ Individual images docker save -o imagefile.tar image_id docker load -i imagefile.tar docker save | bzip2 | ssh user@host 'bunzip2 | docker load' # you may like to insert 'pv' in the pipeline # http://www.ivarch.com/programs/pv.shtml NOTE: use 'repo:tag' for the image name so it will be preserved otherwise you may have to re-tag the image after loading it. All Images #save all images (uses image ids) docker save $(docker images -q) | ... | docker load # get the tages... docker images | sed '1d' | awk '{print $1 " " $2 " " $3}' > mydockersimages.list # set the tags on other system while read REPOSITORY TAG IMAGE_ID; do echo "== Tagging $REPOSITORY $TAG $IMAGE_ID ==" docker tag "$IMAGE_ID" "$REPOSITORY:$TAG" done < mydockersimages.list # --- # As above but use tags... docker save $(docker images --format '{{.Repository}}:{{.Tag}}') \ -o allinone.tar # --- # Pack up all images tar -cvf docker-images.tar /var/lib/docker/aufs /var/lib/docker/image # Unpack on machine tar -xvf docker-images.tar -C /var/lib/docker/ # and restart service docker restart Use a temporary Registry.... https://github.com/brthor/docker-push-ssh # Install pip install docker-push-ssh Use it... docker-push-ssh -i ~/my_ssh_key username@myserver.com my-docker-image You will need to manually add your localhost to Docker's insecure_registries configuration. ------------------------------------------------------------------------------- Docker Image Registry https://docs.docker.com/registry/ Create a Registry docker service create --name registry \ --publish published=5000,target=5000 \ registry:2 docker service ls # see if it is running curl http://localhost:5000/v2/ # test docker service rm registry # remove it Using Registry... docker login localhost:5000 # login to a registry # so you can push/pull images docker push {image} # add an image docker pull {image} # get an image docker search # look for images Example docker tag nginx:latest daviddavistx/nginx:latest # tag image in docker docker push daviddavistx/nginx:latest # push it to registry docker search docker search unbuntu docker search - -filter “is-official=true” ubuntu ------------------------------------------------------------------------------- Docker Volumes (mounts into container/service) docker volume ls docker volume inspect {volume_name} Checking volumes on a running container docker container run -d --mount source={vol name},target=/app/new-vol docker container inspect {container_id} docker volume inspect {volume_name} ------------------------------------------------------------------------------- Watching Docker! # report all the happening for docker. docker events =============================================================================== Map a PID on host to container trace the parent container process from the pid you are interested in pps 2078 pstree -lspa 2078 The containerd-shim is the container process... for example... systemd,1 --switched-root --system --deserialize 22 └─containerd-shim,1868 -namespace moby -id f51dbabff82482f928cd52c4c631dcd2a274cb35ce3e07b79b65245119cc0f7f -address /run/containerd/containerd.sock └─nginx,1889 └─nginx,2078 The ID is the container ID... docker ps | grep f51dbabff824 f51dbabff824 virtual-griffith-ers-docker.docker.itc.griffith.edu.au/speechtotext-client:0.3.57 "/docker-entrypoint.…" 13 days ago Up 13 days 80/tcp ers_speech_to_text_acoe_web.2.poj0qimy53pnbaj22dunsvehq At this point you can kill the container docker kill ers_speech_to_text_acoe_web.2.poj0qimy53pnbaj22dunsvehq Docker swarm will hopefully rescheduale it ------------------------------------------------------------------------------- Docker exec problem (minor annoyance) When you login to a docker container using "exec", ** on exit ** it will kill off all processes belonging to the started process group, of that exec. docker exec --tty --interactive .... {container} /bin/bash This means a backgrounded program started in the ".bashrc" will also be killed. While the exact same program started in exactly the same way from the command line will continue to run! For example... nohup program >/tmp/program.log 2>&1 & From the ".bashrc" the program will have the same PGID & SID as the bash, (PGID is also the PID of the bash). And it will be killed with the shell. HOWEVER: From a normal command line, the program will have the same SID as the launching bash shell, but it will have a different PGID, and thus will NOT be killed on the shell (and "docker exec" exit). Solution is to start the background process using "setsid" instead of "nohup". -------------------------------------------------------------------------------