Home / server / Questions / 999096
Accepted
Alain O'Dea
Asked: 2020-01-16 09:53:06 +0800 CST

Why does this script not reliably log the latest file?

  • 772

This script fails sometimes and succeeds sometimes (a very annoying situation):

#!/bin/bash
set -Eeuxo pipefail

test_dir="$(mktemp -d)"
touch "$test_dir/file"{001..312}
latest_file="$(find $test_dir -type f | sort -r | head -n1)"
echo $latest_file

When it succeeds, it logs something like this:

++ mktemp -d
+ test_dir=/tmp/tmp.yWelcpBYB7
+ touch /tmp/tmp.yWelcpBYB7/file001 /tmp/tmp.yWelcpBYB7/file002 ... /tmp/tmp.yWelcpBYB7/file312
++ find /tmp/tmp.yWelcpBYB7 -type f
++ head -n1
++ sort -r
+ latest_file=/tmp/tmp.yWelcpBYB7/file312
+ echo /tmp/tmp.yWelcpBYB7/file312
/tmp/tmp.yWelcpBYB7/file312

When it fails, it logs something like this:

++ mktemp -d
+ test_dir=/tmp/tmp.VzTqmgpZyG
+ touch /tmp/tmp.VzTqmgpZyG/file001 /tmp/tmp.VzTqmgpZyG/file002 ... /tmp/tmp.VzTqmgpZyG/file312
++ find /tmp/tmp.VzTqmgpZyG -type f
++ sort -r
++ head -n1
+ latest_file=/tmp/tmp.VzTqmgpZyG/file312

Notice that echo $latest_file line is not executed here even though it appears in xtrace

I cannot get a successful run if I use 10,000 files, so I suspect it has something to do with head stopping find early.

#!/bin/bash
set -Eeuxo pipefail

test_dir="$(mktemp -d)"
touch "$test_dir/file"{0000..9999}
latest_file="$(find $test_dir -type f | sort -r | head -n1)"
echo $latest_file

If I suppress stop on error (using set +e), it succeeds:

#!/bin/bash
set -Eeuxo pipefail

test_dir="$(mktemp -d)"
touch "$test_dir/file"{0000..9999}
set +e
latest_file="$(find $test_dir -type f | sort -r | head -n1)"
set -e
echo $latest_file

Why does this script not reliably log the latest file?

bash

1 Answers

  1. Best Answer

    The problem is the -e. Why? -e makes bash abort in case a process exits with a non-zero exit code (the full rules are a bit more complicated). If there is a pipe, just the last command counts.

    Your head -n1 creates an error situation, internally, because it has to kind of break the pipe (you can check it with strace) to ignore the rest of the output from sort.

    So, to make your script work reliably with -e, you can add a cat to the end of the pipe. head will still break the pipe, but since it no longer is the last command in it, it will not be taken into account by -e. cat is a no-op for the pipe:

    #!/bin/bash
set -Eeuxo pipefail

test_dir="$(mktemp -d)"
touch "$test_dir/file"{0000..9999}
latest_file="$(find $test_dir -type f | sort -r | head -n1 | cat)"
echo $latest_file

    Please check Why doesn't set -e (or set -o errexit, or trap ERR) do what I expected? to know why -e is such an unstable feature and the kind of problems it might bring about. There are many examples at the end. My favorite:

    #!/bin/bash

set -e
foo=$(expr 1 - 1)
echo survived

    It will not print survived, that line will not be executed. But, if you had foo=$(expr 2 - 1), then the echo would be executed!

    You'd be better off implementing your own error checking, -e isn't the best solution out there.

    • 3