Home / server / Questions / 199596
Accepted
Arthur Ulfeldt
Asked: 2010-11-09 13:48:56 +0800 CST

is gunzip atomic?

  • 772

I have two scripts that watch a directory (postgres warm-standby)

one script copies compressed files into the directory and then uncompresses them.

the other script watches for the uncompressed files and then ingests them into postgres.

Im wondering if i could his a case where the ingestor script could read teh output of gunzip while it was writing it?

in short: does gunzip

  1. decompress and then change name
  2. copy into new name as it uncompresses
  3. something else
compression postgresql gzip

2 Answers

  1. It's probably not a good idea to base the correctness of your scripts on assumptions about how a particular utility (such as gunzip) works, since the internal behaviour of a utility can change in subsequent releases. I recommend, instead, that you use the proper locking methods (as provided by your particular OS) to ensure mutual exclusion. Most Linux distros provide flock for use in shell scripts.

    • 1
  2. Best Answer

    It is non-atomic, as you can see if you do an ls during a large compression:

    dfranke@alatar:~$ dd if=/dev/zero of=bigfile bs=1M count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 39.1106 s, 82.4 MB/s
dfranke@alatar:~$ gzip bigfile &
[1] 19876
dfranke@alatar:~$ ls -l bigfile*
-rw-r--r-- 1 dfranke staff 3221225472 2010-11-08 17:16 bigfile
-rw------- 1 dfranke staff     114688 2010-11-08 17:18 bigfile.gz
dfranke@alatar:~$ ls -l bigfile*
-rw-r--r-- 1 dfranke staff 3221225472 2010-11-08 17:16 bigfile
-rw------- 1 dfranke staff     212992 2010-11-08 17:18 bigfile.gz
    • 1