Home / server / Questions / 244917
Accepted
Ben McCormack
Asked: 2011-03-09 14:03:07 +0800 CST

Combine multiple unix commands into one output

  • 772

I need to search our mail logs for a specific e-mail address. We keep a current file named maillog as well as a week's worth of .bz2 files in the same folder. Currently, I'm running the following commands to search for the file:

grep [email protected] maillog
bzgrep [email protected] *.bz2

Is there a way combine the grep and bzgrep commands into a single output? That way, I could pipe the combined results to a single e-mail or a single file.

linux unix bash grep

5 Answers

  1. Best Answer

    Another way is

    { grep ...; bzgrep ...;} >file

    && has the difficulty that the bzgrep wouldn't be run if the grep failed.

    Note the mandatory space after the opening curly brace and semicolon after the last command. Alternatively, you can use the subshell syntax (parentheses instead of curly braces), which isn't as picky:

    (grep ...; bzgrep ...) >file
    • 24

  2. bzgrep automatically defaults to regular grep if a file isn't bzip-compressed. Thus the following should be sufficient:

    bzgrep [email protected] maillog *bz2 | mail -s "logs yay" someuser@blah

    oh also of course here's my obligatory GNU Parallel solution too:

    parallel -m bzgrep [email protected] ::: maillog* *bz2 | mail -s "logs yay" someuser@blah

    which could be a lot faster if you are checking a lot of files.

    • 11

  3. Here's another way to do it (assuming you're running bash, which you probably are):

    cat <(bzgrep ...) <(grep ...)

    Here bash is transparently feeding the output of the bzgrep and grep commands into cat as if they were files (and they sort of are under the hood, details in url at the bottom).

    In your particular case I'd recommend Phil's solution, but the above is a good trick to keep in your bag.

    If you're interested, you can read more here: http://www.tldp.org/LDP/abs/html/process-sub.html

    • 2

  4. At the time of my writing this, the accepted answer's syntax was wrong for most, if not all, Bourne-derived shells, including bash. I suggested an edit to the top and accepted answer to fix that, but I was also inclined to add all this other information, and this would've been more of a rewrite instead of an edit.

    You can use compound commands:

    { grep ...; bzgrep ...; } >file

    ..or subshells (note the parentheses instead of curly braces):

    (grep ...; bzgrep ...) >file

    ..to group the commands. The subshell way has nicer syntax (more forgiving of lack of whitespace and allows you to omit the last semicolon), but it either forks a new process, or "pretends" to by having the commands run in an cleaned up environment. Both have advantages depending on what you want to do, which don't matter here, but are worth looking up if you want more proficiency with the shell.

    Note: You can use pipelining with these tricks too, so you could do something like this:

    { grep ...; bzgrep ...; } | less

    P.S. if you don't care about the ordering of the matches in your combined output, you could use a single & between the two commands, like so: { grep ... & bzgrep ...; }. Then the two commands run simultaneously: the grep gets launched and the shell puts it in the background, then the shell will run the bzgrep. (But there's a small caveat with that, with the explanation involving file redirection and file stream buffering potentially causing a very small portion of the lines in the output file to get split/mangled: whether you'd see this would depend on how your grep, bzgrep, and libc stdio.h functions are implemented. In most implementations, I believe piping the command before redirecting to a file will avoid the problem, so you could do { foo & bar; } | cat - >file as a workaround.)

    • 1

  5. You can tie the commands together with && which will allow you to run each command.

    you could also add >> textfile.txt to the end of each command and have the output hit a file then mail that file out.

    • 0