Home / ubuntu / Questions / 1039476
Accepted
Olimjon
Asked: 2018-05-24 07:59:08 +0800 CST

How to get total download size of a list?

  • 772

Imagine that, I have got download-list.txt. It contains some file urls:

http://example.com/a.txt
http://example.com/b.txt
http://example.com/c.txt
http://example.com/d.txt
http://example.com/e.txt

When I run wget -i download-list.txt --spider, it shows like:

http://example.com/a.txt
...
Length: 128 (128B) [text/txt]
...
http://example.com/b.txt
...
Length: 120 (120B) [text/txt]
...
http://example.com/c.txt
...
Length: 100 (100B) [text/txt]
...
http://example.com/d.txt
...
Length: 90 (90B) [text/txt]
...
http://example.com/e.txt
...
Length: 80 (80B) [text/txt]
...

But, I do not need individual file size, I need total file size: 

518 (128+120+100+90+80)

How to achieve this result? (Any other methods without wget is also appreciated)

command-line files wget

1 Answers

  1. Best Answer

    I would suggest to use curl:

    <download-list.txt xargs -n1 curl -sI |
  awk '/[cC]ontent-[lL]ength/{total_size+=$2} END{ print total_size " bytes" }'

    The curl will print the header information (which file?) as like below that contains "Content-Length" in bytes (if the remote server can provide that); then with awk we are summing-up the second column which is the file size for the line if matched with Content-Length and at the END print the total size saved in total_size variable. 

    HTTP/1.1 200 OK
Content-Length: 1921843200
Content-Type: application/x-iso9660-image
ETag: "728d0000-56ac4c63786e2"
Server: Apache/2.4.18 (Ubuntu)
Last-Modified: Thu, 26 Apr 2018 18:44:15 GMT
Connection: keep-alive
Date: Wed, 23 May 2018 16:45:08 GMT

    from man curl

    -I, --head
    (HTTP/FTP/FILE) Fetch the HTTP-header only! HTTP-servers feature the command HEAD which this uses to get nothing but the header of a document. When used on an FTP or FILE file, curl displays the file size and last modification time only.

    -s, --silent
    Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it.

    • 5