Home / server / Questions / 84815
Accepted
Garret Heaton
Asked: 2009-11-15 16:40:45 +0800 CST

How can I get the size of an Amazon S3 bucket?

  • 772

I'd like to graph the size (in bytes, and # of items) of an Amazon S3 bucket and am looking for an efficient way to get the data.

The s3cmd tools provide a way to get the total file size using s3cmd du s3://bucket_name, but I'm worried about its ability to scale since it looks like it fetches data about every file and calculates its own sum. Since Amazon charges users in GB-Months it seems odd that they don't expose this value directly.

Although Amazon's REST API returns the number of items in a bucket, s3cmd doesn't seem to expose it. I could do s3cmd ls -r s3://bucket_name | wc -l but that seems like a hack.

The Ruby AWS::S3 library looked promising, but only provides the # of bucket items, not the total bucket size.

Is anyone aware of any other command line tools or libraries (prefer Perl, PHP, Python, or Ruby) which provide ways of getting this data?

amazon-web-services

27 Answers

  1. This can now be done trivially with just the official AWS command line client:

    aws s3 ls --summarize --human-readable --recursive s3://bucket-name/

    Official Documentation: AWS CLI Command Reference (version 2)

    This also accepts path prefixes if you don't want to count the entire bucket:

    aws s3 ls --summarize --human-readable --recursive s3://bucket-name/directory
    • 468
  2. Best Answer

    The AWS CLI now supports the --query parameter which takes a JMESPath expressions.

    This means you can sum the size values given by list-objects using sum(Contents[].Size) and count like length(Contents[]).

    This can be be run using the official AWS CLI as below and was introduced in Feb 2014

     aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"
    • 210

  3. AWS Console:

    As of 28th of July 2015 you can get this information via CloudWatch. If you want a GUI, go to the CloudWatch console: (Choose Region > ) Metrics > S3

    AWS CLI Command:

    This is much quicker than some of the other commands posted here, as it does not query the size of each file individually to calculate the sum.

     aws cloudwatch get-metric-statistics --namespace AWS/S3 --start-time 2015-07-15T10:00:00 --end-time 2015-07-31T01:00:00 --period 86400 --statistics Average --region eu-west-1 --metric-name BucketSizeBytes --dimensions Name=BucketName,Value=toukakoukan.com Name=StorageType,Value=StandardStorage

    Important: You must specify both StorageType and BucketName in the dimensions argument otherwise you will get no results. All you need to change is the --start-date, --end-time, and Value=toukakoukan.com.

    Here's a bash script you can use to avoid having to specify --start-date and --end-time manually.

    #!/bin/bash
bucket=$1
region=$2
now=$(date +%s)
aws cloudwatch get-metric-statistics --namespace AWS/S3 --start-time "$(echo "$now - 86400" | bc)" --end-time "$now" --period 86400 --statistics Average --region $region --metric-name BucketSizeBytes --dimensions Name=BucketName,Value="$bucket" Name=StorageType,Value=StandardStorage
    • 177

  4. s3cmd can do this :

    s3cmd du s3://bucket-name

    • 110

  5. If you download a usage report, you can graph the daily values for the TimedStorage-ByteHrs field.

    If you want that number in GiB, just divide by 1024 * 1024 * 1024 * 24 (that's GiB-hours for a 24-hour cycle). If you want the number in bytes, just divide by 24 and graph away.

    • 26

  6. If you want to get the size from AWS Console:

    1. Go to S3 and select the bucket
    2. Click on "Metrics" tab

    enter image description here

    By default you should see Total bucket size metrics on the top

    • 25

  7. Using the official AWS s3 command line tools:

    aws s3 ls s3://bucket/folder --recursive | awk 'BEGIN {total=0}{total+=$3}END{print total/1024/1024" MB"}'

    This is a better command, just add the following 3 parameters --summarize --human-readable --recursive after aws s3 ls. --summarize is not required though gives a nice touch on the total size.

    aws s3 ls s3://bucket/folder --summarize --human-readable --recursive
    • 22

  8. s4cmd is the fastest way I've found (a command-line utility written in Python):

    pip install s4cmd

    Now to calculate the entire bucket size using multiple threads:

    s4cmd du -r s3://bucket-name
    • 15

  9. You can use the s3cmd utility, e.g.:

    s3cmd du -H s3://Mybucket
97G      s3://Mybucket/
    • 9

  10. So trolling around through the API and playing some same queries, S3 will produce the entire contents of a bucket in one request and it doesn't need to descend into directories. The results then just requiring summing through the various XML elements, and not repeated calls. I don't have a sample bucket that has thousands of items so I don't know how well it will scale, but it seems reasonably simple.

    • 6