Home / server / Questions / 473940
Accepted
ALex_hha
Asked: 2013-01-31 11:16:50 +0800 CST

Caching large amounts of data

  • 772

We need to cache all psd/zip files from our amazone S3 storage. The amount of the cache between 150-200 Gb (we need to store files at least 2 month). Now we use transparent proxy squid without any caching.

So my question: is it possible to configure "main" squid such way that all queries to the s3.amazonaws.com he will proxy to the parent squid. In the parent squid we need to cache all zip and psd files only from s3.amazonaws.com. If there are no such files in the cache the parent squid will be download and put "new" files in the cache.

How will squid works with files 1-2 gb? Is there any limitations?

The main proof of this setup is that all psd/zip files from amazone storage delivered to the local clients with maximal speeds from local cache. Because the same files at the same time used by different employees and we spent a lot of time to download those files by each employees

Can this issue be solved in different ways?

linux

1 Answers

  1. Best Answer

    We have tested our system about month and it's worked fine. if someone is interested, below I have added configuration files (main/parent squid)

    main squid (192.168.100.1) configuration file

    acl local_net src 192.168.0.0/16
acl parent_peer peername PARENT_PEER
acl parent_squid dst 192.168.100.50

acl FILE_TO_CACHE urlpath_regex \.(zip|iso|rar)$
acl STORAGE dstdomain storage.example.net

http_port 192.168.100.1:3128 intercept
icp_port 3130

cache_peer 192.168.100.50 parent 3128 3130 name=PARENT_PEER connect-timeout=7 proxy-only
cache_peer_access PARENT_PEER allow STORAGE FILE_TO_CACHE

# to connect to parent via internal interface
tcp_outgoing_address 192.168.100.1 parent_peer

# to properly get cache digest from parent
tcp_outgoing_address 192.168.100.1 parent_squid

# sent all other packets via ISP2
tcp_outgoing_address xxx.xxx.xxx.xxx local_net

    parent squid (192.168.100.50) configuration file

    acl main_squid src 192.168.100.1

acl FILE_TO_CACHE urlpath_regex -i \.(zip|iso|rar)$

cache allow FILE_TO_CACHE
cache deny all

http_access allow main_squid
http_access allow localhost
http_access deny all

icp_port 3130
icp_access allow main_squid
icp_access deny all

http_port 192.168.100.50:3128

cache_mem 12 GB
maximum_object_size_in_memory 64 MB
minimum_object_size 16384 KB
maximum_object_size 1024 MB

cache_swap_low 93
cache_swap_high 98

refresh_pattern \.iso$ 129600 100 129600 override-lastmod override-expire ignore-reload
refresh_pattern \.zip$ 129600 100 129600 override-lastmod override-expire ignore-reload
refresh_pattern \.rar$ 129600 100 129600 override-lastmod override-expire ignore-reload

cache_effective_user squid
cache_effective_group squid
    • 3