Questions[web-proxy](server)

I have a setup where I need a corporate upstream proxy, as well as several differnt proxies that are set up as SSH Socks proxies.

In the past I used px on the windows host machine to parse a self-made proxy.pac (which then depending on the host uses either the corporate proxy or a suitable local proxy via ssh).

Unfortunately they introduced a new security solution that blocks px somehow. Tried to build px as a docker image but it appears to me that px is 100% intended to be run on windows, so it looks like I cannot build it on linux (VM or docker image).

Do you know about any other proxy alternative which I can configure to read a proxy.pac and forward a web request according to this proxy.pac?

This way I could just set the http_proxy variables to this service and let it decide on the upstream instead of changing http_proxy over and over manually.

I use nginx as a reverse proxy for my Node.js application(s) which is running on localhost:3000. Both nginx itself and my application are two separate docker containers. (Separate because later I plan to add more apps to it.)

In my app container I have some .webp alternative images. E.g.

$ ls
 
/app/assets/img/image.jpeg
/app/assets/img/image.jpeg.webp
/app/assets/img/other.jpeg

I've tried using try_files with nginx but as far as I know this is not possible because try_files doesn't "see" my files inside another docker container unless they share a volume. Which they cant. The App container is built somewhere else with everything inside.

What I have right now it this:

upstream app {
  server localhost:3000;
}

[...]

location ~* .+\.(jpg|jpeg|svg|png)$ {
  proxy_pass http://app$uri$webp_suffix;
}

[...]

location / {
  proxy_pass http://app;
}

This is in fact working. The client requests .jpg image and gets .jpg.webp (because he can handle it). The problem is he always gets .jpg.webp. What I need is some if to check if the requested file is available in my app.

So something similar like this maybe? But this is not working :-(

location ~* .+\.(jpg|jpeg|svg|png)$ {
  if (-f http://app$uri$webp_suffix) {
    proxy_pass http://app$uri$webp_suffix;
  }
  proxy_pass http://app$uri;
}

There are many posts how to do that, most for older versions.

According to this post, to setup squid as transparent server for port 80 it should be as simple as this:

//squid.config
http_port 3128 transparent
http_port 80 accel #vhost option is deprecated
http_port 443 accel #not forget https 

but can't put it to work. i can't figure out if is my ip tables configuration, or the squid configuration.

Squid runs on the same pc as the gateway and dhcp server.

The iptables configuration:

# Generated by xtables-save v1.8.2 on Sun Feb  2 14:05:24 2020
*mangle
:PREROUTING ACCEPT [512813:147258305]
:INPUT ACCEPT [505693:146550975]
:FORWARD ACCEPT [5559:319150]
:OUTPUT ACCEPT [485369:200427691]
:POSTROUTING ACCEPT [486362:200704832]
COMMIT
# Completed on Sun Feb  2 14:05:24 2020
# Generated by xtables-save v1.8.2 on Sun Feb  2 14:05:24 2020
*filter
:INPUT ACCEPT [505677:146546207]
:FORWARD DROP [5532:317382]
:OUTPUT ACCEPT [485351:200422918]
-A FORWARD -p icmp -j ACCEPT
-A FORWARD -p tcp -m tcp --dport 80 -j ACCEPT
-A FORWARD -p tcp -m tcp --dport 443 -j ACCEPT
-A FORWARD -p tcp -m tcp --dport 25 -j ACCEPT
-A FORWARD -p tcp -m tcp --dport 110 -j ACCEPT
-A FORWARD -p tcp -m tcp --dport 993 -j ACCEPT
-A FORWARD -p icmp -j ACCEPT
COMMIT
# Completed on Sun Feb  2 14:05:24 2020
# Generated by xtables-save v1.8.2 on Sun Feb  2 14:05:24 2020
*nat
:PREROUTING ACCEPT [55608:3516287]
:INPUT ACCEPT [48615:2816337]
:POSTROUTING ACCEPT [4525:272304]
:OUTPUT ACCEPT [17086:1270605]
-A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128
-A PREROUTING -i wlan0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 127.1.1.0:3128
-A PREROUTING -i wlan0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128
-A POSTROUTING -o eth0 -j MASQUERADE
-A POSTROUTING -o wlan0 -j MASQUERADE
COMMIT

The squid.config file:

acl LocalNet src 192.168.4.0/24 #my network . 
http_access allow LocalNet

acl localnet src 0.0.0.1-0.255.255.255
#acl localnet src 10.0.0.0/8
#acl localnet src 100.64.0.0/10
#acl localnet src 169.254.0.0/16
#acl localnet src 172.16.0.0/12
#acl localnet src 192.168.0.0/16
acl localnet src fc00::/7
acl localnet src fe80::/10
acl SSL_ports port 443
acl Safe_ports port 80  #http
acl Safe_ports port 21  #ftp
acl Safe_ports port 443 #https
acl Safe_ports port 70  #gopher
acl Safe_ports port 210 #wais
acl Safe_ports port 1025-65535  #unregisted ports
acl Safe_ports port 280 #http-mgmt
acl Safe_ports port 488 #gss-http
acl Safe_ports port 591 #filemaker
acl Safe_ports port 777 #multilining http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
#include /etc/squid/conf.d/*
http_access allow localhost
http_access allow all

#for explicit proxy
#http_port 3128
#http_port 3129 tproxy

#for transparent proxy
http_port 3128 transparent
http_port 80 accel
http_port 443 accel

#cache
cache_dir ufs /var/spool/squid 100 16 256
coredump_dir /var/spool/squid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320
cache_effective_user proxy
cache_effective_group proxy

#log file
logformat timeread %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt
access_log daemon:/var/log/squid/access.log timeread

The squid start log:

squid[23403]: Starting Squid Cache version 4.6 for arm-unknown-linux-gnueabihf...
squid[23403]: Service Name: squid
squid[23403]: Process ID 23403
squid[23403]: Process Roles: worker
squid[23403]: With 1024 file descriptors available
squid[23403]: Initializing IP Cache...
squid[23403]: DNS Socket created at [::], FD 5
squid[23403]: DNS Socket created at 0.0.0.0, FD 9
squid[23403]: Adding domain Home from /etc/resolv.conf
squid[23403]: Adding nameserver 192.168.1.254 from /etc/resolv.conf
squid[23403]: Logfile: opening log daemon:/var/log/squid/access.log
squid[23403]: Logfile Daemon: opening log /var/log/squid/access.log
squid[23403]: Unlinkd pipe opened on FD 15
squid[23403]: Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
squid[23403]: Store logging disabled
squid[23403]: Swap maxSize 102400 + 262144 KB, estimated 28041 objects
squid[23403]: Target number of buckets: 1402
squid[23403]: Using 8192 Store buckets
squid[23403]: Max Mem  size: 262144 KB
squid[23403]: Max Swap size: 102400 KB
squid[23403]: Rebuilding storage in /var/spool/squid (clean log)
squid[23403]: Using Least Load store dir selection
squid[23403]: Set Current Directory to /var/spool/squid
squid[23403]: Finished loading MIME types and icons.
squid[23403]: HTCP Disabled.
squid[23403]: Pinger socket opened on FD 22
squid[23403]: Squid plugin modules loaded: 0
squid[23403]: Adaptation support is off.
squid[23403]: Accepting NAT intercepted HTTP Socket connections at local=[::]:3128 remote=[::] FD 18 flags=41
squid[23403]: Accepting reverse-proxy HTTP Socket connections at local=[::]:80 remote=[::] FD 19 flags=9
squid[23403]: Accepting reverse-proxy HTTP Socket connections at local=[::]:443 remote=[::] FD 20 flags=9
squid[23403]: Done reading /var/spool/squid swaplog (0 entries)
squid[23403]: Store rebuilding is 0.00% complete
squid[23403]: Finished rebuilding storage from disk.
squid[23403]:         0 Entries scanned
squid[23403]:         0 Invalid entries.
squid[23403]:         0 With invalid flags.
squid[23403]:         0 Objects loaded.
squid[23403]:         0 Objects expired.
squid[23403]:         0 Objects cancelled.
squid[23403]:         0 Duplicate URLs purged.
squid[23403]:         0 Swapfile clashes avoided.
squid[23403]:   Took 0.05 seconds (  0.00 objects/sec).
squid[23403]: Beginning Validation Procedure
squid[23403]:   Completed Validation Procedure
squid[23403]:   Validated 0 Entries
squid[23403]:   store_swap_size = 0.00 KB
squid[23403]: storeLateRelease: released 0 objects

Update

iptables configuration update below, according to @Piotr P. Karwasz. Also i add log that can be see at /var/log/messages.

# Generated by xtables-save v1.8.2 on Wed Feb  5 18:05:20 2020
*mangle
:PREROUTING ACCEPT [26643:2488050]
:INPUT ACCEPT [26417:2421007]
:FORWARD ACCEPT [203:57945]
:OUTPUT ACCEPT [46685:14572738]
:POSTROUTING ACCEPT [47031:14661263]
COMMIT
# Completed on Wed Feb  5 18:05:20 2020
# Generated by xtables-save v1.8.2 on Wed Feb  5 18:05:20 2020
*filter
:INPUT ACCEPT [26404:2417368]
:FORWARD ACCEPT [7:448]
:OUTPUT ACCEPT [46671:14569146]
-A INPUT -j LOG
-A FORWARD -j LOG
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o eth0 -j ACCEPT
-A OUTPUT -j LOG
COMMIT
# Completed on Wed Feb  5 18:05:20 2020
# Generated by xtables-save v1.8.2 on Wed Feb  5 18:05:20 2020
*nat
:PREROUTING ACCEPT [612:69658]
:INPUT ACCEPT [577:59792]
:POSTROUTING ACCEPT [26:2979]
:OUTPUT ACCEPT [543:44473]
-A PREROUTING -j LOG
-A PREROUTING -i wlan0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128
-A POSTROUTING -j LOG
-A POSTROUTING -o eth0 -j MASQUERADE
-A OUTPUT -j LOG
-A OUTPUT -o lo -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128
COMMIT
# Completed on Wed Feb  5 18:05:20 2020
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

For clarity, here the command i use to filter for dns query hits:

tail -f /var/log/messages |grep -wi --color  -w 'UDP.*DPT=53'

Update 2

Bind 9 was blocking the dns querys. even stoped. Guess is not a regular service. I changed bind9 settings to allow dns querys from anyplace.

Now i got hits like this on the log, for port 53:

Feb  5 18:36:43 MyLinuxBox kernel: [11771.211903] IN=wlan0 OUT= MAC=b8:27:eb:35:e4:5b:48:43:7c:04:7e:55:08:00 SRC=192.168.42.19 DST=192.168.42.1 LEN=59 TOS=0x00 PREC=0x00 TTL=255 ID=43761 PROTO=UDP SPT=63906 DPT=5

It seams the dns query pass through the linux-box, from wlan0 to eth0, as is supposed too, but the response doesn't reach the client, resulting in a dns query timeout.

It is possible bind9 is still messing with the query.

We have several PCs all connected to a switch; What I need from Squid is to share the internet between everyone and at the same time gives me control over what PC can access to internet (password authentication or MAC filtering), with what bandwidth and daily data quota.

First Q: Can Squid provides these requirements?

I don't want the end user to bother with any configurations like entering the proxy IP and port number (I mean the proxy configurations (IP and port number) in Internet Explorer and other softwares) for accessing the proxy.

Besides most of the softwares lack such settings and also this proxy configurations can be very hard to apply especially if we want a new comer to use the network or a smart phone and we also cant use any VPN software (or any software that lacks proxy configurations) on the systems.

Second Q: Can squid run without configurations on the part of end users?

Does there exist a forward proxy server that will lookup and obey robots.txt files on remote internet domains and enforce them on behalf of requesters going via the proxy?

e.g. Imagine a website at www.example.com that has a robots.txt file that restricts certain URLs and applies Crawl-Delays to others.

Multiple automatic clients (e.g. crawlers, scrapers) could then, going via the proxy, access the website at www.example.com without violating the robots.txt directives AND without having to access the file themselves (=> simpler clients and less requests to get robots.txt)

(Specifically, I am looking at the "GYM2008" version of the spec - http://nikitathespider.com/python/rerp/#gym2008 - because it's in wide use)