Timothy R. Butler's questions

I'm trying to add etag based cache verification to a web app I've developed that runs on Apache via an NGINX reverse proxy. The web app is written in Perl. In the Perl script, I check for the $ENV{'HTTP_IF_NONE_MATCH'} header and then test the etag in it, if it exists. If I access the app via cURL, it works:

timothybutler@timothys-MacBook-Pro ~ % curl -I https://server/resource/podcast/20211101.mp3
HTTP/1.1 200 OK
Server: nginx/1.21.4
Date: Fri, 12 Nov 2021 18:06:37 GMT
Content-Type: audio/mpeg
Content-Length: 16665729
Connection: keep-alive
Surrogate-Control: no-cache
Cache-Control: private,max-age=864000
Etag: "1635825322"

Giving that etag back via cURL, I get the expected result:

timothybutler@timothys-MacBook-Pro ~ % curl -H 'If-None-Match: "1635825322"' -I https://server/resource/podcast/20211101.mp3
HTTP/1.1 304 Not Modified
Server: nginx/1.21.4
Date: Fri, 12 Nov 2021 18:19:29 GMT
Connection: keep-alive
Cache-Control: private, max-age=864000
Etag: "1635825322"

However, the process breaks down if I try it in Chrome instead. Chrome reports sending If-None-Match: "1635825322" as one of the headers when I refresh the page, but my server does not set the HTTP_IF_NONE_MATCH environmental variable for that request contrary to the one that came in via cURL. If I dump all of the %ENV hash, the If-Match-None header and its etag is simply not present in any form when Chrome issues the request.

I know Vary: Cookie is not well liked, since it causes caching of every variation of cookies a user might report. On the other hand, without it, I've had to use Cache-Control: no-cache to tell my NGINX caching server to skip caching content entirely that might have a user logged in (every page of my site, incidentally, since there's an account box at the top of the page).

I only send cookies if the user logs in, so most page views would benefit from caching. I would like to narrow it down so that there are only two variations: a cached version for users whose "variance" is that they have no cookies and a non-cached version for those who have cookies (e.g. the user is logged in and has authentication cookies).

If I add "Vary: Cookie" to my Perl program, is there a way for me to tell NGINX that in this case it should only pay attention to whether a sessionId cookie exists or not? Something along the lines of -- to make a pseudo-header -- Vary: Cookie('sessionId')? If there is a way to implement this in the header produced by the script that NGINX would then respect, I think I'd prefer that over putting it in the NGINX configuration, but any solution would be appreciated.

I'm not entirely sure how to test if I'll cause NGINX to go crazy with caching, but I wonder if something along these lines could work: on a viewer who is not logged in, the script sends Vary: Cookie, but on a viewer who is logged in, it sends both that header and Cache-Control: no-cache. Would NGINX understand this to be telling it to cache the first response but not the variant? Or would I mess something up in ways I'm not catching just now?

Update: Here's a sample of one of my server blocks:

server {
    server_name myservername.tld
    listen 80;

    location / {
        proxy_cache $PROXY_CACHE;

        proxy_cache_valid 200 302 60m;
        proxy_cache_valid 404 1m;
        proxy_cache_use_stale error timeout http_429 http_500 http_502 http_503 http_504;
        proxy_cache_background_update on;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 1;
        proxy_cache_lock on;

        proxy_pass $scheme://$APACHE_PROXY_REQ_IP:$APACHE_PROXY_REQ_PORT;
    }

}

I have a program that creates optimized versions of pictures that are uploaded to the /images/ folder on the web server. It traverses subfolders and in each one, creates a .optimized folder that holds the optimized version, if it is at least a certain size smaller than the original. My goal is to check if such an optimized version exists, serve it if it does, and serve the original otherwise (in some sense like how gzip_static serves a .gz version of a file if it exists).

I'm running NGINX as a proxy in front of Apache, so while I'm accustomed to handling issues like this using htaccess, I am trying to do it natively in NGINX to avoid the server having to hand the request over to Apache. In .htaccess, I could do something like this:

RewriteCond %{REQUEST_FILENAME} ^(/images/(?:.*/)?)(.*?)$ [OR]
RewriteCond $1.optimized/$2 -f
RewriteRule .* $1/.optimized/$2 [L]

Is there a good way to handle this directly in NGINX? Most similar use cases I've found kept all the cached/optimized files in a single cached folder, as opposed to the structure I'm describing.

I have a couple of podcasts I host on my site and I've noticed a disturbing trend the last couple of months: my site's bandwidth usage has gone up by 10x, but it appears most of it was a series of Google App Server instances, not an incredible increase in listeners. Digging through the logs, I think most of them appear to identify themselves as "Spotify/1.0," which is hitting my server several times every minute and downloading different old podcast episodes.

I'm wondering if this has something to do with me recently implementing NGINX as a caching server in front of Apache, because the spike roughly coincided with that change. Is there some issue with how I'm answering Spotify's bot? Is this a known issue with their indexer I need to somehow deal with?

For example:

 35.240.121.201 - - [16/May/2021:12:16:14 -0500] "GET /littlehillschurch/resources/podcasts/20201214.mp3 HTTP/1.1" 200 15004716 "-" "Spotify/1.0" "-"
 35.195.247.67 - - [16/May/2021:12:16:15 -0500] "GET /littlehillschurch/resources/podcasts/20200803.mp3 HTTP/1.1" 200 14141931 "-" "Spotify/1.0" "-"
 146.148.19.22 - - [16/May/2021:12:16:15 -0500] "GET /littlehillschurch/resources/podcasts/20210503.mp3 HTTP/1.1" 200 14243142 "-" "Spotify/1.0" "-"
 35.240.121.201 - - [16/May/2021:12:16:15 -0500] "GET /littlehillschurch/resources/podcasts/20210125.mp3 HTTP/1.1" 200 15050067 "-" "Spotify/1.0" "-"
 35.195.91.128 - - [16/May/2021:12:16:15 -0500] "GET /littlehillschurch/resources/podcasts/20200817.mp3 HTTP/1.1" 200 15266593 "-" "Spotify/1.0" "-"
 35.187.181.74 - - [16/May/2021:12:16:15 -0500] "GET /littlehillschurch/resources/podcasts/20200921.mp3 HTTP/1.1" 200 14607340 "-" "Spotify/1.0" "-"
 35.195.247.67 - - [16/May/2021:12:16:15 -0500] "GET /littlehillschurch/resources/podcasts/20210222.mp3 HTTP/1.1" 200 15279536 "-" "Spotify/1.0" "-"
 35.195.91.128 - - [16/May/2021:12:16:15 -0500] "GET /littlehillschurch/resources/podcasts/20210208.mp3 HTTP/1.1" 200 15480738 "-" "Spotify/1.0" "-"
 35.189.225.190 - - [16/May/2021:12:16:15 -0500] "GET /littlehillschurch/resources/podcasts/20210510.mp3 HTTP/1.1" 200 16093457 "-" "Spotify/1.0" "-"
 35.205.135.106 - - [16/May/2021:12:16:15 -0500] "GET /littlehillschurch/resources/podcasts/20200907.mp3 HTTP/1.1" 200 14715203 "-" "Spotify/1.0" "-"
 35.205.135.106 - - [16/May/2021:12:16:16 -0500] "GET /littlehillschurch/resources/podcasts/20200720.mp3 HTTP/1.1" 200 14100420 "-" "Spotify/1.0" "-"

The actual files being requested seem to report correct modified dates, etc. Here's the output from curl -v that seems relevant if I request one of the files:

* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx/1.20.0
< date: Sun, 16 May 2021 17:38:40 GMT
< content-type: audio/mpeg
< content-length: 14100420
< strict-transport-security: max-age=16070400; includeSubDomains
< last-modified: Tue, 21 Jul 2020 00:52:50 GMT
< accept-ranges: bytes
< access-control-allow-origin: *

It appears most discussions about FastCGI here are frozen from nearly a decade ago and so as someone who just decided to start making use of it, I'm wondering if I am making a mistake with unforeseen security or compatibility issues ahead of me. Here are a few things that prompted my concern:

• It appears the libraries for it such as the Perl CPAN module, have not been updated in years.
• The Perl module complains about UTF-8 output years after that became normal and the lack of updates don't seem to offer a lot of hope for a fix.
• I also noted that Mojolicious spun off its FastCGI support some years ago for not meeting the quality standards of the codebase. I don't presently use Mojolicious, but I found that notable.

Have a chosen a dead end by finally waking up to FastCGI this year?

I have clients who use PHP, so I'm interested in PHP support, but I'm primarily interested in if there is a better future-oriented alternative to use that supports Perl, which is the language I use for most of the CGI programs on the server. Maybe I'm overthinking it and FastCGI will be just fine. So far, I'm using mod_fcgid with Apache, although my goal is to get it running directly with Nginx, which I'm already using to serve static content.

Yesterday, I tried to upgrade my FreeNAS 9.10.2 system to FreeNAS 11. The upgrade failed to set the boot environment to FreeNAS 11, so I did that manually after seeing a reboot simply took me back to FreeNAS 9.10.2. That was a mistake: now it boots into what appears to be FreeNAS 11, but when I try to go to the web GUI, I simply get "Internal Server Error." I am able to SSH into the server, at which point I tried to trigger another upgrade by typing sudo freenas-update -v update. When I try this, here is what I get back:

[freenasOS.Configuration:692] TryGetNetworkFile(['http://update-master.ixsystems.com/FreeNAS/FreeNAS-9.10-STABLE/LATEST'])
[freenasOS.Configuration:822] TryGetNetworkFile(['http://update-master.ixsystems.com/FreeNAS/FreeNAS-9.10-STABLE/LATEST']):  Read 2360 bytes total
[freenasOS.Configuration:692] TryGetNetworkFile(['http://update-master.ixsystems.com/updates/ix_crl.pem'])
[freenasOS.Configuration:84] CheckFreeSpace(path=/tmp/tmpa9lojvfr.pem, pool=None, required=1028)
[freenasOS.Configuration:822] TryGetNetworkFile(['http://update-master.ixsystems.com/updates/ix_crl.pem']):  Read 1028 bytes total
[freenasOS.Update:977] Going to try checking cached manifest /var/db/system/update/MANIFEST
[freenasOS.Update:1001] Got this exception: [Errno 2] No such file or directory: '/var/db/system/update/MANIFEST'
[freenasOS.Configuration:692] TryGetNetworkFile(['http://update.ixsystems.com/FreeNAS/Validators/ValidateUpdate-dAn5lU.txt', 'http://update-master.ixsystems.com/FreeNAS/Validators/ValidateUpdate-dAn5lU.txt'])
[freenasOS.Configuration:84] CheckFreeSpace(path=/var/db/system/update/ValidateUpdate, pool=None, required=187)
[freenasOS.Configuration:822] TryGetNetworkFile(['http://update.ixsystems.com/FreeNAS/Validators/ValidateUpdate-dAn5lU.txt', 'http://update-master.ixsystems.com/FreeNAS/Validators/ValidateUpdate-dAn5lU.txt']):  Read 187 bytes total
[freenasOS.Update:1034] DownloadUpdate:  No update available
No updates available

Is there any way to force (from the command line) for the system to go through the FreeNAS 11 install process again so that it fill in whatever clearly didn't install properly in the previous version? If I can do it by SSH that would be idea -- the system is headless, so if I can take advantage of that SSH connection to repair the damage without starting from scratch, that'd be very helpful.

(I tried editing /boot/grub/grub.cfg to set default to the previous, FreeNAS 9.x entry so that the system would boot to the previous version, but that didn't seem to have an effect when I rebooted. Of course, that still wouldn't solve the improper FreeNAS 11 install anyway.)

I have an RHEL 6.8 x86_64 server that has had a long running problem with the SSH server. It has openssh-5.3p1-118.1.el6_8.x86_64 installed. Every so many attempts to connect to SSH/SFTP, the server gives back the error:

ssh_exchange_identification: read: Connection reset by peer

It doesn't happen every time, so it is very hard to debug. When it does happen, even with more verbose logging, nothing shows up in the SSH logs about what happened. What I have noticed is that I have to restart sshd every few weeks or it quits working all together.

Can you offer any suggestions on how I could diagnose this issue and fix it?

Update (27-Aug-16): Here's the results of ssh -vvv on a failed connection:

timothys-MacBook-Pro:~ tbutler$ ssh -vvv tbutler@myserver
OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to myserver [xx.xx.xx.xx] port 22.
debug1: Connection established.
debug1: identity file /Users/tbutler/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tbutler/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tbutler/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tbutler/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tbutler/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tbutler/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tbutler/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tbutler/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
ssh_exchange_identification: read: Connection reset by peer

I tried logging in again and received the same result. Finally on the third try, I was able to connect, which resulted in the debug information looking much the same, but then continuing with:

debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000

I've resized many partitions over the years on Linux, Windows and Mac OS X -- but always using a GUI. However, the time has come where the preset partition sizes my data center placed on my server aren't the right sizes and I need to resize a production server's disks. I could fiddle with it and probably do OK, but given that it is a production server, I wanted to get some advice about the right way to do this. I do have KVM over IP access, so if it is best to take the server offline and boot off a rescue partition, I can do that.

root [/var/lib/mysql]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       9.9G  2.1G  7.3G  23% /
tmpfs           7.8G     0  7.8G   0% /dev/shm
/dev/sda1        99M   77M   18M  82% /boot
/dev/sda8       884G  463G  376G  56% /home
/dev/sda3       9.9G  8.0G  1.5G  85% /usr
/dev/sda5       9.9G  9.1G  308M  97% /var
/usr/tmpDSK     2.0G   38M  1.8G   3% /tmp

As you can see /var and /usr are quite close to being full and I've actually had to symlink some logs on /usr to directories in /home to balance things out. What I would like to do is to add 6-10 GB each to /usr and /var, presumably taking the space from /home.

As I think about how the disk is arranged, the best thought I've come up with is to reduce /home by 16 GB, say, and move /var to the spot freed up, then allocating /var's space to /usr. However, that would put /var at the far end of the disk, which seems less than idea, given that MySQL has all of its data on that partition. I'd love to take the space out of the closer end of /usr, but I assume that would take a very arduous (and perhaps risky) process of moving all of the data in /usr around. I seem to recall having such a process fail for me on a computer in the past.

The other option might be to merge / and /usr since / is underutilized, though I'm not sure if that's a good idea.

Do you have any suggestions both on the best reallocation plan and the commands to use to accomplish it?

UPDATE: I should add -- here's the partition table. There's one unused partition, which, if memory serves, was the original tmp location before I created a tmp image:

    Name        Flags      Part Type  FS Type          [Label]      Size (MB)
 ------------------------------------------------------------------------------
                                      Unusable                      1.05*
    sda1        Boot        Primary   Linux ext2                    106.96*
    sda2                    Primary   Linux ext3                    10737.42*
    sda3                    Primary   Linux ext3                    10737.42*
    sda5        NC          Logical   Linux ext3                    10738.47*
    sda6        NC          Logical   Linux swap / Solaris          2148.54*
    sda7        NC          Logical   Linux ext3                    1074.80*
    sda8        NC          Logical   Linux ext3                    964098.53*

I have a long Ethernet run (about 280 ft/85 m) that goes through a conduit between two buildings. On each end, I have an APC Ethernet surge protector that is grounded to the main ground for each building. While I had some trouble getting it to work at first, once I stumbled across Zyxel's very fine 24-port switches, it has worked really, really well.

Then, a storm hit and the link quit working. As far as I can tell the switches were fine and I've tried replacing both APC surge protectors. It appeared the cable in the conduit -- which has a water leak in it -- was damaged. However, I tried new, better insulated cable and it didn't work (either through the conduit or running on the ground). On the other hand, I had three spare 100ft (30m) pre-terminated cables that I coupled together and that worked just fine running across the ground. I can't imagine why three unshielded cables patched together would work while nicely terminated, shielded direct burial grade Cat5e cable would not.

There is one difference I just observed after putting everything away for the evening: I've been using the TIA 568A wire pattern to the RJ45 connectors and the pre-terminated cable used the TIA 568B pattern.

Could that make the difference in signal loss? Can you think of anything else that would cause 3 patched together cables to work but one solid run not to work?

Someone who has a lot of experience with fiber optic cable told me that somehow the cable needs to be pushed through conduits rather than pulled like I would with a Cat 6. Is this true? If so, what is the best way to push the cable through?

I am trying to get an Gig-E network between two buildings that are approximately 260 ft. away. While some TRENDnet switches failed to be able to connect to each other over Cat 6 at that distance, two Netgear 5-port Gig-E switches do so just fine. However, it still fails after I put in place APC PNET1GB ethernet surge protectors at each end before the line connects to the respective switches. So I find myself wondering if I simply need to find a better surge protector that doesn't degrade the signal as much (if so, what kind would you recommend?) or if I should give up on copper and use fiber between the buildings.

If I opt to go the latter route, I could really use some pointers. It looks like LC connectors are the most common, but I keep running into some others as well. A media converter on each end seems like the simplest solution, but perhaps a Gig-E switch with an SFP port would make more sense? Given a very limited budget, sticking with my existing copper seems best, but if it is bound to be a headache, a 100 meter fiber cable is something I think I can swing cost wise.

I have just set up an HP Micro Server N40L as a FreeNAS with 4 2tb drives in a RAIDZ. I am getting around 30-40MB/sec, with occasional bursts of 50MB/sec., reads and writes that are similar to the low end of that range. As far as I can tell the CPU is not overworked and when I did a file copy internally from one folder in the RAIDZ to another, I hit over 80MB/sec.

I am wondering what might be the best approach to improving read and write speeds. Tweaking my GigE network, adding a ZFS cache drive, etc.?

I am trying to figure out how to get FreeNAS 8 to sleep when inactive and, ideally, wake on lan activity (or, less ideally, wake on a WOL magic packet). However, as I've tried to search for information on how to do this, almost all discussions seem to be centered on FreeNAS 7. Also, the tools included in FreeBSD to do this seem to be missing (i.e. acpiconf, etc.).

Is there a way to get FreeNAS 8 to sleep and wake so that I don't have to leave the server running all the time? Given its usage level, it seems a waste to have the server running constantly.

I just set up a FreeNAS 8.0.3 server and am trying to use its AFP (Netatalk) service to access it via a Mac OS X Lion system. I created the ZFS volume, set its permissions to include my user in its owner group (and set group write permissions), created an AFP share with AFP3 and told that share to "allow" @uninet (my group). I have a user on the server named tbutler, matching the user on my Mac.

I can see the server, "Beatrice," in Finder. When I try to login in Finder using "Connect As...," the user "tbutler" and the proper password, I am returned to the main Finder window with the black bar now saying "Connection Failed."

Here's the most recent data from /var/messages on the server, which shows me trying to login both as a "Registered User" and a "Guest":

Jul 30 00:29:07 freenas afpd[8972]: AFP3.3 Login by nobody
Jul 30 00:29:08 freenas afpd[8972]: AFP logout by nobody
Jul 30 00:29:08 freenas afpd[8972]: dsi_stream_read: len:0, unexpected EOF
Jul 30 00:29:08 freenas afpd[8972]: afp_over_dsi: client logged out, terminating DSI session
Jul 30 00:29:08 freenas afpd[8972]: AFP statistics: 0.14 KB read, 0.12 KB written
Jul 30 00:29:14 freenas afpd[8975]: AFP3.3 Login by tbutler
Jul 30 00:29:14 freenas afpd[8975]: AFP logout by tbutler
Jul 30 00:29:14 freenas afpd[8975]: dsi_stream_read: len:0, unexpected EOF
Jul 30 00:29:14 freenas afpd[8975]: afp_over_dsi: client logged out, terminating DSI session
Jul 30 00:29:14 freenas afpd[8975]: AFP statistics: 0.62 KB read, 0.48 KB written
Jul 30 00:29:20 freenas afpd[8978]: AFP3.3 Login by tbutler
Jul 30 00:29:20 freenas afpd[8978]: AFP logout by tbutler
Jul 30 00:29:20 freenas afpd[8978]: dsi_stream_read: len:0, unexpected EOF
Jul 30 00:29:20 freenas afpd[8978]: afp_over_dsi: client logged out, terminating DSI session
Jul 30 00:29:20 freenas afpd[8978]: AFP statistics: 0.62 KB read, 0.48 KB written
Jul 30 00:29:27 freenas afpd[8983]: AFP3.3 Login by nobody

(My clock is clearly not properly set, but be that as it may...)

Any suggestions?

UPDATE: Apparently this problem occurs if one gives the AFP share a password in the AFP share settings box. When I removed the password and tried to login using a user account again, it worked just fine.

I just purchased an HP Microserver I was planning to use with FreeNAS. It needed more RAM, so I tried to follow the instructions to pull out the system board draw to access the DIMM slots. Doing so requires removing all of the cables connected to the board, including a "Mini-SAS" cable. The cable looks like it has an oversized USB-style connector with a metal clip to lock it into place similar to an RJ-45 connector's plastic clip. Try as I might, I couldn't get it to unlock, so I called HP tech support and they had me tug on it a good deal more still to no avail.

I'm thinking maybe the system shipped with a damaged connector, based on the HP video about disconnecting the cable. Has anyone worked with one of these systems? Can you comment on how hard it should be to disconnect that cable?

(Now the system won't start up and HP advised me to do an exchange, which I intend to do, but I found myself wondering if I'd have the same problem with the next unit.)

I am planning on setting up a FreeNAS ZFS pool using two 2 tb drives and two 1tb drives. I would like to have data redundancy somehow. I read this post which seems to suggest my two options are to lose 1tb per 2tb drive using RAIDZ or to use this command (which doubles the space required for data redundancy):

zfs set copies=2 poolname

Is there a better way to handle this and acheive something more like Drobo's Beyond RAID or Netgear's XRAID? Alternately, could I partition each 2tb drive into 1 tb partitions and put both partitions in the pool?

I am troubleshooting a computer that is running Windows 7 within a domain. The "look in" box on save and open dialogs is missing all but one entry. I'm guessing that someone flipped on a policy setting that disabled the other entries, because I am now noticing that entries such as "Computer" and "C:" are missing from all of the computers affected by the system policies.

Can anyone point me to where such a setting would appear within Windows group policy controls?