vezult's questions

If requests are delayed by nginx rate limiting (rate limit exceeded, but within burst rate), is this delay included in the nginx $request_time total?

The nginx docs state that $request_time is "time elapsed since the first bytes were read from the client"

If a request is delayed, is that before or after request bytes are read from the client? I assume after, since rate limiting can be based on request headers, etc.

Is there a way to separate total request time and time spent specifically on sending/receiving network communication to/from the client?

Note: I am aware of $upstream_response_time, and am logging that. I specifically am concerned with differentiating specific operations within nginx (caching, rate limiting, etc), and client-side network communications.

While doing some auditing of a database, I found that some attachment content did not match the hashes given in the document's _attachments map.

I tested this by downloading the document and calculating its hash. Comparing that to couchdb showed that they did not match. I then noticed that the mismatched attachments were ones that couchdb was configured to compress. It appears that my couch id configured to use snappy compression:

foobox# grep -E 'file_compression|compressible_types' /etc/couchdb/{default,local}.ini 
/etc/couchdb/default.ini:file_compression = snappy
/etc/couchdb/default.ini:compressible_types = text/*, application/javascript, application/json, application/xml

However, when I attempt to compress the attachment content using snappy, and calculate the hash of the compressed data, it still does not match couchdb hash. In my example below, document-25977 is uncompressed (type application/pdf), and the uncompressed hash matches that provided by couchdb. The 2nd, document-78608, is a compressible type (text/plain), and the hashes do not match:

foobox$ python hashcompare.py
document-25977
couch len:  142918
couch hash:  028540dd92e1982bcb65c29d32e9617e (md5)
local uncompressed len:  142918
local uncompressed hash:  028540dd92e1982bcb65c29d32e9617e
local compressed len:  132333
local compressed hash:  3157583223dc1a53e1a3386d6abc312d

document-78608
couch len:  2180
couch hash:  e613ab6d7f884b835142979489170499 (md5)
local uncompressed len:  2180
local uncompressed hash:  0ab2516c820f5d7afb208e3be7b924dd
local compressed len:  1382
local compressed hash:  d9e79232662f57e6af262fc9f867eaf2

This is the script I used to do the comparison:

import couchdb
import snappy
import md5
import base64

server = couchdb.Server('http://localhost:9999')

db = server['program1']

for doc_id in ['document-25977', 'document-78608']:
    print doc_id
    doc = db[doc_id]
    att_stub = doc['_attachments'][doc_id]
    hash_type, tmpdigest = att_stub['digest'].split('-', 1)
    att = db.get_attachment(doc, doc_id)
    data = att.read()
    # CouchDB is using snappy compression
    compressed_data = snappy.compress(data)

    print 'couch len: ', att_stub['length']
    print 'couch hash: ', base64.b64decode(tmpdigest).encode('hex'), '(%s)' % hash_type
    print 'local uncompressed len: ', len(data)
    print 'local uncompressed hash: ', md5.md5(data).digest().encode('hex')
    print 'local compressed len: ', len(compressed_data)
    print 'local compressed hash: ', md5.md5(compressed_data).digest().encode('hex')
    print

I've verified that the documents are uncorrupted when fetched. So what am I missing? I'm not versed enough in Erlang to read the couchdb source and figure out what is going on. Why would the documents have a digest that does not match its contents compressed or other wise?

I've been using request tracker v4.0.8, over a relatively high latency internet connection, and I noticed that many (all?) of the supposedly static resources (css, javascript, images) are located in a /NoAuth/ directory.

The CSS and javascript for each page are generated by RT on each request, and are served with Cache-Control: no-cache. The images and additional requests generated by those files (misc images, more css, etc) are also no-cache. This all seems rather wasteful, to me.

Is there some trick or configuration that I can use to make RT operate in a less cache-hostile way?

I've noticed a bunch of errors that just recently appeared in /var/log/messages on one of our servers (below). However, the mce client seems to be less certain of the error source than the decoded entries in syslog. Is there some sort of key to use in order to interpret the MCE output?

Nov 12 04:19:19 areion kernel: [14698753.176035] Machine check events logged
Nov 12 04:19:19 areion mcelog: HARDWARE ERROR. This is *NOT* a software problem!
Nov 12 04:19:19 areion mcelog: Please contact your hardware vendor
Nov 12 04:19:19 areion mcelog: MCE 0
Nov 12 04:19:19 areion mcelog: CPU 0 BANK 8
Nov 12 04:19:19 areion mcelog: MISC 640738dd0009159c ADDR 96236c6c0
Nov 12 04:19:19 areion mcelog: TIME 1352711959 Mon Nov 12 04:19:19 2012
Nov 12 04:19:19 areion mcelog: MCG status:
Nov 12 04:19:19 areion mcelog: MCi status:
Nov 12 04:19:19 areion mcelog: MCi_MISC register valid
Nov 12 04:19:19 areion mcelog: MCi_ADDR register valid
Nov 12 04:19:19 areion mcelog: MCA: MEMORY CONTROLLER RD_CHANNELunspecified_ERR
Nov 12 04:19:19 areion mcelog: Transaction: Memory read error
Nov 12 04:19:19 areion mcelog: STATUS 8c0000400001009f MCGSTATUS 0
Nov 12 04:19:19 areion mcelog: MCGCAP 1c09 APICID 20 SOCKETID 1
Nov 12 04:19:19 areion mcelog: CPUID Vendor Intel Family 6 Model 44

All errors seem to be connected with the same memory bank:

areion:~# awk -F'mcelog:' '/mcelog:.*BANK/{ print $2; }' < /var/log/messages |uniq
 CPU 0 BANK 8 

I have the mcelog daemon running, and when I check for error information, it doesn't seem to know where the errors are coming from. Only that they are associated with CPU0 (we only have one CPU in this box):

Memory errors
SOCKET 1 CHANNEL any DIMM any
corrected memory errors:
        77 total
        77 in 24h
uncorrected memory errors:
        0 total
        0 in 24h
Per page corrected memory statistics:
359ffc000: total 2 2 in 24h online

3b93cc000: total 2 2 in 24h online

3ce45c000: total 2 2 in 24h online

96236c000: total 20 20 in 24h online triggered

96545c000: total 9 9 in 24h online

96a82c000: total 9 9 in 24h online

96a8ec000: total 1 1 in 24h online

96fb6c000: total 15 15 in 24h online triggered

9c2edc000: total 15 15 in 24h online triggered

9c5eac000: total 1 1 in 24h online

9c6a1c000: total 1 1 in 24h online

It's not at all clear how I am to interpret this information. On one hand, the mce client doesn't indicate channel or DIMM, but the decoded message indicates the errors occur on DIMM 8. dmesg seems to indicate that only 42 messages were logged:

[14698753.176035] Machine check events logged
[14698753.629174] Machine check events logged
[14698815.338595] __ratelimit: 38 callbacks suppressed
[14698815.338628] Machine check events logged
[14698816.020797] Machine check events logged

I seem to be getting mixed messages, which makes me wonder what assumptions to make based on the info reported from the various sources.

Misc info:

areion:~# grep 'model name' /proc/cpuinfo |uniq
model name      : Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz

areion:~# apt-cache policy mcelog |grep Installed
  Installed: 1.0~pre3-3

areion:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 6.0.6 (squeeze)
Release:        6.0.6
Codename:       squeeze

I'm attempting to create an ssh tunnel, spin it off into a background process, and write a pid file which I can later use to kill it. I CAN run ssh with -f, but then I don't have a handle on its PID.

However, when I attempt to run the script below via nohup or setsid, control is never returned to the shell from which I attempt to daemonize the script.

#!/bin/sh                                                 

while getopts ":v" flag; do
    case $flag in
        v|verbose) verbose=1;;
        :) echo "option -$lastflag requires an argument"; exit 1;;
    esac 
    lastflag="$flag"
    shift $((OPTIND - 1)); OPTIND=1
done

ssh  -p '2222'  -nNL '5984:localhost:5984' '[email protected]' &
pid=$!
sleep 2;
ps -p $pid >/dev/null 2>&1 || exit 1
echo $pid > "var/couchdb-tunnel.pid"
wait

Invocation:

nohup script.sh
setsid script.sh

How I do convince ssh (I assume) to let go of the terminal? Is there some other way I can accomplish what I want?

I'm attempting to set up Apache such that various error pages display with headers/footers appropriate to the context of the error. If the user was requesting a corporate page, I want to show the error with the corporate look and feel. If the error occurs in our application, I want to show it with the application look. Unfortunately, Apache does not want to play along.

In my testing, I've setup Apache to use a particular document when displaying an HTTP 503 error:

# Service Unavailable
ErrorDocument 503 /errors/error_503.shtml

In that document, using SSI directives and environment variables, I have constructed some if/else logic, which could conditionally include this or that header based on the content of an environment variable. This is is my test file:

<!--#if expr="${is_corporate}" -->
corporate here.
<!--#else -->
I'M NOT CORPORATE.
<!--#endif -->
<!--#echo var="is_corporate" --> 
<!--#echo var="REQUEST_URI" --> 
This is a 503 error message! Check it out!

However, I can't seem to match the request URI using SetEnvIf in the apache config, for no reason that I can understand.

   <Location />
      # Allow error documents to use the application or corporate header as appropriate
      SetEnvIfNoCase Request_URI /corporate/home/ is_corporate   # this does not match
      SetEnvIfNoCase Request_URI ^/$ is_corporate                # this does not match
      SetEnvIfNoCase Request_URI ^/c is_corporate                # this does not match
      SetEnvIfNoCase Request_URI ^/C is_corporate                # this does not match
      SetEnvIfNoCase Request_URI .*/ is_corporate                # this matches
      SetEnvIfNoCase Request_URI .*  is_corporate                # this matches
    </Location>

In this particular case, my error document, printing the contents of REQUEST_URI displays: /Corporate/Home/, and always indicates that is_corporate is empty. Using SetEnvIfNoCase should make this a case insensitive match, but trying uppercase/lowercase doesn't seem to make a difference.

Since the SetEnvIf regex is supposed to be pcre compatible, I tested the regex in perl, just to make sure my regex was legit, and should work the way I intend:

vezult@zapazoid:~$ echo '/Corporate/Home' |perl -e 'while(<>) { if (/^\/c/i) { print("match!!: $_\n");} else { print("no match: $_\n"); }}'
match!!: /Corporate/Home

It does. Anyway, so I'm stumped. What might I be missing that is preventing my SetEnvIf statement to behaving as I expect?

I have a Dell r710 (running Debian GNU/Linux 6.0) which is currently in a datacenter hundreds of miles away, and the ethernet card is stuck at 10baseT half duplex. 10baseT is OK, but I want it to be full duplex.

The card in question is a broadcom BCM5709:

areion:~# lspci |grep Ethernet
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

using the bnx2 driver/firmware:

areion:~# apt-cache policy firmware-bnx2
firmware-bnx2:
  Installed: 0.28
  Candidate: 0.28
  Version table:
 *** 0.28 0
        500 http://ftp.us.debian.org/debian/ squeeze/non-free amd64 Packages
        100 /var/lib/dpkg/status

And running the stock kernel for squeeze:

areion:~# uname -a
Linux areion 2.6.32-5-amd64 #1 SMP Wed Jan 12 03:40:32 UTC 2011 x86_64 GNU/Linux

I've tried changing the ethernet settings:

ethtool -s eth0 speed 10 duplex full autoneg off

But then I lose all connectivity, and have to reboot the server in order to connect to it again. To avoid having to constantly reboot the server, and to find out if the ethtool command was hanging, I switched to using this script:

#!/bin/bash

ethtool -s eth0 speed 10 duplex full autoneg off 2>&1 > ethtool.log &

sleep 10
if [ $(jobs -r |wc -l) -gt 0 ]; then
   kill %1 || kill -9 %1
else
  echo '-----------' >> ethtool.log
  ethtool eth0 >> ethtool.log
fi
dmesg |tail > dmesg-recent.log

ethtool -s eth0 autoneg on

This allows me to see the status of the interface after running the command:

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: off
        MDI-X: Unknown
        Supports Wake-on: g
        Wake-on: d
        Link detected: no

From dmesg:

[  442.865157] bnx2: eth0 NIC Copper Link is Down

It seems that either autonegotation is failing, or the switch that I am plugged into is advertising 10Mbit/half duplex. I am told that the switch is running at 10M full duplex.

Might this be a driver issue, hardware issue, cable issue, switch issue? Is there anything I can do remotely to identify the source of the problem, or at least eliminate possibilities?

Update:

Turning autonegotiation off, and setting the options to 10/HD, 10/FD, 100/HD, 100/FD, etc (anything other than autonegotiated 10/HD) results in NO-CARRIER, as reported by ip link show dev eth0. The same is true for leaving autonegotiation on, and advertising anything other than 10/HD. Is there any way this could be something other than an issue with the switch?