Jason Cohen's questions

I'm using the nginx X-Accel-Redirect feature where a proxied backend (A) can "internally redirect" nginx to reprocess a new URI (B) as if that were the one originally given by the client browser.

It works great in making request B (also proxied)and returning B's resulting headers back to the client browser. So far so good.

But also I have some headers from A which I'd like to pass along too. In other words, merge a handful of headers from A in with the result from B.

How can I do this? I've tried

add_header  X-My-Header  $http_x_my_header;

But that $http_x_my_header variable is empty, presumably because the original request didn't have it. I've also tried $upstream_x_my_header but that's empty too. I've tried things like proxy_pass_header X-My-Header; but no.

I have CPU I/O wait steady around 50%, but when I run iostat 1 it shows little to no disk activity.

What causes wait without iops?

NOTE: There no NFS or FUSE filesystems here, but it is using Xen virtualization.

We're seeing a weird statistic on the number of table-opens. I've appended the mysqltuner dump below.

It says we've open 94,000 tables in the past 24 hours. But there's only 15,000 tables total and we know that on this particular read-slave replicant only about 100 tables are being accessed.

Furthermore it says we're caching only 2k of those 94k.

How do I make sense of these numbers? And how to fix the caching issue -- sure we could "increase the table cache" but that doesn't sound like the root issue.

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.1.41-3ubuntu12.9-log
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster 
[--] Data in MyISAM tables: 4G (Tables: 14960)
[--] Data in InnoDB tables: 225M (Tables: 570)
[!!] Total fragmented tables: 2115

-------- Performance Metrics -------------------------------------------------
[--] Up for: 2h 13m 36s (647K q [80.798 qps], 28K conn, TX: 247B, RX: 542M)
[--] Reads / Writes: 63% / 37%
[--] Total buffers: 2.5G global + 2.7M per thread (1000 max threads)
[OK] Maximum possible memory usage: 5.2G (66% of installed RAM)
[OK] Slow queries: 0% (783/647K)
[OK] Highest usage of available connections: 5% (57/1000)
[OK] Key buffer size / total MyISAM indexes: 1.0G/1.4G
[OK] Key buffer hit rate: 100.0% (763M cached / 74K reads)
[OK] Query cache efficiency: 55.5% (335K cached / 604K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (312 temp sorts / 113K sorts)
[!!] Temporary tables created on disk: 47% (59K on disk / 124K total)
[OK] Thread cache hit rate: 99% (63 created / 28K connections)
[!!] Table cache hit rate: 2% (2K open / 94K opened)
[OK] Open file limit used: 75% (3K/5K)
[OK] Table locks acquired immediately: 99% (577K immediate / 583K locks)
[!!] Connections aborted: 45%
[OK] InnoDB data size / buffer pool: 225.3M/256.0M

-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    MySQL started within last 24 hours - recommendations may be inaccurate
    Temporary table size is already large - reduce result set size
    Reduce your SELECT DISTINCT queries without LIMIT clauses
    Increase table_cache gradually to avoid file descriptor limits
    Your applications are not closing MySQL connections properly
Variables to adjust:
    table_cache (> 2048)

nullmailer logs when a message is sent out, but includes no information like the to-addresses, subject lines, which machine it's sending to, etc..

I can't find any documentation about how to change logging info. Is there a way?

I'm able to use limit_req to rate-limit all requests to my server.

However I'd like to remove the rate restriction for certain IP addresses (i.e. whitelist) and use a different rate restriction for certain others (i.e. certain IPs I'd like as low as 1r/s).

I tried using conditionals (e.g. if ( $remote_addr = "1.2.3.4" ) {}) but that seems to work only with rewrite rules, not for rate-limit rules.

I'm looking for a queuing system which is fast and reliable and potentially can be spread across machines. Platform is Linux. Open source is preferable.

Both RabbitMQ and ZeroMQ look good but I have no experience with these or any other queuing systems. Can you point me in the right direction?

Various processes log in various formats to various locations with various roll-over logic in /var/log.

I'd like a log-viewer that can handle a decent quantity of data, join roll-over log files, and ideally could even interleave log records to get a timeline of what happened on the machine.

For example, I saw a CPU usage spike at a certain time this morning and I'd like to see if there's anything in any of the logs that explains it.

P.S. Yes I realize there's configuration issues like Apache logs can be "anything" so you'd need to tell the log viewer how to parse, and yes I understand that interleaving is hard between products because few columns (except possible "date" and "general message") would necessarily line up.