We use pg_dump nightly to make a snapshot of our database. We did for a long time with a simple command
pg_dump -Fc database_name
This takes about an hour and produces a file of 30+GByte.
How can we speed up things?
We had never any problems with nginx. We use 5 nginx server as loadbalancers in front of many spring boot application servers.
We were running them for years on debian 9 with the default nginx package 1.10.3. Now we switched three of our loadbalancers to debian 10 with nginx 1.14.2. First everything runs smoothly. Then, on high load we encountered some problems. It starts with
2020/02/01 17:10:55 [crit] 5901#5901: *3325390 SSL_write() failed while sending to client, client: ...
2020/02/01 17:10:55 [crit] 5901#5901: *3306981 SSL_write() failed while sending to client, client: ...
In between we get lots of
2020/02/01 17:11:04 [error] 5902#5902: *3318748 upstream timed out (110: Connection timed out) while connecting to upstream, ...
2020/02/01 17:11:04 [crit] 5902#5902: *3305656 SSL_write() failed while sending response to client, client: ...
2020/02/01 17:11:30 [error] 5911#5911: unexpected response for ocsp.int-x3.letsencrypt.org
It ends with
2020/02/01 17:11:33 [error] 5952#5952: unexpected response for ocsp.int-x3.letsencrypt.org
The problem does only exits for 30-120 seconds on high load and disappears afterwards.
In the kernel log we have sometimes: Feb 1 17:11:04 kt104 kernel: [1033003.285044] TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters.
But on other occasions we don't see any kernel.log messages
On both debian 9 and debian 10 servers we use the identical setup and had some TCP Tuning in place:
# Kernel tuning settings
# https://www.nginx.com/blog/tuning-nginx/
net.core.rmem_max=26214400
net.core.wmem_max=26214400
net.ipv4.tcp_rmem=4096 524288 26214400
net.ipv4.tcp_wmem=4096 524288 26214400
net.core.somaxconn=1000
net.core.netdev_max_backlog=5000
net.ipv4.tcp_max_syn_backlog=10000
net.ipv4.ip_local_port_range=16000 61000
net.ipv4.tcp_max_tw_buckets=2000000
net.ipv4.tcp_fin_timeout=30
net.core.optmem_max=20480
The nginx config is exactly the same, so I just show the main file:
user www-data;
worker_processes auto;
worker_rlimit_nofile 50000;
pid /run/nginx.pid;
events {
worker_connections 5000;
multi_accept on;
use epoll;
}
http {
root /var/www/loadbalancer;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
types_hash_max_size 2048;
server_tokens off;
client_max_body_size 5m;
client_header_timeout 20s; # default 60s
client_body_timeout 20s; # default 60s
send_timeout 20s; # default 60s
include /etc/nginx/mime.types;
default_type application/octet-stream;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:100m;
ssl_buffer_size 4k;
ssl_dhparam /etc/nginx/dhparam.pem;
ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_session_tickets on;
ssl_session_ticket_key /etc/nginx/ssl_session_ticket.key;
ssl_session_ticket_key /etc/nginx/ssl_session_ticket_old.key;
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/ssl/rapidssl/intermediate-root.pem;
resolver 8.8.8.8;
log_format custom '$host $server_port $request_time $upstream_response_time $remote_addr "$ssl_session_reused" $upstream_addr $time_iso8601 "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent";
access_log /var/log/nginx/access.log custom;
error_log /var/log/nginx/error.log;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_path /var/cache/nginx/ levels=1:2 keys_zone=imagecache:10m inactive=7d use_temp_path=off;
proxy_connect_timeout 10s;
proxy_read_timeout 20s;
proxy_send_timeout 20s;
proxy_next_upstream off;
map $http_user_agent $outdated {
default 0;
"~MSIE [1-6]\." 1;
"~Mozilla.*Firefox/[1-9]\." 1;
"~Opera.*Version/[0-9]\." 1;
"~Chrome/[0-9]\." 1;
}
include sites/*.conf;
}
The upstream timeout signals some problems with our java machines. But at the same time the debian9 nginx/loadbalancer is running fine and has no problems connecting to any of the upstream servers. And the problems with letsencrypt and SSL_write are signaling to me some problems with nginx or TCP or whatever. I really don't know how to debug this situation. But we can reliable reproduce it most of the times we encounter high load on debian10 servers and did never see it on debian 9.
Then I installed the stable version nginx 1.16 on debian10 to see if this is a bug in nginx which is already fixed:
nginx version: nginx/1.16.1
built by gcc 8.3.0 (Debian 8.3.0-6)
built with OpenSSL 1.1.1c 28 May 2019 (running with OpenSSL 1.1.1d 10 Sep 2019)
TLS SNI support enabled
configure arguments: ...
But it didn't help.
It seems to be a network related problem. But we do not encouter it on the application servers. But the load is of course lower as the loadbalancer/nginx machine has to handle external and internal traffic.
It is very difficult to debug as it only happens on high load. We treid to load test the servers with ab, but we could not reproduce the problem.
Can somebody help me and give me some hints how to start further debugging of this situation?
I tried to install kernel 5.x from backports in a frsh installed debian 10 and I failed because I have a 10G NIC:
echo deb http://deb.debian.org/debian buster-backports main contrib non-free > /etc/apt/sources.list.d/buster-backports.list
apt-get update
apt-get -t buster-backports upgrade
apt-get install -t buster-backports firmware-linux firmware-linux-nonfree linux-headers-5.4.0-0.bpo.2-amd64 linux-image-5.4.0-0.bpo.2-amd64
update-initramfs: Generating /boot/initrd.img-5.4.0-0.bpo.2-amd64
W: Possible missing firmware /lib/firmware/bnx2x/bnx2x-e2-7.13.11.0.fw for module bnx2x
W: Possible missing firmware /lib/firmware/bnx2x/bnx2x-e1h-7.13.11.0.fw for module bnx2x
W: Possible missing firmware /lib/firmware/bnx2x/bnx2x-e1-7.13.11.0.fw for module bnx2x
It doesn't find the firmware files. And the machine is not booting.
kernel log says:
Jan 29 11:06:00 Debian-102-buster-64-minimal kernel: [ 3.609341] ipmi_si IPI0001:00: IPMI message handler: Found new BMC (man_id: 0x003c0a, prod_id: 0x1001, dev_id: 0x20)
Jan 29 11:06:00 Debian-102-buster-64-minimal kernel: [ 3.672134] ipmi_si IPI0001:00: IPMI kcs interface initialized
Jan 29 11:06:00 Debian-102-buster-64-minimal kernel: [ 3.677407] ipmi_ssif: IPMI SSIF Interface driver
Jan 29 11:06:01 Debian-102-buster-64-minimal kernel: [ 5.476563] bnx2x 0000:61:00.1: firmware: failed to load bnx2x/bnx2x-e2-7.13.11.0.fw (-2)
Jan 29 11:06:01 Debian-102-buster-64-minimal kernel: [ 5.476763] firmware_class: See https://wiki.debian.org/Firmware for information about missing firmware
Jan 29 11:06:01 Debian-102-buster-64-minimal kernel: [ 5.476967] bnx2x 0000:61:00.1: Direct firmware load for bnx2x/bnx2x-e2-7.13.11.0.fw failed with error -2
Jan 29 11:06:01 Debian-102-buster-64-minimal kernel: [ 5.477174] bnx2x: [bnx2x_func_hw_init:6002(eth1)]Error loading firmware
Jan 29 11:06:01 Debian-102-buster-64-minimal kernel: [ 5.477331] bnx2x: [bnx2x_nic_load:2730(eth1)]HW init failed, aborting
Jan 29 11:06:02 Debian-102-buster-64-minimal kernel: [ 5.672256] bnx2x 0000:61:00.0: firmware: failed to load bnx2x/bnx2x-e2-7.13.11.0.fw (-2)
Jan 29 11:06:02 Debian-102-buster-64-minimal kernel: [ 5.672495] bnx2x 0000:61:00.0: Direct firmware load for bnx2x/bnx2x-e2-7.13.11.0.fw failed with error -2
Jan 29 11:06:02 Debian-102-buster-64-minimal kernel: [ 5.672736] bnx2x: [bnx2x_func_hw_init:6002(eth0)]Error loading firmware
Jan 29 11:06:02 Debian-102-buster-64-minimal kernel: [ 5.672895] bnx2x: [bnx2x_nic_load:2730(eth0)]HW init failed, aborting
With the Debian stock kernel everything runs fine.
How can I install a kernel 5.x in debian10 from backports or another ppa (without compiling my own kernel)?
I have installed nginx on multiple servers (our load balancers).
When i check my SSL installation with https://www.ssllabs.com/ I get A+ results for each server but the first server has always
Session resumption (caching) No (IDs assigned but not accepted)
while all other servers have
Session resumption (caching) Yes
All servers are configured automatically, so identical. I additionally checked the nginx configuration with
/usr/sbin/nginx -T > lb1.nginx
/usr/sbin/nginx -T > lb2.nginx
...
But there are no differences.
ssl_session is configured like this on every single server:
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:100m;
Now comes the interesting part:
I have ten load balancers and the first one is marked by ssllabs with 'session Resumption: No'.
Now I moved the IP Address of the first server to the second server. The first request of ssllabs which is now going to the second server and is showing 'session Resumption: No'. The second request which is going to the second server, too, is showing 'session Resumption: Yes'. But both queries are arriving at the same server.
It seems to me that ssl labs is doing something different with the first request.
Can there be any other reason why ssllabs shows NO session resumption only for the first request no matter to which server is "first"?
We are running nginx on a debian system as our loadbalancer
nginx version: nginx/1.9.10
built with OpenSSL 1.0.2e 3 Dec 2015 (running with OpenSSL 1.0.2g 1 Mar 2016)
We had http2 activated in our server like this:
server {
listen 443 ssl http2;
....
}
All requests are proxied to our application server. We are logging the request_time and upstream_time from nginx.
We switched recently our iOS hybrid app from UIWebView to WKWebView in Week 13, see what happened:
The upstream_time stayed the same but our request time suddenly raised as more and more users updated their app to the new version.
Today we decided to give it a try and deactivate http2. We only removed the word "http2" from our server config:
server {
listen 443 ssl;
....
}
See what happened:
It seems http2 is not production ready. I don't know if it is server side or client side. Maybe even the request_time is not properly logged inside nginx.
Does anybody has more info about using http2 with nginx and/or WKWebView?
We are running nginx/1.9.10 as a frontend server with multiple application server as upstream. We are using plain http, mostly https and switched to http/2 in the last weak.
We are logging like this:
log_format custom '$host $server_port $request_time '
'$upstream_response_time $remote_addr '
'"$http2" $upstream_addr $time_iso8601 '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
We suddenly see a larger difference between $request_time and $upstream_response_time . A difference here is quite natural as the $request_time depends on the users network while the upstream_response_time does not.
So usually you should not care too much about $request_time as long as $upstream_response_time is stable.
But I still wanted to check what is happening because it strted to getting worse with http/2 So I compared average response times for https/1.1 and https/2.0
First I cecked all http/1.1 request and calculated average response_time and average upstream_time:
grep ' 443 ' access.log|grep 'HTTP/1.1'|\
cut -d ' ' -f 3,4 | awk '{r+=$1; u+=$2} END {print r/NR; print u/NR}'
0.0139158 # average response time for https/1.1
0.00691421 # average upstream time for https/1.1
Now I did the same with https/2.0:
grep ' 443 ' access.log|grep 'HTTP/2.0'| \
cut -d ' ' -f 3,4 | awk '{r+=$1; u+=$2} END {print r/NR; print u/NR}'
0.0828755 # average response time for https/1.1
0.00606643 # average upstream time for https/2.0
As you see the upstream time is nearly identical but the request time is slower for http/2 by factor 7! Wow! Isn't http/2 expected to be faster?
Now I checked all request which have a huge difference between these two values and nearly all of the top 500 has been a status code of 302
grep ' 443 ' access.log | grep 'HTTP/1.1' | \
awk '{ if ( $3 != $4 && $4 != "-" ) { \
gsub(/\./,"",$3);gsub(/\./,"",$4); \
print $3-$4,$4,$6,$9,$11,$12 }}' | \
sort -n | tail -n 10000 | grep 'POST HTTP/1.1" 302' | wc -l
9008
# of 10000 (or 90%) request ordered by difference between
# response and request time have status code 302
So 90% of all requests with the highest difference between response and upstream time are status code 302. This is strange
On http/2 it is even worse:
grep ' 443 ' access.log | grep 'HTTP/2.0' | \
awk '{ if ( $3 != $4 && $4 != "-" ) { \
gsub(/\./,"",$3);gsub(/\./,"",$4); \
print $3-$4,$4,$6,$9,$11,$12 }}' | \
sort -n | tail -n 10000 | grep 'POST HTTP/2.0" 302' | wc -l
9790
# of 10000 (or 98%) request ordered by difference between
# response and request time have status code 302
So here are 98% of these request are 302 status.
Why does http/2 appear to be slower than http/1.1? Why are so many 302 status codes involved in requests were upstream and response time differ extremly (in HTTP/1.1 and HTTP/2.0)?
I want my server with debian 8.1 not to use cpu scaling but always run at the highest possible frequency.
I have installed cpufrequtils
$ dpkg -l | grep cpufreq
ii cpufrequtils 008-1 amd64 ...
ii libcpufreq0 008-1 amd64 ...
I set the governor to performance with
$ cat /etc/default/cpufrequtils
ENABLE="true"
GOVERNOR="performance"
MAX_SPEED="0"
MIN_SPEED="0"
and called /etc/init.d/cpurequtils restart
When I list my cpu info with cpufreq-info I get results like this:
analyzing CPU 2:
driver: intel_pstate
CPUs which run at the same hardware frequency: 2
CPUs which need to have their frequency coordinated by software: 2
maximum transition latency: 0.97 ms.
hardware limits: 1.20 GHz - 3.80 GHz
available cpufreq governors: performance, powersave
current policy: frequency should be within 1.20 GHz and 3.80 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency is 1.73 GHz (asserted by call to hardware).
The last line shows that the cpu is not running at full speed. When I check the values from /sys I get the same results:
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq
echo "--"
cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq
1198203
1199707
2001015
3048828
1551210
1358847
2953808
1982832
1523867
1200253
1654296
3446132
--
1198203
1199707
2001015
2643730
1772695
1358847
2953808
1982832
1523867
1200253
1654296
3446132
What I wonder about is that I do not have any file named 'scaling_available_frequencies' in /sys which is mentioned in many howtos
ls -1 /sys/devices/system/cpu/cpu0/cpufreq/
affected_cpus
cpuinfo_cur_freq
cpuinfo_max_freq
cpuinfo_min_freq
cpuinfo_transition_latency
related_cpus
scaling_available_governors
scaling_cur_freq
scaling_driver
scaling_governor
scaling_max_freq
scaling_min_freq
scaling_setspeed
The appropiate kernel modules are all loaded:
$ lsmod | grep cpufre
cpufreq_powersave 12454 0
cpufreq_userspace 12525 0
cpufreq_conservative 14184 0
cpufreq_stats 12782 0
Now I set scaling_min like this:
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
Now I get Infos like sees, which say frequency should be between 3.80 GHz and 3.80 GHz but actually is not:
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: 0.97 ms.
hardware limits: 1.20 GHz - 3.80 GHz
available cpufreq governors: performance, powersave
current policy: frequency should be within 3.80 GHz and 3.80 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency is 2.84 GHz (asserted by call to hardware).
The governor "performance" doesn't seem to do the job. So how to I force my CPUs to run at full speed?
We had a customer which has set a CNAME Record for his domain. Somehow he managed it to set an A Record too, which should be not possible and is forbidden by DNS. But the result was:
$ dig @ns1.your-server.de tippspiel-bl1.unternehmen-frische.de
...
;; ANSWER SECTION:
tippspiel-bl1.unternehmen-frische.de. 7200 IN CNAME www.kicktipp.de.
tippspiel-bl1.unternehmen-frische.de. 7200 IN A 78.46.10.156
The second record is illegal. But this led to some confusion of other caching DNS Server which returned 78.46.10.156 when they were asked about www.kicktipp.de. But this is completely wrong.
The other DNS server used both answers and were mixing them. End result: Users visiting www.kicktipp.de were send to 78.46.10.156 which is the IP of unternehmen-frische.de
It seems that I can hijack a domain when setting DNS for a domain with a CNAME and an A Record. Is this a known bug? How can I protect my domain against it?
I would like to monitor my peak network usages. I already use munin to monitor network traffic in a 5 minute sample. But as we have lot of peaks of http traffic which lasts only a few seconds I want to know the peak network usage per second in the last 5 minutes not the average.
At the moment I use iftop which is an easy to see the peak usage. But iftop can only run interactively. I use screen to get it running and can look into it on a regular basis.
How can I use iftop or a similar tool to just write out the peak network usage in the last 5 minutes every 5 minutes. I just need a file with lines like this:
2014-06-17 15:43:12 TX: 14,3 MBit/s RX: 16,2 MBit/s Sum: 29,6 MBit/s
2014-06-17 15:48:12 TX: 11,3 MBit/s RX: 12,2 MBit/s Sum: 22,3 MBit/s
and so on.
I tried ntop but it is quite a bit of overhead and I can't get the data easily into munin. And it does not write out peaks per 5 Minutes.
Does anybody know a tool or efficient script for this?
We have a cluster of tomcat servers, just upgraded to debian7/openjdk7/tomcat7 (all from stock debian). Now we had two JVM crash on different machines. Both servers having identical hardware and are configured exactly the same (besides IP Addresses of course)
First Crash:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fd0b582123a, pid=6542, tid=140533723084544
#
# JRE version: 7.0_25-b30
# Java VM: OpenJDK 64-Bit Server VM (23.7-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x62623a] LinkResolver::runtime_resolve_interface_method(CallInfo&, methodHandle, KlassHandle, Handle, KlassHandle, bool, Thread*)+0x5a
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
#
--------------- T H R E A D ---------------
Current thread (0x00007fd0948b0800): JavaThread "catalina30" daemon [_thread_in_vm, id=6670, stack(0x00007fd08e94b000,0x00007fd08ea4c000)]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000000
Second Crash:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f0a3a22421d, pid=3454, tid=139681826494208
#
# JRE version: 7.0_25-b30
# Java VM: OpenJDK 64-Bit Server VM (23.7-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x71021d] ParRootScanWithBarrierTwoGensClosure::do_oop(unsigned int*)+0x6d
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
#
--------------- T H R E A D ---------------
Current thread (0x000000000088c000): GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=3781]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000018
We never had any JVM crash before.
I really don't know where to start. Does it look like a hardware fault? JDK Bug? Tomcat Bug?
What is the most probably reason for this?
We have a few dedicated server rented in a data center with debian 7, kernel 3.2.
We use one of those servers as a database server. The network between our application server and database server is not dedicated to us but is used by other customers of the data center.
From time to time we recognize TCP retransmissions on this line. We think it is due to congestion or ddos attacks. Our provider tries to prevent attacks but is not always successful immediately of course.
Anyway. Usually our application server gets results from the database within 20 milliseconds as the database servers are very fast and the round-trip-time (RTT) average is 0.3 ms (so under 1 ms).
When a TCP packet is lost on this line the retransmission time out (RTO) kicks in. It is calculated by the round-trip-time but is at least 200ms. So when one packet needs to be retransmitted we have 220 milliseconds before our application server gets its data just because of the RTO.
For me rto_min=200ms seems to be way to high for a link with rtt under 1ms.
It is possible to set the rto_min with ip like this:
ip route change default via 144.76.176.65 dev eth0 rto_min 5ms
RTO is still calculated but can get down to 5ms as our RTT is very small.
Should I consider this or are there other TCP pitfalls I will fall into setting the rto_min so small? What is a resonable value for rto_min or is it better not to touch it?
In our application we use Hibernate and PostgreSQL to store data. In one of our database tables we have a discriminator column which says for example "TIPPSPIEL". It is a fixed string and can not be manipulated by any user.
Suddenly we had one entry in this huge table where we had "TIPPQPIEL" instead of "TIPPSPIEL". We have no clue how this can happen.
Is it possible by any means that our hard disk is switching one bit, so our letter "S" is no longer encoded as "1010001" but suddenly becomes a "Q" on the hard disk with one bit switched like this: 1010011?
I am not an expert on hard disk an bit physics but I guess an OS or a disk has checksums and other stuff to ensure that this can't happen.
Is it possible that just one bit switches so my file shows me a letter "Q" instead of a "S"?
UPDATE: We made further analyse. Our slave database gets its WAL Records from master (PostgreSQL feature). Whatever: our slave server should be in sync. But the slave wasn't in sync regarding this particular row. We could see that it happened a few days ago without any interaction from a user on this particular entry. So it MUST be a bit flipping around. scary!
We are using apache2.2 as a front-end server with application servers as reverse proxies behind apache.
We are using mod_cache for some images and enabled it like this:
<IfModule mod_disk_cache.c>
CacheEnable disk /
CacheRoot /var/cache/apache2/mod_disk_cache
CacheIgnoreCacheControl On
CacheMaxFileSize 2500000
CacheIgnoreURLSessionIdentifiers jsessionid
CacheIgnoreHeaders Set-Cookie
</IfModule>
The image urls vary completely and have no common start pattern, but they all end in ".png". Thats why we used the root in CacheEnable /
If not served from the cache, the request is forwarded to an application server via reverse proxy. So far so good, cache is working fine.
But I really only need to cache all image request ending in ".png". My above configuration still works as my application server send an appropriate Cache-Control: no-cache header on the way back to apache. So most pages send a no-cache header back and they get not cached at all. My ".png" responses doesn't send a Cache-Control header so apache is only going to cache all urls with ".png". Fine.
But when a new request enters apache, apache does not know that only .png requests should be considered, so every request is checking a file on disk (recorded with strace -e trace=file -p pid):
[pid 19063] open("/var/cache/apache2/mod_disk_cache/zK/q8/Kd/g6OIv@woJRC_ba_A.header", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
I don't want to have apache going to disk every request, as the majority of requests are not cached at all. And we have up to 10.000 request/s at peak time. Sometimes our read IO wait spikes. It is not getting really slow, but we try to tweak it for better performance.
In apache 2.4 you can say:
<LocationMatch .png$>
CacheEnable disk
</LocationMatch>
This is not possible in 2.2 and as I see no backports for debian I am not going to upgrade.
So I tried to tweak apache2.2 to follow my rules:
<IfModule mod_disk_cache.c>
SetEnvIf Request_URI "\.png$" image
RequestHeader unset Cache-Control
RequestHeader append Cache-Control no-cache env=!image
CacheEnable disk /
CacheRoot /var/cache/apache2/mod_disk_cache
#CacheIgnoreCacheControl on
CacheMaxFileSize 2500000
CacheIgnoreURLSessionIdentifiers jsessionid
CacheIgnoreHeaders Set-Cookie
</IfModule>
The idea is to let apache decide to serve request from cache based on Cache-control header (CacheIgnoreCacheControl default to off). And before simply set a RequestHeader based on the request. If it is not an image request, set a Cache-control header, so it should bypass the cache at all.
This does not work, I guess because of late processing of RequestHeader directive, see https://httpd.apache.org/docs/2.2/mod/mod_headers.html#early
I can't add early processing as "early" keyword can't be used together with a conditional "env=!image"
I can't change the url requesting the images and I know there are of course other solutions. But I am only interested in configuring apache2.2 to reach my goal.
Does anybody has an idea how to achieve my goal?
I run a debian squeeze standard Apache installation (2.2) and make use of SSLClientCertificates to authorize users. This works fine so far.
But we noticed a slow down of some parallel requests and tried to check if my SSLSessionCache is working.
So I checked my localhost/server-status and it reads like this:
SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 0
subcaches: 32, indexes per subcache: 133
index usage: 0%, cache usage: 0%
total sessions stored since starting: 0
total sessions expired since starting: 0
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 0 hit, 0 miss
total removes since starting: 0 hit, 0 miss
Seems to be running but whatever SSL request I make, all counters stay at 0, so no sessions are cached.
I tried to set KeepAlive Off, to let every request establish a new SSL connection, but still I see no numbers counting up in my SSLSessionCache Status.
This is my SSLSessionCache Configuration from standard debian mods-enabled/ssl.conf:
SSLSessionCache shmcb:${APACHE_RUN_DIR}/ssl_scache(512000)
SSLSessionCacheTimeout 300
SSLMutex file:${APACHE_RUN_DIR}/ssl_mutex
in my ${APACHE_RUN_DIR} I see no files at all, no ssl_mutex and no ssl_cache file. When I switch my SSLSessionCache to
SSLSessionCache dbm:${APACHE_RUN_DIR}/ssl_scache
I can see a file in this directory, but all status numbers are still zero.
I tried to set LogLevel to debug. The only messages I get about the ssl cache are:
$ grep cache /var/log/apache2/error.log
ssl_scache_shmcb.c(253): shmcb_init allocated 512000 bytes of shared memory
ssl_scache_shmcb.c(272): for 511920 bytes (512000 including header), recommending 32 subcaches, 133 indexes each
ssl_scache_shmcb.c(306): shmcb_init_memory choices follow
ssl_scache_shmcb.c(308): subcache_num = 32
ssl_scache_shmcb.c(310): subcache_size = 15992
ssl_scache_shmcb.c(312): subcache_data_offset = 3208
ssl_scache_shmcb.c(314): subcache_data_size = 12784
ssl_scache_shmcb.c(316): index_num = 133
Shared memory session cache initialised
ssl_scache_shmcb.c(452): [client xyz] inside shmcb_status
ssl_scache_shmcb.c(512): [client xyz] leaving shmcb_status
(removed date an loglevel for readability, replaced IP for privacy)
So here are my questions:
- Is it correct to have no files for mutex and sessionCache in the given directory?
- If yes, how to prove my SessionCache is working?
we are runing an apache reverse proxy in front of several tomcats. sometimes we like to know to which server the request was forwarded. as we use mod_proxy_balancer.
i tried
\"%{X-Forwarded-Server}i\"
but this doesn't work. it shows the name of my loadbalancer (apache)
How can i log which worker was selected?
I'm getting nuts writing a simple plugin to query my ehcache statistics. Look what i have so far:
$ munin-run ehcache
standardquerycache_cachehits.value 1779609
standardquerycache_cachemisses.value 153404
Now I want to draw only a cache hit ratio graph but i fail to achieve this. I am even not getting it to work to draw a graph of the total sum (just for exercise):
$ munin-run ehcache config
graph_category Ehcache
graph_title Cache Hit ratios
graph_vlabel percent %
graph_order standardquerycache_total standardquerycache_ratio
standardquerycache_cachehits.label standardquerycache_cachehits
standardquerycache_cachehits.graph no
standardquerycache_cachehits.type COUNTER
standardquerycache_cachemisses.label standardquerycache_cachemisses
standardquerycache_cachemisses.graph no
standardquerycache_cachemisses.type COUNTER
standardquerycache_total.sum standardquerycache_cachehits standardquerycache_cachemisses
standardquerycache_total.label standardquerycache_total
standardquerycache_ratio.label standardquerycache
standardquerycache_ratio.cdef standardquerycache_cachehits,standardquerycache_total,1,standardquerycache_total,0,EQ,IF,/,100,*,UNKN,standardquerycache_total,0,EQ,IF,UNKN,standardquerycache_total,UNKN,EQ,IF
This is my output in munin-update.log:
2011/08/07 16:36:39 Opened log file
2011/08/07 16:36:39 [INFO]: Starting munin-update
2011/08/07 16:36:39 [INFO]: Config update, ensuring type of '...total-g.rrd' is 'GAUGE'.
2011/08/07 16:36:39 [INFO]: Config update, ensuring max of '...rrd' is 'U'.
2011/08/07 16:36:39 [INFO]: Config update, ensuring min of '..total-g.rrd' is 'U'.
2011/08/07 16:36:39 [WARNING] Service ehcache on ... returned no data for label standardquerycache_total
2011/08/07 16:36:39 [WARNING] Service ehcache on ... returned no data for label standardquerycache_ratio
2011/08/07 16:36:39 [INFO] Reaping Munin::Master::UpdateWorker<...>. Exit value/signal: 0/0
2011/08/07 16:36:39 [INFO]: Munin-update finished (0.14 sec)
Maybe I am doing something completely wrong, but munin seems to ask for a value which should be calculated by cdef.
Anything wrong with my plugin configuration?
We had some trouble with one of our image servers last week and need some help. See our munin monitoring graph:
We are runing debian squeeze and we have lots of requests because this is one of our image servers. We do not use keep-alive (maybe we should, but that's another topic)
These numbers are request counts per minute from our log files:
- 17:19: 66516
- 17:20: 64627
- 17:21: 123365
- 17:22: 111207
- 17:23: 58257
- 17:24: 17710
- ... and so on
So you see, we have lots of requests per minute but as most request are served in 0-1ms everything runs fine usually.
Now as you see in our munin graphic munin didn't manage to connect to this server on munin port and ask the relevant numbers. The connection simply failed. As the server is not overloaded by any means (cpu, memory, network). it must has something to do with our firewall/tcp stack. At the time the munin plugin failed to connect we had only 17MBit of incoming and outgoing traffic on a 100MBit connection.
you often here a limit of 65k of tcp connections, but this is normally misleading as it refers to the 16bit tcp header and belongs to 65k per ip/port combination.
our time_wait timeout is set to
net.ipv4.tcp_fin_timeout = 60
we could lower this to drop more TIME_WAIT connections earlier, but first i want to know what limits the network from being reachable.
we are using iptables with state module. But we already raised the max_conntrack parameter.
net.ipv4.netfilter.ip_conntrack_max = 524288
does anybody know what kernel parameters to look at or how to diagnose this problem next week when we have our next peek?
we are on a debian system and are trying to tune the tcp/ip stack to our needs. We all know that you can set the maximum tcp buffer size with some kernel parameters like this:
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.udp_wmem_min = 4096
net.core.wmem_max = 261071
To calculate the maximum buffer size for your needs you "just" have to calculate it. (see http://fasterdata.es.net/TCP-tuning/ )
But as we do not know the round-trip-time of our users, it is quite difficult. It might be ok to assume something between 20 and 60 ms. But for an mobile network it is something like 100-300 ms (tested with my phone). So it is quite difficult to know how many data might be "on the line".
We would like to see the actual buffer size and the utilization of it.
Does anybody know how to sneek into the actual tcp write and receive buffers?
We encounter some strange packet loss and want to know the reason for this.
We have an imageserver and a server for stressing the imageserver. Both are located in the same datacenter
First we run a load test like this (command shortened for readability):
ab -n 50 -c 5 http://testserver/img/de.png
The image has only about 300 Bytes. Results are very fast responses:
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 0
Processing: 1 3 0.7 3 4
Waiting: 1 3 0.7 3 3
Total: 1 3 0.7 3 4
When we increase concurrency we see some lags (command shortened for readability):
sudo ab -n 500 -c 50 http://testserver/img/de.png
Results with concurrency 50:
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 1
Processing: 2 35 101.6 12 614
Waiting: 2 35 101.6 12 614
Total: 3 36 101.7 12 615
so we see most requests are pretty fast, a few of them are pretty slow.
We dumped the whole network traffic with tcpdump and saw some strange retransmissions.
alt text http://vygen.de/screenshot1.png
this dump was taken on the imageserver!
so you see that the initial package (No. 306) containing the GET request is arriving on the imageserver, but it seems the package get lost after tcpdump has logged it. It seems to me that this package does not arrive at the tomcat image server.
the retransmission is triggered by the requesting server 200ms later and everything runs fine afterwards.
Do you know any reason why a package can get lost after it was received?
Our machines are both:
- Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz
- 8 GB RAM
- Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)
- Debian version 5.0.5
So we do not have any problems concerning memory or cpu load.
We had some problems with our nic controller a while ago. We handled it by using a different driver, we are using now r8168 instead of r8169
But we had the same problems of lost packets with an Intel NIC - Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
So we see the same problem with equal machines but different ethernet cards.
Till now i thought packet loss would only happen between servers on the line, when the packet gets corrupted or things like that.
We really want to know what reasons there might be for those packet loss after tcpdump has logged them.
Your help is very appreciated.
We are running a firewall with iptables on our Debian Lenny system. I show you only the relevant entries of our firewall.
Chain INPUT (policy DROP 0 packets, 0 bytes)
target prot opt in out source destination
ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 state NEW
Chain OUTPUT (policy DROP 0 packets, 0 bytes)
target prot opt in out source destination
ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
LOGDROP all -- * * 0.0.0.0/0 0.0.0.0/0
Some packets get dropped each day with log messages like this:
Feb 5 15:11:02 host1 kernel: [104332.409003] dropped IN= OUT=eth0 SRC=<OWN_IP> DST=<REMOTE_IP> LEN=1420 TOS=0x00 PREC=0x00 TTL=64 ID=18576 DF PROTO=TCP SPT=80 DPT=59327 WINDOW=54 RES=0x00 ACK URGP=0
for privacy reasons I replaced IP Addresses with <OWN_IP> and <REMOTE_IP>
This is no reason for any concern, but I just want to understand what's happening. The web server tries to send a packet to the client, but the firewall somehow came to the conclusion that this packet is "UNRELATED" to any prior traffic.
I have set a kernel parameter ip_conntrack_ma to a high enough value to be sure to get all connections tracked by iptables state module:
sysctl -w net.ipv4.netfilter.ip_conntrack_max=524288
What's funny about that is I get one connection drop every 20 minutes:
06:34:54
06:52:10
07:10:48
07:30:55
07:51:29
08:10:47
08:31:00
08:50:52
09:10:50
09:30:52
09:50:49
10:11:00
10:30:50
10:50:56
11:10:53
11:31:00
11:50:49
12:10:49
12:30:50
12:50:51
13:10:49
13:30:57
13:51:01
14:11:12
14:31:32
14:50:59
15:11:02
That's from today, but on other days it looks like this, too (sometimes the rate varies).
What might be the reason?
Any help is greatly appreciated. kind regards Janning