Theron Luhn's questions

I have some simple connection and connection rate limiting in HAProxy:

    # Store IPs to limit connection rate
    stick-table type ip size 200k expire 5m store gpc0,conn_cur,conn_rate(10s)
    tcp-request connection track-sc0 src

    # Abusers are immediately rejected
    tcp-request connection reject if { sc0_get_gpc0 gt 0 }

    # Test connection count and rate
    acl connabuse sc0_conn_cur gt 20
    acl connkill  sc0_inc_gpc0 gt 0
    tcp-request connection reject if connabuse connkill

    acl rateabuse sc0_conn_rate gt 30
    acl ratekill  sc0_inc_gpc0 gt 0
    tcp-request connection reject if rateabuse ratekill

Unfortunately, this has been causing some problems. I'd like to temporarily stop blocking people, but log the time and IP when those limits are hit, so I can play around with the rules and see what works and what doesn't. How can I do this?

I'm trying to get monit to monitor a program and start it if it goes down. Here's my basic config:

check process maitre_d_8000 with pidfile /home/tango/env/maitre_d/8000/pid.pid
    start program = "/bin/bash -c 'cd /home/tango/env/maitre_d/; ../bin/python maitre_d/main.py 8000/config.json &'" as uid tango and gid tango
    stop program = "/bin/bash -c 'cd /home/tango/env/maitre_d/; ../bin/python kill.py 8000/pid.pid'" as uid tango and gid tango

However, whenever I kill the program, I get this in the logs:

[EDT Apr 17 21:30:25] error    : 'maitre_d_8000' process is not running
[EDT Apr 17 21:30:27] info     : 'maitre_d_8000' trying to restart
[EDT Apr 17 21:30:27] info     : 'maitre_d_8000' start: /bin/bash
[EDT Apr 17 21:30:57] error    : 'maitre_d_8000' failed to start

Not very helpful at all. And the bash command should work just fine, running su tango -c "/bin/bash -c 'cd /home/tango/env/maitre_d/; ../bin/python maitre_d/main.py 8000/config.json &'" in the terminal results in successfully starting the program.

Any ideas of what could be going wrong?

I wrote a quick Python server to serve resampled images. For example, a URL might look something like http://images.domain.com/resample/100x100/9f362e1994264321.jpg. Being that resampling images is expensive, a caching layer is necessary. It seems like an nginx reverse-proxy would be a good option for this, and here and here seem like good places to start.

However, there's a problem. There are millions of images, so by storing http://images.domain.com/resample/100x100/9f362e1994264321.jpg in the filesystem as /home/nginx/cache/resample/100x100/9f362e1994264321.jpg (or something of the sort), eventually cache/resample/100x100/ will have millions of files in it, which will make file lookups very inefficient.

I deal with this problem while storing the original images by distributing them among many subdirectories, eg, 9f/36/9f362e1994264321.jpg. However, I'm not sure how I might do the same with nginx. I could change the URL to do likewise, and I will if that's the only solution, but I'd rather keep the URL as pretty as possible.

Can I do this with nginx? If not with nginx, can I do it something else, like varnish?

I get emails from CSF every time it blocks someone. Here's one I got this morning:

Time:    Sat Jan 19 10:17:24 2013 -0800
IP:      <REDACTED> (US/United States/-)
Hits:    21
Blocked: Temporary Block

Sample of block hits:
Jan 19 10:16:28 red kernel: [6727856.279786] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=<REDACTED> DST=<REDACTED> LEN=48 TOS=0x00 PREC=0x00 TTL=52 ID=47609 DF PROTO=TCP SPT=40954 DPT=587 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 19 10:16:30 red kernel: [6727858.716317] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=<REDACTED> DST=<REDACTED> LEN=64 TOS=0x00 PREC=0x00 TTL=52 ID=40975 DF PROTO=TCP SPT=43688 DPT=587 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 19 10:16:31 red kernel: [6727859.809269] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=<REDACTED> DST=<REDACTED> LEN=64 TOS=0x00 PREC=0x00 TTL=52 ID=56375 DF PROTO=TCP SPT=43688 DPT=587 WINDOW=65535 RES=0x00 SYN URGP=0
.....  (It continues on with more of the same)

The way I'm reading this is that this particular IP (SRC, right?) got blocked because of excessive hits to port 587 (DPT, right?) (which is SMTP). Am I correct? Are there any other important bits I should know how to read? Google wasn't helping when I tried to learn how to decipher these things.

And then the matter of interpreting: This happens to be my business partner's IP. He says he wasn't sending any emails at the time. Any theories on what could be going on?

I have a CentOS server that came installed with WHM and cPanel and the corresponding LAMP stack. I disabled Apache and MySQL and replaced them with nginx and PostgreSQL.

...Or at least I thought I did. Apache insists on starting up on boot, blocking the ports nginx needs. So every time the server goes down for maintenance or is restarted for some other reason, Apache starts serving an old "under construction" page we had put up. After being alerted to the issue via a few angry phone calls, I have to shutdown Apache and start up nginx.

chkconfig --list httpd returns:

httpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off

So I don't know what's booting up Apache. Any thoughts on what could be happening?

I'm running a Python Pyramid app on a CentOS server using uWSGI and nginx. I'm using SQLAlchemy as an ORM, MySQLdb as the API, and MySQL as a database. The site hasn't gone live yet, so the only traffic is me and some other employees of the company. We purchased some data to populate the database, so the largest (and most frequently queried) table is ~150,000 rows.

Yesterday I opened four new tabs of the website in quick succession, and I got back a couple 502 Bad Gateway errors. I looked in the uWSGI log, and found the following:

sqlalchemy.exc.OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT ge...

Important note: This error is not due to MySQL's wait_timeout. Been there, done that.

I wondered if the issue was caused by concurrent requests being served simultaneously. I made myself a poor man's load tester:

for i in {1..10}; do (curl -o /dev/null http://domain.com &); done;

Sure enough, within those ten requests at least one would throw a 2006 error, oftentimes more. Sometimes the errors would get even stranger, for example:

sqlalchemy.exc.NoSuchColumnError: "Could not locate column in row for column 'table.id'"

When the column most definitely exists and worked fine on all the other identical requests. Or, this one:

sqlalchemy.exc.ResourceClosedError: This result object does not return rows. It has been closed automatically.

When, once again, it worked fine for all the other requests.

To further verify that the problem stemmed from concurrent database connections, I set uWSGI to a single worker and multi-threading disabled, forcing the requests to be processed one at a time. Sure enough, the problems disappeared.

In an attempt to find the problem, I set up an error log for MySQL. With the exception of some notices during MySQL starting up, it remains empty.

Here is my MySQL config:

[mysqld]
default-storage-engine = myisam
key_buffer = 1M
query_cache_size = 1M
query_cache_limit = 128k
max_connections=25
thread_cache=1
skip-innodb
query_cache_min_res_unit=0
tmp_table_size = 1M
max_heap_table_size = 1M
table_cache=256
concurrent_insert=2
max_allowed_packet = 1M
sort_buffer_size = 64K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
thread_stack = 64K
innodb_file_per_table=1
log-error=/var/log/mysql/error.log

Heavy Googling on the error revealed little, but suggested that I increase max_allowed_packet. I increased it to 100M and restarted MySQL, but that didn't help at all.

To summarize: Concurrent connections to MySQL cause 2006, 'MySQL server has gone away' and some other strange errors. There is nothing of relevance in MySQL's error log.

I have been working at this for hours and haven't made any progress. Can somebody help me out?

I'm trying to run Google's Closure Compiler on my server (CentOS 6). I installed it, but only got errors when I tried to run it.

java.lang.NoSuchMethodError: method java.util.regex.Pattern.quote with signature (Ljava.lang.String;)Ljava.lang.String; was not found. (Closure::Error)
   at com.google.javascript.jscomp.RhinoErrorReporter.replacePlaceHolders(RhinoErrorReporter.java:68)
   at com.google.javascript.jscomp.RhinoErrorReporter.<init>(RhinoErrorReporter.java:74)
   at com.google.javascript.jscomp.RhinoErrorReporter.<init>(RhinoErrorReporter.java:32)
   at com.google.javascript.jscomp.RhinoErrorReporter$OldRhinoErrorReporter.<init>(RhinoErrorReporter.java:143)
   at com.google.javascript.jscomp.RhinoErrorReporter$OldRhinoErrorReporter.<init>(RhinoErrorReporter.java:139)
   at com.google.javascript.jscomp.RhinoErrorReporter.forOldRhino(RhinoErrorReporter.java:108)
   at com.google.javascript.jscomp.Compiler.<init>(Compiler.java:151)
   at com.google.javascript.jscomp.CommandLineRunner.createCompiler(CommandLineRunner.java:755)
   at com.google.javascript.jscomp.AbstractCommandLineRunner.doRun(AbstractCommandLineRunner.java:693)
   at com.google.javascript.jscomp.AbstractCommandLineRunner.run(AbstractCommandLineRunner.java:334)
   at com.google.javascript.jscomp.CommandLineRunner.main(CommandLineRunner.java:871)

It seems I'm running GIJ instead of Oracle's Java, so I think that could be causing my problems. (Or maybe not, feel free to correct me if this isn't likely.) How do I remove GIJ and install Oracle Java in it's stead?

I did the obvious: Downloaded the RPM from Oracle, and ran rpm -Uvh jre-7u5-linux-x64.rpm as root. Ran java --version and it's still GIJ.

I've been Google for half an hour now, but haven't found anything helpful.

I want to redirect some of my old sites to my new sites. Some Googling suggest pasting this into my .htaccess:

redirect 301 / http://new.com/

However, the new site does not have the same structure as the old sites, so it's just pointing to a bunch of 404s. How can I configure my old sites to redirect to the root of my new site, regardless of path? eg:

old.com/foo -> new.com
old.com/bar/foo.html -> new.com
old.com -> new.com

Thank you!

I have to transfer a website to a new IP address on the same server in order to install an SSL cert. Apparently it will take 4-24 hours for the DNS changes to propagate. I asked my host's support if I could serve the website to both IP addresses and avoid downtime, but they said that's not possible. I'm calling BS, it seems to me it should be very possible. So, is it possible? If so, how do I do it?

My server is a VPS running CentOS with Apache and cPanel/WHM.

This question is a followup to this one. I got Apache running on port 80 on one of my four IPs pretty easily. Now I need to run Maven on port 80 on another IP. Running Maven on port 80 is already difficult enough; I have no idea how to define a specific IP. Any ideas on how to do this? Perhaps ipchains or iptables would be the easiest?

I need to run Apache and Maven on my server, and because of cross-domain security restrictions they must both be running on port 80. Originally I achieved this using HAProxy, but this got in the way of the other websites I host on my VPS. I do have four IPs, however, and it would be very nice to have Apache running port 80 on one IP and Maven running port 80 on another. Then I could easily route a few subdomains to Maven using a CNAME record. Is this possible to do?