Termos's questions

I have Nginx working as reverse proxy and Django application behind it. Django app is connected to Nginx through UWSGI. Django app is deployed using Docker and gets down for a second or so during deployment. I would like to instruct Nginx to return custom 503 error page during that period.

Below is my Nginx config:

upstream server_django {
    server 0.0.0.0:8000;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name server.com;

    location / {
        # required to serve static assets
        alias    /data/server.com/website_root/;
        try_files $uri @django;
    }

    location @django {
        uwsgi_pass  server_django;
        include     /etc/nginx/uwsgi_params;
    }
}

I have tried adding following config inside "server" block:

uwsgi_intercept_errors on;
error_page 502 503 504 =503 @django_is_down;
location @django_is_down {
    root /error_pages/server.com/;
    add_header Retry-After 10 always;
    index 503.html;
}

but it didn't help since Nginx stared serving default 404 page when backend is down.

How do I detect when Django is down in Nginx and serve custom error page when that happens?

I run a website which is being abused every day with bots trying to send spam, not respecting robots.txt and so on. The biggest problem, however is repeating attempts to massively harvest website. Right now i have my own script which uses Nginx to block misbehaving ips. This list contains like 100000+ ips and it's still growing.

Now the problem: I am migrating my website to a new server. I'd like to ask for advice on how to block such amount of ips. The most popular solution is Fail2Ban, however it is able to ban using iptables which have their own limitations (around 4500 rules, see How many rules can iptables support?) So, blocking maximum of 4500 ips is definitely not enough for me.

I'd like to know what is the best way to overcome this limitation.

Should i add rules to fail2ban to block ips using nginx or should i use another approach ?

I am trying to write .htacess file that would catch urls like:

http://www.site.com/pageone.html

http://www.site.com/pagetwo.html

and redirect them to index.php so it would receive 'pageone' and 'pagetwo' as GET parameters.

Below is my htacess:

Options +FollowSymlinks

AuthUserFile /dev/null

AuthGroupFile /dev/null

RewriteEngine On

RewriteOptions inherit

RewriteRule ^((?:.[^/])+).html$ index.php?page=$1 [L]

AddHandler php5-script .php

The problem with this regexp is that it catches some pages while showing 404 error on others. And i don't understand this behavior.

For example

http://www.site.com/draw.html

works fine. I get 'draw' as GET parameter in my script.

But

http://www.site.com/draws.html

redirects to 404 page

Could anyone explain what is wrong with my htacess ?

p.s. - i am seeing this on a local Apache server under typical MAMP config.

I have a site running on Nginx (v1.0.14) serving as reverse proxy which proxies requests to Apache (v2.2.19). So Nginx runs on port 80, Apache is on 8080.

Overall site works fine except that i cannot block access to certain directories with .htaccess file.

For example i have 'my-protected-directory' on 'www.site.com' Inside it i have htaccess with following code:

<Files *>
order deny,allow
deny from all
allow from 1.2.3.4 <--- my ip address here
</Files>

When i try to access this page with my ip (1.2.3.4) i get 404 error which is not what i expect:

http://www.site.com/my-protected-directory

However everything works as expected when this page is served directly by Apache. I can see this page, everyone else can't.

http://www.site.com:8080/my-protected-directory

Update. Nginx config (7.1.3.7 is site ip.):

user  apache;
worker_processes  4;
error_log  logs/error.log;
pid        logs/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    sendfile        on;
    keepalive_timeout  65;
    gzip  on;
    gzip_min_length 1024;
    gzip_http_version 1.1;
    gzip_proxied any;
    gzip_comp_level 5;
    gzip_types    text/plain text/css
                  application/x-javascript text/xml
                  application/xml application/xml+rss
                  text/javascript image/x-icon;

    server {
    listen       80;
    server_name  www.site.com site.com 7.1.3.7;
    access_log  logs/host.access.log  main;

    # serve static files
    location ~* ^.+.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js)$ {
      root    /var/www/vhosts/www.site.com/httpdocs;
      proxy_set_header Range "";
      expires 30d;
    }

    # pass requests for dynamic content to Apache
    location / {
      proxy_redirect               off;
      proxy_set_header X-Real-IP  $remote_addr;
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Range "";
      proxy_pass      http://7.1.3.7:8080;
    }
  }

Update 2: mod_rpaf is installed and ip address get determined correctly

Could please anyone tell me what is wrong and how this can be fixed ?

I've created a linux bash file in a windows text editor. This file contained commands for moving files to another location under linux.

After i ran this file i discovered that destination files have "\r" symbol in their end. (I guess this is because linux and windows use different line end symbols)

So 'Dir' command shows that i have files like these: "httpd.conf\r" while 'ls' shows the same filename as: "httpd.conf?"

How do i delete these files ?

These commands do not work:

Rm httpd.conf\r
Rm httpd.conf\\r
rm 'httpd.conf\r'

I receive the following error:

rm: cannot lstat `httpd.confr': No such file or directory

The problem:

I manage a website which has lot's of dynamically generated pages. Every day bots from Google, Yahoo and other search engines download like 100K+ of pages. And sometimes i have problems with "hackers" trying to massively download the whole website.

I would like to block ip addresses of "hackers" while keeping search engine bots crawling the pages. What is the best way to do this ?

Note:
Right now i am solving the problem as follows. I save ip of each page request to a file every X seconds. And i have a crontab script which counts repetitive ip's every 30 minutes. For ip's which are repeated too many times, the script checks a hostname- if it doesn't belong to Google/Yahoo/Bing/etc then we have a candidate for banning. But i don't really like my solution and think that auto banning could be done better or using some out of the box solution.