Is there a way to filter request that look like "GET http://example.com" but not filter one that look like "GET /index.html"? In other words what is the Nginx variable that can be used in the "If" clause?
The reason for asking this is that I see in my web site log multiple requests that look like: "GET http://somesite.com/proxyheader" from users that search for free proxy servers.
I want to block all request from yandex.ru search bot. It is very traffic intensive (2GB/day). I first blocked one C class IP range, but it seems this bot appear from different IP ranges.
For example:
spider31.yandex.ru -> 77.88.26.27 spider79.yandex.ru -> 95.108.155.251 etc..
I can put some deny in robots.txt but not sure if it respect this. I am thinking of blocking a list of IP ranges.
Can somebody suggest some general solution.