If I have a url that is used for getting messages and I create it like so: http://www.mydomain.com/somelonghash123456etcetc and this URL allows for other services to POST messages to. Is it possible for a search engine robot to find it? I don't want to have it in my robots.txt because that will expose it already to anybody that sees the robots file.
Of course I will put in other authentication in the app, but not having anyone discovering that URL is the first step.
Any common methods?
I'm searching for a intranet search engine which is capable of spider our intranet websites and network shares like SMB, NFS and optionaly AFP.
Ever better for us would be a search engine which is extendable via plugins like the Spotlight framework on Mac OS X.
Does somebody know something like this? Most of the results I've found so far are only capable of spidering websites.
If the robots.txt file is missing in the root directory of a website, how are things treated as:
It should logically be the second one according to me. I ask in reference to this question.
We experience a lot of traffic and server load on a web server.
All I can find out is majestic12 accessing pages all the time.
I wonder how I can prevent majestic12 from indexing the site
Do they respect any robots.txt entry and how do I write such an enty?
I want to block all request from yandex.ru search bot. It is very traffic intensive (2GB/day). I first blocked one C class IP range, but it seems this bot appear from different IP ranges.
For example:
spider31.yandex.ru -> 77.88.26.27 spider79.yandex.ru -> 95.108.155.251 etc..
I can put some deny in robots.txt but not sure if it respect this. I am thinking of blocking a list of IP ranges.
Can somebody suggest some general solution.