Home / server / Questions / 142281
Accepted
user12096
Asked: 2010-05-16 15:56:01 +0800 CST

How to prevent majestic 12 from indexing a site

  • 772

We experience a lot of traffic and server load on a web server.

All I can find out is majestic12 accessing pages all the time.

I wonder how I can prevent majestic12 from indexing the site

Do they respect any robots.txt entry and how do I write such an enty?

apache-2.2 search-engine

2 Answers

  1. Best Answer

    According to Majestic 12's own page about robots.txt, they fully respect robots exclusion (3rd answer from the top). The robots.txt file is a plain text file in the root of your website, i.e. you place it at:

    http://www.yourdomain.com/robots.txt

    and have these lines in the file:

    User-agent: MJ12bot
Disallow: /

    So if you want to block that bot, I see no problem -- unless you're getting hammered by one of the fake bots they mention.

    • 6

  2. For the op's follow up question:

    I wonder if they have mentioned all fake bots. Why would someone claim to be majestic12?

    That would be a false flag operation. Virus disguises itself as a legit bot/process to crawl ip's. The explanation is on the majestic12 faq way down.

    • 3