Home / server / Questions / 319353
Accepted
lamp_scaler
Asked: 2011-10-07 23:37:33 +0800 CST

How to prevent discovery of a secure URL?

  • 772

If I have a url that is used for getting messages and I create it like so: http://www.mydomain.com/somelonghash123456etcetc and this URL allows for other services to POST messages to. Is it possible for a search engine robot to find it? I don't want to have it in my robots.txt because that will expose it already to anybody that sees the robots file.

Of course I will put in other authentication in the app, but not having anyone discovering that URL is the first step.

Any common methods?

security web-applications robots.txt search-engine

3 Answers

  1. If your authentication is working there is no reason to hide the url. Id rather focus on that.

    • 5
  2. Best Answer

    It is not possible for search engines to find it, if it is not linked anywhere (as this is the only way search engines find other information). In order to make sure a search engine does not find it via a robots.txt and have the secrecy, use a double-hash:

    http://example.com/asdfghjk/12345678

    Your robots.txt would disallow anything below asdfghjk:

    User-agent: *
Disallow: /asdfghjk/

    But anyone that does not know the full path will still not be able to get the second url part from looking at it.

    • 4

  3. you could put in something identifiable such as /secretpage[hash]123456etcetc then grep the apache logs which an automated script that block access if one ip tries to access a secretpage url too often.

    In reality though, miademora is correct. Get your authentication secure (2 factor?) and ditch the security by obscurity.

    • 0