Basj's questions

I have a service example.com that needs to allow users to host a few files in their own subdomain userpage-userabc.example.com, similarly as does Github with userabc.github.io.

I currently use Apache, and each time I want to configure a new domain or subdomain, I have to create manually a new <VirtualHost>, restart the Apache server, etc. This is not possible here.

Example: if a visitor creates a new account userabc (added in a database), how to make that the userpage-userabc.example.com subdomain is available, and automatically serving files from /www/userabc/ or serving http://userpage-userabc.example.com/ with /www/main/index.php?user=userabc?

Note: *.example.com already has a wildcard A DNS-record to redirect to my server

What's a simple lightweight tool to do this "dynamic subdomain creation"? (I currently use Apache, PHP, Python, but I can use other tools).

Or is it possible with a single <VirtualHost> that catches all sudomains *.example.com and a RewriteRule including the subdomain?

Let's say I own example.com, that I bought by ExampleRegistrar.
Let's say that, by default, this registrar uses their own nameservers ns1.exampleregistrar.com, ns2.exampleregistrar.com.

Thus, when a client wants to browse to www.example.com, it probably goes like this:

• root DNS servers looks for .com nameserver
• .com nameserver looks for example.com, it sees ns1.exampleregistrar.com and passes the request to this nameserver
• ns1.exampleregistrar.com looks for www.example.com and finally finds IP 203.0.113.5 which it gives to the client.

Question: how does it work if I configure a custom nameserver ns1.othernameserver.com in the registrar domain's settings?

Does this mean the registry for .com will directly have ns1.othernameserver.com in its database?

Is it correct that, then, any request from a client to www.example.com will never go through any of the registrar's servers?

Around 2000-2010, shared hosting was extremely popular as a cheap solution (sometimes a few $ / month, or sometimes even free for just a few MB) for people starting blogs, small websites, e.g. using Wordpress.

There was usually:

• just Apache + PHP + MySQL
• no SSH, only (s)FTP access
• something like 100 MB
• as far as I remember, they probably didn't create a new virtual machine for each account

Question: before containerization / Docker went popular, how did major shared hosting providers ensure user isolation?

Did they just used ChrootDirectory in sshd_config + different users like in How to create an isolated/jailed SFTP user? + <VirtualHost> config with open_basedir to prevent PHP code to access other accounts' files?

More generally, what were the main isolation techniques, preventing user1234 to access user5678's files on the same server with some malicious PHP code?

I have read Multiple wildcard dns entries and know the limitations/drawbacks of wildcard DNS entries, but I have a question about the order in which it is listed by my registrar:

example.com        A      192.0.2.1    (1)
test.example.com   A      192.0.2.2    (2)
*.example.com      A      192.0.2.3    (3)
test2.example.com  A      192.0.2.4    (4)

More precisely:

• Am I correct that (3) won't impact (1) and so the only A record for example.com is (1)?

• Is rule (2) overriden by rule (3), or will both rules apply for test.example.com (thus two A records for this subdomain)?

• As (4) is listed after (3), does this mean test2.example.com will have only one A record (192.0.2.4) or two A records (both 192.0.2.3 and 192.0.2.4)?

Let's say I rent 6 dedicated servers by a server provider for my website http://example.com:

#1: 203.2.113.1
#2: 203.2.113.2
#3: 203.2.113.3
...
#6: 203.2.113.6

each of them having a maximum upload rate of 10 MB/s.

I decide to use machine #1 as load-balancer, so by my registrar, I set up a DNS "A" record: example.com A 203.2.113.1.

Two cases:

• 50 clients want to receive a video stream @200KB/s from my service at the same time: everything seems fine:

  They connect http://example.com in their browser, the DNS gives them IP 203.2.113.1, their browser makes a HTTP request to this IP, the load-balancer (machine #1) receives the HTTP request, gives the task to machine #2, #3, ..., #6, and these machines send the stream to the client by passing the data first through the load balancer. Am I correct that the client wouldn't accept a response from 203.2.113.2 because their request was sent to 203.2.113.1?

• 200 clients want to receive a video stream @200KB/s from my service: it does not work:

  The clients connect http://example.com in their browser, the DNS gives them IP 203.2.113.1, their browser makes a HTTP request to this IP, the load-balancer (machine #1) receives the HTTP request, gives the task to machine #2, #3, ..., #6. Each of these 5 machines process 1/5th of the 200 clients, i.e. 40 clients * 200KB/s = 8 MB/s.

  If #2, #3, ..., #6 could send the data directly to the clients, it would be fine (8 MB/s).

  But since the data has to transit via the load-balancer machine #1, it does not work because machine #1 cannot upload at 40 MB/s.

How to configure the load-balancer (machine #1) to solve this problem?

Or would it be possible that the load-balancer passes the requests to machines #2, #3, ..., #6 and that then, they answer to the clients directly, without going through machine #1 for the responses?

I'd like to obtain one certificate working for all my subdomains *.example.com.

This works:

certbot-auto certonly --webroot --webroot-path /home/www/example/ --domain example.com 
                                --domain www.example.com --email [email protected]

but this (with *.):

certbot-auto certonly --webroot --webroot-path /home/www/example/ --domain example.com 
                                --domain *.example.com --email [email protected]

fails with:

Obtaining a new certificate
Performing the following challenges:
Client with the currently selected authenticator does not support any combination of challenges that will satisfy the CA. You may need to use an authenticator plugin that can do challenges over DNS.

How to use certbot-auto to generate a certificate for *.example.com?

I have read What permissions should my website files/folders have on a Linux webserver? and here is a linked question. Let's say:

• /var/www/bobweb/ is owned by bob:bob

• /var/www/eveweb/ is owned by eve:eve

and that each site has an Apache VirtualHost, linking to bobweb.com and eveweb.com.

Each website needs to be writable by Apache (there is an image uploader on both websites), so I often read:

"Just do chgrp -R www-data /var/www/bobweb/, idem for Eve, and that's all!"

This gives:

drwxrwx---  3 bob    www-data 4096 Jan  9 17:45 bobweb
drwxrwx---  5 eve    www-data 4096 Feb 26  2019 eveweb

This indeed prevents Bob to visit Eve's files from shell/SSH and vice-versa.

But it seems that Eve could write a PHP script (run by www-data) to read (or even modify) /var/www/bobweb/wp-config.php (thus stealing Bob's DB password!).

Question: how to modify user+group ownerships and permissions to prevent Bob to read Eve's files and vice-versa, but still allow Apache+PHP to write data on these websites?

(typical shared hosting situation)

I am using SendGrid to send emails and I have properly configured my domain (SPF, etc.). Usually everything works, except that Hotmail/Outlook often rejects the emails I send:

reason": "550 5.7.1 Unfortunately, messages from [x.89.100.x] weren't sent. Please contact your Internet service provider since part of their network is on our block list (S3140). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. [AM6EUR05FT024.eop-eur05.prod.protection.outlook.com]"

I already contacted SendGrid, but their action did not help, so there is probably something to do on my end?

Quoting https://sendersupport.olc.protection.outlook.com/pm/troubleshooting.aspx#errors:

Why did I receive a "550 command rejected due to Sender ID validation failure." SMTP Non-Delivery Report (NDR) when I attempt to send mail to Outlook.com users? Outlook.com will not allow delivery of email sent from a domain where the Sender ID record was configured by the domain owner to NOT allow ANY IP to send mail from that domain. Sender ID allows a domain owner to protect domains that aren't intended for sending email in order to help protect their domain from being spoofed. This can be done by publishing a simple TXT record in DNS like the following example (note: the organization would replace example.com with their own domain and or sub-domain name):

example.com IN TXT "v=spf1 -all"

If the domain is repurposed to send mail, the administrator of the DNS record should update the Sender ID record to include the IP address(s) that are authorized to send mail from that domain. Note that updates to your Sender ID record can take up to 48 hours to propagate through the Internet, so it's a good idea to wait 48 hours after making a change to your record before you initiate any new email activities. In addition, Microsoft strongly recommends that you conduct email testing prior to sending live communications to your users/customers.

The double negation "Outlook.com will not allow delivery of email sent from a domain where the Sender ID record was configured by the domain owner to NOT allow ANY IP..." makes it a little bit unclear.

What should be done on my end (e.g. in DNS records) to ensure emails to Hotmail to be delivered?

With Apache, if I want to slightly modify a "Rewrite rule" for a website on my server, I can just edit a few lines in /var/www/website5/.htaccess, save and it will be immediately active.

Benefit: it's not required to service apache2 restart, thus it won't interrupt ongoing connections on other websites that are on the same server (let's imagine someone is currently downloading a big 1 GB, I would prefer not to interrupt this!).

Question: I'm now testing nginx, I read that there is no "natural" equivalent of .htaccess (of course, there are add-ons that allow this, but here I'm really focusing on understanding the original philosophy of nginx, that does not have this feature).

How to slightly modify the nginx configuration (e.g. URL rewrite rules) of a website, without requiring the whole server (hosting many websites) to restart?

PS:

• if there exists a "graceful reload" feature that waits for requests to be terminated before restarting nginx, does this mean that if a client just begun the download of a 2GB file that will last 2 hours, nginx will wait 2 hours (for the request to be completed) before reloading the new configuration?
  Then it would be very difficult to modify the configuration if we have to potentially wait minutes/hours to test that config modifications are done.

• if the latter problem doesn't exist, how does it work? (is there a documentation source about this specific point?)

I know how to setup these solutions:

1. Let's say the website www.example.com has a single machine for it (IP 203.0.113.123), using a Linux + Apache server. It works if the bandwidth is not too high.

2. Let's say we now have more and more requests for this website, thus the computer with IP 203.0.113.123 will now only be a load-balancing server, dispatching the requests to many workers (other computers), which serve the actual webpages.

but I have a question about this case:

3. Let's say now there are sometimes peaks of tens or hundreds of thousands of requests per second for this website. Having a single load-balancing machine 203.0.113.123 is not possible anymore.

If we have 10 load-balancing machines: 203.0.113.123, 203.0.113.124, 203.0.113.125, 203.0.113.126, 203.0.113.127, ..., 203.0.113.132 (that dispatch the requests to tens of other workers), how to have the clients arrive to all of them, randomly / Round-robin-style?

The DNS records usually display only one A record, even for big well-known websites (see here the DNS records for Google, there is only one A record), such as:

www.example.com 3600    A   0   203.0.113.123

How to make that clients' requests arrive to the ten machines randomly and not only 203.0.113.123?

(Obviously, we don't want a single machine to dispatch the requests to the ten load-balancing machines, because then we would be back to case 2.)

For each website I have, I noticed that having just this A DNS record:

*.example.com   3600    A   0   192.1.2.3

will make http://example.com unavailable and having just this A DNS record:

example.com 3600    A   0   192.1.2.3

will make http://www.example.com unavailable.

Question: is it mandatory to have two A DNS records to support www and non-www

*.example.com   3600    A   0   192.1.2.3
example.com     3600    A   0   192.1.2.3

or is there a way to define both in one A DNS record?

PS: If it's mandatory to have two records, would you use:

www.example.com 3600    A   0   192.1.2.3
example.com     3600    A   0   192.1.2.3

or would you do it this way:

www.example.com 3600    CNAME example.com
example.com     3600    A   0   192.1.2.3

?

As mentioned here, having a Reverse PTR record is key for successful email sending and delivery from one's own server. In order to check if it's correctly set for my server, I tried:

dig -x 111.111.111.111    #  replace with the IP of a server

but the answer is:

; <<>> DiG 9.9.5-3ubuntu0.1-Ubuntu <<>> -x 111.111.111.111
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 29789
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;xxx.xxx.xxx.xxx.in-addr.arpa.     IN      PTR

;; Query time: 27 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Sep 10 14:51:32 CEST 2017
;; MSG SIZE  rcvd: 54

I don't see anything that looks like a Reverse PTR record.

I tried with various IPs obtained from well-known websites: dig -x 151.101.1.69 (obtained from ping stackoverflow.com), and it looks the same.

Why can't I see any Reverse PTR record with dig?

I have set this in /etc/postfix/main.cf:

inet_protocols=ipv4 
#I also tried with smtp_address_preference=ipv4           

But still, the received message in Gmail shows an IPV6 address next to SPF. Why?

When I look at my Apache log other_vhosts_access.log, I see many many attempts, from a few different IP per month like this:

www.example.com:80 91.200.x.x - - [25/Jun/2017:17:20:19 +0200] "POST /wp-login.php HTTP/1.1" 
www.example.com:80 91.200.x.x - - [25/Jun/2017:17:20:19 +0200] "POST /wp-login.php HTTP/1.1" 
www.example.com:80 91.200.x.x - - [25/Jun/2017:17:20:20 +0200] "POST /wp-login.php HTTP/1.1" 

It seems to be brute-force attacks.

Is there a simple way to auto-ban (no link with Autobahn) these attackers?

• If I do it with a Wordpress plugin, the traffic will still go into PHP, wasting resources, etc.

• If I do it at Apache level, it will surely still waste resource

• Should I do it at lowest level possible? i.e. IP tables?

Is there a tool that looks for such attackers in other_vhosts_access.log and automatically ban them in iptables?

On a 1&1 shared hosting, I'm using mail(...); from a PHP script (which itself calls /usr/sbin/sendmail -t -i) and finally I get an error message (see below).

More or less my mail (I send less than 5 mails per day) is considered as spam by SORBS. It's not spam (it is usually an important mail to a customer with their download link after a purchase) nor a virus.

I'm having a nightmare since days to solve this problem.

Questions:

• what is the flow diagram of my email ?

local PHP ==> local sendmail ==> ... ? SORBS ? ==> recipient server ? => error message? (who sends it? sorbs or recipient or local sendmail?)

• where is SORBS working in this story ? By my shared hosting (1&1)? Somewhere in the middle? By who is this service called? By the recipient's server?

• how to typically solve such problems? SORBS seems to be difficult to reach (no phone, difficult to contact support via email, etc.) and very opaque

Note: I've asked SORBS for delisting, but the problem continues.

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed:

[email protected]: SMTP error from remote server for RCPT TO command, host: mail.forwarder.lexsynergy.com (212.20.251.xx) reason: 550-212.227.126.xx is listed at dnsbl.sorbs.net (127.0.0.xx, 127.0.0.xx: 550 Virus Transmitting Server See: http://www.sorbs.net/lookup.shtml?212.227.126.xx)

--- The header of the original message is following. ---

Received: from xx.kundenserver.de (infong1124.kundenserver.de [212.227.18.xx]) by mrelayeu.kundenserver.de (node=mreue006) with ESMTP (Nemesis) id 0LzUX8-1b2ynI3IFW-014gcV; Tue, 06 Sep 2016 14:34:58 +0200 Received: from 173.0.81.1 (IP may be forged by CGI script) by icpu2490.kundenserver.de with HTTP id 4AibFb-1bfKiR2kcj-002Zae; Tue, 06 Sep 2016 14:34:58 +0200 X-Sender-Info: xxx