Altimus Prime's questions

If this is a duplicate, just point me to the answer and I'll remove this.

I have a locally hosted email server using the ISP for the nameserver. In the email headers I get the Spam Assassin message URIBL_BLOCKED. When I google URIBL_BLOCKED the first result is https://help.directadmin.com/item.php?id=666 which says to change your resolve.conf to have nameserver 127.0.0.1, but this assumes you have a dns server running and listening on the localhost.

Moving down the Google results I find http://uribl.com/refused.shtml which has an interesting section which reads:

Low Volume Workaround? If you are low volume user, you have a few options. Possibly changing your nameservers from a public dns provider (ie opendns/google) to your local ISP may solve it. If your local ISP is also effected because they are very large (ie cox/att/comcast/etc), you may need to use your own recursive DNS solution. If your company has DNS servers, point to them for resolution. Alternatively, you could setup a caching nameserver on the loopback of the machine doing the spam checking, and point the DNS to localhost.

Can someone point me to instructions on how to do this? I've read that bind9 is a linux program that can do this but wherever I find bind instructions it's regarding how to host the dns records of my sites, not how to route my Spam Assassin URBL requests through my local nameserver.

Using Certbot to install an R3 Let's Encrypt certificate on an nginx webserver causes all the other domains in the nginx configuration to be included under "Subject Alternative Name" on the certificate. This is undesirable for my use case.

I read the man page here and some other Stack Exchange posts here and here.

Regarding the use of alternative names, the man page says (and I don't fully understand):

-d DOMAIN, --domains DOMAIN, --domain DOMAIN Domain names to apply. For multiple domains you can use multiple -d flags or enter a comma separated list of domains as a parameter. The first domain provided will be the subject CN of the certificate, and all domains will be Subject Alternative Names on the certificate. The first domain will also be used in some software user interfaces and as the file paths for the certificate and related material unless otherwise specified or you already have a certificate with the same name. In the case of a name collision it will append a number like 0001 to the file path name. (default: Ask)

How can I specify or omit the Subject Alternative Names entirely when using Certbot to install a Let's Encrypt certificate? If Certbot can't, is there a different way while still using R3 Let's Encrypt certificates?

In Postfix we can apply some basic header, body, client checks using regex. I wanted to use flags.

I added the following to the header_checks file:

/some(\s|_|-)words(\s|_|-)here/gmi REJECT

and tested it by sending myself and email with those words in the title, but I get the error in my maillog stating unknown regexp option "g": skipping this rule

Is there a proper way to apply flags or does postfix regex simply not use or acknowlege certain flags?

Wouldn't setting the DNS server to 127.0.0.1 prevent my system from actually resolving remote hosts?

My business email server has the DNS set to the ISP given DNS servers. My Spamassassin is giving the message URIBL_BLOCKED. The solution given here indicates that I should set the DNS server to the localhost at 127.0.0.1 and another similar solution was here.

Wouldn't that prevent my system from resolving other remote hosts? Is there anything in a basic linux (centos 7) install that would actually be able to resolve whatever remote host this system might need to reach?

EDIT: Having tried it, it kind of seems broken when I make the change (emails sent from it didn't arrive when they otherwise have), so if someone can fill me in on the missing details, I'd be very appreciative.

The Problem:

How do you diagnose an ISP specific connectivity problem?

I host a few sites on local machine. I pay for a business class account with Centurylink and have a lot of 5 static ips. The sites are accessible from anywhere in the world except other Centurylink users. If my customers didn't live in the sticks like me, I wouldn't have ever known. Cox users, Verizon users, Cricket, At&t, Comcast, and a huge number of US and internationally based internet service providers can access my sites and ip addresses, EXCEPT other people using Centurylink.

What I've done:

While on location at another person's home with Centurylink internet, I attempted to access each ip address on ports 80 and 443. I attempted to ssh into the server at an arbitrary port that I've selected for that service. I ran tracert on both the domains and the ip addresses and got the following:

1   2ms 1ms 1ms modem.Home [192.168.0.1]
2   30ms    30ms    25ms    rbflxyza84.centurylink.net [x.x.123.45]
3   29ms    30ms    33ms    asdf-ghjk.inet.qwest.net [x.x.122.34]
4   29ms    30ms    33ms    x.x.x.18
5   200ms   79ms    59ms    x-x-45-67.orlf.qwest.net [x.x.45.67]
6   *   *   *   Request timed out.
7   *   *   *   Request timed out.
8   *   *   *   Request timed out.
9   *   *   *   Request timed out.
10  *   *   *   Request timed out. 

Coincidentally x.x.45.67 is the ip address I get when I Google What's my ipv4 into Google. If this were working the static ip address would be the next result, so I would think that the request is being blocked at the router.

Using the originating public ip address I searched the access logs and the firewall logs on the router. There are no records of the remote ip address. I am able to find records when packets are dropped according to the rules I've set for OTHER ip addresses.

Just in case I checked the server logs and found no record of the remote address.

Where else could I check to diagnose this problem? I would have asked on SO Network Engineering, but they restrict their questions to only enterprise level issues and solutions.

I've checked blacklists, but my ip addresses don't come up. I've checked the router firewall, but when something there is blocked by one of my rules, it gets logged and I can find the record.

My worst case solution

Before someone suggests it, my worst case scenario is to reverse proxy the traffic through some remote servers I also administer.

Edit

I set up the reverse proxy last night in less time than the average hold time at Centurylink. I'm loathe to call them because they took a month just to provision my static IP addresses. It was like no one there understood why a business needs a static IP address, or even what that is.

Solution

Michael Hampton's comment about the netmask being incorrect is exactly right. All the clients who can't access my address share at least the the first 8 bits of the ip address. The netmask was 255.0.0.0 when it should have been 255.255.255.248. NetworkManager was overwriting my settings, so even after setting it correctly (probably for the second time), it was reverting to the incorrect netmask.

PHPMailer worked fine until June 1st 2020. The SSL certificate for the site sending the email is valid from June 1st 2019 to June 1st 2021 and is accepted fine in the browser, but postfix suddenly has a TLS problem. I have access to both the email server and the web server attempting to send the email and they are running on the same network. I get the following messages when the site attempts to send the email.

log/maillog on the email server:
Jun  9 12:26:39 12-34-56-78 postfix/submission/smtpd[3299]: connect from unknown[12.34.56.80]
Jun  9 12:26:39 12-34-56-78 postfix/submission/smtpd[3299]: SSL_accept error from unknown[12.34.56.80]: 0
Jun  9 12:26:39 12-34-56-78 postfix/submission/smtpd[3299]: warning: TLS library problem: 3299:error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:s3_pkt.c:1493:SSL alert number 45:
Jun  9 12:26:39 12-34-56-78 postfix/submission/smtpd[3299]: lost connection after STARTTLS from unknown[12.34.56.80]
Jun  9 12:26:39 12-34-56-78 postfix/submission/smtpd[3299]: disconnect from unknown[12.34.56.80]

I checked that none of the configuration files have changed for either postfix or PHPMailer. They're still dated as last modified in 2019.

I don't want to disable 'verify peer' because it really ought to verify the peer. I just can't tell why it stopped working when the certificate hasn't changed, but I'm sure the dates are more than coincidental. How do I fix this and how do I prevent this in the first place?

** EDIT **

In my case the problem was that the CA bundle was expired, even though the certificate was valid. I think they have to be together in the same file for postfix config.

I completely misunderstood which side has issue with which. In this case the client had issue with the email server.

To fix:

1. Check postfix/main.cf (or whatever your file is named)

2. Look for the line specifying smtpd_tls_cert_file

3. Replace that file with the current cert (including the CA bundle)

EDIT:

I had a cert issued by the folling people, whose root CA bundle expired https://www.ssls.com/knowledgebase/sectigo-root-certificate-expiring-may-30-2020/

I've gotten a lot of strange http requests in my access logs before, like calls to nonexistent WordPress login scripts and application specific locations. I've even gotten a few wise guy requests like ...GET /your-site-sucks HTTP/1.1 400....

I'm curious about some that have varying lengths of octal characters. Here's an example:

x.x.x.x - - [31/Aug/2019:15:20:13 -0400] "\x01A\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" 400 157 "-" "-"

I understand this as Start of heading start of text null null null null null..... Looking up the ip address, I found it was labeled as an abusive ip.

What kind of exploit is this particular request looking for?

When provisioning a Windows Server 2008 R2 Datazenter VPS in Azure there is a question,

Already have a Windows Server Licence?

I selected "no" and expected that the cost for the instance would include a license because the info bubble says,

If you own Windows licenses with active Software Assurance (SA) or have an active Windows Server subscription, use Azure Hybrid Benefit to save compute cost.

Upon accessing the server via remote desktop connection, the instance indicates that windows is not activated and needs a product key.

https://azure.microsoft.com/en-us/pricing/licensing-faq/ states:

The license to run Windows Server in the Azure environment is by default included in the per-minute cost of your Windows Virtual Machine.

So where is the key?

Need clarification for upstream SSL on an nginx reverse proxy server

I've been reading the nginx docs regarding reverse proxy and securing ssl connections to upstream servers but I'm still confused about which ssl certificates go where. Many of the examples I find have nginx proxying localhost, but my situation has the endpoints on different servers, ports and physical locations.

I'd like to have several domains resolved at the nginx server. Each of those domains has an ssl certificate on its current server for its actual domain name.

Right now I have each server running at it's own network location and physical location, but I'd like to have a single point to manage those endpoints.

My end result should look like

                             client
                               |
                             nginx
                       https://example1.com
                       https://example2.com 
                       https://example3.com
                             x.x.x.x
                               |
               -----------------------------------------
               |                |                      |
https://example1.com    https://example2.com    https://example3.com
   a.b.c.d:1234             e.f.g.h:5678            i.j.k.l:9012

Right now https://example1.com resolves to a.b.c.d:1234 which has it's own ssl certificate installed. Because I need to represent to clients that the nginx server is serving for the domain example1.com I think I need the example1.com ssl moved to the frontrunning nginx server, right? If I do that, what ssl certificate do I use on a.b.c.d:1234 to maintain a secure upstream connection?

The nginx docs say client.crt and server.crt, but the CA uses a domain to register these. What is client and server in a reverse proxy situation? To me the client is the browser making the request.

Which ssl certificates go where on an reverse proxy?

Edit:

I already know you can look like you have a secure connection by simply placing the url based certificates on the proxy server. What I'm hoping to know is what ssl certificates to put on the backend servers. Just reuse their respective certificates? Could example1.com.crt go on both the proxy server and the backend server?

I'm moving my server to a co-location center and they'r not concerned with the actual wattage, which is what I've tracked, but they are very concerned with the peak amperage. They charge by the amps made available to the machine. Is there some industry standard way I can test that? The person I spoke with in the data center is a sales guys, so he's not sure of the technical aspects that he's asking me about.

If there's a software solution, my system is an HP DL580 G7 running centos 7.

What I've tried:

I have a UPS on it now that gives wattage outputs which bounce all over the place. The highest I've seen is 800 watts, so my guess is 800watts/120volts should be six and two thirds amps. Do I provision 7 amps? Doesn't sound very precise.

Powerstat says "Device does not have any RAPL domains, cannot power measure power usage." so I don't think it's compatible with my system.

Please let me know what the industry standards for this are.

I have over 2.5 million ssh attempts and fail2ban and only a few thousand ip's blocked. The find time was set to 600, max retries 4, and bantime -1. The server is physically present. I changed the jail.local and tried restarting systemctl restart fail2ban and it starts with a whole bunch of errors. I reverted jail.local back how it was and tried again and still get the same errors. I deleted some of the secure logs and tried again and it seemed to start but when I reviewed the status the errors were still there and almost nothing is being blocked.

I get the error Fail2ban ipset create fail2ban-sshd hash: ip timeout -l followed by a whole bunch of subsequent failures that I think are precipitated by the first.

I tried stopping, uninstalling and reinstalling fail2ban and still have the same initial error followed by a whole bunch of errors.

My questions are:

1. Why/how does ipset create fail2ban-sshd hash timeout?

2. How can I properly reinstall fail2ban so that maybe it might block something?

3. If there's no working answer on the first two, can't the ip blocking rules be programmed outside of fail2ban?

fail2ban.log:

2019-05-12 21:08:35,823 fail2ban.action         [1730]: ERROR   ipset create fail2ban-sshd hash:ip timeout -1
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- stdout: ''
2019-05-12 21:08:35,823 fail2ban.action         [1730]: ERROR   ipset create fail2ban-sshd hash:ip timeout -1
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- stderr: "ipset v6.29: Syntax error: '-1' is out of range 0-4294967\n\x1b[91mError: COMMAND_FAILED\x1b[00m\n"
2019-05-12 21:08:35,824 fail2ban.action         [1730]: ERROR   ipset create fail2ban-sshd hash:ip timeout -1
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- returned 13
2019-05-12 21:08:35,824 fail2ban.actions        [1730]: ERROR   Failed to start jail 'sshd' action 'firewallcmd-ipset': Error starting action
2019-05-12 21:08:35,825 fail2ban.actions        [1730]: NOTICE  [sshd] Ban 218.92.0.147
2019-05-12 21:08:46,771 fail2ban.transmitter    [1730]: WARNING Command ['start', 'sshd'] has failed. Received OperationalError('database is locked',)
2019-05-12 21:08:51,787 fail2ban                [1730]: CRITICAL Unhandled exception in Fail2Ban:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fail2ban/server/jailthread.py", line 66, in run_with_except_hook
    run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.py", line 290, in run
    self.jail.putFailTicket(ticket)
  File "/usr/lib/python2.7/site-packages/fail2ban/server/jail.py", line 195, in putFailTicket
    self.database.addBan(self, ticket)
  File "/usr/lib/python2.7/site-packages/fail2ban/server/database.py", line 96, in wrapper
    return f(self, self._db.cursor(), *args, **kwargs)
OperationalError: database is locked
2019-05-12 21:08:56,800 fail2ban.actions        [1730]: ERROR   Failed to get all bans merged, jail 'sshd': database is locked
2019-05-12 21:09:01,812 fail2ban.actions        [1730]: ERROR   Failed to get jail bans merged, jail 'sshd': database is locked
2019-05-12 21:09:01,941 fail2ban.action         [1730]: ERROR   ipset add fail2ban-sshd 218.92.0.147 timeout -1 -exist -- stdout: ''
2019-05-12 21:09:01,941 fail2ban.action         [1730]: ERROR   ipset add fail2ban-sshd 218.92.0.147 timeout -1 -exist -- stderr: 'ipset v6.29: The set with the given name does not exist\n'
2019-05-12 21:09:01,942 fail2ban.action         [1730]: ERROR   ipset add fail2ban-sshd 218.92.0.147 timeout -1 -exist -- returned 1

systemctl status fail2ban

fail2ban.service - Fail2Ban Service
   Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-05-12 21:08:46 EDT; 38min ago
     Docs: man:fail2ban(1)
  Process: 1462 ExecStop=/usr/bin/fail2ban-client stop (code=exited, status=255)
  Process: 1726 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS)
 Main PID: 1730 (fail2ban-server)
   CGroup: /system.slice/fail2ban.service
           ├─1599 /usr/bin/python2 -s /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b
           └─1730 /usr/bin/python2 -s /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b

May 12 21:08:24 myHost systemd[1]: Starting Fail2Ban Service...
May 12 21:08:24 myHost fail2ban-client[1726]: 2019-05-12 21:08:24,579 fail2ban.server         [1728]: INFO    Starting Fail2ban v0.9.7
May 12 21:08:24 myHost fail2ban-client[1726]: 2019-05-12 21:08:24,580 fail2ban.server         [1728]: INFO    Starting in daemon mode
May 12 21:08:29 myHost fail2ban-client[1726]: ERROR  NOK: ('database is locked',)
May 12 21:08:34 myHost fail2ban-client[1726]: ERROR  NOK: ('database is locked',)
May 12 21:08:46 myHost fail2ban-client[1726]: ERROR  NOK: ('database is locked',)
May 12 21:08:46 myHost systemd[1]: Started Fail2Ban Service.
[root@myHost /]# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 7
|  |- Total failed:     46
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned: 1
   |- Total banned:     1
   `- Banned IP list:   218.92.0.147

I was installing a new cert like I've done dozens of times before. I copied the names crt, key, bundle files as backup and gave the new files the names of the old files. Then I restarted apache. It failed. It gave an error like what's below.

I tried rebooting the server. Checking again, no difference. I removed the newly uploaded files and replaced with the backup copies and tried restarting again. It still gives the same message. I've never seen this error before. There have not been any changes to the config files.

1) Someone who understands the inner workings of Apache, why would it try to kill a process that doesn't exist and that it can't find?

2) Being that it's not there, why would that prevent Apache from starting?

sudo systemctl status httpd:

● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2018-12-06 23:12:20 EST; 9s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 3615 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 3613 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 3613 (code=exited, status=1/FAILURE)

Dec 06 23:12:20 myHostsystemd[1]: Starting The Apache HTTP Server...
Dec 06 23:12:20 myHostsystemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Dec 06 23:12:20 myHostkill[3615]: kill: cannot find process ""
Dec 06 23:12:20 myHostsystemd[1]: httpd.service: control process exited, code=exited status=1
Dec 06 23:12:20 myHostsystemd[1]: Failed to start The Apache HTTP Server.
Dec 06 23:12:20 myHostsystemd[1]: Unit httpd.service entered failed state.
Dec 06 23:12:20 myHostsystemd[1]: httpd.service failed.

sudo journalctl -xe just says the same thing.

The server is running centOS 7. Apache version is 2.4.6.

In my installation of MySQL the database files are located in ./var/lib/mysql. I wanted to move one of those databases to another drive, so I followed instructions to move one database to a different directory and copied the data files for the database to a different drive, made a symlink to that directory with the same name as the original database folder. Chowned both the directory recusively and the symlink to mysql, same as the other directories in ./var/lib/mysql.

When I restarted MySQL, everything ran fine, but the database at issue is missing (Not critical because this is a dry run.) I checked the logs and had these errors:

2018-07-28T02:43:59.449437Z 0 [Warning] InnoDB: Ignoring tablespace `sf_master/tablename` because it could not be opened.
2018-07-28T02:43:59.449465Z 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2018-07-28T02:43:59.449474Z 0 [ERROR] InnoDB: Cannot open datafile for read-only: './sf_master/tablename.ibd' OS error: 71

I set selinux to permissive to see if that was the problem but the errors are the same.

I confirmed that in my.cnf that symbolic-links=1 and that on the restart I had innodb_force_recovery = 1

The data folder looks like:

-rw-r-----. 1 mysql mysql         56 Apr 24  2017 auto.cnf
drwxr-x---. 2 mysql mysql       4096 Apr 24  2017 mysql
drwxr-x---. 2 mysql mysql       4096 Dec  6  2017 operators
lrwxrwxrwx. 1 mysql mysql         16 Jul 27 22:33 sf_master -> ./nvme/sf_master
drwxr-x---. 2 mysql mysql       8192 Apr 24  2017 sys

NOTE:

I'm posting this problem on serverfault because I think the problem has to do with my understanding of symlinks and the way they are used on directories in linux file systems, or possibly permissions. I'm running a centos 7 server.

UPDATE:

Per Wilson Hauck's request the corrected directory is:

-rw-r-----. 1 mysql mysql         56 Apr 24  2017 auto.cnf
drwxr-x---. 2 mysql mysql       4096 Apr 24  2017 mysql
drwxr-x---. 2 mysql mysql       4096 Dec  6  2017 operators
lrwxrwxrwx. 1 mysql mysql         16 Jul 27 22:33 sf_master -> /nvme/sf_master
drwxr-x---. 2 mysql mysql       8192 Apr 24  2017 sys

This was really just a very dumb typo error.

Short question:

How do you enable port forwarding on only a single host ip address?

Backstory:

My Centos 7 server has 5 ip addresses. Previously I had apache listening on all of them and various domains assigned to those ip addresses which were resolved with virtual hosts.

I changed the Listen directive in httpd.conf so that now apache only listens to 4 of the ip addresses

Using node.js I created another server instance, but it won't let me listen on the standard port 80 without elevated permissions. I don't want to run it with elevated permissions.

I'd like to port forward port 80 to something like 8080, but only on the one ip address without affecting traffic directed at the other 4 ip addresses. It's important that traffic on other ip addresses are not affected by the rule.

I think the solution will look similar to:

firewall-cmd --zone=public --add-masquerade --permanent
firewall-cmd --zone=public --add-forward-port=port=80:proto=tcp:toport=8080 --permanent

The other questions and answers I've found have to do with source ip addresses instead of host ip addresses.

How do you set the outbound ip address of sendmail?

There are dozens of tutorials but most just tell you what to set without explaining anything and most of them assume you're running a virtual machine and only have a static ip.

Setup The server is a physical machine running centos 7 with port 25 opened on the LAN and in SElinux with the option to send emails from apache enabled. I keep getting timeout errors when I attempt to send outbound emails. I've tested on gmail, yahoo and godaddy emails.

When I tested with nmap --traceroute and cURL I see that my outbound connections are going over the dynamic ip address which has port 25 blocked by the isp.

I need to still connect from the LAN because ssh and most other ports are closed to the outside world. Port 25, 80, 443 and a couple others are open to the outside world. All but port 25 seem to be properly handled by their associated applications.

Output of ip show

$ ip route show
default via 192.168.1.254 dev enp4s0f0 proto static metric 100
default via 108.223.xxx.222 dev enp4s0f1 proto static metric 101
default via 108.223.xxx.222 dev enp4s0f3 proto static metric 102
108.223.xxx.217 dev enp4s0f1 proto kernel scope link src 108.223.xxx.217 metric 100
108.223.xxx.217 dev enp4s0f3 proto kernel scope link src 108.223.xxx.217 metric 101
108.223.xxx.218 dev enp4s0f1 proto kernel scope link src 108.223.xxx.218 metric 100
108.223.xxx.218 dev enp4s0f3 proto kernel scope link src 108.223.xxx.218 metric 101
108.223.xxx.219 dev enp4s0f1 proto kernel scope link src 108.223.xxx.219 metric 100
108.223.xxx.219 dev enp4s0f3 proto kernel scope link src 108.223.xxx.219 metric 101
108.223.xxx.220 dev enp4s0f1 proto kernel scope link src 108.223.xxx.220 metric 100
108.223.xxx.220 dev enp4s0f3 proto kernel scope link src 108.223.xxx.220 metric 101
108.223.xxx.221 dev enp4s0f1 proto kernel scope link src 108.223.xxx.221 metric 100
108.223.xxx.221 dev enp4s0f3 proto kernel scope link src 108.223.xxx.221 metric 101
108.223.xxx.222 dev enp4s0f1 proto static scope link metric 100
108.223.xxx.222 dev enp4s0f3 proto static scope link metric 101
192.168.1.0/24 dev enp4s0f0 proto kernel scope link src 192.168.1.71 metric 100
192.168.1.71 dev enp4s0f3 proto kernel scope link src 192.168.1.71 metric 100

Errors from maillog

Dec 16 18:14:53 localhost sendmail[77855]: vBGNErI1077873: [email protected], ctladdr=cdata (1000/1000), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30014, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (vBGNErDq077874 Message accepted for delivery)
Dec 16 18:24:12 localhost sendmail[77855]: vBGGdu99077513: to=<[email protected]>, ctladdr=<[email protected]> (1000/1000), delay=06:44:16, xdelay=00:10:00, mailer=esmtp, pri=750469, relay=alt4.gmail-smtp-in.l.google.com. [108.177.15.26], dsn=4.0.0, stat=Deferred: Connection timed out with alt4.gmail-smtp-in.l.google.com.
Dec 16 18:24:12 localhost sendmail[77855]: vBGGAGt4077433: to=<[email protected]>, ctladdr=<[email protected]> (1000/1000), delay=07:13:56, xdelay=00:00:00, mailer=esmtp, pri=840469, relay=alt4.gmail-smtp-in.l.google.com., dsn=4.0.0, stat=Deferred: Connection timed out with alt4.gmail-smtp-in.l.google.com.
Dec 16 18:24:12 localhost sendmail[77855]: vBGFweLX077340: to=<[email protected]>, ctladdr=<[email protected]> (1000/1000), delay=07:25:32, xdelay=00:00:00, mailer=esmtp, pri=930469, relay=alt4.gmail-smtp-in.l.google.com., dsn=4.0.0, stat=Deferred: Connection timed out with alt4.gmail-smtp-in.l.google.com.

Output of netstat -tuplen | grep 25 to check that port 25 is open

$ sudo netstat -tuplen | grep 25
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      0          93443      10849/sendmail: acc

EDIT: The answer that this is marked as a duplicate of talks about relaying a gmail email. I don't think that's what I'm asking for. I'm trying to originate an email from my server, on my domain and ip address, and send that to a remote email server. I read that answer and have tried to glean some insight, but I don't think that it's related. If it is, please explain how it's related.

EDIT AFTER ACCEPTING ANSWER: The answer at CentOS 6.2 sendmail: how to control the ip number used when relaying mail to mx.google.com actually is the right answer, but I would have never recognized it as such because I didn't think that I was relaying an email. Thank you to everyone for helping me get it done.