xendi's questions

If in my automation script, I need to do something like install oh-my-zsh, I will have a .sh script like this:

#!/bin/sh

runuser -l user -c 'sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"'

This works and will install oh-my-zsh as I expect but it then puts the session into a zsh shell. If I add the command exit to the end of the script, it does nothing.

How can I either disallow switching to zsh (Or any other shell) or exit from it in my script?

My proposed setup:

• An EC2 server running Apache or nginx.
• An S3 Bucket in the same region/availability zone mounted with S3FS.
• The Apache or nginx document root set to the S3FS mount.

My question: Is this a tenable and scalable solution for hosting both static AND dynamic content?

I setup an iredmail server so that my website can send registration emails and all that sort of thing. Problem is, it can send mail just fine (And even inbox) but it can't receive. Port 25 is open, the daemon is running, mx records are set but no go. I used this tool at mxtoolbox.com and it shows this:

with raw connection output of:

Connecting to 167.**.***.***

220 mail.*******.com ESMTP Postfix [658 ms]
EHLO EC2AMAZ-CT1LM3F.mxtoolbox.com
250-mail.*******.com
250-PIPELINING
250-SIZE 15728640
250-ETRN
250-STARTTLS
250-AUTH PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-DSN
250 SMTPUTF8 [720 ms]
MAIL FROM:<[email protected]>
250 2.1.0 Ok [719 ms]
RCPT TO:<[email protected]>
450 4.7.1 <EC2AMAZ-CT1LM3F.mxtoolbox.com>: Helo command rejected: Host not found [704 ms]

LookupServerv2 3426ms

Hostname redacted from raw log but you can see it in the screenshot if you have some test to run. The email address I'm trying to receive to is site@.

So again, looks good but can't receive to inbox at the domain. I would consider reinstalling everything or maybe using something other than iredmail but when you have a setup that's inboxing (Often difficult to achieve) at major providers, it's a tough decision to tear it down.

In my Python script I needed a fast/efficient way to set a max filesize on a file I'm constantly writing to. Rather than bring the whole thing into py's RAM, I ran this shell command:

sed -i '1d' file.csv

I monitor the filesize periodically and run the command as needed. Problem is that now if I tail -f file.csv, tail stops tailing the file as soon as sed removes a line from it. Any solution?

I have a rather long bash script for setting up a server cluster. I'd like to start switching over to fish interpreter but I'd like to do it in steps. Can I keep my bash script as is and add the fish command to switch to fish prompt and the rest of the bash script will execute the rest of the commands from the fish interpreter, even though it's a #!/bin/bash script?

I have my digitalocean droplets setup so I can login to ssh via keyfiles. This works fine for the root account but if I create a secondary account via useradd, the only way I'm able to get into the account is via su. If I try to login with ssh using the same keyfile, I get:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic) 

even though I added my public key to /home/user/.ssh/authorized_keys. What to do?

This is a CentOS 7 server.

Note: I also tried adding AllowUsers exampleuser root to /etc/ssh/sshd_config.

I have a mission-critical production CentOS server running PHP7.1 from the remi repo. I need to downgrade PHP to 7.0. How can I do safely do this (What commands) with no downtime? Here's some output to provide some context.

# yum info php
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.linux.duke.edu
 * epel: fedora-epel.mirrors.tds.net
 * extras: mirror.linux.duke.edu
 * remi-php71: remi.mirror.wearetriple.com
 * remi-safe: remi.mirror.wearetriple.com
 * updates: mirror.es.its.nyu.edu
Installed Packages
Name        : php
Arch        : x86_64
Version     : 7.1.12
Release     : 4.el7.remi
Size        : 9.4 M
Repo        : installed
From repo   : remi-php71
Summary     : PHP scripting language for creating dynamic web sites

How I originally upgraded to PHP7.1 was by enabling the remi php7.1 repo for PHP so now printing info for just php package shows the php7.1 package info as seen above. What I need is for yum info php to return the php 7.0 package that can be found in remi's php70 package:

# yum info php70
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.linux.duke.edu
 * epel: fedora-epel.mirrors.tds.net
 * extras: mirror.linux.duke.edu
 * remi-php71: remi.mirror.wearetriple.com
 * remi-safe: remi.mirror.wearetriple.com
 * updates: mirror.es.its.nyu.edu
Available Packages
Name        : php70
Arch        : x86_64
Version     : 1.0
Release     : 5.el7.remi
Size        : 2.9 k
Repo        : remi-safe
Summary     : Package that installs PHP 7.0
License     : GPLv2+
Description : This is the main package for php70 Software Collection,
            : that install PHP 7.0 language.

What commands would accomplish that and replace php7.1 with php7.0?

My nginx keep crashing and reporting "bad gateway" errors in the browser. Nginx and PHP-FPM don't come preconfigured to handle large traffic loads. I had to put a systemctl restart php7.0-fpm cron job in place each hour just to make sure my sites don't stay down for too long when they go. Let's just get down to it.

Some errors I get from /var/log/php7.0-fpm.log:

[20-Sep-2017 12:08:21] NOTICE: [pool web3] child 3495 started
[20-Sep-2017 12:08:21] NOTICE: [pool web3] child 2642 exited with code 0 after 499.814492 seconds from start

[20-Sep-2017 12:32:28] WARNING: [pool web3] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 7 idle, and 57 total children

Nothing jumps out at me in the nginx log. If I leave it running for too long without restarting it (PHP-FPM), I will get gateway errors. I've tried following tutorials 3 times now tweaking settings but it's still no good. Right now I've got all kinds of settings probably way off but it never works either way I do it.

/etc/nginx/nginx.conf:

user www-data;
worker_processes auto;
pid /run/nginx.pid;

worker_rlimit_nofile 100000;

events {
        worker_connections 4096;
        use epoll;
        multi_accept on;
}


http {
        sendfile on;
        reset_timedout_connection on;
        client_body_timeout 10;
        send_timeout 2;
        keepalive_timeout 30;
        keepalive_requests 100000;
        tcp_nopush on;
        tcp_nodelay on;
        types_hash_max_size 2048;
        fastcgi_read_timeout 300000;
        client_max_body_size 9000m;
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        gzip on;
        gzip_disable "msie6";
        gzip_vary on;
        gzip_proxied any;
        gzip_comp_level 6;
        gzip_buffers 16 8k;
        gzip_http_version 1.1;
        gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
        open_file_cache max=200000 inactive=20s;
        open_file_cache_valid 30s;
        open_file_cache_min_uses 2;
        open_file_cache_errors on;

        access_log off;
}

/etc/php/7.0/fpm/php-fpm.conf:

    [www]

    pm = dynamic
    pm.max_spare_servers = 200
    pm.min_spare_servers = 100
    pm.start_servers = 100
    pm.max_children = 300

    [global]
    pid = /run/php/php7.0-fpm.pid
    error_log = /var/log/php7.0-fpm.log
    include=/etc/php/7.0/fpm/pool.d/*.conf

/etc/php/7.0/fpm/pool.d/www.conf:

[www]

user = www-data
group = www-data
listen = /run/php/php7.0-fpm.sock
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 300
pm.start_servers = 100
pm.min_spare_servers = 100
pm.max_spare_servers = 200
pm.max_requests = 500

One of my sites (/etc/php/7.0/fpm/pool.d/web3.conf):

[web3]

listen = /var/lib/php7.0-fpm/web3.sock
listen.owner = web3
listen.group = www-data
listen.mode = 0660

user = web3
group = client1

pm = dynamic
pm.max_children = 141
pm.start_servers = 20
pm.min_spare_servers = 20
pm.max_spare_servers = 35
pm.max_requests = 500

chdir = /

env[HOSTNAME] = $HOSTNAME
env[TMP] = /var/www/clients/client1/web3/tmp
env[TMPDIR] = /var/www/clients/client1/web3/tmp
env[TEMP] = /var/www/clients/client1/web3/tmp
env[PATH] = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Resource/proc usage from htop:

I'm currently using nginx rewrites to pass server variables. I'm able to pass a variable like this: http://example.com/test such that it passes that as: http://example.com/?p=test. I do that with this:

location / {
  rewrite ^/([^/\.]+)$ /?p=$1 break;
}

Now I want to be able to pass variables using subdirectories so that mappings occur like this:

http://example.com/profile/user1 => http://example.com/?p=user1

and I want to have multiple rules like this so that on the same site I also have a rule like:

http://example.com/play/vid1 => http://example.com/?v=vid1

I tried this but it didn't work:

location /profile/ {
  rewrite ^/([^/\.]+)$ /?p=$1 break;
}
location /play/ {
  rewrite ^/([^/\.]+)$ /?v=$1 break;
}

I also tried without the trailing slashes like:

location /profile {
  rewrite ^/([^/\.]+)$ /?p=$1 break;
}
location /play {
  rewrite ^/([^/\.]+)$ /?v=$1 break;
}

How can I do this?

For all the sites on my new server, it seems as though once nginx has served some amount of traffic, it starts serving "ERROR 502 - Bad Gateway!".

Let's take a look at some logs and config...

• OS: Ubuntu 16.04 (Server)

• Hardware: 6 cores, 12GB RAM

• Panel: ISPConfig

• nginx version: 1.10.3

From /var/www/example/log/error.log:

11: Resource temporarily unavailable) while connecting to upstream, client: 216.170.123.179, server: example.com, request: "GET /KHFxygk9 HTTP/1.1", upstream: "fastcgi://unix:/var/lib/php7.0-fpm/web3.sock:", host: "example.com", referrer: "http://example.com/"
2017/08/10 06:01:06 [error] 15408#15408: *31372 connect() to unix:/var/lib/php7.0-fpm/web3.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 192.3.114.12, server: example.com, request: "GET /jiPNQjwp HTTP/1.1", upstream: "fastcgi://unix:/var/lib/php7.0-fpm/web3.sock:", host: "example.com", referrer: "http://example.com/"

Here are my config files:

• /etc/php/7.0/fpm/pool.d/www.conf

[www]    
user = www-data
group = www-data

listen = /run/php/php7.0-fpm.sock

listen.owner = www-data
listen.group = www-data

pm = dynamic
pm.max_children = 12    
pm.start_servers = 2    
pm.min_spare_servers = 1    
pm.max_spare_servers = 3    
pm.max_requests = 500

• /etc/nginx/nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
        worker_connections 768;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";


        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

systemctl status php7.0-fpm

● php7.0-fpm.service - The PHP 7.0 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.0-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-08-10 04:44:16 UTC; 1h 30min ago
  Process: 20310 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
  Process: 20302 ExecReload=/usr/lib/php/php7.0-fpm-checkconf (code=exited, status=0/SUCCESS)
  Process: 13060 ExecStartPre=/usr/lib/php/php7.0-fpm-checkconf (code=exited, status=0/SUCCESS)
 Main PID: 13070 (php-fpm7.0)
   Status: "Processes active: 20, idle: 12, Requests: 6874, slow: 0, Traffic: 0.3req/sec"
   CGroup: /system.slice/php7.0-fpm.service
       ├─13070 php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)                      
       ├─13072 php-fpm: pool apps-{fpm_domain}                                              
       ├─13073 php-fpm: pool apps-{fpm_domain}                                              
       ├─13074 php-fpm: pool ispconfig                                                      
       ├─13075 php-fpm: pool ispconfig                                                      
       ├─13076 php-fpm: pool web1                                                           
       ├─13077 php-fpm: pool web1                                                           
       ├─13079 php-fpm: pool web2                                                           
       ├─13080 php-fpm: pool web3                                                           
       ├─13081 php-fpm: pool web3                                                           
       ├─13082 php-fpm: pool web4                                                           
       ├─13083 php-fpm: pool web4                                                           
       ├─13084 php-fpm: pool web5                                                           
       ├─13085 php-fpm: pool web5                                                           
       ├─13087 php-fpm: pool www                                                            
       ├─13088 php-fpm: pool www                                                            
       ├─16186 php-fpm: pool web3                                                           
       ├─16603 php-fpm: pool web3                                                           
       ├─16604 php-fpm: pool web3                                                           
       ├─16605 php-fpm: pool web3                                                           
       ├─16609 php-fpm: pool web3                                                           
       ├─16614 php-fpm: pool web3                                                           
       ├─16629 php-fpm: pool web3                                                           
       ├─16630 php-fpm: pool web3                                                           
       ├─18019 php-fpm: pool web2                                                           
       ├─18026 php-fpm: pool web2                                                           
       ├─18027 php-fpm: pool web2                                                           
       ├─18028 php-fpm: pool web2                                                           
       ├─18369 php-fpm: pool web2                                                           
       ├─18370 php-fpm: pool web2                                                           
       ├─18410 php-fpm: pool web2                                                           
       ├─18411 php-fpm: pool web2
       └─18413 php-fpm: pool web2                                                           
Aug 10 04:44:16 web1.example.com systemd[1]: Stopped The PHP 7.0 FastCGI Process Manager.
Aug 10 04:44:16 web1.example.com systemd[1]: Starting The PHP 7.0 FastCGI Process Manager...
Aug 10 04:44:16 web1.example.com systemd[1]: Started The PHP 7.0 FastCGI Process Manager.

I'd like to implement IP-based authentication on my proxy servers. Consider a user of my service called user1. Here's what my ACL currently looks like for that:

acl user11 proxy_auth [-i] user11
acl user12 proxy_auth [-i] user12
acl user13 proxy_auth [-i] user13

with a corresponding outgoing IP address assignment:

tcp_outgoing_address 175.25.11.25 user11
tcp_outgoing_address 175.25.11.26 user12
tcp_outgoing_address 175.25.11.27 user13

This way, user1 can use multiple outbound IP addresses by appending a number to their username when authenticating.

I want user1 to have access to many outgoing IP addresses but use IP-based authentication. As I understand it, I would do IP-based authentication like this:

acl user11 10.0.0.1
acl user12 10.0.0.1
acl user13 10.0.0.1

But that won't work because then the user has no way of using/specifying a different outgoing IP address. This must mean that I have to use a different port for each outbound IP address.

Suppose my server's main IP was 175.25.11.1 (What the user will connect to) and I wanted each port they connect to to have a different outbound IP address. Let's also assume the user's IP is 10.0.0.1 and we want to use IP-based authentication. The way I understand it, this is how I would do that:

http_port 175.25.11.1:3128 name=3128
http_port 175.25.11.1:3129 name=3129
http_port 175.25.11.1:3130 name=3130

acl user13128 myportname 3128 src 10.0.0.1 http_access allow user13128 tcp_outgoing_address 175.25.11.25 user13128
acl user13129 myportname 3129 src 10.0.0.1 http_access allow user13129 tcp_outgoing_address 175.25.11.26 user13129
acl user13130 myportname 3130 src 10.0.0.1 http_access allow user13130 tcp_outgoing_address 175.25.11.27 user13130

Please correct me if I'm wrong. My question is, can I setup the IP-based authentication so that I can change it in 1 place in my squid.conf. That way, if the user changes their IP, I don't have to rewrite a huge list of ACL. Consider that this server has 10,000 IP addresses bound to it. Can I use some sort of wildcard that says:

acl user1* src 10.0.0.1 http_access allow

Please correct any error I may have here. This is my first attempt at IP-based authentication.

Squid'd default is to operate as an HTTP proxy; however, this FAQ https://docstore.mik.ua/squid/FAQ-4.html#ss4.10 says "No changes are necessary to use Squid with socks5. Simply add the usual -Dbind=SOCKSbind etc., to the compile line and -lsocks to the link line."

That sounds great and all except that I don't code c/c++ beyond knowing the basic syntax so typically when I compile something, it's ./configure and ./make. Occasionally I have to modify a makefile's compile line but I'm not sure about linking here (Though I get the concept of linking).

It's also a bit confusing to interpret exactly what to do by "add the usual X etc.," so if someone could point out the exact steps here it would be a great help since they don't seem to be detailed anywhere. It doesn't sound hard.

I already have squid compiled and running properly for HTTP connections but SOCKS support is a must.

I need to make a program that logs bandwidth usage per-user of a squid proxy. In my access.log I see output such as...

1481377143.042      0 104.123.25.183 TCP_DENIED/407 3829 CONNECT scontent.xx.fbcdn.net:443 - HIER_NONE/- text/html
1481377143.403      0 104.123.25.183 TCP_DENIED/407 3793 CONNECT cx.atdmt.com:443 - HIER_NONE/- text/html
1481377277.865 134623 104.123.25.183 TCP_MISS/200 4118 CONNECT scontent.xx.fbcdn.net:443 star7 HIER_DIRECT/2a03:2880:f022:b:face:b00c:0:3 -
1481377277.868 134626 104.123.25.183 TCP_MISS/200 4116 CONNECT scontent.xx.fbcdn.net:443 star7 HIER_DIRECT/2a03:2880:f022:b:face:b00c:0:3 -

I believe the response size in bytes is supposed to be in there but which entry is it? I realize header size isn't logged so the accuracy won't be perfect but it should be accurate enough. I don't know any better way to keep track of squid user bandwidth usage so this will have to do but I need to know which is the size entry.