larshie's questions -server

larshie

Asked: 2020-11-10 03:08:45 +0800 CST

nginx error - 4718#4718 Connect() failed (111:Connection refused)

nginx error:

 2020/12/01 06:54:05 [error] 4718#4718: *1 connect() failed (111:Connection refused while connecting to upstream, client 192.168.1.1, server: www.some-place.org, request: "Get /bigbluebutton/api HTTP/1.1", upstream: "http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org"

Question Updated Sunday 06 December 2020 at 09:36 CET.

Here is the problem:

Cannot reach bigbluebutton/api (our online classroom platform) on 'http://127.0.0.1:8090'

Check if port 8090 is active: (already port-forwarded in router)

(Classroom public IP address: XXX.X.XX.XX)

sudo telnet XXX.X.XX.XX 8090

telnet: Unable to connect to remote host: Connection refused

Check error logs (2 logs):

sudo bbb-conf --debug

Errors found in /var/log/nginx/error.log: (10 errors, same as in title above)

2020/12/01 06:54:05 [error] 4718#4718: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.1, server: www.some-place.org, request: "GET /bigbluebutton/api HTTP/1.1", upstream: "http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org"
2020/12/01 06:54:05 [error] 4718#4718: *1 open() "/var/www/nginx-default/50x.html" failed (2: No such file or directory), client: 192.168.1.1, server: www.some-place.org, request: "GET /bigbluebutton/api HTTP/1.1", upstream: "http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org"
2020/12/01 06:54:06 [error] 4718#4718: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.1, server: www.some-place.org, request: "GET /bigbluebutton/api HTTP/1.1", upstream: http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org"
2020/12/01 06:54:06 [error] 4718#4718: *3 open() "/var/www/nginx-default/50x.html" failed (2: No such file or directory), client: 192.168.1.1, server: www.some-place.org, request: "GET /bigbluebutton/api HTTP/1.1", upstream: "http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org"
2020/12/01 06:54:07 [error] 4718#4718: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.1, server: www.some-place.org, request: "GET /bigbluebutton/api HTTP/1.1", upstream: "http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org"
2020/12/01 06:54:07 [error] 4718#4718: *5 open() "/var/www/nginx-default/50x.html" failed (2: No such file or directory), client: 192.168.1.1, server: www.some-place.org, request: "GET /bigbluebutton/api HTTP/1.1", upstream: "http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org" 2020/12/01 06:54:08 [error] 4718#4718: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.1, server: www.some-place.org, request: "GET /bigbluebutton/api HTTP/1.1", upstream: "http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org"
2020/12/01 06:54:08 [error] 4718#4718: *7 open() "/var/www/nginx-default/50x.html" failed (2: No such file or directory), client: 192.168.1.1, server: www.some-place.org, request: "GET /bigbluebutton/api HTTP/1.1", upstream: "http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org"
2020/12/01 06:54:09 [error] 4718#4718: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.1, server: www.some-place.org, request: "GET /bigbluebutton/api HTTP/1.1", upstream: "http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org"
2020/12/01 06:54:09 [error] 4718#4718: *9 open() "/var/www/nginx-default/50x.html" failed (2: No such file or directory), client: 192.168.1.1, server: www.some-place.org, request: "GET /bigbluebutton/api HTTP/1.1", upstream: "http://127.0.0.1:8090/bigbluebutton/api", host: "www.some-place.org"

Errors found in /var/log/syslog (2 errors):

Dec 1 06:53:44 bbb-server red5-shutdown.sh[4167]: Exception connecting to 127.0.0.1
Dec 1 06:53:44 bbb-server red5-shutdown.sh[4167]: java.lang.ArrayIndexOutOfBoundsException: 0

Check if all applications are running:

sudo bbb-conf --status

14 checked active (nginx; freeswitch; redis-server; bbb-apps-akka; bbb-transcode-akka; bbb-fesl-akka; red5; tomcat7; mongod; bbb-html5; bbb-webrtc-sfu; kurento-media-server; etherpad; bbb-web).

Next, check if any firewalls are active:

sudo ufw status

Status: inactive

Next, check if bbb-web is listening on port 8090:

sudo netstat -atnp ¦ grep 8090

tcp6 0 0 127.0.0.1:8090 :::* LISTEN 1464/java

Next, perform an nginx dump and pipe the result to the nano editor:

sudo nginx -T ¦ nano

As the output from this command is greater than the 30 000 characters allowed in this body, I have posted the second half today. I will then replace it with the first half again in a couple of days for those who missed it.

# configuration file /etc/bigbluebutton/nginx/presentation-slides.nginx:
#
# BigBlueButton open source conferencing system - http://www.bigbluebutton.org/
#
# Copyright (c) 2012 BigBlueButton Inc. and by respective authors (see below).
#
# This program is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the 
# Free Software Foundation; either version 3.0 of the License, or (at your option) any later version.
#
# BigBlueButton is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License along with BigBlueButton; if not, see <http://www.gnu.org/licenses/>.
#
# Have nginx serve the presentation slides instead of tomcat as large files causes tomcat to OOM. (ralam sept 20, 2018)
location ~^\/bigbluebutton\/presentation\/(?<meeting_id_1>[A-Za-z0-9\-]+)\/(?<meeting_id_2>[A-Za-z0-9\-]+)\/(?<pres_id>[A-Za-z0-9\-]+)\/svg\/(?<page_num>\d+)$ {
default_type image/svg+xml;
    alias /var/bigbluebutton/$meeting_id_2/$meeting_id_2/$pres_id/svgs/slide$page_num.svg;
}

location ~^\/bigbluebutton\/presentation\/(?<meeting_id_1>[A-Za-z0-9\-]+)\/(?<meeting_id_2>[A-Za-z0-9\-]+)\/(?<pres_id>[A-Za-z0-9\-]+)\/slide\/(?<page_num>\d+)$ {
    alias /var/bigbluebutton/$meeting_id_2/$meeting_id_2/$pres_id/slide-$page_num.swf;
}

location ~^\/bigbluebutton\/presentation\/(?<meeting_id_1>[A-Za-z0-9\-]+)\/(?<meeting_id_2>[A-Za-z0-9\-]+)\/(?<pres_id>[A-Za-z0-9\-]+)\/thumbnail\/(?<page_num>\d+)$ {
default_type image/png;
    alias /var/bigbluebutton/$meeting_id_2/$meeting_id_2/$pres_id/thumbnails/thumb-$page_num.png;
}

location ~^\/bigbluebutton\/presentation\/(?<meeting_id_1>[A-Za-z0-9\-]+)\/(?<meeting_id_2>[A-Za-z0-9\-]+)\/(?<pres_id>[A-Za-z0-9\-]+)\/textfiles\/(?<page_num>\d+)$ {
default_type text/plain;
    alias /var/bigbluebutton/$meeting_id_2/$meeting_id_2/$pres_id/textfiles/slide-$page_num.txt;
}

# configuration file /etc/bigbluebutton/nginx/presentation.nginx:
#
# BigBlueButton open source conferencing system - http://www.bigbluebutton.org/
#
# Copyright (c) 2012 BigBlueButton Inc. and by respective authors (see below).
#
# This program is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the 
# Free Software Foundation; either version 3.0 of the License, or (at your option) any later version.
#
# BigBlueButton is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License along with BigBlueButton; if not, see <http://www.gnu.org/licenses/>.
#
location /playback/presentation/playback.html {
return 301 /playback/presentation/0.81/playback.html?$query_string;
# If you have recordings from 0.9.0 beta versions and are sure that you will never want to play recordings made with BigBlueButton 0.81, 
#comment the line above and uncomment the following line: return 301 /playback/presentation/0.9.0/playback.html?$query_string;
}

location /playback/presentation {
    root /var/bigbluebutton;
    index index.html index.htm;
}

location /presentation {
    root /var/bigbluebutton/published;
    index index.html index.htm;
}

# configuration file /etc/bigbluebutton/nginx/screenshare.nginx:
# Handle desktop sharing tunneling.  Forwards requests to Red5 on port 5080.
location /screenshare {
    proxy_pass http://127.0.0.1:5080;
    proxy_redirect default;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    client_max_body_size 10m;
    client_body_buffer_size 128k;
    proxy_connect_timeout 90;
    proxy_send_timeout 90;
    proxy_read_timeout 90;
    proxy_buffer_size 4k;
    proxy_buffers 4 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    include fastcgi_params;
}

# configuration file /etc/nginx/fastcgi_params:
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param REQUEST_SCHEME $scheme;
fastcgi_param HTTPS $https if_not_empty;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param REDIRECT_STATUS 200;

# configuration file /etc/bigbluebutton/nginx/sip.nginx:
location /ws {
    proxy_pass https://192.168.1.51:7443;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_read_timeout 6h;
    proxy_send_timeout 6h;
    client_body_timeout 6h;
    send_timeout 6h;
    auth_request /bigbluebutton/connection/checkAuthorization;
    auth_request_set $auth_status $upstream_status;
}

# configuration file /etc/bigbluebutton/nginx/verto.nginx:
location /verto {
    proxy_pass https://127.0.0.1:8082;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_read_timeout 6h;
    proxy_send_timeout 6h;
    client_body_timeout 6h;
    send_timeout 6h;
}

# configuration file /etc/bigbluebutton/nginx/web.nginx:
# Handle request to bbb-web running within a SpringBoot Tomcat embedded servlet container.  This is for BBB-API and Presentation.
location /bigbluebutton {
proxy_http_version 1.1;
location /bigbluebutton {
    proxy_pass http://127.0.0.1:8090;
    proxy_redirect default;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # Workaround IE refusal to set cookies in iframe
    add_header P3P 'CP="No P3P policy available"';
}

location ~ "^\/bigbluebutton\/presentation\/(?<prestoken>[a-zA-Z0-9_-]+)/upload$" {
    proxy_pass http://127.0.0.1:8090;
    proxy_redirect default;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    # Workaround IE refusal to set cookies in iframe
    add_header P3P 'CP="No P3P policy available"';
    # Allow 30M uploaded presentation document.
    client_max_body_size 30m;
    client_body_buffer_size 128k;
    proxy_connect_timeout 90;
    proxy_send_timeout 90;
    proxy_read_timeout 90;
    proxy_buffer_size 4k;
    proxy_buffers 4 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    include fastcgi_params;
    proxy_request_buffering off;
    # Send a sub-request to allow bbb-web to refuse before loading
    auth_request /bigbluebutton/presentation/checkPresentation;
}

location /bigbluebutton/presentation/download {
    return 404;
}

location ~ "^/bigbluebutton/presentation/download\/[0-9a-f]+-[0-9]+/[0-9a-f]+-[0-9]+$" {
    if ($arg_presFilename !~ "^[0-9a-f]+-[0-9]+\.[0-9a-zA-Z]+$") {
    return 404;
    }

    proxy_pass http://127.0.0.1:8090$uri$is_args$args;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    # Workaround IE refusal to set cookies in iframe
    add_header P3P 'CP="No P3P policy available"';
}

location = /bigbluebutton/presentation/checkPresentation {
    proxy_pass http://127.0.0.1:8090;
    proxy_redirect default;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Presentation-Token $prestoken;
    proxy_set_header X-Original-URI $request_uri;
    proxy_set_header Content-Length "";
    proxy_set_header X-Original-Content-Length $http_content_length;
    # Allow 30M uploaded presentation document.
    client_max_body_size 30m;
    client_body_buffer_size 128k;
    proxy_pass_request_body off;
    proxy_request_buffering off;
}

# To check connection authentication, include:
# auth_request /bigbluebutton/connection/checkAuthorization; auth_request_set $auth_status $upstream_status;
#
# and make sure to add sessionToken param in the request URI
location = /bigbluebutton/connection/checkAuthorization {
    internal;
    proxy_pass http://127.0.0.1:8090;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header X-Original-URI $request_uri;
}

    location ~ "^/bigbluebutton\/textTrack\/(?<textTrackToken>[a-zA-Z0-9]+)\/(?<recordId>[a-zA-Z0-9_-]+)\/(?<textTrack>.+)$" {
        # Workaround IE refusal to set cookies in iframe
        add_header P3P 'CP="No P3P policy available"';
        # Allow 30M uploaded presentation document.
        client_max_body_size 30m;
        client_body_buffer_size 128k;
        proxy_connect_timeout 90;
        proxy_send_timeout 90;
        proxy_read_timeout 90;
        proxy_buffer_size 4k;
        proxy_buffers 4 32k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;
        include fastcgi_params;
        proxy_request_buffering off;
        # Send a sub-request to allow bbb-web to refuse before loading
        auth_request /bigbluebutton/textTrack/validateAuthToken;
        default_type text/plain;
        alias /var/bigbluebutton/captions/$recordId/$textTrack;
    }

    location = /bigbluebutton/textTrack/validateAuthToken {
        internal;
        proxy_pass http://127.0.0.1:8090;
        proxy_redirect default;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-textTrack-token $textTrackToken;
        proxy_set_header X-textTrack-recordId $recordId;
        proxy_set_header X-textTrack-track $textTrack;
        proxy_set_header X-Original-URI $request_uri;
    }
}

# configuration file /etc/bigbluebutton/nginx/webrtc-sfu.nginx:
location /bbb-webrtc-sfu {
    proxy_pass http://127.0.0.1:3008;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_read_timeout 6h;
    proxy_send_timeout 6h;
    client_body_timeout 6h;
    send_timeout 6h;
    auth_request /bigbluebutton/connection/checkAuthorization;
    auth_request_set $auth_status $upstream_status;
}

# configuration file /etc/nginx/sites-enabled/default:
##
# You should look at the following URL's in order to grasp a solid understanding of Nginx configuration files in order to fully unleash the power of 
# Nginx. http://wiki.nginx.org/Pitfalls http://wiki.nginx.org/QuickStart http://wiki.nginx.org/Configuration
#
# Generally, you will want to move this file somewhere, and start with a clean file but keep this around for reference. Or just disable in 
# sites-enabled.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##
# Default server configuration
#
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    # SSL configuration
    #
    # listen 443 ssl default_server; listen [::]:443 ssl default_server;
    #
    # Note: You should disable gzip for SSL traffic. See: https://bugs.debian.org/773332
    #
    # Read up on ssl_ciphers to ensure a secure configuration. See: https://bugs.debian.org/765782
    #
    # Self signed certs generated by the ssl-cert package Don't use them in a production server!
    #
    # include snippets/snakeoil.conf;
    root /var/www/html;
    # Add index.php to the list if you are using PHP
    index index.html index.htm index.nginx-debian.html;
    server_name _;
    location / {
        # First attempt to serve request as file, then as directory, then fall back to displaying a 404.
        try_files $uri $uri/ =404;
    }
    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #       include snippets/fastcgi-php.conf;
    #
    #       # With php7.0-cgi alone:
    #       fastcgi_pass 127.0.0.1:9000;
    #       # With php7.0-fpm:
    #       fastcgi_pass unix:/run/php/php7.0-fpm.sock;
    #}

    # deny access to .htaccess files, if Apache's document root concurs with nginx's one
    #
    #location ~ /\.ht {
    #       deny all;
    #}
}

# Virtual Host configuration for example.com
#
# You can move that to a different file under sites-available/ and symlink that to sites-enabled/ to enable it.
#
#server {
#       listen 80; listen [::]:80;
#
#       server_name example.com;
#
#       root /var/www/example.com; index index.html;
#
#       location / {
#           try_files $uri $uri/ =404;
#       }
#}

We use BigBlueButton (Ver. 2.2.30) as our online classroom platform. It seems that this 'Permission denied' in trying to connect to port 8090 is the reason why users are now getting 'ICE error 1007' when trying to connect to the audio bridge.

Continuing with the search for solutions, ICE error 1007 happens from 2 possibilities: 1) connection blocked by a firewall (there are no active firewalls on our production servers), and 2) connection blocked by NAT.

In case NAT is the source of the blockage, I have included the contents of the current NAT iptables:

Chain PREROUTING (policy ACCEPT)
target      prot opt source         destination
DOCKER      all  --  0.0.0.0/0      0.0.0.0/0      ADDRTYPE match dst-type LOCAL
    
Chain INPUT (policy ACCEPT)
target      prot opt source         destination
    
Chain OUTPUT (policy ACCEPT)
target      prot opt source         destination
DOCKER      all  --  0.0.0.0/0      !127.0.0.0/8   ADDRTYPE match dst-type LOCAL
    
Chain POSTROUTING (policy ACCEPT)
target      prot opt source         destination
MASQUERADE  all  --  172.17.0.0/16  0.0.0.0/0
MASQUERADE  all  --  172.18.0.0/16  0.0.0.0/0
MASQUERADE  tcp  --  172.18.0.2     172.18.0.2     tcp dpt:80
MASQUERADE  tcp  --  172.18.0.3     172.18.0.3     tcp dpt:5432
    
Chain DOCKER (2 references)
target      prot opt source         destination
RETURN      all  --  0.0.0.0/0      0.0.0.0/0
RETURN      all  --  0.0.0.0/0      0.0.0.0/0
DNAT        tcp  --  0.0.0.0/0      127.0.0.1      tcp dpt:5000 to:172.18.0.2:80
DNAT        tcp  --  0.0.0.0/0      127.0.0.1      tcp dpt:5432 to:172.18.0.3:5432

I hope this helps...

nginx error - 4718#4718 Connect() failed (111:Connection refused)

Can you pass user/pass for HTTP Basic Authentication in URL parameters?

Ping a Specific Port

Check if port is open or closed on a Linux server?

How to automate SSH login with password?

How do I tell Git for Windows where to find my private RSA key?

What's the default superuser username/password for postgres after a new install?

What port does SFTP use?

Command line to list users in a Windows Active Directory group?

What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats?

How to determine if a bash variable is empty?