Micromegas's questions

In the process of my question here Production server much slower then local test server when sending large request I am trying to understand more profoundly what a webserver - especially nginx - actually does. Basically I want to understand better the architecture that I took for granted a long time. Maybe it's trivial but please hear me out...

I am running a django app with nginx and gunicorn and I am sending post-requests to the server. Since I am trying to understand where my performance problems are I want to understand in detail how my requests are being treated.

So I am sending a POST-request via the library requests. As I understand it, every data sent in the internet is using the HTTP-protocol, so basically the data I am sending is transmitted in text-format. Correct? There is no code interpretation of python objects/data-structures whatsoever? So the size of a request reaching nginx is only the size of the request converted to text-format?

As I further understand this, nginx now receives this request in text-format and then decides what to do with this request... It forwards this request to my web (django) application (or maybe better: my backend server) where my code is being executed and where the data is written into the DB.

So does nginx solely forward the request and that's it? Or does it do anything else?

Then the code or the Database throw an error or give back a success message to nginx which nginx returns as server output (like 404 or 405 or 201 or whatever code...).

SO my question is: How is nginx treating these API requests and how are they interpreted?

I am running a django app and I am trying to send an API POST-request which is pretty big to my production server.

Now the problem is that what takes 4 seconds locally takes 1 minute in production. That might not sound like much but I am planning on sending this request 100 of times and every second counts.

I circled in on the problem and I think it might be an nginx configuration problem but I can't be certain. Here is my troubleshooting process with respective code:

I have a big dictionary dataset_dict = {1:1234, 2:1244 ... 525600: 124345662}, so that means roughly half a million entries.

I send this and measure the time of my post-request:

            dataset_dict = {1:1234, 2:1244 ... 525600: 124345662}

            data = {
                "element": name_element,
                "description": f"description of element",
                "type_data": "datatype",
                "data_json": dataset_dict,
            }

            start = datetime.datetime.now()
            requests.request("post", url="myendpoint", json=data)
            end = datetime.datetime.now()
            runtime = end - start
            print("time-post-request:", runtime)

This takes 4 seconds locally and 50 seconds in production.

So I keep going and I measure the time of only the server code. With that I mean only the code that is executed in my view. I use raw-SQL to achieve maximum performance

    start_time = datetime.datetime.now()
    cursor = connection.cursor()
    data_json = json.dumps(request.data["data_json"])
    
    ##......code shortened for clarity

    cursor.execute(
        "INSERT INTO sql_table(data_json) VALUES ('{}')".format(data_json)
    )
    end_time = datetime.datetime.now()
    runtime = end_time - start_time
    
    print("success, time needed", runtime)

    msg = {"detail": "Created successfully"}
    return Response(msg, status=status.HTTP_201_CREATED)

This code on the server needs 3seconds locally and only 2seconds in production.

So my question is now: Where do the 56 seconds go?

I am inferring that I can exclude postgreSQL settings since the data injection seems to work quite fast.

Nginx settings could be a good start to look, so I was monitoring the server log and I got warn] 28#28: *9388 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000060,

In the official doc I read:

If the request body size is more than the buffer size, then the entire (or partial) request body is written into a temporary file

So I assume I lose time because nginx writes the request to disk. I adjust the request body size to 4GB. I run it again and it takes almost the same amount of time (55 seconds), and I still get the warning.... Should I go even higher?

Is there any other screws I could adjust to get the performance up and the creation time down to what I have locally?? Shouldn't production servers in general be faster than local dev servers?? Maybe it's the internet connection?

So my main question: How could I increase the performance so I could get to a comparable time (seconds) for my data creation?

Specs:

Linux server 
RAM: 16GB 
CPUS: 4

I will post my nginx settings. I am really no expert with nginx so any help how I could increase my performance for this use-case is highly appreciated also if it doesn't solve the problem.

Nginx:

worker_processes auto;

events {
    worker_connections 1024;
    use epoll;
    multi_accept on;
}


http {

    ...
    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay    on;
    send_timeout       90;
    keepalive_timeout  90;
    fastcgi_read_timeout 120;
    proxy_read_timeout 120;
    fastcgi_buffers 8 128k;
    fastcgi_buffer_size 128k;
    client_body_timeout 120;
    client_body_buffer_size 4G;
    client_header_buffer_size 1k;
    large_client_header_buffers 4 8k;
    client_header_timeout 120;
    client_max_body_size 5G;

    reset_timedout_connection on;
    types_hash_max_size 2048;
    server_tokens off;

    gzip  on;
    gzip_static on;
    gzip_min_length 512;

}

Any more info needed, I'll be happy to post it.

I have a perfectly working nginx configuration that serves a django app which looks like this:

server {

    listen 443 ssl;
    server_name domain.de;
    proxy_max_temp_file_size 0;
    proxy_buffering off;
    charset utf-8;
    ssl_stapling off;
    ssl_stapling_verify off;

    ssl_certificate            /etc/letsencrypt/live/domain.de/fullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/domain.de/privkey.pem;


    set $my_host $http_host;
    if ($http_host = "domain.de") {
          set $my_host "domain.de";
    }


    location / {
        proxy_pass http://django:5000;
        proxy_set_header Host $my_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

}

server {
        listen 80 ;
        server_name domain.de;
        return 301 https://domain.de$request_uri;
}
server {
        listen 80 ;
        server_name www.domain.de;
        return 301 https://domain.de$request_uri;
}
server {
        listen 443 ;
        server_name www.domain.de;
        return 301 https://domain.de$request_uri;
        ssl_stapling off;
        ssl_stapling_verify off;

        ssl_certificate           /etc/letsencrypt/live/domain.de/fullchain.pem;
        ssl_certificate_key       /etc/letsencrypt/live/domain.de/privkey.pem;
}

What I want to do is to add a new domain (let's say domain.com). I already got the certificates. No matter how I add another serverblock I get a 400 response telling me that the certificate is not valid.

EDIT: I am running nginx in a docker-container:

nginx:
    container_name: 'nginx'
    image: nginx:latest
    ports:
      - "80:80"
      - "443:443"
    volumes: ##The following files are found on the linux server.
       - /root/nginx-conf/hbe.conf:/etc/nginx/conf.d/default.conf
       - /root/nginx-conf/main.conf:/etc/nginx/nginx.conf
       - /etc/letsencrypt/live/domain.de/fullchain.pem:/etc/letsencrypt/live/domain.de/fullchain.pem
       - /etc/letsencrypt/live/domain.de/privkey.pem:/etc/letsencrypt/live/domain.de/privkey.pem
       - /etc/letsencrypt/live/domain.com/fullchain.pem:/etc/letsencrypt/live/domain.com/fullchain.pem
       - /etc/letsencrypt/live/domain.com/privkey.pem:/etc/letsencrypt/live/domain.com/privkey.pem
    depends_on:
      - django

I tried or example to copy the main server-block adjusting the domain-name and the certificate location:

server {

    listen 443 ssl;
    server_name domain.de;
    proxy_max_temp_file_size 0;
    proxy_buffering off;
    charset utf-8;
    ssl_stapling off;
    ssl_stapling_verify off;

    ssl_certificate            /etc/letsencrypt/live/domain.de/fullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/domain.de/privkey.pem;


    set $my_host $http_host;
    if ($http_host = "domain.de") {
          set $my_host "domain.de";
    }


    location / {
        proxy_pass http://django:5000;
        proxy_set_header Host $my_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

}

server {

    listen 443 ssl;
    server_name domain.com;
    proxy_max_temp_file_size 0;
    proxy_buffering off;
    charset utf-8;
    ssl_stapling off;
    ssl_stapling_verify off;

    ssl_certificate            /etc/letsencrypt/live/domain.com/fullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/domain.com/privkey.pem;


    set $my_host $http_host;
    if ($http_host = "domain.com") {
          set $my_host "domain.com";
    }


    location / {
        proxy_pass http://django:5000;
        proxy_set_header Host $my_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

}

I also tried copying the rest of the server blocks, adjusting the URL, but no matter how I do it, I still get a 400 bad request.

I'm really not an expert with nginx, would be very grateful if someone could lend me a hand. Thanks so much in advance!

I am running a django app with nginx as webserver on an EC2 instance. The website gets a decent amount of API calls per minute depending on how many testpipelines I run. My problem is that the website times out sporadically. I can't reach it in the browswer and all my tests fail of course. Also I can't login to my instance anymore but have to wait until it is back up.... SHouldn't I at least be able ssh into it even though the website times out??

Now I am wondering: Is that rather an nginx configuration error or could it be that my EC2 does not have enough memory? I am deploying to an t2 micro instance on EC2 which has 1 vCPU and also only 1GB of memory. Let's say I get 20-30 API calls, all trying to download files between 10 and 50 MB. Do I need a bigger instance or should it be no problem to handle for the EC2?

Or could it be my nginx configuration? Right now I use this (I am not at all an nginx expert, please tell me if there are stupid configs set):

main config

user  nginx;
worker_processes auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections 1024;
    use epoll;
    multi_accept on;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay    on;
    send_timeout       90;
    keepalive_timeout  90;
    fastcgi_read_timeout 120;
    proxy_read_timeout 120;
    fastcgi_buffers 8 128k;
    fastcgi_buffer_size 128k;
    client_body_timeout 120;
    client_body_buffer_size 128K;
    client_header_buffer_size 1k;
    large_client_header_buffers 4 8k;
    client_header_timeout 120;
    client_max_body_size 5G;

    types_hash_max_size 2048;
    server_tokens off;

    gzip  on;
    gzip_static on;
    gzip_min_length 512;


    include /etc/nginx/conf.d/*.conf;
}

And server config

server {

    listen 443 ssl;
    server_name mywebsite.de;
    proxy_max_temp_file_size 0;
    proxy_buffering off;
    charset utf-8;
    ssl_stapling off;
    ssl_stapling_verify off;

    ssl_certificate            /etc/letsencrypt/live/mywebsite.de/fullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/mywebsite.de/privkey.pem;


    set $my_host $http_host;
    if ($http_host = "mywebsite.de") {
          set $my_host "mywebsite.de";
    }


    location / {
        proxy_pass http://django:5000;
        proxy_set_header Host $my_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 3600; 
  }

}

server {
        listen 80 ;
        server_name mywebsite.de;
        return 301 https://mywebsite.de$request_uri;
}
server {
        listen 80 ;
        server_name www.mywebsite.de;
        return 301 https://mywebsite.de$request_uri;
}
server {
        listen 443 ;
        server_name www.mywebsite.de;
        return 301 https://mywebsite.de$request_uri;
        ssl_stapling off;
        ssl_stapling_verify off;

        ssl_certificate           /etc/letsencrypt/live/mywebsite.de/fullchain.pem;
        ssl_certificate_key       /etc/letsencrypt/live/mywebsite.de/privkey.pem;
}

It could also be something else, but what else could crash my machine? Maybe Redis? I am not sure. I am looking for possible reasons this could happen and for possible solutions to have this as stable as possible.

Very grateful to the community for any help! Thanks in advance!

I am running a django app with nginx in docker. I can download files via an API using the library requests. When I want to download a file with 55 MB somehow the connection breaks down and the download doesn't finish.

I tried tweaking nginx configs like this but still the download does not succeed:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
        worker_connections 1024;
        use epoll;
        multi_accept off;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    send_timeout       7200;
    keepalive_timeout  7200;
    fastcgi_read_timeout 7200;
    proxy_read_timeout 7200;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

I also tried to set the workers higher but it is not recommended in the docs. I set the values of

send_timeout       7200;
    keepalive_timeout  7200;
    fastcgi_read_timeout 7200;
    proxy_read_timeout 7200;

very high so the server doesn't close the connection. But the download still breaks after like 27 MB.

Is there any config that would cause this?

Help is very much appreciated. Thanks so much in advance.

EDIT: I can download the file with a fast internet connection in the browser. But when the connection is slow it breaks. But I need to be able to download files with a slower connection too..

EDIT2: Trailing the logs in real time I got a [CRITICAL] WORKER TIMEOUT (pid:18) . Is that nginx related?

I am running a django app with nginx in a docker container and everything was running perfectly fine up until I rebuild the container. Suddenly I get a 502 (Bad Gateway) error and the logs say

(111: Connection refused) while connecting to upstream, client: 216.245.221.82

I have no idea what changed to be honest and I am a new with nginx. The django/python code should not have changed. I am pulling the latest nginx image so maybe there was a change which doesn't work well with my config. But I tried wit older versions too and get the same error. Maybe my app crashes since there is no port it is listening to? (see screenshot)

I tried adjusting the ALLOWED_HOSTS to ['*'], setting SSL_NO_VERIFY to True and False, still same error. I am out of options.... I don't know how to debug it better. Locally the app works well, so setting debug to True didn't help me.

I just don't know. I am deploying my app on an EC2 instance with nginx in a docker container.

Here are my configs and my compose file:

server {

    listen 443 ssl;
    server_name mywebsite.de;
    client_max_body_size 0;
    charset utf-8;
    ssl_stapling off;
    ssl_stapling_verify off;

    ssl_certificate            /etc/letsencrypt/live/mywebsite.de/fullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/mywebsite.de/privkey.pem;


    set $my_host $http_host;
    if ($http_host = "mywebsite.de") {
          set $my_host "mywebsite.de";
    }


    location / {
        proxy_pass http://django:5000;
        proxy_set_header Host $my_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

}

server {
        listen 80 ;
        server_name mywebsite.de;
        return 301 https://mywebsite.de$request_uri;
}
server {
        listen 80 ;
        server_name www.mywebsite.de;
        return 301 https://mywebsite.de$request_uri;
}
server {
        listen 443 ;
        server_name www.mywebsite.de;
        return 301 https://mywebsite.de$request_uri;
        ssl_stapling off;
        ssl_stapling_verify off;

        ssl_certificate           /etc/letsencrypt/live/mywebsite.de/fullchain.pem;
        ssl_certificate_key       /etc/letsencrypt/live/mywebsite.de/privkey.pem;
}

version: '3'

services:
  django:
    build:
      context: .
      dockerfile: ./compose/production/django/Dockerfile
    image: my_website_production_django
    env_file:
      - ./.envs/.production/.django
      - ./.envs/.production/.postgres
    command: /start

  nginx:
    container_name: 'nginx'
    image: nginx:latest
    ports:
      - "80:80"
      - "443:443"
    volumes: ##The following files are found on AWS EC2 instance.
       - /home/ubuntu/nginx-conf/my.conf:/etc/nginx/conf.d/default.conf
       - /etc/letsencrypt/live/mydomain.de/fullchain.pem:/etc/letsencrypt/live/mydomain.de/fullchain.pem
       - /etc/letsencrypt/live/mydomain.de/privkey.pem:/etc/letsencrypt/live/mydomain.de/privkey.pem
    depends_on:
      - django

  redis:
    image: redis:5.0

Happily providing more info or code if needed. Truly desperate and grateful for help. Thank you.

I am running an app with a dockerized nginx on an EC2 instance and configured my ssl with Let’s Encrypt. Now I would like another primary domain to that certificate but I fail in doing so. When I run

sudo docker run -it --rm -v /docker-volumes/etc/letsencrypt:/etc/letsencrypt -v /docker-volumes/var/lib/letsencrypt:/var/lib/letsencrypt -v /docker/letsencrypt-docker-nginx/src/letsencrypt/letsencrypt-site:/data/letsencrypt -v "/docker-volumes/var/log/letsencrypt:/var/log/letsencrypt" certbot/certbot certonly --webroot --email myemail --agree-tos --no-eff-email --webroot-path=/data/letsencrypt -d mydomain.de -d www.mydomain.de -d anotherdomain.com -d www.anotherdomain.com

it fails. If I run the command without anotherdomain.com -d www.anotherdomain.com it succeeds in renewing my certificates.

Can I extend the certificate for primary domains in letsencrypt or is that only possible for subdomains? Or could there be a problem with the .dev domain because of strict https? Or do I need to change my configs of my nginx somehow?

My config: server {

listen 443 ssl; server_name mydomain.de; charset utf-8; ssl_stapling off; ssl_stapling_verify off;

ssl_certificate /etc/letsencrypt/live/mydomain.de/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/mydomain.de/privkey.pem; set $my_host $http_host; if ($http_host = "mydomain.de") { set $my_host "mydomain.de"; }

location / {
    proxy_pass http://django:5000;
    proxy_set_header Host $my_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

}

server { listen 80 ; server_name mydomain.de; return 301 https://mydomain.de$request_uri; } server { listen 80 ; server_name www.mydomain.de; return 301 https://mydomain.de$request_uri; } server { listen 443 ; server_name www.mydomain.de; return 301 https://mydomain.det$request_uri; ssl_stapling off; ssl_stapling_verify off;

    ssl_certificate  /etc/letsencrypt/live/ mydomain.de/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ mydomain.de/privkey.pem;

}

Really appreciate any kind of help on this. Thanks so much in advance.

My goal is to have a django app running on an EC2 instance using nginx inside of a docker container.

I have the app. I have a domain. I have an EC2 instance. I'm trying to bring it all together.

To configure my app with nginx I need ssl certificates. I want to use let's encrypt. To use let's encrypt I need to validate my domain.

But my domain, or rather the A-Record of my domain, right now is not pointing to anything. Thus, trying to obtain the certificates with certbot sudo certbot certonly --standalone --preferred-challenges http -d mydomain.dev gives me:

http://mydomain.dev/.well-known/acme-challenge/ko3tofGIoo93e-VFPX9CB__CqcNp1ZPazcQR1G5LsGE: Timeout during connect (likely firewall problem) To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.

Ok. So my first question to understand this is: Where do I point my A record? I tried pointing it to my own IP and to my EC2 instance, which is an ubuntu server.

But on this server there is also nothing running at the moment. So it makes sense to me that it doesn't work. Is it even possible what I am trying to do?

When that works another doubt I have is about nginx within the docker container... Once I have those certs and put it into my configuration and spin it up on my EC2 instance, how does my domain interact with it? Is the nginx container the same as if I had it on the server installed? Sorry if those questions are too simple, but I am not a network expert trying to figure out what is going on under the hood.

Any explanation, help or hints I am very grateful for.

If needed:

This is how my nginx conf should look like at the end:

services:
  django:
    build:
      context: .
      dockerfile: ./compose/production/django/Dockerfile
    image: my_webiste_production_django
    env_file:
      - ./.envs/.production/.django
      - ./.envs/.production/.postgres
    command: /start

  nginx:
    image: nginx
    restart: always
    ulimits:
      nproc: 65535
      nofile:
        soft: 49999
        hard: 65535
    volumes:
      - /some/path/www/html:/usr/share/nginx/html
      - /some/path/log:/var/log/nginx
      - /some/path/compose/production/nginx/sites-enabled/sitefitnesshq.conf:/etc/nginx/conf.d/default.conf
      - /etc/letsencrypt/live/mydomain.com/fullchain.pem:/etc/letsencrypt/live/mydomain.com/fullchain.pem
      - /etc/letsencrypt/live/yourdomain.com/privkey.pem:/etc/letsencrypt/live/mydomain.com/privkey.pem
      - /etc/letsencrypt/live/yourdomain.com/chain.pem:/etc/letsencrypt/live/mydomain.com/chain.pem
    ports:
      - "80:80"
      - "443:443"
    depends_on:
      - django
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        window: 60s

I want to configure my dockerized django app with nginx and run it on an EC2 instance and I can't get it to work. I am a beginner with nginx.

My error when I do docker-compose -f production.yml up is:

client sent invalid method while reading client request line, client: 80.153.184.19, server: ec2-XXXXX.eu-central-2.compute.amazonaws.com, request: "�U�F'��Io�q�+!�P��

And I'm like whuat? Isn't there any basic config for this to make it run smoothly?

My nginx config myconf:

server {

    set $my_host $host;
    if ($host ~ "\d+\.\d+\.\d+\.\d+") {
        set $my_host "ec2-XXXXX.eu-central-2.compute.amazonaws.com";
    }

    listen 80;
    server_name ec2-XXXXX.eu-central-2.compute.amazonaws.com;
    charset utf-8;
    error_log /dev/stdout info;
    access_log /dev/stdout;

    location / {
        proxy_pass http://django:5000;
        proxy_set_header Host $my_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

}

My docker-compose:

version: '3'

#volumes:
#  production_caddy: {}

services:
  django:
    build:
      context: .
      dockerfile: ./compose/production/django/Dockerfile
    image: heatbeat_website_production_django
    env_file:
      - ./.envs/.production/.django
      - ./.envs/.production/.postgres
    command: /start

  nginx:
    build:
      context: .
      dockerfile: ./compose/production/nginx/Dockerfile
    ports:
      - "0.0.0.0:8080:80"
    depends_on:
      - django

  redis:
    image: redis:5.0

my dockerfile for nginx:

FROM tutum/nginx
RUN rm /etc/nginx/sites-enabled/default
COPY ./compose/production/nginx/myconf.conf /etc/nginx/sites-enabled/default

I am extremely grateful for any help... I'm so stuck... Thanks in advance!

I am trying to launch a new EC2 instance with docker installed on it on AWS. Later on I need to ssh into this instance. I can create it without a problem doing:

docker-machine create --driver amazonec2 --amazonec2-region=eu-central-1 machine-ec2

This creates a new machine and also a keypair. Problem is it doesn't download the .pem file I need later on to ssh into it.

Thus, I tried to create a new machine with an existing keypair. I created a keypair, then I downloaded it, copied it to my ./ssh/aws folder and ran:

docker-machine create --driver amazonec2 --amazonec2-keypair-name=machine-aws-keypair --amazonec2-ssh-keypath=~/.ssh/aws/ --amazonec2-region=eu-central-1 machine-ec2

that gives me the error:

Error creating machine: Error in driver during machine creation: unable to create key pair: open ~/.ssh/aws/: no such file or directory
equally: docker-machine create --driver amazonec2 --amazonec2-keypair-name=machine-aws-keypair --amazonec2-ssh-keypath=~/.ssh/aws/nameofmykeyfile.pem --amazonec2-region=eu-central-1 machine-ec2

Is there sth I am doing wrong or is this a problem of AWS? I think the former... Any ideas on how to fix this? Help is very much appreciated. Thanks in advance!

EDIT: It would be enough to know how to get my privatekey.pem when creating a new instance via terminal....

I am running a django app which connects to heroku. My database is configured to use AWS RDS.

Now I needed to change db instances. So I created a new instance on AWS and changed the config vars in Heroku.

The weird thing is that it has no effect at all. Heroku still connects to my old AWS database.

Can it be that Heroku doesn't overwrite the DATABASE_URL or sth similar?

I did:

heroku config:set DATABASE_URL="postgres://postgresusername@awsendpoint/databasename?sslrootcert=rds-combined-ca-bundle.pem" -a appname

heroku config:set POSTGRES_DB=nameofdatabae
heroku config:set POSTGRES_DB=nameofdb.somenumbers.eu-central-1.rds.amazonaws.com

When I do heroku config I can see that my config vars are updated. I also restarted the app and deployed anew. But the changes still don't take effect.

Is there anything I am missing here? Or how can I debug this?

Help is greatly appreciated. Thanks in advance

I am trying to move my static assets to AWS. I am following many blog posts but most of all this one: https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html

Now I can either make it work so that it all publicly visible or nothing is.

If I add this policy to my bucket everyone can see everything even though I have my config as explained below:

    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::mybucket/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::accountnbr:user/user"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::mybucket/*"
        }
    ]
}

That should mean everyone can see, only I can edit.

Now I configured my static assets and a private Storage where I want to put my media files. In my static assets I have my css files and so on.

Here is my config:

AWS_ACCESS_KEY_ID = 'mykey'
AWS_SECRET_ACCESS_KEY = 'mysecretkey'
AWS_STORAGE_BUCKET_NAME ='mybucket'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'mysite/static'),
]
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)
STATICFILES_STORAGE = 'config.settings.storage_backends.StaticStorage'
AWS_DEFAULT_ACL = 'public-read'
AWS_PRIVATE_MEDIA_LOCATION = 'media'
PRIVATE_FILE_STORAGE = 'config.settings.storage_backends.PrivateMediaStorage'

then I have this in my storage_backends.py

class StaticStorage(S3Boto3Storage):
    location = 'static'
    default_acl = 'public'

class PrivateMediaStorage(S3Boto3Storage):
    location = settings.AWS_PRIVATE_MEDIA_LOCATION
    default_acl = 'private'
    file_overwrite = False
    custom_domain = False

I didn't configure a public media storage because I only need private media files. And public static files which I think go to my StaticStorage....

Like this unfortunately also my private files can be downloaded via the link.....

I red about pre-signed URLs but shouldn't it work as expected with the config I have? Or am I missing something?

Thanks in advance, very much appreciated!

I have private Gitlab instance running on a private IP. I want to host a website with Gitlab pages that is only visible inside my network and not accessible from the outside. Also I need my private Gitlab instance to not be accessible from the outside.

Gitlab pages can only be used when having the following prerequisites:

Have an exclusive root domain for serving GitLab Pages. Note that you cannot use a subdomain of your GitLab’s instance domain.
Configure a wildcard DNS record.
(Optional) Have a wildcard certificate for that domain if you decide to serve Pages under HTTPS.
(Optional but recommended) Enable Shared runners so that your users don’t have to bring their own.
(Only for custom domains) Have a secondary IP.

I would like to configure it that way but it only serves me if that doesn't open up my Gitlab instance to the public and if my site could be hosted within my network.

Is that possible? Or will it open up my server to the public?

Thanks very much in advance, really appreciate it.

I have a Cookiecutter django project which uses Docker. I write my tests using pytest and I run my tests using docker-compose -f local.yml run django pytest. My local.yml file is in my root dir (where manage.py is)

I want to run these tests automatically with Gitlab CI. For that I set up my own gitlab server, installed and configured the runners. As executor I am using docker.

Then I tried to configure my gitlab-ci.yml file but I am not entirely sure how to do it correctly. This is what I have so far:

image: docker

services:
 - docker:dind

stages:
 - test

tests-website:
 stage: test
 before_script:
   - apk update
   - apk upgrade
   - apk add python python-dev py-pip build-base libffi-dev openssl-dev libgcc
   - pip install docker-compose
 script:
   - docker-compose -f local.yml build
   - docker-compose -f local.yml run django pytest

 tags:
   - docker

My pipeline runs fine until docker-compose -f local.yml build. Then I get

couldn't connect to Docker daemon at http://docker:2375 - is it running? If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

I assume that this is a misleading error since I am struggling a bit to understand how I can execute my local.yml file. Or how I can run my tests in the pipeline. Maybe there are also more configurations I have to do to get the tests running? Or maybe what I have so far is completely wrong? I am not sure...

Can someone help me out with this one? Help is very much appreciated! Thanks so much in advance!

I set up my private Gitlab server. It works fine and when I enter my IP I can access it. I do not have a custom domain and I also do not want one since I don't want to make the server accessible from the outside.

Most of the guides only show how to set up https with Let's Encrypt for which - as far as I can tell - I need to be in control of a domain and configure an A record for it. That's not really an option.

Some rare other guides show me how to create my self signed certificates to use it with GL (using Open ssl). My problem with that is that browsers will still tell me that the connection is insecure and treat self-signed certs as http and not https. So whtat's the sense of doing it in the first place.

So that leaves me with my question: Is there a way to set up https without a custom domain but with a real certificate (best without buying it). Or is that not even possible? If it is possible how could I start to configure my GL intance?

I was thinking maybe using cloudflare could help? I am not sure though.

Thanks very much in advance for help, hints and tipps. Very much appreciated.

I am trying to get a vpn connection set up to connect to my router but very unsuccessfully. I am on Ubuntu and am very lost.

The server is set up to use Ipsec, IKEv2, AES256 and SHA2 256 with preshared key. Bit depth is 14.

So the server is all set up I guess. I hope correctly. I am struggling to set up the client with my ubuntu. I tried to use L2TP over Ipsec but I couldn't connect. I also tried strongswan but I I don't know how to configure it correctly.

Is there a straightforward way to set that up? Which would be the best client? Strongswan, vpnc, openvpn?? I can't even figure that out...

And how could I set this up? On a Mac the following works fine:

Server address: mystaticip
Remote ip: localIdinMyrouter
LocalID: peeridinmyrouter

For any help or hint I am very grateful. Thanks in advance