Anadi Misra's questions

Is there a way I can make an EKS Faragte node or a EC2 instance running in a private subnet connect to an ec2 instance running in the public subnet in the same VPC?

When I try a test from the VPC "Reachability Analyzer" for path from the ENI attached to NAT Gateway to the EC2 instance the test succeeds.

When I run curl from inside the EKS POD (which is essentially running as a node in the Private Subnet of the VPC) to the instance running the public subnet of the same VPC the command times out.

I'm running jetty bound to 0.0.0.0 port 28980 on that target EC2 public subnet instance.

they both seem like good add-ons when you're running node groups; I'm assuming given Fargate mandates ALB's would register IP only for services, and other similar fargate requirements, do we still need these two add-ons?

Problem with installing fargate profiles and coreddns addon; I'm using terraform for some parts and kubetctl for others, the fargate profiles are created via terraform:

fargate_profiles = {
  kube-system-profile = {
    name = "kube-system-profile"
    selectors = [
      {
        namespace = "kube-system"
        labels = {
          name = "kube-system"
          k8s-app = "kube-dns"
        }
      }
    ]
    tags = {
      Cost = "DaCost"
      Environment = "dev"
      Name = "coredns-fargate-profile"
    }
  },
  swiftalk-dev-profile = {
    name = "dev-profile"
    selectors = [
      {
        namespace = "dev"
        labels = {
          name = "dev"
        }
      }
    ]
    tags = {
      Cost = "DaCost"
      Environment = "dev"
      Name = "dev-profile"
    }
  },
}

I then install coredns addon again using terraform

resource "aws_eks_addon" "core_dns" {
  addon_name        = "coredns"
  addon_version     = "v1.8.3-eksbuild.1"
  cluster_name      = "${var.eks_cluster_name}-dev"
  resolve_conflicts = "OVERWRITE"
  tags              = { "eks_addon" = "coredns", name = "kube-system" }
  depends_on        = [kubernetes_namespace.dev]
}

I patched the coredns deployment for fargate

kubectl patch deployment coredns \
  --namespace kube-system \
  --type=json \
  -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'

and then restarted

kubectl rollout restart -n kube-system deployment/coredns

however, the coredns pods are still in pending state

kubectl get pods -n kube-system
NAME                      READY   STATUS    RESTARTS   AGE
coredns-5766d4545-g6nxn   0/1     Pending   0          46m
coredns-5766d4545-xng48   0/1     Pending   0          46m
coredns-b744fccf4-hb726   0/1     Pending   0          77m

And the cloud watch logs point out to the pods looking for nodes to deploy instead of fargate

I0723 10:24:38.059960       1 factory.go:319] "Unable to schedule pod; no nodes are registered to the cluster; waiting" pod="kube-system/coredns-b744fccf4-hb726"
I0723 10:24:38.060078       1 factory.go:319] "Unable to schedule pod; no nodes are registered to the cluster; waiting" pod="kube-system/coredns-5766d4545-xng48"

I cannot delete the bucket from an IAM user account which uses a virtual MFA device profile

I have generated session toekns and added it to the profile section of ~/.aws/credentials file. and the profile config is added to the ~/.aws/config file

❯ cat config
[default]
output = json
region = us-east-1
[mfa]
output = json
region = us-east-1

and

[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
[mfa]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXX
aws_session_token = XXXXXXXXXXXXXXXXXXXXX

When I run the command to delete this bucket (it is empty)

❯ aws s3 rm s3://iac-bucket --recursive --region us-east-1 --endpoint-url https://s3.us-east-1.amazonaws.com --profile mfa
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Also, the bucket does not show up in Management Console nor on ls command

❯ aws s3 mb s3://iac-bucket --profile mfa

gives no output, and

❯ aws s3 ls s3://iac-bucket --profile mfa --region us-east-1 --endpoint-url https://s3.us-east-1.amazonaws.com

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

I have the following policies attached to this user via a group

How do I delete this bucket? Why doesn't it show up at all? I know it exists because

❯ aws s3 mb s3://iac-bucket --profile mfa --region us-east-1 --endpoint-url https://s3.us-east-1.amazonaws.com
make_bucket failed: s3://iac-bucket An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

I have an application running as a docker container mapped to port 8080; On this same server I also have nginx also configured to serve a Laravel application which has some URLs that have api at context root https://example.com/api/news for example. The docker application URLs starts with either web/ or api/ so to avoid URL proxy confusions I'm trying to serve all docker container application /comments context path, thereby Laravel requests continue from host/api and docker app URL host/comments/api etc. I have following locations in the configuration (in the order described here)

upstream remark42 {
    server 127.0.0.1:8080 weight=100 max_fails=5 fail_timeout=5;
}

server {
    listen 80;
    server_name www.example.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    access_log /var/www/example.com/storage/logs/access.log;
    error_log /var/www/example.com/storage/logs/error.log;

    ssl_certificate /etc/nginx/ssl/example_com_chain.crt;
    ssl_certificate_key /etc/nginx/ssl/example_com.key;

    root /var/www/example.com/public/;
    index index.php index.html index.htm;

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass    unix:/run/php/php7.4-fpm.sock;
        fastcgi_index   index.php;
        fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param   APP_ENV  production;
        include         fastcgi_params;
    }

    location /comments {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        rewrite ^/comments(.*) /$1 break;
        proxy_pass http://remark42/;
    }

    location ~* \.(?:css|js)$ {
        access_log        on;
        etag              on;
        if_modified_since exact;
        add_header Pragma "public";
        add_header        Cache-Control "max-age=31557600, public, must-revalidate, proxy-revalidate";
    }

And then at the end of file

location / {
  try_files $uri $uri/ /index.php?$args;
}

I get a 404 error on accessing https://www.example.com/comments/web/embed.js or https://www.example.com/comments/api/v1/user?blah

with the JS file giving following error in logs

2020/05/18 18:05:54 [error] 3047#3047: *5035 open() "/var/www/example.com/public/comments/web/last-comments.js" failed (2: No such file or directory), client: 49.207.48.221, server: , request: "GET /comments/web/last-comments.js HTTP/2.0", host: "www.example.com", referrer: "https://www.example.com/blogs/yet-another-svn-change-log-tool"

So the proxy pass doesn't work and it tries to fetch js files from disk.

I have a wordpress installation using Nginx as the web server, now there is a need of adding moodle as the LMS to the same site, as a subdirectory, for example; www.mysite.com is where the wordpress site works then moodle would be www.mysite.com/learn.

This moodle runs in the same machine in a docker container which uses the bitnami moodle image; port 8081 is mapped to port 80 of the docker container i.e.

docker run -d -p 8081:80 -p 4443:443 --name moodle

I added a location block before wordpress configuration to the nginx config

location /learn {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $server_name;
    proxy_pass http://0.0.0.0:8081;
}

the php configuration for wordpress is as follows

# Pass the PHP scripts to FastCGI server (locally with unix: param to avoid network overhead)
location ~ \.php$ {
    # Prevent Zero-day exploit
    try_files $uri =404;

    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    #NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini

    fastcgi_pass    unix:/run/php/php7.1-fpm.sock;
    fastcgi_index   index.php;
    fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param   APP_ENV  production;
    include         fastcgi_params;
}

However this still gives a 404 response when I access www.amysite.com/learn

I checked the docker proxy is running bound to all IP addresses

/usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8081 -container-ip 172.19.0.3 -container-port 80

also a wget to localhost:8081 gives the moodle installation home page so I'm led to belive it is definitely a problem with my location block; or it is the Zero Day exploit config try_files $uri =404; causing the issue, even if so, I still can't remove that line.

Update

This configuration worked

location ~ ^/apply/(.*)$ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $server_name;
    proxy_pass http://0.0.0.0:8081/$1;
}

I can reach Moodle home page but all further URLs break because Moodle is not aware of generating links with the /learn context root; think I'll have to reconfigure Moodle to generate /learn context URLS

I have been searching on how to configure wordpress to connect to mysql db over ssl, however most of the example are about three to five years old e.g. this wordpress support topic

my wordpress version is 4.5.2 and obviusly the code and settings is nothing like what I in examples all over forums, is there a more relevant example on how to fix settings in wp-settings.php and wp-includes/wp-db.php to ensure secure connection to a remote MySql?

I get these error messages when logging in to phpmyadmin instance

Error during session start; please check your PHP and/or webserver log file and configure your PHP installation properly. Also ensure that cookies are enabled in your browser.

mysqli_query(): SSL operation failed with code 1. OpenSSL Error messages: error:0607A082:digital envelope routines:EVP_CIPHER_CTX_set_key_length:invalid key length error:0607A082:digital envelope routines:EVP_CIPHER_CTX_set_key_length:invalid key length

mysqli_query(): MySQL server has gone away

mysqli_query(): Error reading result set's header

I have setup mysql certificated "self signed" on the hosts and from the maching running phpMyAdmin I can connect to remote MySql through the mysql client

expro_app@ubuntu-app:/etc/mysql$ mysql --ssl-ca=ca-cert.pem --ssl-cert=client-cert.pem --ssl-key=client-key.pem -h XX.XXX.X.103 -P 7306 -u admin_secure -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 45
Server version: 5.5.49-0ubuntu0.14.04.1 (Ubuntu)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> status;
--------------
mysql  Ver 14.14 Distrib 5.5.49, for debian-linux-gnu (x86_64) using readline 6.3

Connection id:      45
Current database:   
Current user:       admin_secure@ubuntu-app
SSL:            Cipher in use is DHE-RSA-AES256-SHA
Current pager:      stdout
Using outfile:      ''
Using delimiter:    ;
Server version:     5.5.49-0ubuntu0.14.04.1 (Ubuntu)
Protocol version:   10
Connection:     XX.XXX.X.103 via TCP/IP
Server characterset:    latin1
Db     characterset:    latin1
Client characterset:    utf8
Conn.  characterset:    utf8
TCP port:       7306
Uptime:         2 days 8 hours 7 min 31 sec

Threads: 1  Questions: 126  Slow queries: 0  Opens: 48  Flush tables: 1  Open tables: 41  Queries per second avg: 0.000

Here's the phpMyAdmin settings

/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'cookie';
/* Server parameters */
$cfg['Servers'][$i]['host'] = 'XX.XXX.X.103';
$cfg['Servers'][$i]['port'] = 'XXXX';
$cfg['Servers'][$i]['connect_type'] = 'tcp';
$cfg['Servers'][$i]['compress'] = false;
$cfg['Servers'][$i]['AllowNoPassword'] = false;
$cfg['Servers'][$i]['ssl']=true;
$cfg['Servers'][$i]['ssl_key'] = '/etc/mysql/client-key.pem';
$cfg['Servers'][$i]['ssl_cert'] = '/etc/mysql/client-cert.pem';
$cfg['Servers'][$i]['ssl_ca'] = '/etc/mysql/ca-cert.pem';
$cfg['Servers'][$i]['ssl_ciphers'] = 'DHE-RSA-AES256-SHA';
$cfg['Servers'][$i]['ssl_verify'] = false;

I am setting ssl_verify = false to avoid checking self-signed ceritifcates, given the little hack in libraries/dbi/DBIMysqli.php

if ($cfg['Server']['ssl']) {
            mysqli_ssl_set(
                $link,
                $cfg['Server']['ssl_key'],
                $cfg['Server']['ssl_cert'],
                $cfg['Server']['ssl_ca'],
                $cfg['Server']['ssl_ca_path'],
                $cfg['Server']['ssl_ciphers']
            );
            /*
             * disables SSL certificate validation on mysqlnd for MySQL 5.6 or later
             * @link https://bugs.php.net/bug.php?id=68344
             * @link https://github.com/phpmyadmin/phpmyadmin/pull/11838
             */
            if (! $cfg['Server']['ssl_verify']) {
                mysqli_options(
                    $link,
                    MYSQLI_OPT_SSL_VERIFY_SERVER_CERT,
                    $cfg['Server']['ssl_verify']
                );
                $client_flags |= MYSQLI_CLIENT_SSL_DONT_VERIFY_SERVER_CERT;
            }
        }

The error logs dont really help either, no OpenSSL messages there. On the remote server, nothing related to the phpMyAdmin machine's IP address on the MySql server machine either.

What am I missing here?

I have added public hosted zone foo.mydomian.io in Route 53 and got my DNS provider to add name-servers for the sub-domain, however any records I create for mapping instances be it Type - A or CNAME don't work. If I run a dig +trace ns I get a response which shows the AWS NS records for my subdomain, but I still reach instances using the names XXXX.foo.mydomain.io

I am not sure how to further troubleshoot this, or whom to seek support in this case, is it my DNS provider or Amazon?

What I also notice in the dig output is something like this in the answer section

foo.mydoamin.io.    3600    IN  NS  ns-1143.awsdns-14.org.mydomain.io.
foo.mydoamin.io.    3600    IN  NS  ns-403.awsdns-50.com.mydomain.io.
foo.mydoamin.io.    3600    IN  NS  ns-1764.awsdns-28.co.uk.mydomain.io.
foo.mydoamin.io.    3600    IN  NS  ns-775.awsdns-32.net.mydomain.io.

I am wondering if the answer should atually be

foo.mydoamin.io.    3600    IN  NS  ns-1143.awsdns-14.org.

only?

Running graphite under apache httpd, with slqite database, I have the correct folder permissions

[root@liaan55 httpd]# ls -ltr /var/lib | grep graphite
drwxr-xr-x. 2 apache        apache        4096 Aug 23 19:36 graphite-web

and

[root@liaan55 httpd]# ls -ltr /var/lib/graphite-web/
total 68
-rw-r--r--. 1 apache apache 65536 Aug 23 19:46 graphite.db

syncdb also seems to have gone fine

[root@liaan55 httpd]# sudo -su apache
bash-4.1$ whoami
apache
bash-4.1$ python /usr/lib/python2.6/site-packages/graphite/manage.py syncdb
/usr/lib/python2.6/site-packages/graphite/settings.py:231: UserWarning: SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security
  warn('SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security')
/usr/lib/python2.6/site-packages/django/conf/__init__.py:75: DeprecationWarning: The ADMIN_MEDIA_PREFIX setting has been removed; use STATIC_URL instead.
  "use STATIC_URL instead.", DeprecationWarning)
/usr/lib/python2.6/site-packages/django/core/cache/__init__.py:82: DeprecationWarning: settings.CACHE_* is deprecated; use settings.CACHES instead.
  DeprecationWarning
Creating tables ...
Creating table account_profile
Creating table account_variable
Creating table account_view
Creating table account_window
Creating table account_mygraph
Creating table dashboard_dashboard_owners
Creating table dashboard_dashboard
Creating table events_event
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table django_session
Creating table django_admin_log
Creating table django_content_type
Creating table tagging_tag
Creating table tagging_taggeditem

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'apache'): root
E-mail address: [email protected]
Password: 
Password (again): 
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
bash-4.1$ exit

and the local-settings.py file is as follows

STORAGE_DIR = '/var/lib/graphite-web'

INDEX_FILE = '/var/lib/graphite-web/index' 

DATABASES = {
    'default': {
        'NAME': '/var/lib/graphite-web/graphite.db',
        'ENGINE': 'django.db.backends.sqlite3',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': ''
    }
}

I still get this error

[Sat Aug 23 19:47:17 2014] [error] [client 10.42.33.238]   File "/usr/lib/python2.6/site-packages/django/db/backends/sqlite3/base.py", line 344, in execute
[Sat Aug 23 19:47:17 2014] [error] [client 10.42.33.238]     return Database.Cursor.execute(self, query, params)
[Sat Aug 23 19:47:17 2014] [error] [client 10.42.33.238] DatabaseError: attempt to write a readonly database

not sure what is missing in this configuration

the pip command

pip search statsd --proxy=http://anadi.misra:[email protected]:8080

Now my password contains an @ too so I give the encoded form in password; i.e.

some@pass is supplied as some%40pass, but I get exception name or service not known which makes me feel there is still some syntax issue here, the exception log:

Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pip/basecommand.py", line 139, in main
    status = self.run(options, args)
  File "/usr/lib/python2.6/site-packages/pip/commands/search.py", line 38, in run
    pypi_hits = self.search(query, index_url)
  File "/usr/lib/python2.6/site-packages/pip/commands/search.py", line 52, in search
    hits = pypi.search({'name': query, 'summary': query}, 'or')
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1235, in request
    self.send_content(h, request_body)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1349, in send_content
    connection.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
    self.connect()
  File "/usr/lib64/python2.6/httplib.py", line 720, in connect
    self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
gaierror: [Errno -2] Name or service not known

Storing complete log in /root/.pip/pip.log

The same proxy configuration works for yum in /etc/yum.conf though

this is in continuation of the question asked here about host only adapter on dhcp

I upgraded to vagrant 1.6.3 and the updated Vagrantfile to following setting for multiple adapters

# add additional adapter for inter machine networking
dev.vm.network :private_network, :type => "dhcp", :adapter => "2", :netmask => "255.255.255.0"

it goes through creating adapters but then fails bringing up the mic on vm

Anadis-MacBook-Pro:full-stack-env anadi$ vagrant up
Bringing machine 'full-stack-env' up with 'virtualbox' provider...
==> full-stack-env: Clearing any previously set forwarded ports...
==> full-stack-env: Clearing any previously set network interfaces...
==> full-stack-env: Preparing network interfaces based on configuration...
    full-stack-env: Adapter 1: nat
    full-stack-env: Adapter 2: hostonly
==> full-stack-env: Forwarding ports...
    full-stack-env: 22 => 4223 (adapter 1)
    full-stack-env: 8080 => 8090 (adapter 1)
==> full-stack-env: Running 'pre-boot' VM customizations...
==> full-stack-env: Booting VM...
==> full-stack-env: Waiting for machine to boot. This may take a few minutes...
    full-stack-env: SSH address: 127.0.0.1:4223
    full-stack-env: SSH username: vagrant
    full-stack-env: SSH auth method: private key
    full-stack-env: Warning: Connection timeout. Retrying...
    full-stack-env: Warning: Connection timeout. Retrying...
    full-stack-env: Warning: Remote connection disconnect. Retrying...
==> full-stack-env: Machine booted and ready!
==> full-stack-env: Checking for guest additions in VM...
==> full-stack-env: Setting hostname...
==> full-stack-env: Configuring and enabling network interfaces...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

ARPCHECK=no /sbin/ifup eth 2> /dev/null

Stdout from the command:

Device eth does not seem to be present, delaying initialization.


Stderr from the command:

how ever when I log in to the environment I see two network interfaces as expected

Anadis-MacBook-Pro:full-stack-env anadi$ vagrant ssh
Last login: Wed Jun  4 12:54:47 2014 from 10.0.2.2
[vagrant@full-stack-env ~]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:BD:39:57  
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:febd:3957/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:511 errors:0 dropped:0 overruns:0 frame:0
          TX packets:360 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:54574 (53.2 KiB)  TX bytes:46675 (45.5 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:A3:86:C9  
          inet addr:172.28.128.3  Bcast:172.28.128.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fea3:86c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1360 (1.3 KiB)  TX bytes:894 (894.0 b)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

I am bit confused here on why it is trying to add another mic (eth2)? In the VM I used for creating this vagrant box, I had added two NICs already.

I am using the standard auth.conf provided in puppet install for the puppet master which is running through passenger under Nginx. However for most of the catalog, files and certitifcate request I get a 403 response.

### Authenticated paths - these apply only when the client
### has a valid certificate and is thus authenticated

# allow nodes to retrieve their own catalog
path ~ ^/catalog/([^/]+)$
method find
allow $1

# allow nodes to retrieve their own node definition
path ~ ^/node/([^/]+)$
method find
allow $1

# allow all nodes to access the certificates services
path ~ ^/certificate_revocation_list/ca
method find 
allow *

# allow all nodes to store their reports
path /report
method save
allow *

# unconditionally allow access to all file services
# which means in practice that fileserver.conf will
# still be used
path /file
allow *

### Unauthenticated ACL, for clients for which the current master doesn't
### have a valid certificate; we allow authenticated users, too, because
### there isn't a great harm in letting that request through.

# allow access to the master CA
path /certificate/ca
auth any
method find
allow *

path /certificate/
auth any
method find
allow *

path /certificate_request
auth any
method find, save
allow *

path /facts
auth any
method find, search
allow *

# this one is not stricly necessary, but it has the merit
# of showing the default policy, which is deny everything else
path /
auth any

Puppet master however does not seems to be following this as I get this error on client

[amisr1@blramisr195602 ~]$ sudo puppet agent --no-daemonize --verbose --server bangvmpllda02.XXXXX.com
[sudo] password for amisr1: 
Starting Puppet client version 3.0.1
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /certificate_revocation_list/ca [find] at :110
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [search] at :110
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [find] at :110 Could not retrieve file metadata for puppet://devops.XXXXX.com/plugins: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [find] at :110
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /catalog/blramisr195602.XXXXX.com [find] at :110
Using cached catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /report/blramisr195602.XXXXX.com [save] at :110

and the server logs show

XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/certificate_revocation_list/ca? HTTP/1.1" 403 102 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/file_metadatas/plugins?links=manage&recurse=true&&ignore=---+%0A++-+%22.svn%22%0A++-+CVS%0A++-+%22.git%22&checksum_type=md5 HTTP/1.1" 403 95 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/file_metadata/plugins? HTTP/1.1" 403 93 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:53 +0530] "POST /production/catalog/blramisr195602.XXXXX.com HTTP/1.1" 403 106 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:53 +0530] "PUT /production/report/blramisr195602.XXXXX.com HTTP/1.1" 403 105 "-" "Ruby"

thefile server conf file is as follows (and goin by what they say on puppet site, It is better to regulate access in auth.conf for reaching file server and then allow file server to server all)

[files]
  path /apps/puppet/files
  allow *
[private]
  path /apps/puppet/private/%H
  allow *
[modules]
  allow *

I am using server and client version 3

Nginx has been compiled using the following options

nginx version: nginx/1.3.9
built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) 
TLS SNI support enabled
configure arguments: --prefix=/apps/nginx --conf-path=/apps/nginx/nginx.conf --pid-path=/apps/nginx/run/nginx.pid --error-log-path=/apps/nginx/logs/error.log --http-log-path=/apps/nginx/logs/access.log --with-http_ssl_module --with-http_gzip_static_module --add-module=/usr/lib/ruby/gems/1.8/gems/passenger-3.0.18/ext/nginx --add-module=/apps/Downloads/nginx/nginx-auth-ldap-master/

and the standard nginx puppet master conf

server {
ssl                on;
listen                     8140 ssl;
server_name        _;

passenger_enabled          on;
passenger_set_cgi_param    HTTP_X_CLIENT_DN $ssl_client_s_dn; 
passenger_set_cgi_param    HTTP_X_CLIENT_VERIFY $ssl_client_verify; 
passenger_min_instances    5;

access_log                 logs/puppet_access.log;
error_log                  logs/puppet_error.log;

root                       /apps/nginx/html/rack/public;

ssl_certificate            /var/lib/puppet/ssl/certs/bangvmpllda02.XXXXXX.com.pem;
ssl_certificate_key        /var/lib/puppet/ssl/private_keys/bangvmpllda02.XXXXXX.com.pem;
ssl_crl                    /var/lib/puppet/ssl/ca/ca_crl.pem;
ssl_client_certificate     /var/lib/puppet/ssl/certs/ca.pem;
ssl_ciphers                SSLv2:-LOW:-EXPORT:RC4+RSA;
ssl_prefer_server_ciphers  on;
ssl_verify_client          optional;
ssl_verify_depth           1;
ssl_session_cache          shared:SSL:128m;
ssl_session_timeout        5m;
}

Puppet is picking up the correct settings from the files mentioned because config print command points to /etc/puppet

[amisr1@bangvmpllDA02 puppet]$ sudo puppet config print | grep conf
async_storeconfigs = false
authconfig = /etc/puppet/namespaceauth.conf
autosign = /etc/puppet/autosign.conf
catalog_cache_terminus = store_configs
confdir = /etc/puppet
config = /etc/puppet/puppet.conf
config_file_name = puppet.conf
config_version = ""
configprint = all
configtimeout = 120
dblocation = /var/lib/puppet/state/clientconfigs.sqlite3
deviceconfig = /etc/puppet/device.conf
fileserverconfig = /etc/puppet/fileserver.conf
genconfig = false
hiera_config = /etc/puppet/hiera.yaml
localconfig = /var/lib/puppet/state/localconfig
name = config
rest_authconfig = /etc/puppet/auth.conf
storeconfigs = true
storeconfigs_backend = puppetdb
tagmap = /etc/puppet/tagmail.conf
thin_storeconfigs = false

I checked the firewall rules on this VM; 80, 443, 8140, 3000 are allowed. Do I still have to tweak any specifics to auth.conf for getting this to work?

Update

I added verbose logging to the puppet master and restarted nginx; here's the additional info I see in logs

Mon Dec 10 18:19:15 +0530 2012 Puppet (err): Could not resolve 10.209.47.31: no name for 10.209.47.31
Mon Dec 10 18:19:15 +0530 2012 access[/] (info): defaulting to no access for 10.209.47.31
Mon Dec 10 18:19:15 +0530 2012 Puppet (warning): Denying access: Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :111
Mon Dec 10 18:19:15 +0530 2012 Puppet (err): Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :111
10.209.47.31 - - [10/Dec/2012:18:19:15 +0530] "GET /production/file_metadata/plugins? HTTP/1.1" 403 93 "-" "Ruby"

On the agent machine facter fqdn and hostname both return a fully qualified host name

[amisr1@blramisr195602 ~]$ sudo facter fqdn
blramisr195602.XXXXXXX.com

I then updated the agent configuration to add

dns_alt_names = 10.209.47.31

cleaned all certificates on master and agent and regenerated the certificates and signed them on master using the option --allow-dns-alt-names

[amisr1@bangvmpllDA02 ~]$ sudo puppet cert sign blramisr195602.XXXXXX.com

Error: CSR 'blramisr195602.XXXXXX.com' contains subject alternative names (DNS:10.209.47.31, DNS:blramisr195602.XXXXXX.com), which are 
disallowed. Use `puppet cert --allow-dns-alt-names sign blramisr195602.XXXXXX.com` to sign this request.

[amisr1@bangvmpllDA02 ~]$ sudo puppet cert --allow-dns-alt-names sign blramisr195602.XXXXXX.com

Signed certificate request for blramisr195602.XXXXXX.com
Removing file Puppet::SSL::CertificateRequest blramisr195602.XXXXXX.com at '/var/lib/puppet/ssl/ca/requests/blramisr195602.XXXXXX.com.pem'

however, that doesn't help either; I get same errors as before. Not sure why in the logs it shows comparing access rules by IP and not hostname. Is there any Nginx configuration to change this behavior?

On the server

[root@bangvmpllDA02 logs]# ruby -v
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

[root@bangvmpllDA02 logs]# puppet --version
3.0.1

and

[root@bangvmpllDA02 logs]# service nginx configtest
nginx: the configuration file /apps/nginx/nginx.conf syntax is ok
nginx: configuration file /apps/nginx/nginx.conf test is successful
[root@bangvmpllDA02 logs]# service nginx status
nginx (pid 25923 25921 25920 25917 25908) is running...
[root@bangvmpllDA02 logs]# 

however none of my agents are able to connect to the master, they all fail with errors like so

[amisr1@blramisr195602 ~]$ puppet agent --test --verbose --server bangvmpllda02.XXX.com
Info: Creating a new SSL certificate request for blramisr195602.XXX.com
Info: Certificate Request fingerprint (SHA256): 26:EB:08:1F:82:32:E4:03:7A:64:8E:30:A3:99:93:26:E6:66:B9:B0:49:B6:08:F9:67:CA:1B:0C:00:B9:1D:41
Error: Could not request certificate: Error 405 on SERVER: <html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>

Exiting; failed to retrieve certificate and waitforcert is disabled

when I check logs on puppet master

[root@bangvmpllDA02 logs]# tail puppet_access.log
[05/Dec/2012:17:45:18 +0530] "GET /production/certificate/ca? HTTP/1.1" 404 162 "-" "Ruby"
[05/Dec/2012:18:32:23 +0530] "PUT /production/certificate_request/sl63anadi.XXX.com HTTP/1.1" 405 166 "-" "-"
[05/Dec/2012:18:33:33 +0530] "GET /production/certificate/sl63anadi.XXX.com? HTTP/1.1" 404 162 "-" "-"
[05/Dec/2012:18:33:33 +0530] "GET /production/certificate_request/sl63anadi.XXX.com? HTTP/1.1" 404 162 "-" "-"
[05/Dec/2012:18:33:33 +0530] "PUT /production/certificate_request/sl63anadi.XXX.com HTTP/1.1" 405 166 "-" "-"

and the error logs show that nginx is not really able to process the request well

2012/12/05 18:33:33 [error] 25920#0: *23 open() "/etc/puppet/rack/public/production/certificate/sl63anadi.XXX.com" failed (2: No such file or directory), client: 10.209.47.26, server: , request: "GET /production/certificate/sl63anadi.XXX.com? HTTP/1.1", host: "bangvmpllda02.XXX.com:8140"
2012/12/05 18:33:33 [error] 25920#0: *24 open() "/etc/puppet/rack/public/production/certificate_request/sl63anadi.XXX.com" failed (2: No such file or directory), client: 10.209.47.26, server: , request: "GET /production/certificate_request/sl63anadi.XXX.com? HTTP/1.1", host: "bangvmpllda02.XXX.com:8140"
2012/12/05 18:47:56 [error] 25923#0: *27 open() "/etc/puppet/rack/public/production/certificate/ca" failed (2: No such file or directory), client: 10.209.47.31, server: , request: "GET /production/certificate/ca? HTTP/1.1", host: "bangvmpllda02.XXX.com:8140"
2012/12/05 18:47:56 [error] 25923#0: *28 open() "/etc/puppet/rack/public/production/certificate_request/blramisr195602.XXX.com" failed (2: No such file or directory), client: 10.209.47.31, server: , request: "GET /production/certificate_request/blramisr195602.XXX.com? HTTP/1.1", host: "bangvmpllda02.XXX.com:8140"

Passenger does not show any application groups either

[root@bangvmpllDA02 nginx]# passenger-status 
----------- General information -----------
max      = 15
count    = 0
active   = 0
inactive = 0
Waiting on global queue: 0

----------- Application groups -----------
[root@bangvmpllDA02 nginx]#

here's my nginx configuration

[root@bangvmpllDA02 logs]# cat ../nginx.conf

user  puppet;
worker_processes  4;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    use epoll;
    worker_connections  1024;
}


    http {
        include       mime.types;
        default_type  application/octet-stream;

        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';

        access_log  logs/access.log  main;

        sendfile        on;
        #tcp_nopush     on;
        server_tokens off;
        #keepalive_timeout  0;
        keepalive_timeout  120;

        gzip  on;
        gzip_http_version 1.1;
        gzip_disable "msie6";
        gzip_vary on;
        gzip_min_length 1100;
        gzip_buffers 64 8k;
        gzip_comp_level 3;
        gzip_proxied any;
        gzip_types text/plain text/css application/x-javascript text/xml application/xml;

        server {
            listen       80;
            server_name  bangvmpllda02.XXXX.com;

            charset utf-8;

            #access_log  logs/http.access.log  main;

            location / {
                root   html;
                index  index.html index.htm index.php;
            }

            #error_page  404              /404.html;

            # redirect server error pages to the static page /50x.html
            #
            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
                root   html;
            }

            # proxy the PHP scripts to Apache listening on 127.0.0.1:80
            #
            #location ~ \.php$ {
            #    proxy_pass   http://127.0.0.1;
            #}

            # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
            #
            location ~ \.php$ {
                root           html;
                fastcgi_pass   unix:/var/run/php-fpm/php-fpm.sock;
                fastcgi_index  index.php;
                fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
                fastcgi_param  SCRIPT_NAME  $fastcgi_script_name;
                include        fastcgi_params;
            }

            # deny access to .htaccess files, if Apache's document root
            # concurs with nginx's one
            #
            location ~ /\.ht {
            access_log off;
            log_not_found off; 
                deny  all;
            }

        location ~* \.(jpg|jpeg|gif|png|css|js|ico|xml)$ {
            access_log        off;
            log_not_found     off;
            expires           2d;
        }   
        }

        # Passenger needed for puppet
        passenger_root  /usr/lib/ruby/gems/1.8/gems/passenger-3.0.18;
        passenger_ruby  /usr/bin/ruby;
        passenger_max_pool_size 15;

        server {
        ssl                on;
        listen                     8140 default ssl;
            server_name                bangvmpllda02.XXXX.com; 
        passenger_enabled          on;
        passenger_set_cgi_param    HTTP_X_CLIENT_DN $ssl_client_s_dn; 
        passenger_set_cgi_param    HTTP_X_CLIENT_VERIFY $ssl_client_verify; 
        passenger_min_instances    5;

        access_log                 logs/puppet_access.log;
        error_log                  logs/puppet_error.log;

        root                       /etc/puppet/rack/public;

        ssl_certificate            /var/lib/puppet/ssl/certs/bangvmpllda02.XXX.com.pem;
        ssl_certificate_key        /var/lib/puppet/ssl/private_keys/bangvmpllda02.XXX.com.pem;
        ssl_crl                    /var/lib/puppet/ssl/ca/ca_crl.pem;
        ssl_client_certificate     /var/lib/puppet/ssl/certs/ca.pem;
        ssl_ciphers                SSLv2:-LOW:-EXPORT:RC4+RSA;
        ssl_prefer_server_ciphers  on;
        ssl_verify_client          optional;
        ssl_verify_depth           1;
        ssl_session_cache          shared:SSL:128m;
        ssl_session_timeout        5m;
        }
    }

and the puppet.conf

[main]
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet
    dns_alt_names = devops.XXXX.com,devops
    confdir = /etc/puppet
    vardir = /var/lib/puppet
    storeconfigs = true
    storeconfigs_backend = puppetdb
    thin_storeconfigs = false
    async_storeconfigs = false
    ssl_client_header = SSL_CLIENT_S_D
    ssl_client_verify_header = SSL_CLIENT_VERIFY

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl

any ideas where am I going wrong? I checkthe directory permissions; /usr/share/puppet, /etc/puppet and /var/lib/puppet (and files inside them) are owned by puppet user.

Solved

The simple solution to my complicated problem was that I had placed the config.ru in wrong place

moved it to /etc/puppet/rack , it was in /etc/puppet/rack/public

Well!!! :-/

I get the error

[anadi@bangda ~]# tail -f /var/log/nginx/error.log 
[ pid=19741 thr=23597654217140 file=utils.rb:176 time=2012-09-17 12:52:43.307 ]: *** Exception LoadError in PhusionPassenger::Rack::ApplicationSpawner (no such file to load -- puppet/application/master) (process 19741, thread #<Thread:0x2aec83982368>):
    from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
    from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
    from config.ru:13
    from /usr/local/lib/ruby/gems/1.8/gems/rack-1.4.1/lib/rack/builder.rb:51:in `instance_eval'
    from /usr/local/lib/ruby/gems/1.8/gems/rack-1.4.1/lib/rack/builder.rb:51:in `initialize'
    from config.ru:1:in `new'
    from config.ru:1

when I start nginx server with passenger module configured, puppet master configured to run through rack.

here is the config.ru

[anadi@bangda ~]# cat /etc/puppet/rack/config.ru
# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
#$:.unshift('/usr/share/puppet/lib')

$0 = "master"

# if you want debugging:
# ARGV << "--debug"

ARGV << "--rack"
require 'puppet/application/master'
# we're usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Application[:master].run

and the nginx configuration for puppet master is as follows

[anadi@bangda ~]# cat /etc/nginx/conf.d/puppet-master.conf 
server {
  listen                     8140 ssl;
  server_name                bangda.mycompany.com;

  passenger_enabled          on;
  passenger_set_cgi_param    HTTP_X_CLIENT_DN $ssl_client_s_dn; 
  passenger_set_cgi_param    HTTP_X_CLIENT_VERIFY $ssl_client_verify; 

  access_log                 /var/log/nginx/puppet/master.access.log;
  error_log                  /var/log/nginx/puppet/master.error.log;

  root                       /etc/puppet/rack/public;

  ssl_certificate            /var/lib/puppet/ssl/certs/bangda.mycompany.com.pem;
  ssl_certificate_key        /var/lib/puppet/ssl/private_keys/bangda.mycompany.com.pem;
  ssl_crl                    /var/lib/puppet/ssl/ca/ca_crl.pem;
  ssl_client_certificate     /var/lib/puppet/ssl/certs/ca.pem;
  ssl_ciphers                SSLv2:-LOW:-EXPORT:RC4+RSA;
  ssl_prefer_server_ciphers  on;
  ssl_verify_client          optional;
  ssl_verify_depth           1;
  ssl_session_cache          shared:SSL:128m;
  ssl_session_timeout        5m;
}

however when I run puppet through the ususal puppetmasterd daemon it works perfect with no errors.

I can see somehow the nginx+passenger+rack setup fails to initialize while the same works when running the natvie puppetmaster daemon.

Any configuration that I am missing?

Update:Solved got it to work thanks to @Shane Madden's comment

puppet is located in

/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb

while ruby libs expect it to load from

/usr/local/lib/ruby/site_ruby/1.8/

hence changed the config.ru like so

# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
$:.unshift('/usr/lib/ruby/site_ruby/1.8/')

$0 = "master"

# if you want debugging:
# ARGV << "--debug"

ARGV << "--rack"
require 'puppet/application/master'
# we're usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Application[:master].run

Works now.