Ben Davis's questions

We host a web-based IDE that runs in a container. I'm using ACI to host the container instances. I need to mount the user's files in the container, and make sure the user cannot access other users' files. An individual user is not expected to go above 1GB usage.

Right now I'm using Azure Files in the Standard/Hot performance tier, w/ ZRS replication. This can be expensive due to per-transaction costs (users saving files often, using git, etc).

I've considered using Azure's premium tier file shares, which are priced on size, not transactions. But premium file shares have a minimum size of 100GB, which would be too expensive if each user had their own share.

I've also considered having one monolithic Premium share for all users, however I would need a way to mount that share while restricting access to other users' files. This can't be done with Samba mounts as they do not support file permissions.

I've also looked into using Azure's new NFS shares, but unfortunately these are not available with ACI.

Are there any other ways I can get around having to pay transaction costs for user storage?

Take the following simplified example:

services:
  app:
    image: alpine
    env_file: config.env
    entrypoint:
      - /bin/sh
      - -c
      - 'echo "Var is: $$MY_VAR"'

I have config.env locally with my local dev settings, and a config.env on the server with production settings.

When I use docker-compose -H [email protected] up --build -d, it uses my local config.env instead of the one on the server. I'd rather not store environment-specific configuration in my repository, however the methods recommended in the docker documentation all involve making separate .yml files or .env files on the same machine on which you're using docker-compose.

So, my question is, how can I store configuration on my server, and use docker-compose on my dev machine to deploy? Or am I better off just uploading the repo to the server and running docker-compose there so I can use the remote env file?

I'm registered for the Azure Files NFSv4 preview, and I cannot seem to find a way to mount NFS file shares on ACI. I have tried the following:

1. Created a virtual network in eastus with two subnets, one specific to aci
2. Changed NFS storage account networking to only allow connections from above vnet:
3. Deployed container with the following json:

    {
      "type": "Microsoft.ContainerInstance/containerGroups",
      "apiVersion": "2019-12-01",
      "location": "eastus",
      "name": "nfs-test",
      "properties": {
        "osType": "Linux",
        "containers": [
          {
            "name": "sshd",
            "properties": {
              "image": "boldidea.azurecr.io/ssh-test",
              "ports": [
                {
                  "port": 22,
                  "protocol": "TCP"
                }
              ],
              "resources": {
                "requests": {
                  "cpu": 1,
                  "memoryInGB": 1.5
                }
              },
              "volumeMounts": [
                {
                  "mountPath": "/mnt/user-storage",
                  "name": "user-storage"
                }
              ]
            }
          }
        ],
        "volumes": [
          {
            "azureFile": {
              "shareName": "ide-user-storage",
              "storageAccountKey": "(redacted)",
              "storageAccountName": "ideworkspacestoragenfs"
            },
            "name": "user-storage"
          }
        ],
        "imageRegistryCredentials": [
          {
            "server": "(redacted)",
            "username": "(redacted)",
            "password": "(redacted)"
          }
        ],
        "ipAddress": {
          "type": "Private",
          "ports": [
            {
              "port": 22,
              "protocol": "TCP"
            }
          ]
        },
        "networkProfile": {
          "id": "/subscriptions/(redacted)/resourceGroups/ideResourceGroup/providers/Microsoft.Network/networkProfiles/aci-network-profile-ide-vnet-eastus-aci"
        }
      }
    }
   

The deployment hangs for roughly 30 minutes before failing with the error "Failed to mount Azure File Volume". What am I doing wrong?

I'm deploying an application called Theia (a cloud-based IDE) on Azure Container Instances. The app is throwing errors about not being able to watch project files for changes. I did some research and this happens when fs.notify.max_user_watches is too low. I checked on the container instance, and apparently it's set to the default of 8192.

I tried running sysctl -w fs.inotify.max_user_watches=524288, but I get an error saying that /sys is mounted read-only.

Is there any other way for me to change this setting in ACI?

I would like to find the number of seconds between when the pod was assigned to when the service becomes ready. When i look at the event log for a pod, there is no event that signals when the service becomes ready. Is this something that I need to write a custom script for, or is there somewhere else I can look?

The pods in my application scale with 1 pod per user (each user gets their own pod). I have the limits for the application container set up like so:

  resources:
    limits:
      cpu: 250m
      memory: 768Mi
    requests:
      cpu: 100m
      memory: 512Mi

The nodes in my nodepool have 8GB of memory each. I started up a bunch of user instances to begin testing, and watched my resource metrics go up as I started each one:

CPU:

Memory:

At 15:40, I saw the event logs show this error (note: the first node is excluded using a taint):

0/2 nodes are available: 1 Insufficient memory, 1 node(s) didn't match node selector.

Why did this happen when the memory/cpu requests were still well below the total capacity (~50% for cpu, ~60% mem)?

Here is some relevant info from kubectl describe node:

Non-terminated Pods:          (12 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  ide                         theia-deployment--ac031811--football-6b6d54ddbb-txsd4              110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    13m
  ide                         theia-deployment--ac031811--footballteam-6fb7b68794-cv4c9          110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    12m
  ide                         theia-deployment--ac031811--how-to-play-football-669ddf7c8cjrzl    110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    14m
  ide                         theia-deployment--ac031811--packkide-7bff98d8b6-5twkf              110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    9m54s
  ide                         theia-deployment--ac032611--static-website-8569dd795d-ljsdr        110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    16m
  ide                         theia-deployment--aj090111--spiderboy-6867b46c7d-ntnsb             110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    2m36s
  ide                         theia-deployment--ar041311--tower-defenders-cf8c5dd58-tl4j9        110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    14m
  ide                         theia-deployment--np091707--my-friends-suck-at-coding-fd48ljs7z    110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    4m14s
  ide                         theia-deployment--np091707--topgaming-76b98dbd94-fgdz6             110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    5m17s
  kube-system                 csi-azurefile-node-nhbpg                                           30m (1%)      400m (21%)  60Mi (1%)        400Mi (7%)     12d
  kube-system                 kube-proxy-knq65                                                   100m (5%)     0 (0%)      0 (0%)           0 (0%)         12d
  lens-metrics                node-exporter-57zp4                                                10m (0%)      200m (10%)  24Mi (0%)        100Mi (1%)     6d20h

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                       Requests      Limits
  --------                       --------      ------
  cpu                            1130m (59%)   3750m (197%)
  memory                         4836Mi (90%)  7988Mi (148%)
  ephemeral-storage              0 (0%)        0 (0%)
  hugepages-1Gi                  0 (0%)        0 (0%)
  hugepages-2Mi                  0 (0%)        0 (0%)
  attachable-volumes-azure-disk  0             0

My application has one pod per user, which is created/destroyed programmatically. I have no need to automatically replicate individual pods, as each pod has a different config depending on user settings. I know deployments are the recommended way to create pods, but I don't see any advantages other replication features. Is there any reason why I shouldn't just create standalone pods if I don't need pod replication?

I did some testing today with my autoscaling setup on Azure Kubernetes. I noticed that when an autoscale was triggered, it took a while for the next node to spin up, so the last pod had to wait a long time to be scheduled. I would like to make it so that when my server reaches a certain threshold, new nodes are added, but pods can still be scheduled on the already-running nodes. Is that possible?

The kubernetes documentation states the following under "Accessing the API from within a Pod":

The easiest way to use the Kubernetes API from a Pod is to use one of the official client libraries. These libraries can automatically discover the API server and authenticate.

I have pods running theia-ide, a cloud-based IDE, where each user can access the console for their container. Can they access the Kubernetes API from that container without authenticating?

While setting up a new cluster I removed an Azure storage account expecting the controller to re-create it, however, now when I try to use an azurefile PVC I get an error that says it can't resolve the file share address.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: 'workspace-storage-j5vzpx4z'
  namespace: ide
  labels:
    component: user-storage
    userid: !f 'j5vzpx4z'
spec:
  storageClassName: azurefile
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

The pod logs show:

  Warning  FailedMount  <invalid>  kubelet, aks-nodepool1-22436891-vmss000000  MountVolume.SetUp failed for volume "pvc-3e7e8869-ecce-4fef-8440-0594f5546962" : mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/9e9828bc-622f-4bb4-bd27-8c2051876700/volumes/kubernetes.io~azure-file/pvc-3e7e8869-ecce-4fef-8440-0594f5546962 --scope -- mount -t cifs -o username=f7e7421469e4443b786ad1c,password=wl+/E9DeEClxfshVccI6FrjjAHDfWviACj0jtHyyyaIoXka6BXm3BpKsNwD/P5/zdZO94tduFN0vuvNQsxI8Lw==,file_mode=0777,dir_mode=0777,vers=3.0 //f7e7421469e4443b786ad1c.file.core.windows.net/kubernetes-dynamic-pvc-3e7e8869-ecce-4fef-8440-0594f5546962 /var/lib/kubelet/pods/9e9828bc-622f-4bb4-bd27-8c2051876700/volumes/kubernetes.io~azure-file/pvc-3e7e8869-ecce-4fef-8440-0594f5546962
Output: Running scope as unit run-r29ba9063eb1c457e993a2e740ca2e7dd.scope.
mount error: could not resolve address for f7e7421469e4443b786ad1c.file.core.windows.net: Unknown error

How can I fix this?

I'm trying to troubleshoot a 404 message on my ingress. When I view logs using kubectl logs -n ingress-nginx ingress-nginx-controller-xxxxxx -f I don't see any output when making a request to the URL. Is there a specific setting that allows me to see access/error logs?

I'm looking for what I would normally see when viewing /var/log/nginx/error.log or /var/log/nginx/access.log.

I'm using the python kubernetes client to control deployment of resources. I'd like to poll the status of multiple resources at once to provide near-realtime feedback on when a given set of resources is available. Right now I have to query each resources individually, which seems excessive on a system where each set of resources scales by N users (so if there are 6 resources there would be N(6) requests for a status check).

Is there way to query the API and get the status of all resources that match a given label selector? If not, is there another approach I should consider?

I have the following in my nginx config:

location / {
    # CORS
    add_header 'Access-Control-Allow-Origin' '*';
    add_header 'Access-Control-Allow-Methods' 'GET, POST' always;

    proxy_pass http://localhost:8080/guacamole/;  # systemd: tomcat9
    proxy_buffering off;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $http_connection;
    proxy_cookie_path /guacamole/ /;
    access_log off;
}

When I send a POST request to the server, the Access-Control-Allow-Origin header is not present in the response. It is, however, when I send a GET request. Any ideas why?

We're using Open LDAP for our directory. Apparently it allows case-insensitive bind auth, and this is causing some issues on our systems. I would like for authentication to fail for logins that are not the correct case. Is there a configuration option for this?

I had ldap authentication working great last night, then today it doesn't seem to work. I can authenticate as a user, but the client can't seem to look up info about the user:

Example logging in as ldap user "ts121207":

$ su - ts121207
Password:
$ id -u
5003
$ whoami
whoami: cannot find name for user ID 5003

I'm able to bind as the user and make queries from the client without issue:

$ ldapsearch -h ldap.example.org -D cn=ts121207,ou=students,ou=users,dc=example,dc=org -b ou=students,ou=users,dc=example,dc=org -w secret '(uid=ts121207)' uid givenName sn
# extended LDIF
#
# LDAPv3
# base <ou=students,ou=users,dc=example,dc=org> with scope subtree
# filter: (uid=ts121207)
# requesting: uid givenName sn 
#

# ts121207, students, users, example.org
dn: cn=ts121207,ou=students,ou=users,dc=example,dc=org
uid: ts121207
givenName: Test
sn: Student

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

I've tried restarting sssd, as well as clearing out /var/lib/sss/db/*.

The interesting thing is when I look at the slapd logs on the ldap server, it doesn't seem to bind as the user when looking up info. This is what is logged as soon as I run whoami:

Nov 19 17:57:52 example.org slapd[14150]: conn=1332 op=0 BIND dn="" method=128                                                                                                                                                                                                             
Nov 19 17:57:52 example.org slapd[14150]: conn=1332 op=0 RESULT tag=97 err=0 text=                                                                                                                                                                                                         
Nov 19 17:57:52 example.org slapd[14150]: conn=1332 op=1 SRCH base="dc=example,dc=org" scope=2 deref=0 filter="(&(objectClass=posixAccount)(uidNumber=5003))"                                                                                                                             
Nov 19 17:57:52 example.org slapd[14150]: conn=1332 op=1 SRCH attr=uid userPassword uidNumber gidNumber cn homeDirectory loginShell gecos description objectClass                                                                                                                          
Nov 19 17:57:52 example.org slapd[14150]: conn=1332 op=1 SEARCH RESULT tag=101 err=0 nentries=0 text=                                                                                                                                                                                      

Shouldn't the client by binding as the user in order to perform that query so it can read the attributes? Or am I misinterpreting that log line?

I'm would like to have a filesystem for network-mounted home directories that can scale easily (per user) at a low cost. EFS seems well-suited to this (especially with Infrequent Access enabled). However, I cannot see a way to set quotas on a per-user basis.

I know typically quotas are set in kernel-space for physical volumes, but I don't think it's possible for NFS volumes (the quotas have to be set on the NFS server, which is the EFS service).

Are there any other things I might explore to restrict how much space users can store in their EFS-mounted homedirs? Or should I be looking to another service?

I'm a contract web developer, and my client doesn't know anything about managing AWS, but wants to control the billing for the account. So, he wants to give me full access to set up EC2, S3, SES,and RDS.

I told him to create an IAM role for me (IAM > Create Role > Another AWS Account), and I told him which permissions to grant, but from there I'm not sure how to switch to that role.

Is there something else he needs to do to give me access?

I teach an after-school program where we're using MIT App Inventor, which requires a connection between the mobile device and laptop. The network at the facility uses isolation, which prevents devices from communicating with each other.

My idea was to set up a basic VPN server that the devices could connect to and then communicate on virtual LAN. I thought I might just spin up a cloud Linux server, but obviously I can't allow connected VPN clients to access anything outside of the server.

Is there a way to create a "virtual LAN" on the server with its own set of 192.168.0.* addresses, but are "jailed" to the sever (and of course still be able to access the internet)?

I'm getting the following in my mail logs on my app server:

550-5.7.1 [67.23.15.78] The IP you're using to send mail is not authorized to 550-5.7.1 send email directly to our servers. Please use the SMTP relay at your 550-5.7.1 service provider instead. Learn more at 550 5.7.1 http://support.google.com/mail/bin/answer.py?answer=10336

I've checked and the IP is not listed on any PBL/SBL/etc that I know of. The domain name in the FROM: email address is a CNAME to our app server's primary domain (djangostage.farstarserver.com). We have a PTR record for the server's IP, so reverse DNS resolves to the same domain (djangostage.farstarserver.com). We also have the following SPF record set up for the domain to allow our vhost domains on that server to send email under their own domains:

"v=spf1 a a:django.farstarserver.com -all"

Any ideas why Google is blocking our server?