dimisjim's questions

Just built a new AMI recently, based on the latest 18.04. Did also an apt upgrade when provisioning it, in order to have the latest packages as well.

Problem is that this new AMI has no swapfile in it, checked via swapon --show. It uses 5.3.0-1030-aws kernel. Same image built 6 months ago with the same way (but no apt upgrade done on top of it) which uses 4.15.0-1052-aws, does have a swapfile as expected.

The swapfile can be created at provisioning time too, but I am just curious on why this has changed since December.

What am I missing?

I know my question may be too generic but please bear with me for a moment:

I use ubuntu 18.04 LTS, and had docker-ce installed (using the official install procedure described in https://docs.docker.com/engine/install/ubuntu/) on a modern thinkpad. Nothing fancy about my setup.

A day or two ago, my system hanged when trying to poweroff my machine. Then, when I tried to use docker, the daemon was unresponsive, just hanged. systemd unit was loaded, but had not started. I assumed at that point that the poweroff hang was due to this docker daemon issue.

So I tried to purge and reinstall docker from the system. sudo apt-get purge docker-ce docker-ce-cli containerd.io Even this command hanged.

So I tried to manually unlock dpkg by:

sudo systemctl disable docker
pgrep -a apt
sudo kill -9 <apt process>
sudo rm /var/lib/apt/lists/lock
sudo rm /var/cache/apt/archives/lock
sudo rm /var/lib/dpkg/lock

also deleted all docker and containerd related files + disabled and removed systemd unit:

sudo systemctl disable docker
sudo rm -rf /etc/docker
sudo rm -rf /run/containerd/containerd.sock
sudo rm -rf /usr/bin/dockerd
sudo rm -rf /lib/systemd/system/docker.service
sudo rm -rf /var/lib/docker-engine
sudo rm -rf /var/lib/docker
sudo rm -rf /var/run/docker/
sudo rm -rf /var/run/docker.pid
sudo rm -rf /var/run/docker.sock
sudo rm /etc/apt/sources.list.d/docker.list
sudo rm -rf /var/lib/containerd
sudo rm -rf /opt/containerd
sudo systemctl daemon-reload

and restarted which then allowed me to finally run the purge command successfully: sudo apt-get purge docker-ce docker-ce-cli containerd.io

Then when I try to reinstall docker (using again the official docs method, stable channel), it hangs at enabling the systemd service:

...
Preparing to unpack .../5-docker-ce_5%3a19.03.8~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce (5:19.03.8~3-0~ubuntu-bionic) ...
Setting up aufs-tools (1:4.9+20170918-1ubuntu1) ...
Setting up containerd.io (1.2.13-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up cgroupfs-mount (1.4) ...
Setting up docker-ce-cli (5:19.03.8~3-0~ubuntu-bionic) ...
Setting up pigz (2.4-1) ...
Setting up docker-ce (5:19.03.8~3-0~ubuntu-bionic) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.

which I need to suspend (ctrl+z) and then clean up everything again as described above.

Here's also a screenshot on how the systemd service looks while installation is stuck:

I have also tried to do apt clean and/or apt autoclean then reboot and try again, but nothing has worked so far, it still hangs when installing.

We are currently using EMR for easy job submission for our spark jobs. Recently I came across the "FSx lustre + S3" solution that is being advertised as ideal for HPC situations. EMRFS however is also said to be optimized for this particular scenario, making S3 look like a local hadoop filesystem.

So I am wondering, why would anyone choose either one of these 2 in terms of cost and performance?

This question could be a follow up to AWS S3 costs for when AWS EMR uses it but unfortunately I don't have the reputation to post a comment there.

Thanks in advance for the help.

I remember trying s3fs a year back, trying to use some S3 bucket as a FUSE filesystem. I remember it being quite laggy, especially when coupled with git operations on it (an oblivious system architect had a weird requirement like this).

Fast forward a few months, I discovered Lustre and AWS FSx for Lustre coupled with S3, which allegedly provides a very similar use case for it:

(aws fsx faq)

Amazon FSx for Lustre works natively with Amazon S3, making it easy for you process cloud data sets with high performance file systems. When linked to an S3 bucket, an FSx for Lustre file system transparently presents S3 objects as files and allows you to write results back to S3.

So how does Lustre compare performance-wise with FUSE filesystems for S3 such as s3fs (and other like goofys)?

Thanks :)

My architecture in AWS is as follows:

There are 2 identical zabbix agents (based on zabbix/zabbix-agent:centos-4.0.11) each one running on a different EC2 instance. Zabbix server runs on a third instance (also dockerized with dockbix using 4.0 version as well), all three of them inside the same VPC.

The idea is to have a Network Load Balancer that listens to the port that both agents run (10050) and have those 2 aforementioned instances being registered on the target group. Then, the DNS of this NLB would be provided to the Zabbix host configuration as the interface. The goal is to have multiple zabbix hosts targeting the same NLB and their requests being routed according to traffic load to the different agent. There is a zabbix agent item in each host that invokes a UserParameter (a python script) that is defined in each one of the two zabbix agent conf file.

My problem is as follows: zabbix_get (and the equivalent call made automatically according to the interval set in the host conf) timeouts occasionally. One time I get a successful response

{"response":"success","info":"processed: 4; failed: 0; total: 4; seconds spent: 0.000106"}

(python script used is pretty fast, it just takes 1 second) and other times I get a response such as:

zabbix_get [4515]: Timeout while executing operation.

This happens one after another. So one succeeded and the next timeouts, then the next succeeds and so on.

I have tried to test the connection with telnet, and it works all the time. I have even tried to use a simple tcp echo container, which also worked fine all the time.

Any ideas on what might be wrong would be greatly appreciated :)

EDIT: Just wanted to note that this behavior occurs not just with my custom UserParameter defined script, but also with built-in agent calls such as agent.version or agent.ping or net.tcp.port[<serverIp>, 10051] etc

EDIT2: With tcpdump src <serverIp> run inside the agent instances it seems there is similar traffic happening with a successful and a timed out response

Is there a way to see which IAM did a modification on a aws s3 bucket object (with versioning enabled)?

I know that there is the aws s3api list-object-versions, but it only shows the owner's DisplayName and ID (owner of aws organisation), not the underlying IAM that performed the modification.

"Owner": {
    "DisplayName": "my-username",
    "ID": "7009a8971cd660687538875e7c86c5b672fe116bd438f46db45460ddcd036c32"

Thanks in advance for the help.