Taha Jahangir's questions

I have a ceph-rgw installation with a large bucket (~60M objects) and 16 osds, the bucket index is sharded into 997 shards. In this environment single directory listing takes more than 30 seconds:

$ time rclone lsd t:bucket/non/existent/path/ --contimeout=1h --timeout=1h
real    0m34.816s

This is very annoying, and may clients (e.g. rclone itself) do a list-dir op before a PUT to check/verify something. (Preventing clients from a sending list_objects/list_bucket is not a good option)

The log of rgw daemon is normal. Part of log is:

08:57:45.267+0000 7f0492db2700  1 ====== starting new request req=0x7f05039a9620 =====
08:57:45.267+0000 7f0492db2700 20 req 412648 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/bucket
08:57:45.267+0000 7f0492db2700 10 req 412648 0.000000000s canonical request = GET
08:57:45.267+0000 7f0492db2700  2 req 412648 0.000000000s s3:list_bucket verifying op params
08:57:45.267+0000 7f0492db2700  2 req 412648 0.000000000s s3:list_bucket pre-executing
08:57:45.267+0000 7f0492db2700  2 req 412648 0.000000000s s3:list_bucket executing
08:57:45.267+0000 7f0492db2700 20 req 412648 0.000000000s s3:list_bucket RGWRados::Bucket::List::list_objects_ordered starting attempt 1
08:57:45.267+0000 7f0492db2700 10 req 412648 0.000000000s s3:list_bucket RGWRados::cls_bucket_list_ordered: :bucket[e6fb9c7c-74a2-4819-a0ed-e740d4eb590c.4751590.1]) start_after="[]", prefix="/non/existent/path/" num_entries=1001, list_versions=0, expansion_factor=1
08:57:45.271+0000 7f0492db2700 10 req 412648 0.004000000s s3:list_bucket RGWRados::cls_bucket_list_ordered request from each of 997 shard(s) for 8 entries to get 1001 total entries
08:58:07.495+0000 7f04efe6c700 10 librados: Objecter returned from call r=0
08:58:08.779+0000 7f04cd627700  4 rgw rados thread: no peers, exiting
08:58:18.803+0000 7f0492db2700  2 req 412648 33.535980225s s3:list_bucket completing
08:58:18.803+0000 7f047bd84700  2 req 412648 33.535980225s s3:list_bucket op status=0
08:58:18.803+0000 7f047bd84700  2 req 412648 33.535980225s s3:list_bucket http status=200
08:58:18.803+0000 7f047bd84700  1 ====== req done req=0x7f05039a9620 op status=0 http_status=200 latency=33.535980225s ======
08:58:18.803+0000 7f047bd84700  1 beast: 0x7f05039a9620: 192.168.1.1 - rgwuser [10/Nov/2021:08:57:45.267 +0000] "GET /bucket?delimiter=%!F(MISSING)&max-keys=1000&prefix=non%!F(MISSING)existent%!F(MISSING)path%!F(MISSING) HTTP/1.1" 200 413 - "rclone/v1.57.0" - latency=33.535980225s

The environment detail is: Ceph Version: 16.2.5 Installed with rook, Each OSD is about ~4T with a 256G SSD Metadata device.

There is two main options for load-balancing in OPNsense (and pfSense): relayd and haproxy. pfSense has removed relayd in favour of haproxy [1,2], but OPNsense still supports it. In what usecases we should prefer using haproxy over relayd (or vice versa)?

• [1] https://redmine.pfsense.org/issues/9386
• [2] https://forum.netgate.com/topic/140790/heads-up-relayd-deprecated-on-pfsense-2-5-0/14

P.S. Personally (despite the notes in [1]) I think relayd is a more fittable solution for a firewall, because it uses NAT to load balance, and thus consuming less resources (both CPU and RAM), also states are synced in a HA setup. haproxy gives more options to load balance, but it's more suspectable to misfunction in a high-load environment.

Consider we have a public website example.com resolving to 1.1.1.1. The actual web-server is behind a router/firewall and is listening on port 8080.

Simple setup is OK for internet users, but not for other clients in 192.168.1.0 range. A client in 192.168.1.3 cannot connect to 1.1.1.1:80. We have hairpinning NAT problem, (described also in mikrotik wiki).

Suppose, we solve issue not by configuring router, but with iptables in web-server itself. This commands instruct all outgoing packets on port 8080 to be sent via router:

iptables -t mangle -A OUTPUT -p tcp -m tcp --sport 8080 -j MARK --set-xmark 1
ip rule add fwmark 1 table natreflect
ip route add default via 192.168.1.1 table natreflect

The above config solves all clients in 192.168 range, but not clients in 192.168.1.2 itself.

When a client 192.168.1.2 wants to connect to 1.1.1.1:80, it sends a packet like 192.168.1.2:34567->1.1.1.1:80 to router. Router does NAT and sends a packet 192.168.1.2:34567->192.168.1.2:8080 to web server (e.g. on its eth0 interface). Web servers receives this packet. The packet traverses chains mangle.PREROUTING and nat.PREROUTING, but it doesn't reach to mangle.INPUT (and connection does not establish).

What's the problem? (rp_filter is set to 0 on all interfaces)

The best way to hide switching of master/slave redis servers, is to use a proxy over master server.

Among proxy candidates, there is two more important (and used) choices: twemproxy and haproxy(>=1.5)

If we are not interested of some twemproxy features (like sharding) (and intrested only on proxing job), which one is better (and why?)

Ansible tags can be used to run only a subset of tasks/roles. This means that by default all tasks are executed and we can only prevent some tasks to execute.

Can we limit a task to be exectued only when "foo" tag is specified? Can we use current tags in when section of a task?