Questions[ceph-rgw](server)

I am trying to run a new Ceph cluster with Rados GW using the last software version 16.2.7, but when I set up RGW nodes I found out there are some changes in the CLI comparing with a version 16.2.4 I tested before.

The next commands are missed in the 16.2.7 version:

ceph dashboard set-rgw-api-user-id $USER
ceph dashboard set-rgw-api-access-key ...
ceph dashboard set-rgw-api-secret-key ...

they don't exist in ceph dashboad -h output on the 16.2.7 version:

# ceph dashboard -h | grep set-rgw-api | grep -v reset
dashboard set-rgw-api-access-key                     Set the RGW_API_ACCESS_KEY option value read from -i
dashboard set-rgw-api-admin-resource <value>         Set the RGW_API_ADMIN_RESOURCE option value
dashboard set-rgw-api-secret-key                     Set the RGW_API_SECRET_KEY option value read from -i
dashboard set-rgw-api-ssl-verify <value>             Set the RGW_API_SSL_VERIFY option value

But on the 16.2.4 version everything is on place:

# ceph dashboard -h | grep set-rgw-api | grep -v reset
dashboard set-rgw-api-access-key                                                                          Set the RGW_API_ACCESS_KEY option value read from -i <file>
dashboard set-rgw-api-admin-resource <value>                                                              Set the RGW_API_ADMIN_RESOURCE option value
dashboard set-rgw-api-host <value>                                                                        Set the RGW_API_HOST option value
dashboard set-rgw-api-port <value>                                                                        Set the RGW_API_PORT option value
dashboard set-rgw-api-scheme <value>                                                                      Set the RGW_API_SCHEME option value
dashboard set-rgw-api-secret-key                                                                          Set the RGW_API_SECRET_KEY option value read from -i <file>
dashboard set-rgw-api-ssl-verify <value>                                                                  Set the RGW_API_SSL_VERIFY option value
dashboard set-rgw-api-user-id <value>                                                                     Set the RGW_API_USER_ID option value

Do you know what is a replacement for this commands or where were they moved? Thank you in advance.

UDP: In both cases the host OS is Debian 10.

The list of ceph packages is identical on both RGW setups: 16.2.4:

# dpkg -l | grep ceph
ii  ceph                              16.2.4-1~bpo10+1             amd64        distributed storage and file system                                                                                                
ii  ceph-base                         16.2.4-1~bpo10+1             amd64        common ceph daemon libraries and management tools                                                                                  
ii  ceph-common                       16.2.4-1~bpo10+1             amd64        common utilities to mount and interact with a ceph storage cluster                                                                 
ii  ceph-mgr                          16.2.4-1~bpo10+1             amd64        manager for the ceph distributed storage system                                                                                    
ii  ceph-mgr-modules-core             16.2.4-1~bpo10+1             all          ceph manager modules which are always enabled                                                                                      
ii  ceph-mon                          16.2.4-1~bpo10+1             amd64        monitor server for the ceph storage system                                                                                         
ii  ceph-osd                          16.2.4-1~bpo10+1             amd64        OSD server for the ceph storage system                                                                                             
ii  libcephfs2                        16.2.4-1~bpo10+1             amd64        Ceph distributed file system client library                                                                                        
ii  libsqlite3-mod-ceph               16.2.4-1~bpo10+1             amd64        SQLite3 VFS for Ceph
ii  python3-ceph-argparse             16.2.4-1~bpo10+1             all          Python 3 utility libraries for Ceph CLI                                                                                            
ii  python3-ceph-common               16.2.4-1~bpo10+1             all          Python 3 utility libraries for Ceph                                                                                                
ii  python3-cephfs                    16.2.4-1~bpo10+1             amd64        Python 3 libraries for the Ceph libcephfs library

# dpkg -l | grep rados
ii  librados2                         16.2.4-1~bpo10+1             amd64        RADOS distributed object store client library
ii  libradosstriper1                  16.2.4-1~bpo10+1             amd64        RADOS striping interface
ii  python3-rados                     16.2.4-1~bpo10+1             amd64        Python 3 libraries for the Ceph librados library
ii  radosgw                           16.2.4-1~bpo10+1             amd64        REST gateway for RADOS distributed object store

16.2.7:

# dpkg -l | grep ceph
ii  ceph                              16.2.7-1~bpo10+1             amd64        distributed storage and file system
ii  ceph-base                         16.2.7-1~bpo10+1             amd64        common ceph daemon libraries and management tools
ii  ceph-common                       16.2.7-1~bpo10+1             amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-mgr                          16.2.7-1~bpo10+1             amd64        manager for the ceph distributed storage system
ii  ceph-mgr-modules-core             16.2.7-1~bpo10+1             all          ceph manager modules which are always enabled
ii  ceph-mon                          16.2.7-1~bpo10+1             amd64        monitor server for the ceph storage system
ii  ceph-osd                          16.2.7-1~bpo10+1             amd64        OSD server for the ceph storage system
ii  libcephfs2                        16.2.7-1~bpo10+1             amd64        Ceph distributed file system client library
ii  libsqlite3-mod-ceph               16.2.7-1~bpo10+1             amd64        SQLite3 VFS for Ceph
ii  python3-ceph-argparse             16.2.7-1~bpo10+1             all          Python 3 utility libraries for Ceph CLI
ii  python3-ceph-common               16.2.7-1~bpo10+1             all          Python 3 utility libraries for Ceph
ii  python3-cephfs                    16.2.7-1~bpo10+1             amd64        Python 3 libraries for the Ceph libcephfs library

# dpkg -l | grep rados
ii  librados2                         16.2.7-1~bpo10+1             amd64        RADOS distributed object store client library
ii  libradosstriper1                  16.2.7-1~bpo10+1             amd64        RADOS striping interface
ii  python3-rados                     16.2.7-1~bpo10+1             amd64        Python 3 libraries for the Ceph librados library
ii  radosgw                           16.2.7-1~bpo10+1             amd64        REST gateway for RADOS distributed object store

I'm deploying regular Ceph Pacific using cephadm, including RGW. Now I want to use the ingress-service, which includes specifying a virtual ip - so far so good.

Unfortunately this service now creates a haproxy.cfg-file with a binding to "*:80" (used port 80 for regular HTTP access), though I configured virtual ip 10.90.90.2.

As there will be several IPs with several services binding to port 80 I'm not sure whether I'm at fault here (missing a configuration option) or there is an oversight at cephadm, which forces binding to '*' due to the conditional here not being configurable in the ingress service (ingress.py#L132).

Essentially: why is cephadm forcing a bind for "*:<Frontend Port>" on the RGW ingress service, or how can I force this service to bind to specified virtual ip? I'm not entirely sure that editing the configured haproxy.cfg will survive a reboot of the host, or an orch migration between hosts.

I have a ceph-rgw installation with a large bucket (~60M objects) and 16 osds, the bucket index is sharded into 997 shards. In this environment single directory listing takes more than 30 seconds:

$ time rclone lsd t:bucket/non/existent/path/ --contimeout=1h --timeout=1h
real    0m34.816s

This is very annoying, and may clients (e.g. rclone itself) do a list-dir op before a PUT to check/verify something. (Preventing clients from a sending list_objects/list_bucket is not a good option)

The log of rgw daemon is normal. Part of log is:

08:57:45.267+0000 7f0492db2700  1 ====== starting new request req=0x7f05039a9620 =====
08:57:45.267+0000 7f0492db2700 20 req 412648 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/bucket
08:57:45.267+0000 7f0492db2700 10 req 412648 0.000000000s canonical request = GET
08:57:45.267+0000 7f0492db2700  2 req 412648 0.000000000s s3:list_bucket verifying op params
08:57:45.267+0000 7f0492db2700  2 req 412648 0.000000000s s3:list_bucket pre-executing
08:57:45.267+0000 7f0492db2700  2 req 412648 0.000000000s s3:list_bucket executing
08:57:45.267+0000 7f0492db2700 20 req 412648 0.000000000s s3:list_bucket RGWRados::Bucket::List::list_objects_ordered starting attempt 1
08:57:45.267+0000 7f0492db2700 10 req 412648 0.000000000s s3:list_bucket RGWRados::cls_bucket_list_ordered: :bucket[e6fb9c7c-74a2-4819-a0ed-e740d4eb590c.4751590.1]) start_after="[]", prefix="/non/existent/path/" num_entries=1001, list_versions=0, expansion_factor=1
08:57:45.271+0000 7f0492db2700 10 req 412648 0.004000000s s3:list_bucket RGWRados::cls_bucket_list_ordered request from each of 997 shard(s) for 8 entries to get 1001 total entries
08:58:07.495+0000 7f04efe6c700 10 librados: Objecter returned from call r=0
08:58:08.779+0000 7f04cd627700  4 rgw rados thread: no peers, exiting
08:58:18.803+0000 7f0492db2700  2 req 412648 33.535980225s s3:list_bucket completing
08:58:18.803+0000 7f047bd84700  2 req 412648 33.535980225s s3:list_bucket op status=0
08:58:18.803+0000 7f047bd84700  2 req 412648 33.535980225s s3:list_bucket http status=200
08:58:18.803+0000 7f047bd84700  1 ====== req done req=0x7f05039a9620 op status=0 http_status=200 latency=33.535980225s ======
08:58:18.803+0000 7f047bd84700  1 beast: 0x7f05039a9620: 192.168.1.1 - rgwuser [10/Nov/2021:08:57:45.267 +0000] "GET /bucket?delimiter=%!F(MISSING)&max-keys=1000&prefix=non%!F(MISSING)existent%!F(MISSING)path%!F(MISSING) HTTP/1.1" 200 413 - "rclone/v1.57.0" - latency=33.535980225s

The environment detail is: Ceph Version: 16.2.5 Installed with rook, Each OSD is about ~4T with a 256G SSD Metadata device.