Ceph Blog

v19.2.5 Squid released

2026-07-14T00:00:00Z

This is the fifth backport release in the Squid series. We recommend all users update to this release.

Notable Changes ¶

MDS (Metadata Server) ¶

Fixed a segmentation fault relating to MDSContext completion and request queuing.

OSD (Object Storage Daemon) ¶

Rollback & Vector Fixes: Corrected rollback logic for partial write object information (OI) and optimized Erasure Coding (EC) by ensuring Twiddle creates a full-sized vector.

RGW (RADOS Gateway) ¶

TLS 1.3 Ciphersuites: Introduced the ssl_ciphersuites configuration option for the Beast front-end to support TLS 1.3 cipher customizations.
S3 PutObject ACLs Fix: Fixed a bug where PutObject requests were erroneously rejected under BlockPublicAcls due to an incorrect comparison return value from canned_acl.compare().
Multisite Sync Robustness: Resolved an infinite loop scenario in RGWBucketFullSyncCR when a source bucket was deleted mid-sync by introducing a mechanism to clear stateful data in reused bucket_list_result objects.
SNS Policy Evaluation: Updated ListTopics to use the account root ARN for policy evaluation when invoked by a non-root account user, preventing unexpected implicit denies from identity-based policies.
Presigned URLs: Fixed an internal server error occurring during the authentication process of presigned URLs.
Prerequest Hook: Restructured the order of execution so that the prerequest hook now runs after the authorization process has completed.
Object Tagging: Added handling for plain-text object tags during execution of RGWObjTags::decode().
Beast Frontend Stability: Applied a strand executor to the Beast timeout timer to avoid concurrent socket access and potential race conditions.
Bucket Statistics: Added a fix for bucket stats execution when the underlying bucket index does not exist.

Changelog ¶

[Stretch Mode] mon: restrict changing mon election strategy post stretch mode (pr#65458, Kamoltat Sirivadhna)
Check if HTTP\_X\_AMZ\_COPY\_SOURCE header is empty (pr#66028, Suyash Dongre)
Fix multifs auth caps check (pr#65359, Kotresh HR)
librbd/migration/QCOWFormat: avoid use-after-free in execute_request() (pr#68187, Ilya Dryomov)
librbd: avoid losing sparseness in read_parent() (pr#68462, Ilya Dryomov)
mds: add ref counting to LogSegment (pr#68443, Milind Changire)
mds: use SimpleLock::WAIT_ALL for wait mask (pr#68320, Patrick Donnelly)
mgr/snap_schedule: restrict retention period multipliers set (pr#67267, Milind Changire)
mgr/vol: don't delete user-created pool in "volume create" command (pr#63068, Rishabh Dave)
mgr/volumes: allow disabling async job thread (pr#62433, Rishabh Dave)
mgr: ensure that all modules have started before advertising active mgr (pr#68248, Laura Flores)
mon/AuthMonitor: add osd w cap for superuser client (pr#68318, Venky Shankar, Patrick Donnelly)
os/bluestore: introduce allocator lookup policy (pr#66840, Igor Fedotov)
pybind/mgr/volumes: fix typo in casesensitive vxattr (pr#63023, Patrick Donnelly, Xavi Hernandez)
python-common/cryptotools: stop using the removed X509Req API (pr#69752, Kefu Chai)
qa/multisite: switch to boto3 (pr#68919, Shilpa Jagannath)
qa/radosgw_admin: replace boto2 with boto3 (pr#68849, Adam C. Emerson, Casey Bodley)
qa/rgw/upgrade: symlinks are explicit about distro versions (pr#68058, Casey Bodley)
qa/rgw: remove ragweed from verify subsuite (pr#69114, Casey Bodley)
qa/rgw: Revive crypt kmip (pr#68372, Kyr Shatskyy)
qa/suites/rados: temporarily disable ceph-post-file test (pr#68611, Laura Flores)
qa/tasks/keystone: restart mariadb for rocky and alma linux too (pr#67543, Kyr Shatskyy)
qa/tasks/keystone: upgrade keystone to 2025.2 (pr#67756, Kyr Shatskyy)
qa/workunits/rbd: drop racy assert in test_tasks_recovery() (pr#68189, Ilya Dryomov)
qa: allow multiple mgr sessions during eviction test (pr#68321, Patrick Donnelly)
qa: fix setting rbd_sparse_read_threshold_bytes in test_migration_clone() (pr#68616, Ilya Dryomov)
qa: Remove cephadm e2e tests from teuthology (pr#68819, Afreen Misbah)
qa: resolve py3.12 regression for random.sample (pr#68319, Patrick Donnelly)
rgw/auth: fix internal server error for presigned urls (pr#69049, Tobias Urdin)
rgw/beast: add ssl_ciphersuites option for tls 1.3 (pr#69179, Casey Bodley)
rgw/beast: use strand executor for timeout timer to prevent concurrent socket access (pr#68507, Oguzhan Ozmen)
RGW/multisite: fix bucket-full-sync infinite loop caused by stale bucket_list_result reuse (pr#67921, Oguzhan Ozmen)
rgw/s3: fix PutObject's canned_acl comparisons for BlockPublicAcls (pr#69481, Casey Bodley)
rgw/sns: ListTopics uses account root arn for policy evaluation (pr#69274, Casey Bodley)
rgw: account rm --purge-data can delete users/roles/groups/oidcs too (pr#68060, Casey Bodley)
RGW: Change prerequest hook to run after authorization process (pr#67711, Emin Sunacoglu)
rgw: fix 'bucket stats' when bucket index doesn't exist (pr#68504, Casey Bodley)
RGW: handle plain-text object tags in RGWObjTags::decode() (pr#67926, Oguzhan Ozmen)
rgw: read_obj_policy() consults s3:prefix when deciding between 403/404 (pr#68652, Casey Bodley)
suites/rados/cephadm: add still running warning to the ignore list (pr#62532, Nitzan Mordechai)
test/rgw: remove depracated boto dependency (pr#68470, Yuval Lifshitz)

Recover the Ceph Monitor Store Using OSDs

2026-07-04T00:00:00Z

Introduction ¶

A few weeks ago I helped a Ceph user to recover his broken cluster (see this thread). Basically, after his Monitors stopped working he re-deployed a new cluster with the same Ceph FSID and attached the existing OSDs to the re-deployed hosts. But it’s not that easy to re-activate those OSDs because the new Monitors don’t have the old osdmap, hence they don’t know anything about the existing OSDs. So how did we fix that?

This scenario can be considered a total Monitor store failure. There’s a documented procedure in the upstream docs, unfortunately it’s written for non-cephadm clusters and doesn’t contain many details about subsequent steps. When Ceph daemons run within containers there are more things to consider. So I decided to write this blog post and add some more details to the procedure, targeting mainly clusters managed by cephadm. But the recovery procedure doesn’t specifically require cephadm-specific commands, so it can be considered as a general guideline how to recover from a Monitor store loss, just with some extra details about cephadm deployments.

This procedure only works for non-encrypted OSDs. If you are using dmcrypt (LUKS) OSDs, make sure you have a backup of the keys, otherwise your data is lost forever! But there is some development, a user in Slack pointed me to this PR which adds a backup mechanism for Monitors, probably available in the upcoming Umbrella release. Note that the PR also contains this statement:

Monitor backups complement, but do not replace, the existing Monitor recovery procedures

The procedure covered by said docs is already written in script form, so I used that as a template and extended it for cephadm usage and included some very basic logging. It collects the osdmaps from all OSDs and contains the necessary considerations regarding containers. I decided to automate only the osdmap collection, not all required steps of the store rebuild procedure (e. g. Monitor store rebuild, client auth, etc.) because the result of the collection process must be inspected carefully before proceeding. Here’s the script I used on one of my lab clusters to collect the osdmaps:

#!/bin/bash

: <<'END'
This script does:
1. Login to each OSD host
2. Create a temporary mon-store in /tmp after cleaning it up 
3. Create a temporary mon-store directory within each OSD's
   filesystem (which is then mapped into the cephadm shell) after cleaning it up
4. Collect the osdmaps from each OSD and sync it to each host's temporary mon-store  
5. Sync each host's temporary mon-store to the central mon-store
END

# List of all OSD hosts
hosts="squid1 squid2 squid3"

# Ensure this user has passwordless sudo on all OSD hosts (if not root)
# It's also easier if the user has passwordless login, e.g. via ssh public key
user="root"

# Cluster FSID
ceph_fsid="df500e26-1930-11f1-a79f-fa163e65a168"

# Logging directories
ms_central="/tmp/mon-store-central"
log_central="/tmp/central-osd-logs"
logdir="/tmp/logdir"
ms_collected="/tmp/mon-store-collected"

# Ensure all OSDs are stopped before proceeding

rm -rf $ms_central
rm -rf $log_central
mkdir -p $log_central

for host in $hosts; do
ssh $user@$host <<EOF
logdir="/tmp/logdir"
ms_collected="/tmp/mon-store-collected"
ceph_fsid="df500e26-1930-11f1-a79f-fa163e65a168"
osd_dir="/var/lib/ceph/\$ceph_fsid"
rm -rf \$ms_collected
rm -rf \$logdir
mkdir -p \$logdir
echo -e "\nHost: \$(hostname -s)"

for osd in \$(cephadm ceph-volume lvm list --format json 2>/dev/null | jq -r '.[].[].tags | ."ceph.osd_id"'); do
echo -e "\nProcessing osd.\$osd ..."
rm -rf \$osd_dir/osd.\$osd/mon-store-collect
mkdir -p \$osd_dir/osd.\$osd/mon-store-collect

cephadm shell --name osd.\$osd -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-\$osd --no-mon-config --op update-mon-db --mon-store-path /var/lib/ceph/osd/ceph-\$osd/mon-store-collect 2> >(tee \$logdir/osd.\$osd.\$(hostname -s).log | grep -vEi "quay|infer|using|creating")

rsync -qaz \$osd_dir/osd.\$osd/mon-store-collect/. \$ms_collected
done
EOF
rsync -qaz -e ssh $user@$host:$ms_collected/. $ms_central
rsync -qav -e ssh $user@$host:$logdir/. $log_central
done

If not all osdmaps are collected successfully, the rebuilt Monitor store will be incomplete and some OSDs won’t be able to join the cluster. There are many more potential obstacles before the OSDs can be activated, so it’s critical to review the logs from each host after the osdmap collection script has finished (in case of errors) and then continue step by step.

Collect the osdmaps ¶

There are a couple of assumptions made in the script, you will need to adapt those according to your requirements. For example, the list of hosts, the Ceph FSID, and the user who logs in to each host. All OSDs should be stopped if there are still running processes, otherwise the ceph-objectstore-tool will fail during collection. I recommend to stop all daemons (mgr, MDS, ceph-exporter, etc.) before starting the recovery, otherwise you might get stuck with a stale Manager.

The procedure has been tested on two different Squid lab clusters, one on 19.2.3, the other one on 19.2.4. Here’s an example of the script output:

Host: squid1

Processing osd.1 ...
osd.1   : 0 osdmaps trimmed, 226 osdmaps added.

Processing osd.2 ...
osd.2   : 0 osdmaps trimmed, 228 osdmaps added.

Processing osd.0 ...
osd.0   : 0 osdmaps trimmed, 223 osdmaps added.

Ceph FSID ¶

Depending on how much data is actually lost, there are several ways to retrieve the Ceph FSID (to keep the output brief, I use $FSID instead of the actual UUID):

ceph.conf: If there’s any ceph.conf left on any host (clients also have a ceph.conf) it will contain the FSID. Alternatively, since every daemon also has a copy of ceph.conf, check any daemon config file in /var/lib/ceph/$FSID/osd.X/config if the filesystem is intact. Note that hosts that have been used for prior cluster incarnations may have multiple directories under /var/lib/ceph, so be sure to use only the latest and greatest.
LV tags: On any OSD host run the command and look for the tag ceph.cluster_fsid=$FSID:

    lvs -o tags

ceph-volume: On any OSD host run the command:

    cephadm ceph-volume lvm list | grep "cluster fsid"

Only run "bare" ceph-volume commands if your cluster is not managed by cephadm! Otherwise you’ll most likely end up with more chaos!

OSD FSID ¶

In some failure scenarios you might need the FSID of each OSD to rebuild an OSD’s directory structure. To collect those, you can also use ceph-volume or LV tags:

LV tags: On any OSD host run the command and look for the tag ceph.osd_fsid=$FSID:

    lvs -o tags

ceph-volume: On any OSD host run the command:

    cephadm ceph-volume lvm list | grep "osd fsid"

Only run “bare” ceph-volume commands if your cluster is not managed by cephadm! Otherwise you’ll most likely end up with more chaos!

SSH user ¶

The easiest way is to have a user that can log in to each host without a password and has (passwordless) sudo permissions to execute the necessary commands. Depending on your Ceph deployment, you may log in as the cephadm user which usually has the mentioned permissions. Often the root user is configured as the cephadm user, as it is in my lab cluster in which I tested this procedure.

Rebuild the Monitor store ¶

Assuming the collection of osdmaps was successful and no errors were logged, you can now rebuild the store. The hostname of the first Monitor host in this example is squid1.

# Create a backup of the current store (just in case)
root@squid1:~# rsync -a /var/lib/ceph/$FSID/mon.squid1/store.db/ /root/mon-store-backup/

# Ensure all daemons are stopped!

# Remove current mon-store
root@squid1:~# rm -rf /var/lib/ceph/$FSID/mon.squid1/store.db/

# Copy collected store.db from your management
# node (where you ran the collection script) to the first monitor's store.db
root@squid1:~# rsync -av -e ssh /tmp/mon-store-collected/store.db/ root@squid1:/var/lib/ceph/$FSID/mon.squid1/store.db/

# Rebuild mon-store
root@squid1:~# cephadm shell --name mon.squid1 -- ceph-monstore-tool /var/lib/ceph/mon/ceph-squid1/ rebuild  -- --keyring /etc/ceph/ceph.keyring --mon-ids squid1 squid2 squid3
...
epoch 0
fsid df500e26-1930-11f1-a79f-fa163e65a168
last_changed 2026-06-27T12:18:38.557560+0000
created 2026-06-27T12:18:38.557560+0000
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:192.168.124.107:3300/0,v1:192.168.124.107:6789/0] mon.squid2
1: [v2:192.168.124.222:3300/0,v1:192.168.124.222:6789/0] mon.squid3
2: [v2:192.168.124.230:3300/0,v1:192.168.124.230:6789/0] mon.squid1

### Note: you need to specify all monitors in the correct order (as mentioned in the docs)!

# Change ownership 
root@squid1:~# chown -R 167.167 /var/lib/ceph/$FSID/mon.squid1/store.db/

Start the Monitors ¶

If all above steps were successful, you can try to start the first Monitor, either via systemd or cephadm or podman/docker This also depends on the actual deployment situation. One of these commands should work:

root@squid1:~# systemctl start ceph-$FSID@mon.squid1.service 

root@squid1:~# cephadm unit start --name mon.squid1

root@squid1:~# podman start ... / docker start ...

# Now check the monitor log to see if it starts successfully
root@squid1:~# journalctl -fu start ceph-$FSID@mon.squid1.service

If the first Monitor starts, copy the contents of its store.db directory to the other Monitor daemon directories, ensure the correct ownership, and try to start them as well. Inspect their logs if something goes wrong, and if it does it must be fixed before continuing. You can also simply repeat the rsync and rebuild steps to recreate the Monitor stores for the remaining Monitors.

If the Monitors start you should be able to see a Ceph status now (ceph -s). The Ceph status should now contain the OSDs in the output (just an excerpt):

ceph -s
cluster:
id: df500e26-1930-11f1-a79f-fa163e65a168
health: HEALTH_WARN
mons are allowing insecure global_id reclaim
no active mgr
services:
mon: 3 daemons, quorum squid2,squid3,squid1 (age 57s)
mgr: no daemons active
osd: 6 osds: 1 up (since 52m), 6 in (since 54m)
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:

We now see that the Monitors are aware of six OSDs (which is correct for my lab cluster), but the pool, objects and usage output is not correct since the OSDs are not up yet and haven’t been able to report to the Monitors and Managers.

Before starting OSDs ¶

There are additional steps required before you can activate the OSDs. For example, all custom configuration option settings (ceph config dump) are gone, auth entries are missing, the container_image config value has been reset, and the orchestrator doesn’t work yet: it must be re-enabled first. Not all of these steps have to be performed necessarily before activating the OSDs, but some of them are crucial and others can make life easier when activating the OSDs.

# Disable auth_allow_insecure_global_id_reclaim (to clear the warning)
root@squid1:~# ceph config set mon auth_allow_insecure_global_id_reclaim false

# Recreate and import at least one mgr auth keyring to enable the orchestrator
root@squid1:~# cat mgr.squid1.tkgxvc.keyring
[mgr.squid1.tkgxvc]
key = AQDjJhtqCOWnFRAA4/QHkFo2tnWFG8m37kM5HQ==
caps mds = allow *
caps mon = profile mgr
caps osd = allow *

root@squid1:~# ceph auth import -i mgr.squid1.tkgxvc.keyring

# Recreate and import all OSDs keyrings
# If present, use key from /var/lib/ceph/$FSID/osd.$id/keyring to recreate an OSD keyring
root@squid1:~# cat osd.0.keyring
[osd.0]
key = AQBzGxtqPUu8EBAAC0BqZMcfRJb5ZuGtXFgBCA==
caps mgr = allow profile osd
caps mon = allow profile osd
caps osd = allow *

# Check ceph auth for completeness
root@quid1:~# ceph auth ls

# Start at least one manager daemon to be able to manage the cluster again
root@squid1:~# systemctl start ceph-$FSID@mgr.squid1.tkgxvc.service

# Set the public network(s), otherwise OSDs will refuse to start
root@squid1:~# ceph config set global public_network 192.168.124.0/24

# Enable cephadm module
root@squid1:~# ceph mgr module enable cephadm

# Set orchestrator backend
root@squid1:~# ceph orch set backend cephadm

# Recreate cephadm user/keyring
root@squid1:~# ceph cephadm generate-key

# Get new public key
root@squid1:~# ceph cephadm get-pub-key

# Add new public key to each host's authorized_keys (usually of the root user)

# Re-add hosts to the orchestrator
root@squid1:~# ceph orch host add squid1 192.168.124.230
root@squid1:~# ceph orch host add squid2 192.168.124.107
root@squid1:~# ceph orch host add squid3 192.168.124.222

# Add labels if necessary

# Configure container_image (choose the correct version for your cluster)
root@squid1:~# ceph config set global container_image quay.io/ceph/ceph:v19.2.4

If you don’t change the container_image, some daemons most likely will be deployed using the hard-coded default:

docker.io/ceph/daemon-base:latest-master-devel

Not changing that image could lead to issues when redeploying other daemons such as MDSes.

Activate the OSDs ¶

There’s a wrapper for cephadm to activate existing OSDs. It covers a slightly different scenario: when an OSD host fails due to a broken operating system, its OSDs can be re-activated after reinstalling the operating system. In that case, after all prerequisites are met, you can simply run:

ceph cephadm osd activate <host>

Chances are high that this approach won’t work in a scenario like we’re facing here: total Monitor store loss. So a more manual workaround like the following might be required.

Now assuming that the previously described steps worked and errors were fixed, we can try to start OSDs. Check the contents of the OSD directories first and see if the most important files are present:

root@squid1:~# ls -l /var/lib/ceph/$FSID/osd.0/
total 68
lrwxrwxrwx 1 167 167 93 Jun 19 14:56 block -> /dev/ceph-3d3a6efd-02d7-4abf-ae23-17bc239feb1e/osd-block-8012c0dc-cb7b-4179-908c-df0a94a4227b
-rw-r--r-- 1 167 167 37 Jun 19 14:56 ceph_fsid
-rw------- 1 167 167 181 Jun 19 14:58 config
-rw-r--r-- 1 167 167 37 Jun 19 14:56 fsid
-rw------- 1 167 167 142 Jun 19 14:58 keyring
-rw------- 1 167 167 1875 Jun 19 14:33 unit.poststop
-rw------- 1 167 167 3457 Jun 19 14:33 unit.run
-rw-r--r-- 1 167 167 2 Jun 19 14:56 whoami

Without these files the OSDs will most likely refuse to start. Now let’s give it a try:

# systemd
root@squid1:~# systemctl start ceph-$FSID@osd.0.service

or via

# cephadm
root@squid1:~# cephadm unit start --name osd.0

Alternatively, if the orchestrator works (you could try to deploy node-exporter or other daemons to confirm), you can also just redeploy an OSD:

root@squid1:~# ceph orch daemon redeploy osd.X

This will recreate missing files in the OSD’s directory and hopefully start the OSD successfully.

Inspect the OSD logs after starting them and look out for error messages if they fail to start:

root@squid1:~# journalctl -fu ceph-$FSID@osd.X.service

If you enabled log_to_file you can follow the log file like this:

root@squid1:~# tail -F /var/log/ceph/$FSID/ceph-osd.X.log

Next steps ¶

Let’s assume that we could successfully start the OSDs and recovery is progressing, and we need to access the data now. For RBD pools there’s (probably) not much you need to do: as soon as clients can access the OSDs again, they should be able to map/attach/access the RBD images.

For CephFS it’s a different story. Since the MDS maps are lost as well, all file systems will have to be recreated, including the MDS daemons. The Ceph docs cover this part. Read the instructions very carefully! The required commands are:

root@squid1:~# ceph fs new <fs_name> <metadata_pool> <data_pool> --force --recover
root@squid1:~# ceph fs set <fs_name> joinable true

Since there are no existing MDS daemons yet, you’ll need to deploy them either via .yaml file:

root@squid1:~# ceph orch apply -i mds.yaml

or directly:

root@squid1:~# ceph orch apply mds <fs_name> --placement="label:mds"

Check out the docs for more information on MDS deployment.

I won’t go into more details, as this post is long enough as it is. But you’ll need to deploy required daemons for all services you had running before the crash. So you’ll have to recreate your yaml files for all the services like RGW, Monitoring, NVMe-oF, etc. The existing service specs in a recovered cluster look like this (when no services have been managed yet):

root@squid1:~# ceph orch ls
NAME  PORTS  RUNNING  REFRESHED  AGE  PLACEMENT     
mgr              2/0  9m ago     -    <unmanaged>   
mon              1/0  9m ago     -    <unmanaged>   
osd                0  9m ago     -    <unmanaged>

You’ll need to update those as well to match your actual cluster configuration. I recommend to regularly back up your cluster configuration, auth keyrings, service specs and so on, so you don’t have to recreate everything from scratch in such a failure scenario. Commands to capture the most crucial information include the below:

ceph status
ceph orch ls --export
ceph config dump
ceph osd crush dump
ceph report

One strategy is to emplace a cron job that emails the output files as attachments; another is to store them as S3 objects, say in another Ceph cluster. Be sure that you keep a prudent number of revisions, so that a cluster mishap does not overwrite your good data with bad.

After the recovery has been completed, the cluster status looks like this:

squid1:~ # ceph -s
 cluster:
   id:     df500e26-1930-11f1-a79f-fa163e65a168
   health: HEALTH_OK
 
 services:
   mon: 3 daemons, quorum squid2,squid3,squid1 (age 80m)
   mgr: squid3.fepqsu(active, since 62m), standbys: squid1.pcgzhi
   mds: 2/2 daemons up, 3 standby
   osd: 6 osds: 6 up (since 65m), 6 in (since 65m)
 
 data:
   volumes: 2/2 healthy
   pools:   7 pools, 385 pgs
   objects: 49 objects, 4.3 MiB
   usage:   421 MiB used, 60 GiB / 60 GiB avail
   pgs:     385 active+clean

Potential obstacles ¶

There are things that can go wrong while trying to re-activate the OSDs. This is not an exhaustive list, but it can help in specific scenarios.

Missing OSD path contents ¶

As mentioned above, the contents of the OSD directories must be present and matching. If OSDs don’t start, the logs might point to permission issues, missing files (unit.poststop, type, and so on), a broken symbolic link to the OSD block device, or other failures.

Ceph Dashboard ¶

Most likely you’ll have to enable the dashboard again since all configurations are gone:

root@squid1:~# ceph mgr module enable dashboard

After that, you’ll have to recreate users, roles etc., whatever you had configured before.

Packages ¶

One of the issues during OSD activation in this case was the presence of the ceph-osd package on at least one of the hosts. It’s known to be problematic to have that package present on a cephadm-managed cluster. So make sure that the ceph-osd package is absent from all cephadm-managed hosts. Basically, on a cephadm-managed cluster node you only need the cephadm package: the orchestrator uses a binary stored at /var/lib/ceph/$FSID/cephadm.{some_digest} to remotely execute commands. For comfort reasons the ceph-common package is also useful. Other YUM/DNF/APT packages including ceph-mon, ceph-mgr, etc. are not required and should also be absent.

ceph-volume ¶

As I already pointed out before, don’t use the bare ceph-volume commands to try and fix things when OSDs are managed by the orchestrator. I recommend to remove the ceph-volume package as well, if present. Attempts to fix the OSDs made things worse and caused more chaos, leaving more things to clean up.

Container images ¶

As mentioned above, there’s a hard-coded default image (docker.io/ceph/daemon-base:latest-master-devel) which can make life harder. Make sure to configure the correct Ceph container_image as well as those for other daemons, including nvmeof, grafana, prometheus, node_exporter, etc.):

root@squid1:~# ceph config ls | grep container_image
container_image
mgr/cephadm/container_image_alertmanager
mgr/cephadm/container_image_base
mgr/cephadm/container_image_elasticsearch
mgr/cephadm/container_image_grafana
mgr/cephadm/container_image_haproxy
mgr/cephadm/container_image_jaeger_agent
mgr/cephadm/container_image_jaeger_collector
mgr/cephadm/container_image_jaeger_query
mgr/cephadm/container_image_keepalived
mgr/cephadm/container_image_loki
mgr/cephadm/container_image_node_exporter
mgr/cephadm/container_image_nvmeof
mgr/cephadm/container_image_prometheus
mgr/cephadm/container_image_promtail
mgr/cephadm/container_image_samba
mgr/cephadm/container_image_snmp_gateway

Disclaimer: The procedure above was tested in two lab environments in order to write this blog post, but the individual steps were executed in a real cluster. It worked for me/us but it might not work for you. If anything goes wrong it’s not my fault, but yours. And it’s yours to fix it!

If you have suggestions how to improve the procedure or feedback in case you had to use it, I’d appreciate a comment!

PS: Note that the best way to avoid losing cluster data due to an unrecoverable Monitor store is to not lose the Monitor store in the first place. The abovementioned backup procedure can help, as will ensuring that every production cluster runs five Monitor daemons. While three is a valid number of Monitors to run, five has much greater resilience. Ensure that your Monitors are spread across failure domains (e.g. hosts or racks) and that no more than two can be placed in any one failure domain.

This article has been crossposted from https://heiterbiswolkig.blogs.nde.ag/2026/06/28/cephadm-recover-mon-store-using-osds/

Reclaim Ceph Capacity Through CephFS Transcoding

2026-07-01T00:00:00Z

Data expands to fill available storage (and beyond)! ¶

It used to be that enterprise storage meant 6RU rackmount Fujitsu 2351 Eagles, each holding a mind-boggling 380 MiB of data: enough for a whole company!

Today that 380 MiB can't even hold a 4k pickleball video. Enterprises, educational instutitions, and really just about anyone these days demand storage capacities that start on the order of hundreds of tebibytes and rapidly grow to pebibytes.

As this article is written in the spring of 2026, the memory market, which includes DRAM, SSDs, and legacy HDDs, has experienced a dramatic escalation of pricing. It is not uncommon to be quoted a price four times what the same hardware cost a year ago, and there are signs that it is going to get worse before it gets better.

What's a poor ammonite to do??

Cephers find themselves between the Charybdis of quotes approaching Disaster Area's hypermathematics and the Scylla of hungry users armed with torches and git forks.

Git forks, pitchforks. Get it? Sigh. Tough room. Anyway...

Short of nuking the site from orbit, how do we make everyone happy, or at worst mildly discontented?

Efficiency!

CephFS ¶

CephFS is a popular, highly available and scalable software-defined POSIX-style distributed filesystem that can easily store tens of pebibytes of precious data. Or, alternately, cat videos.

Ceph deployments often begin small, with replicated pools for perceived performance needs. As the cluster grows to more nodes and more data, it may become feasible and desirable to switch to Erasure Coding (EC) to make more efficient use of raw capacity.

An EC pool thus can require substantially less raw storage for a given amount of user data, or store gobs more user data on a given amount of raw capacity

This EC overhead table presents efficiency (space amplification) factors for a spectrum of EC profiles. Replicated pools usually maintain three copies of data, so for comparison they manifest an overhead factor of 3.0. EC 4+2 or 6+3 presents a factor of just 1.5, with some tradeoffs, including some space amplification for the very smallest files.

The use of EC has traditionally been limited to workloads with modest write performance requirements. With SSDs, however, many workloads are actually compatible with the tradeoffs of EC. Moreover, the Ceph Tentacle release delivered the highly anticipated Fast EC which brings EC pool performance much closer to parity with that of replicated pools, enabling its use for workloads that previously demanded replication.

Cephers often think that once data is written to a RADOS pool, the data protection strategy is carved in stone unless one manually and disruptively moves data to a different pool. RGW Lifecycle policies however offer a way to transition Ceph Object Storage data between storageclasses and pools transparently, but we're to talk about the draft ^H^H^H erm .. CephFS

CephFS file systems / volumes typically provision one RADOS pool for metadata (foo.meta) and one for data (foo.data). One can however attach multiple pools to a CephFS volume, which may live on different media (HDD, mainstream NVMe SSD, coarse-IU QLC SSD) and/or employ different data protection strategies (Replica 3, EC 4+2, EC 8+3, EC etc). When one chooses to use EC to hold CephFS data, it's a good idea to provision at least two data pools: the first (default) data pool holds inode backtraces and HEAD objects. When there are no other data pools, the first data pool also holds file data.

When one or more additional (and often EC) data pools are added to a CephFS volume, inode backtraces and HEAD objects are still placed in the first / default data pool. But how do we place bulk file data?

One of the more obscure but very, very useful features of CephFS is the ability to set file layouts. These are somewhat akin to RGW / S3 storage classes in that they can be applied to CephFS volume (sub)directories or individual files to gate data placement. Which is downright smurfy, but when one has pebibyates of data on replicated or expensive storage and wishes to transparently migrate it to the new data pool(s), what to do? Setting the layout on a directory, even the top level directory of a volume, directs new files to the desired pool, but existing files must be copied to new files in order to change their placement. Doing this with a recursive cp, tar, or cpio command can be a lot of work, especially if one does not have sufficient capacity to temporarily store an entire second copy of data. Moreover, we need to consider symlinks, which are stored only in the .meta pool, files with multiple hard links, files that are currently being written, and files that may be read or written while copying.

The solution ¶

Recently as a Ceph Ambassador I crossed paths with a community member storing breathtaking amounts of data, and adding more every day. All of this CephFS data had been traditionally stored on mainstream TLC SSDs in replica 3 pools, because these are the defaults for performance.

Quotes for capacity expansion turned out to be dramatically higher than budgeted, and they were in a bind.

I helped them discover that while their MDS estate is very busy, at the RADOS level their datalake is read-mostly, with modest to moderate client read workload. They do need the reads to be fast, but write speed is of lesser concern. This is a classic workload for EC, and for QLC (or QLC-class) SSDs. I helped them identify additional QLC-class SSD options, and proposed a converstion of the CephFS datalake to EC to reclaim ... well they don't want to disclose numbers, but let's say a many zeros of existing raw capacity to defray and defer spending until such a time that the NAND market cools down.

Enter this ultimate answer to EC, CephFS, and everything: a transcoder script posted by Reddit user marcan42. This script is a terrific inspiration, but bears a number of shortcomings. I worked with the community member to enhance the script: my fork can be found here.

In a nutshell, the script walks a CephFS volume or subdirectory thereof, processing files by creating a copy with the new file layout then atomically renaming the copy over the original. This strategy means that clients can continue to access data without risk of disruption, corruption, or divergence. Care is taken to avoid symlinks and to properly handle files with multiple hard links.

By working on one file at a time, we avoid the need for enough unused capacity for a second if temporary copy of a directory tree or even entire volume. Because if we had that kinda space, we mostly wouldn't be having this conversation. Mostly. This approach also helps avoid divergence and reduce or eliminate the need for laborious cleanup in the event of a network or client host outage.

Painstaking care is taken to detect files that change while being processed so that they are not diverged, corrupted or deleted. Options are present to constrain the files that are transcoded:

To a specific subtree of a volume
To files that are not multiply hardlinked, out of an abundance of caution
To files older and/or newer than a specified number of days, especially when older files are much less likely to be changed or deleted during processing
To files above a specified minimum size.

The script by default processes four files in parallel; this number may be adjusted on the command line to trade off network and client impact against elapsed time.

When using EC, the very smallest files can actually consume more raw capacity than replication. The FastEC improvements in the Ceph Tentacle release reduce but do not eliminate this concern. In releases before Tentacle, RADOS always writes complete stripes. When using conventional media, and with OSDs created on releases newer than Octopus, the default value of bluestore_min_alloc_size is 4KiB. Say we're using an EC 8+3 data pool and a user creates a 1 byte file. A full-stripe write will allocate 11 * 4 = 44KiB of raw storage. For a single file this is negligible, but with millions or billions of files this space amplification adds up.

This fantastic resource from Mark Nelson quantifies and visualizes this space amplification dynamic, which in the Octopus / Pacific area lead Ceph developers to enable lowering the default bluestore_min_alloc_size. Today, however, coarse-IU QLC SSDs are an increasingly compelling alternative to both conventional TLC SSDs and HDDs. They offer cost effectiveness and fantastic density: as of March 2026, U.2 SKUs with the incredible capacity of 245 tebibytes are entering the market.

These however do best when bluestore_min_alloc_size is matched to the IU, which is usually 16 KiB, but for older drives may be as high as 64 KiB. For conventional media and popular EC profiles a minimum size of, say, 500 KiB may be specified to avoid space amplification. For many deployments, much larger files may render a bit of space amp for tiny files more or less moot, but the option is there so that individuals may choose what's right for their file demographics.

It is said that a picture is worth 1024 words, so here's another obigatory contextually appropriate AI-generated octopus:

Oh, you want hard data. ABOUT THE TOPIC. Fine. Be like that.

This first graph shows the raw capacity used by the original Replica 3 data pool of a real-world CephFS volume in green, and the capacity used by the new EC 6+3 pool in blue. The barely-visible red line depicts the obligatory .meta pool. (Ignore the dots, they are Prometheus scraping artifact). One can see readily that as the amount of data consumed by the first pool decreases the amount consumed by the new pool rises -- but at very roughly half the rate. This is expected when transcoding from Replica 3 (space amp 3.0) to EC 6+3 (space amp 1.5). The spans where the lines flatten out somewhat correspond to expanses of files that were too new, too small, symlinks, or multiply linked. The script can handle multiply-linked files, but with as many as 7000 (!) links to some files, those are skipped for the initial runs.

This transcoder run specified:

15 threads
Minimum 500 KiB file size
Minimum 1 day file age
Skipping multiply-linked files
A 6+3 EC policy

With these options, files were processed at 5-15 TiB per hour. Your mileage may vary.

This second graph shows the sum of the series in the above graph, more simply visualizing the aggregate savings in underlying raw capacity. Here we can see that the subject volume consumed roughly 330 TiB before transcoding, but only 185 TiB after, reclaiming 44% of raw capacity. The 7% deviation from the theoretical ideal is the cumulative effect of files too small, too new, or too multiply linked for transcoding.

The beauty of this approach is that the script is idempotent. One may run it again with different, or even the same parameters to pick up files as they pass the minimum age cut-off, or with a smaller threshold. The community member that inspired this post is not yet running Tentacle; once they have upgraded and enabled EC optimizations on their pools, they plan another pass over the data with a smaller threshold.

One does not need to re-run the transcoder to pick up newly-created files, as the process leaves the default layout for the directory (which may be the volume's root) set to the new pool. It is possible, with repeated runs, to arrange arbitrary subdirectory trees to use one layout (pool) or another, with inheritance and overriding. It is even possible to migrate desired subsets of data back to the default pool, or to another pool on different media. The script is written in a way that it does not reprocess files if they are already set to the target layout, ensuring idempotence.

This process does entail the creation of an additional RADOS pool for each CephFS volume transcoded, so be sure to account for that in your manual or autoscaler PG strategy. While not widely known, Ceph currently has a hard limit of 256 CRUSH rules per cluster. The transcoding setup script takes care of creating new EC profiles and CRUSH rules only as necessary, re-using them for additional CephFS volumes with the same profile. When executing, take care to manage pg_num on both old and new pools, via the PG autoscaler or other means. As data is moved off the old pool, the PG budget it was allocated can be reduced, and PGs added to the new pool.

As a rule of thumb, a PG / shard should hold no more than roughly one hundred thousand RADOS objects, or 200 TiB. Remember that the old pool may still hold many RADOS objects, but not much data volume, so avoid the temptation to reduce pg_num too low. The little-used rados df command can help gauge what's right. Also remember that EC PGs are sharded, so in the above example, each EC 6+3 shard is only 16% the size of the previous replica 3 PGs, so don't overdo pg_num on a new EC pool. Depending on the EC profile, the sufficient value for the new pool may be 25% or so of the original replicated pool's value.

The setup script that manages pools and layout xattrs now has an option to create a new pool with a specified number of PGs. It's best to get this at least vaguely right up front, when PG creation is fast and painless on an empty pool. Splitting or merging PGs later when the pool holds tebibytes or pebibytes of data is much slower and more impactful.

When migrating between two EC profiles, use your judgement.

To-do: filter on filename regexes.

This has not been tested on CephFS subvolumes, nor has it been tested transcoding back to a replicated volume. This analysis and code are presented as-is with the goal of delivering value to the Ceph community. No warranty or support should be inferred. It is free, and worth every penny.

Do not taunt the script or stare into it with remaining eye.

Suggestions or improvements are welcome, on a best-effort basis.

The upcoming Umbrella release is expected to introduce transparent pool migration at the RADOS layer. This ability will be very useful and may one day obviate the script discussed here, though one should approach its use carefully after thorough testing. The need for compatible clients will, notably, limit the ability of many CephFS deployments to use that new approach until compatible kernels make it into your Linux distribution(s) in use. The file-layout based approach we've explored here works until the new transparent migration can be rolled out.

About the author, Anthony D'Atri ¶

Ceph at scale since 2014
(Successfully since 2017)
Ceph Ambassador
Ceph Documentation Lead
Author: Learning Ceph, Second Edition
QLC fanboy
Ceph is quite graphically a part of me

Ceph Q2 2026 Newsletter

2026-06-25T00:00:00Z

During the second quarter, the Ceph Foundation focused on strengthening collaboration across the project, expanding community programs, supporting documentation efforts, and investing in regional events. This edition highlights recent governance discussions, community elections, new contributor resources, Ceph Days events, and opportunities for community members to get involved. If you have an idea for a Ceph-related event, outreach effort, or project, we encourage you to submit a funding request and work with the Foundation to help bring it to life.

In This Issue ¶

Board and CSC Alignment on Community Priorities
2026 CSC Election Recap
Meet the New Ceph Technical Writer
Ceph Days Continue to Grow
Latest Ceph Releases
Community Committees and New Ways to Contribute

Board and CSC Alignment on Community Priorities ¶

The Ceph Governing Board and Community Steering Committee (CSC) met this quarter to discuss the long-term health of the project, governance, documentation, community growth, and Foundation-supported initiatives.

The discussion reaffirmed the distinct roles of each organization within the Ceph ecosystem. The CSC continues to provide technical governance for the project, while the Ceph Foundation focuses on community growth, events, outreach, and funding. The Linux Foundation provides operational, legal, and financial support that enables both groups to operate effectively.

A significant area of focus was improving coordination between technical priorities and Foundation-supported programs. Current documentation initiatives include improvements to the documentation backport process, development of a new OSD operations guide, clarification of package and container documentation, and the removal of outdated content.

The Board and CSC also discussed the future of community events and the importance of establishing clear goals and success metrics for Foundation-funded programs. Both groups reaffirmed their commitment to supporting collaboration between users, developers, operators, and organizations across the Ceph ecosystem.

Key Takeaways ¶

Continued alignment between the Board and CSC on project priorities.
Improved coordination around documentation and community outreach efforts.
Ongoing work to define success metrics for Foundation-funded programs and events.
Shared commitment to strengthening collaboration across the community.

2026 CSC Election Recap ¶

This year's CSC election introduced several improvements, including ranked-choice voting and candidate statements, helping create a more transparent and participatory election process.

The election also highlighted opportunities to improve ballot communication and voter outreach in future cycles. Lessons learned from this year's process will help strengthen future elections and improve the overall voter experience.

2026 CSC Co-Chairs ¶

Patrick Donnelly
Neha Ojha
Dan van der Ster

Thank you to all candidates who participated and to every community member who cast a ballot.

Read the full election recap for additional details.

Meet the New Ceph Technical Writer ¶

Earlier this year, the Foundation announced its search for a Technical Writer to help improve the quality and accessibility of Ceph documentation. Following the interview process, the hiring committee selected Emmanuel Ameh for the role.

Emmanuel will work closely with contributors across the community to improve documentation, reduce onboarding friction for new users, and help make Ceph documentation easier to navigate and maintain.

If you are interested in joining the Ceph Tech Writing committee, please contact Emmanuel at eameh@contractor.linuxfoundation.org.

Read Emmanuel's introduction on the Ceph introduction blog to learn more about his background and goals for the role.

Ceph Days Continue to Grow ¶

Community events remain one of the most effective ways for Ceph users and contributors to connect, share experiences, and learn from one another.

During the second quarter, the community gathered for two Ceph Days events:

Ceph Days Seattle 2026 ¶

Date: May 28, 2026
Sponsors: OSNexus and CLYSO
Attendance: 30 community members

John Shubeck, a Senior Storage Technical Specialist for IBM Object Storage, shared a detailed recap of the recent Ceph Days event in a new blog post.

Ceph Days London 2026 ¶

Date: June 17, 2026
Lead Sponsor: IBM
Attendance: 120

Here are a few of the photos of the thriving Ceph community in London!

These events featured technical presentations, real-world deployment stories, community networking, and discussions about the future of Ceph.

The Foundation continues to focus on supporting regional events, meetups, and community-led initiatives that help expand Ceph's reach and introduce new users to the project.

Check the upcoming and tentative Ceph Days events for the rest of 2026.

Ceph, OpenInfra, and KCD Days Korea ¶

Lead Sponsors: Cloud Native Computing Foundation and OpenInfra
Date: Q3
URL: TBD

Ceph Days Zurich 2026 ¶

Lead Sponsor: CLYSO
Date: Oct 14, 2026
URL: https://ceph.io/en/community/events/2026/ceph-days-zurich/

Ceph Days New York ¶

Lead Sponsor: Bloombreg
Date: Q3
URL: TBD

Ceph Days Chicago ¶

Lead Sponsor: CLYSO, IBM, and OSNexus
Date: Q4
URL: TBD

Get Involved ¶

Interested in organizing a meetup, hosting a Ceph Day, speaking at an event, or promoting Ceph in your region? Community participation is what makes these events successful.

If you need support for a Ceph event, meetup, travel, or community campaign, submit a funding request to the Ceph Foundation. Requests are reviewed by the Governing Board and Linux Foundation staff and considered based on available budget and community impact.

Latest Ceph Releases ¶

The community published maintenance releases for both the Squid and Tentacle release series during the quarter.

Thank you to the many contributors, maintainers, testers, and reviewers who continue to support Ceph releases.

Community Committees: More Ways to Contribute ¶

Community committees are coming soon. These committee will provide opportunities for contributors to help shape the future of Ceph beyond code contributions alone. Whether your interests include events, documentation, outreach, onboarding, or community programs, there are many ways to participate.

Committee participation will be shared through our social channel and mailing list. Watch upcoming community communications for additional opportunities to join working groups and contribute to Foundation-led initiatives.

v20.2.2 Tentacle released

2026-06-16T00:00:00Z

This is the second minor release in the Tentacle series. We recommend that all users update to this release.

Release Date ¶

June 16, 2026

Notable Changes ¶

Rocky 10 package-based installs are now supported starting with v20.2.2. Please see the supported platforms for current and planned support in Ceph.

MDS (Metadata Server) ¶

Segmentation fault fixed due to incorrect queueing of request retries.

OSD (Object Storage Daemon) ¶

PGLog Missed List: Fixed a bug to ensure the correct version is attached to the missing list when ignoring log entries.
Data Integrity Asserts: Added assertions to explicitly catch potential corruption in the OSD missing list.

RGW (RADOS Gateway) ¶

Lifecycle Management: Fixed lifecycle transition issues affecting encrypted multipart objects.
REST & Query Handling: RESTArgs::get_string() now properly URL-decodes incoming query parameters.

RADOS / librados / neorados ¶

Linger Operations: Rewrote safety checks to eliminate use-after-free vulnerabilities and LingerOp memory leaks when an unwatch operation returns ENOTCONN. Replaced the flawed is_valid_watch() check with a safe linger_by_cookie() lookup which safely manages LingerOp references using intrusive_ptr. Ensured librados linger callbacks hold a persistent reference to LingerOp to protect against races with simultaneous linger_cancel() requests. Configured librados::IoCtxImpl::aio_unwatch() to asynchronously deliver ENOTCONN to AioCompletion instead of returning the error directly.
Watch/Notify: Fixed neorados notification queue bounds enforcement so that an overflow marker is only appended on the first message exceeding capacity, rather than duplicating it on every subsequent message. Prevented double-cleanup triggers in watch/notify operations when incoming errors arrive after maybe_cleanup() runs. Fixed a bug where notify would lose original error values by ensuring it no longer attempts to decode empty responses. Enhanced io_context shutdown procedures to clear handlers and route cleanly through linger_cancel to avoid use-after-free states.
Async Utilities: Fixed an issue regarding the improper removal of objects from the async service list.

Dashboard ¶

NVMeoF Revamp: Complete UI overhaul with DHCHAP controller key support, namespace encryption, and secure listeners configuration.
Pools & RGW: Added stretch cluster validation for pools, fixed RGW restart/stop issues, storage class restrictions, sync policy fixes, and MSR EC Profile support.
NFS: Toggle visibility for CephFS snapshots, fixed export creation and path value consistency issues.

ceph-volume ¶

Inventory Scanning: Enhanced inventory discovery logic to automatically skip RAM disk devices (/dev/ram*).

External Block Device (extblkdev) ¶

Plugin Stability: Fixed an assertion failure in the FCM plugin when encountering multivolume devices.

Changelog ¶

(tentacle) ceph-volume: backport PRs 67047 and 67240 (pr#67343, Guillaume Abrioux, Parfait Detchenou)
[tentacle] bluestore, extblkdev: Now plugins can raise health warnings (pr#68663, Adam Kupczyk, Igor Fedotov, Martin Ohmacht)
Backporting PRs 67236 and 67419 (pr#67533, Adam King)
Beacon diff + Stretched cluster (pr#68347, Leonid Chernin, Samuel Just)
ceph-volume: include LVM mapper devices in get_devices() (pr#67989, Guillaume Abrioux)
ceph-volume: skip /dev/ram* devices in inventory (pr#68552, Ujjawal Anand)
ceph-volume: skip mkfs discard for LVM NVMe OSDs (pr#68286, Ujjawal Anand)
ceph-volume: skip redundant NVMe mkfs discards (pr#67341, Ujjawal Anand)
ceph-volume: skip virtual cdrom devices in inventory (pr#68108, Ujjawal Anand)
ceph.spec.in: replace golang github prometheus with promtool binary path (pr#68420, Nizamudeen A)
ceph_mon: Fix shutdown order to destroy Monitor before closing mon store (pr#68399, Prashant D)
cephadm: wait for latest osd map after ceph-volume before OSD deploy (pr#68379, Guillaume Abrioux)
cephfs: MDCache request cleanup (pr#66469, Abhishek Lekshmanan)
Check if HTTP\_X\_AMZ\_COPY\_SOURCE header is empty (pr#66027, Suyash Dongre)
client: adjust Fb cap ref count check during synchronous fsync() (issue#73624, pr#65913, Venky Shankar)
client: crash caused by invalid iterator in _readdir_cache_cb (pr#65957, Zhansong Gao)
container/build.sh: add 'rocky-10' suffix if necessary (pr#67895, Dan Mick)
container/build.sh: FROM_IMAGE=rockylinux-10 default for >=tentacle backports (pr#67960, Matan Breizman, David Galloway, John Mulligan, Dan Mick)
debian: package mgr/smb in ceph-mgr-modules-core (pr#67510, Roland Sommer)
debian: remove invoke-rc.d calls from postrm scripts (pr#67354, Kefu Chai)
debian: remove stale distutils override from py3dist-overrides (pr#68276, Thomas Lamprecht, Max R. Carrara, Kefu Chai)
doc: Batch backport for start/hardware-recommendations.rst (pr#67907, Anthony D'Atri, Marc Methot, Pierre Riteau, Ville Ojamo)
extblkdev: Fix FCM plugin asserting on multivolume devices (pr#68877, Adam Kupczyk)
found duplicate series for the match group {fs_id="-1"} (pr#68369, bst2002git)
Implement Admin REST APIs for Setting Account Quota (pr#66905, Nicholas Liu, Jiffin Tony Thottan)
kv/RocksDB: Add instrumentation to BinnedLRUCache (pr#67349, Adam Kupczyk)
libcephsqlite: ensure atexit handlers are registered after openssl (pr#68263, Patrick Donnelly)
librbd/cache/pwl: WriteLogOperationSet::cell can be garbage (pr#67705, Ilya Dryomov)
librbd/migration/QCOWFormat: avoid use-after-free in execute_request() (pr#68188, Ilya Dryomov)
librbd/mirror: detect trashed snapshots in UnlinkPeerRequest (pr#67583, Ilya Dryomov)
librbd: avoid losing sparseness in read_parent() (pr#68463, Ilya Dryomov)
librbd: don't complete ImageUpdateWatchers::shut_down() prematurely (pr#67581, Ilya Dryomov)
librbd: rbd_aio_write_with_crc32c (pr#68038, Alexander Indenbaum)
mds: add ref counting to LogSegment (pr#68439, Milind Changire)
mds: add retry request to MDSRank wait queue rather via finisher (issue#76031, pr#68905, Venky Shankar)
mds: scrub pins more inodes than the mds_cache_memory_limit (pr#67808, Md Mahamudur Rahaman Sajib)
mds: use SimpleLock::WAIT_ALL for wait mask (pr#68313, Patrick Donnelly)
mgr/cephadm: Add KMIP server support for NVMeoF gateway (pr#68086, Gil Bregman)
mgr/cephadm: add rbd_with_crc32c parameter to nvmeof service spec (pr#66933, Alexander Indenbaum)
mgr/cephadm: fix mgmt-gateway startup on IPv6 VIP (pr#68387, Kobi Ginon)
mgr/cephadm: include cluster FSID in root CA Common Name (CN) (pr#64692, Redouane Kachach, Kushal Deb)
mgr/cephadm: serialize OSD class before returning for OSD rm status (pr#69226, Adam King)
mgr/DaemonServer: Implement ok-to-upgrade command (pr#66948, Sridhar Seshasayee)
mgr/DaemonServer: Limit search for OSDs to upgrade within the crush bucket (pr#68350, Sridhar Seshasayee)
mgr/dashboard : Add bottom padding for dashboard screens (pr#68523, Abhishek Desai)
mgr/dashboard : add stretch cluster validation for pools form (pr#68476, Afreen Misbah, Abhishek Desai)
mgr/dashboard : Fix RGW restart/stop issue (pr#68554, Abhishek Desai)
mgr/dashboard : fix-non-versioning-bucket-issue (pr#68520, Abhishek Desai)
mgr/dashboard : Fixes EC profile used pool empty (pr#68730, Abhishek Desai)
mgr/dashboard : Restrict create storage class with existing name (pr#68475, Abhishek Desai)
mgr/dashboard: [RGW-NFS]: User level export creation via UI fails with 500 - Internal Server Error (pr#67013, Dnyaneshwari Talwekar)
mgr/dashboard: [snap-visibility]Edit Client config option remains stuck in loading when nfs user is configured (pr#68542, Dnyaneshwari Talwekar)
mgr/dashboard: [storage-class]: Deleting local storage class from UI does not remove its entry from zone (pr#67949, Dnyaneshwari Talwekar)
mgr/dashboard: Add DHCHAP controller key to NVME host commands (pr#67600, Gil Bregman)
mgr/dashboard: add helper text to bucket form > policy and other spacing fixes (pr#67871, Naman Munet)
mgr/dashboard: Add location to gateway info command in NVMeoF CLI (pr#68345, Gil Bregman)
mgr/dashboard: Add namespace encryption support to NVMeoF CLI (pr#68339, Gil Bregman)
mgr/dashboard: Add nvmeof_top_cli service (pr#67566, Vallari Agrawal)
mgr/dashboard: Add option to edit zone with keys/ (pr#68317, Aashish Sharma)
mgr/dashboard: Add option to set motd via api (pr#68678, Aashish Sharma)
mgr/dashboard: Add overview page (pr#67840, Afreen Misbah, Devika Babrekar, Aashish Sharma, Abhishek Desai, Naman Munet, Nizamudeen A)
mgr/dashboard: Add port and secure-listeners to subsystem add NVMeoF CLI command (pr#68370, Vallari Agrawal, Gil Bregman)
mgr/dashboard: Add restore events in notification screen (pr#67912, pujashahu, pujaoshahu)
mgr/dashboard: Add secure and verify-host-name to "listener add" on NVMeoF CLI (pr#67799, Gil Bregman)
mgr/dashboard: Adding rados ns option into add_ns_req (pr#67470, gadi-didi)
mgr/dashboard: Allow empty port value when adding a listener in NVMEoF CLI (pr#68766, Gil Bregman)
mgr/dashboard: Batch backport nvmeof revamp (pr#67839, Afreen Misbah, Nizamudeen A, Sagar Gopale, pujaoshahu, Puja Shahu, Ville Ojamo)
mgr/dashboard: Bump lodash (pr#68695, Afreen Misbah)
mgr/dashboard: bump nvmeof submodule to 1.6.5 (pr#67326, Vallari Agrawal, Tomer Haskalovitch)
mgr/dashboard: carbonize-osd-flags-modal (pr#68133, Afreen Misbah, Sagar Gopale)
mgr/dashboard: Difference in "path" value observed when rgw user level export created via dashboard vs cli (pr#68583, Dnyaneshwari Talwekar)
mgr/dashboard: fix subvolume group corruption from smb share form (pr#68103, Nizamudeen A)
mgr/dashboard: Fix tags in subvolume list and subvolume groups list (pr#68382, pujaoshahu)
mgr/dashboard: Making 'ISA' as default plugin for EC profiles created through dashboard (pr#68373, Devika Babrekar)
mgr/dashboard: mgr/dashboard: Carbonize Realm Name and Token block in Multi-site Replication Wizard (pr#68546, Sagar Gopale)
mgr/dashboard: Misleading warning when no eligible devices are available for OSD creation (pr#67637, Naman Munet)
mgr/dashboard: nfs export creation fails with obj deserialization (pr#67564, Nizamudeen A)
mgr/dashboard: NFS: Toggle visibility of CephFS snapshots (pr#67636, Afreen Misbah, Dnyaneshwari Talwekar)
mgr/dashboard: Option to select archive option while Import Multi-site Token (pr#68513, Aashish Sharma)
mgr/dashboard: remove sync_from entry when sync_from_all is true (pr#68536, Aashish Sharma)
mgr/dashboard: Rename Alert breadcrumb to Alert Rules (pr#68238, Sagar Gopale)
mgr/dashboard: sync policy created for a bucket in Object >> Multi-site >> Sync-policy, is not reflecting under bucket's replication (pr#68512, Naman Munet)
mgr/dashboard:Adding MSR EC Profile via dashboard (pr#68349, Devika Babrekar)
mgr/smb: fix error handling for fundamental resource parsing (pr#65861, John Mulligan)
mgr/test_orchestrator: fixing daemon_action method signature (pr#69231, Redouane Kachach)
mgr: add config to load modules in main interpreter instead of subinterpreter (pr#67515, Nitzan Mordhai, Nitzan Mordechai, Samuel Just)
mgr: ensure that all modules have started before advertising active mgr (pr#67850, Laura Flores)
mgr: fix continous smb MgrDBNotReady (pr#68598, Pedro Gonzalez Gomez)
mgr: fix PyObject* refcounting in TTLCache and cleanup logic (pr#66482, Nitzan Mordechai)
mgr: guard close_section calls in get_perf_schema_python (pr#69436, Lumir Sliva)
mgr: isolated CherryPy to prevent global state sharing (pr#67465, Nizamudeen A, Anmol Babu)
mon [stretch-mode]: Allow a max bucket weight diff threshold (pr#67790, Kamoltat Sirivadhna, Kamoltat (Junior) Sirivadhna)
mon/AuthMonitor: add osd w cap for superuser client (pr#68314, Venky Shankar, Patrick Donnelly)
monitoring: Fix application overview to show Raw used (pr#68794, Ankush Behl)
mr/dashboard: remove rgw_servers filter from radosgw-sync-overview grafana dashboard (pr#68604, Aashish Sharma)
neorados: Fix Neorados CephContext leak (pr#67513, Adam C. Emerson)
neorados: specify alignments for aligned_storage (pr#67512, Casey Bodley)
neorados: Various Fixes to Watch/Notify (pr#68776, Adam C. Emerson, Casey Bodley)
node-proxy: major refactor and various fixes (pr#67418, Guillaume Abrioux)
nvmeof: Change the NVMEOF image version to 1.6 (pr#68415, Gil Bregman)
nvmeofgw: fix issue of delete all gws from the pool/group (pr#67942, Leonid Chernin)
orch/cephadm: fix osd.default creation (pr#68121, Guillaume Abrioux)
os/bluestore: track compression\_*blob_size* parameters for online update (pr#67888, Igor Fedotov)
os/bluestore:fix bluestore_volume_selection_reserved_factor usage (pr#66837, Igor Fedotov)
osd/scrub: support an operator-abort command (pr#67031, Ronen Friedman)
osd: add pg-upmap-primary to clean_pg_upmaps (pr#67407, Laura Flores)
osd: Allow multiple objects with same version in missing list (pr#69450, Alex Ainscow)
osd: Avoid assertion on empty object read when reading multiple objects (pr#68714, Alex Ainscow)
osd: Avoid pwlc spanning intervals (pr#68708, Bill Scales)
osd: Change rmissing map key from version_t to eversion_t (pr#68716, Alex Ainscow)
osd: Deleting PG should discard pwlc (pr#68709, Bill Scales)
osd: FastEC: always update pwlc epoch when activating (pr#68710, Bill Scales)
osd: Fix bug when calculating min_peer_features (pr#69159, Bill Scales)
osd: Fix incorrect rollback logic for partial write OI (pr#68715, Alex Ainscow)
osd: PGLog Attach correct version to missing list when ignoring log entries (pr#68718, Alex Ainscow)
osd: Twiddle should create a full sized vector for optimized EC (pr#68717, Alex Ainscow)
pybind/mgr: call new _ceph_exit for killpoints (pr#68518, Patrick Donnelly)
pybind/mgr: update modules to use independent CLICommand subtypes with distinct COMMAND attributes (pr#67511, Kefu Chai, Samuel Just)
qa/cephadm: derive container image from cephadm release (pr#68328, Lumir Sliva)
qa/cephfs: lua to respect missing kernel in yaml (pr#67293, Kyr Shatskyy)
qa/cephfs: treat "implicit declaration of function" for blogbench workunit for newer gcc version (issue#75380, pr#68820, Venky Shankar)
qa/distros: add rocky 10.0 as supported distro/container host (pr#68569, Patrick Donnelly, Casey Bodley, Adam King, David Galloway)
qa/radosgw_admin: replace boto2 with boto3 (pr#68739, Adam C. Emerson, Casey Bodley)
qa/rgw/multisite: remove duplicate test_suspended_delete_marker_incremental_sync (pr#68846, Oguzhan Ozmen)
qa/rgw/upgrade: symlinks are explicit about distro versions (pr#68057, Casey Bodley)
qa/rgw: Revive crypt kmip (pr#68371, Kyr Shatskyy)
qa/suites/fs: fix extraneous distro links (pr#69251, Patrick Donnelly)
qa/suites/upgrade: ignore osd in unknown state (pr#69307, Patrick Donnelly)
qa/suites/upgrade: ignore undersized PG during stress splits (pr#69310, Patrick Donnelly)
qa/suites/upgrade: update upgrade paths and exclude rocky10 from non-supported distros (pr#68660, Yaarit Hatuka, Laura Flores)
qa/suites: remove centos restriction from valgrind yaml (issue#18126, issue#20360, pr#67519, Samuel Just)
qa/suites: use tagged version of reef (pr#68357, Laura Flores)
qa/tasks/backfill_toofull.py: Fix assert failures with & without compression (pr#68118, Sridhar Seshasayee)
qa/tasks/keystone: upgrade keystone to 2025.2 (pr#67757, Kyr Shatskyy)
qa/tasks/quiescer: remove racy assertion (pr#68510, Patrick Donnelly)
qa/tasks: capture CommandCrashedError when running nvme list cmd (pr#69232, Redouane Kachach)
qa/tasks: make rbd_mirror_thrash inherit from ThrasherGreenlet (pr#67795, Ilya Dryomov)
qa/tasks: update egrep to 'grep -E' (pr#67518, Nitzan Mordechai, Samuel Just)
qa/workunits/ceph-helpers-root: Add Rocky support for install packages (pr#67507, Nitzan Mordechai)
qa/workunits/rbd: drop racy assert in test_tasks_recovery() (pr#68190, Ilya Dryomov)
qa/workunits: Add updated kernel archive URL (pr#68169, Brad Hubbard)
qa: add 'refresh' config to cephadm.wait_for_service (pr#67038, Vallari Agrawal)
qa: add MDS_INSUFFICIENT_STANDBY to ignorelist (pr#69036, Patrick Donnelly)
qa: Add nvmeof upgrade from v20.2.0 (pr#68149, Vallari Agrawal)
qa: allow multiple mgr sessions during eviction test (pr#68316, Patrick Donnelly)
qa: cephfs suite changes for rocky (pr#68374, Patrick Donnelly)
qa: Fix coredumps caused by udisks (pr#67526, Vallari Agrawal)
qa: Fix nvmeof 'errors during thrashing' (pr#68148, Vallari Agrawal)
qa: fix setting rbd_sparse_read_threshold_bytes in test_migration_clone() (pr#68617, Ilya Dryomov)
qa: fix TypeError in delay (pr#67617, Jos Collin)
qa: fixing cephadm mgmt-gateway test to remove openssl dependency (pr#67820, Redouane Kachach)
qa: ignore cephadm failed daemon warnings during thrashing (pr#69309, Patrick Donnelly)
qa: ignore POOL_FULL for rbd tests exercising full pools (pr#69304, Patrick Donnelly)
qa: install nvme-cli only if distro remains rocky10 (pr#69252, Patrick Donnelly)
qa: krbd_rxbounce.sh: do more reads to generate more errors (pr#67455, Ilya Dryomov)
qa: Leak_StillReachable RocksDB error_handler (pr#68524, Nitzan Mordechai)
qa: rbd_mirror_fsx_compare.sh doesn't error out as expected (pr#67797, Ilya Dryomov)
qa: Remove cephadm e2e tests from teuthology (pr#68818, Afreen Misbah)
qa: resolve py3.12 regression for random.sample (pr#68315, Patrick Donnelly)
qa: suppress false positive delete map mismatch errors (pr#68431, Casey Bodley, Nitzan Mordechai)
qa: suppress MismatchedFree operator delete RocksDB (pr#67508, Nitzan Mordechai)
rgw/beast: use strand executor for timeout timer to prevent concurrent socket access (pr#68506, Oguzhan Ozmen)
rgw/lc: Do not delete DM if its at end of pagination list (pr#67573, kchheda3)
rgw/multisite: check the local bucket's versioning status when replicating deletion from remote (pr#66168, Jane Zhu)
RGW/multisite: fix bucket-full-sync infinite loop caused by stale bucket_list_result reuse (pr#67923, Oguzhan Ozmen)
RGW/Multisite: fix uninitialized LatencyMonitor causing spurious "OSD cluster is overloaded" warning (pr#68803, Oguzhan Ozmen)
rgw/s3: Always include x-amz-content-sha256 header in AWS v4 signatures (pr#66358, Shilpa Jagannath, Matthew N. Heler)
rgw/tests: add os-specific java 1.7 install commands to keycloak task (pr#67982, J. Eric Ivancich)
rgw/website: preserve nameservers for future use in dnsmasq (pr#67061, Kyr Shatskyy)
rgw/zone: remove duplicated startup logic in RGWSI_Zone (pr#66300, Casey Bodley)
rgw: bucket logging fixes (pr#66769, Nithya Balachandran, N Balachandran, Yuval Lifshitz, Casey Bodley)
RGW: Change prerequest hook to run after authorization process (pr#68594, Emin Sunacoglu)
rgw: discard olh\_ attributes when copying object from a versioning-suspended bucket to a versioning-disabled bucket (pr#65557, Jane Zhu)
rgw: fix 'bucket stats' when bucket index doesn't exist (pr#68505, Casey Bodley)
rgw: fix lifecycle transition of encrypted multipart objects (pr#68826, Marcus Watts)
rgw: handle plain-text object tags in RGWObjTags::decode() (pr#67927, Oguzhan Ozmen)
rgw: java s3-tests change setting of JAVA_HOME (pr#68226, J. Eric Ivancich)
rgw: read_obj_policy() consults s3:prefix when deciding between 403/404 (pr#68651, Casey Bodley)
RGW: remove custom copy constructor for RGWObjectCtx and enforce no copy/move (pr#67440, Oguzhan Ozmen)
src/ceph-volume: fast device unavailable as error (pr#67916, Timothy Q Nguyen)
test/rgw/kafka: fix kafka relase to more recent one (pr#67993, Yuval Lifshitz)
test/rgw/lua: ignore hours for zero mtime (pr#67468, Kyr Shatskyy)
test/rgw/notification: do not use netstat in the code (pr#68142, Yuval Lifshitz)
test/rgw/notification: fix the cloudevents package version (pr#68704, Yuval Lifshitz, Adam C. Emerson)
test/rgw: remove depracated boto dependency (pr#68344, Yuval Lifshitz)
test: use json_extract instead of awkward json_tree (pr#67321, Patrick Donnelly)
This change introduces the shared memory communication (SMC-D) for the cluster network (pr#68254, Aliaksei Makarau)
tools/ceph-kvstore-tool: fix crash on db close (pr#68406, Igor Fedotov, Max Kellermann)
upgrade suites update for Rocky10 and missing centos (pr#68733, Nitzan Mordechai)

Ceph Days Seattle 2026 - Cloud Transition to Tape

2026-06-15T00:00:00Z

I'm John Shubeck, an information technology professional with over 44 years of industry experience spanning both the customer and technology provider experience. I'm currently serving as a Senior Storage Technical Specialist for IBM Object Storage platforms across all market segments in the Americas.

On Thursday, May 28th, 30 members, myself included, of the Ceph faithful descended on the SURF business incubator in downtown Seattle for "Ceph Days Seattle 2026". Following the traditional Ceph Days agenda, the Seattle program consisted of a series of "Ceph Talks" designed to introduce new ideas and share lessons learned.

Ceph Days are community-driven events designed to bring together Ceph users, developers, architects, administrators, and anyone interested in open-source storage. Held throughout the year in locations around the world, these one-day gatherings offer a unique opportunity to learn from real-world deployments, explore new technologies, connect with peers, and contribute to discussions that help shape the future of Ceph.

Did you know? ¶

Ceph is a software-defined, hardware-independent storage solution that provides block, file, and object storage on a single unified platform.
Ceph is championed by developers, administrators, users, IT leaders, and Fortune 500 enterprise customers. It is a living, vibrant, and active group that has adopted Ceph in its IT operations.
The Ceph project and community are supported by a Foundation (https://ceph.io/en/foundation/) comprising organizations, stakeholders, and industry leaders who collaborate to coordinate investment, development, and community activities for Ceph.

With the setting, plot, and characters stated above, we turn to today's topic. A key activity of the Ceph community is a series of "Ceph Days" conferences. A Ceph Day is a full-day meetup where participants learn, network, make new acquaintances, and share ideas on how to enhance the value of Ceph. Sometimes IBM participates by providing the venue, a guest speaker or two, or the all-important refreshments. At the same time, make no mistake, Ceph Days are organized by the Ceph community, for the community, to support the community.

Here is where I came in. For the first time, I had the honor of being selected to present one of my favorite topics, Ceph Cloud Transitions. And to raise the stakes even further, Ceph Cloud Transition to S3-enabled TAPE! During my 45 minutes on stage, I covered the theory of cloud transitioners and showed a LIVE demonstration of a Ceph cloud transition to IBM Deep Archive. In this demonstration, I proved that static, unused, and potentially outdated object storage content can be tiered off to tape. In the end, the demonstration posed a radical concept: an object store without artificial boundaries and virtually limitless!

If you would like to see a 10-minute lightning round video of the demonstration, click here:

https://www.youtube.com/watch?v=SVQwkjFF.uWU

I look forward to the Ceph Days Seattle 2026 presentations and other Ceph Talk videos in the near future, which will be posted to:

https://www.youtube.com/@Cephstorage

Feel free to connect with me on LinkedIn:

https://linkedin.com/jshubeck

and on YouTube:

https://youtube.com/@johnshubeck2026

Get Involved ¶

If my experience sounds like something you'd like to try, I encourage you to attend a Ceph Day, join a community meeting, or share your own experiences through a Ceph Tech Talk. The Ceph community welcomes users, developers, and storage enthusiasts of all experience levels. To learn about upcoming events, meetups, and ways to get involved, visit the Ceph Community page at https://ceph.io/community. I hope to see you at a future Ceph Days event.

For whom the door-bell tolls

2026-06-11T00:00:00Z

For whom the door-bell tolls

In a previous post we extolled the benefits of KV caching, a technique to save the KV states from the prefill step of LLM-based inference to reduce time to first token (TTFT) and skip redundant computation. I co-presented this with Tushar Gohad at Cephalocon. Since then I've been thinking a lot about how to improve the state of the art. Really move the needle. We've made strides in a lot of areas in Ceph, especially the work going into Fast EC — if you don't know what I'm talking about you should check it out, it promises huge benefits to a large category of workloads. That's not what we're here for today, though.

Last year there was one paper that stuck in my mind, which is fairly remarkable because I read on the order of 130. That paper was GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture. I struggled with this. It describes a system where a CPU loads a kernel into the GPU that allows the GPU to serve as an NVMe initiator. The struggle was rooted in the fact that block just felt like the wrong interface for KV caching. If you use block, then you need a lookup table that maps the hash of the sequences representing a cache block to a particular (device, offset, length) tuple. It begs for a content-addressable approach with no centralized lookup or coordination. On the other hand, what was described in the paper was just flat-out more electrically efficient. I couldn't have my cake and eat it too.

The idea: RADOS, spoken as NVMe key-value ¶

In 2025 we also saw the first ratified version of the NVMe key-value command set specification. We already have a Ceph implementation of NVMe/TCP that leverages SPDK, and it seemed like we could add support fairly easily for the key-value command set.

For those who don't know the internal mechanics of Ceph, the native API operates against RADOS objects. RADOS is richer than most object stores: it supports reads and writes to arbitrary offsets, deletes, key-value storage (OMAP), watch/notify notifications, and a class system to execute code against objects (cls). How could we find an intersection between the full power of RADOS and GPU-initiated storage?

What I came up with is exposing RADOS via NVMe key-value commands:

NVMe-oF concept	RADOS concept
Subsystem	Pool
Namespace	Namespace (in pool)
Key (1–16 bytes)	Object name (oid)
Value	Object data

The other fun idea — hold onto this one, it comes back — was a vendor-specific command extension to allow the execution of classes against objects. The classic example is thumbnailing an image, but it could be something more interesting, like extracting an embedding. Computational-storage stuff, executed where the data is at rest instead of dragging it across the network to be executed on some other host. We probably don't want random applications running any class against any object, though, so let's have an out-of-band gRPC server that provides a control interface to allowlist particular classes to specific namespaces, and map those allowed classes to opcodes.

Building rados-nkv (overnight, by agents) ¶

It was the afternoon of June 4. I had the start of a plan and the NVMe key-value specification, and it was time to /grill-me-with-docs. After some Q&A with my agentic pals, I decomposed the work into a task DAG, with an adversarial review task paired to each coding task — sub-agents with bounded contexts, arguing with other sub-agents for better quality, a coordinator agent running the merge queue and dispatching work as it unblocked. I set the whole thing on a loop, then went to walk the dog and peek in on the agentic factory from my phone. There's a pleasing recursion here: I was building infrastructure for AI workloads using AI workloads.

The first agent made a surprising find: NVIDIA had recently added an NVMe KV client to SPDK. That undoubtedly made things easier — we had a reference client to work against. I was tapped out, time for me to sleep, while the factory whirs.

The next morning I grabbed my coffee and checked in to see what sort of breakfast had been prepared. I have to say, the dish was perfectly plated, and the flavor was like a breakfast burrito from a San Diego taqueria. On this day rados-nkv was born — a complete SPDK controller exporting a RADOS-backed NVMe key-value namespace over vfio-user. Tested end-to-end. Logically, it looks like this:

The KV control path, proven over a vfio-user loopback — no VM, no network. Key → object, one RADOS object per KV pair, tenancy carried by the RADOS namespace.

We'll need some things to talk to this newfangled key-value namespace, of course. The first order of business is a rados-nkv NIXL plugin, since NIXL is used by both llm-d-kv-cache and LMCache to offload KV-cache blocks from vLLM over the kv-connector interface. For bonus points, we'll whip up rados-nkv-weights to let vLLM stream model weights from rados-nkv.

A flashback to BaM ¶

Around this time a colleague asked me if I was going to SNIA's SDC, which triggered me to scan the sessions on the docket. Serendipitously, I saw Stephen Bates's abstract: GPU-Initiated NVMe: Cutting the CPU Out of the Accelerator-to-Storage Data Path. This gave me an immediate flashback to the BaM paper. The rados-nkv prototype was working, and as-is it would be a big improvement for AI workloads — but what if we could realize GPU-initiated IO to it? DMA-buf changes are landing in the Linux kernel that might just open up a viable path.

I pinged Stephen on LinkedIn, who indicated that ROCm XIO was the droid I was looking for.

Up to this point my mad science had been conducted from the meek confines of a cloud VM. I'd need a physical machine with a real AMD GPU.

A monetary value on cached tokens ¶

I'd been eyeballing Strix Halo Mini PCs as a development machine for x86 builds of Ceph, that would also let me experiment with vLLM. This dovetails into another idea I'd been spitballing with the llm-d folks: adding explicit caching to the serving API and wiring it into the kv-connector. The idea is simple — everyone gets automatic prefix caching, and agentic harnesses can explicitly request caching of prefixes associated with system prompts and other repeated context, with an optional TTL.

Doing so provides a mechanism to associate a monetary value with cached token-hours, whether for internal chargeback or as an additional service offering for model- or token-as-a-service providers. You need controls like this — or some sort of tenant-partitioned cache — to stop enterprising engineers and their agents from setting up a /loop that keeps their prefixes hot and pushes out the contexts of less Machiavellian engineers. If that's not a good enough reason for you, maybe the fact that Google and Anthropic do it is.

Anyway, I started to wonder: could I pass the iGPU from a Strix Halo box into a QEMU guest? I stumbled on a guide by a Proxmox user that suggested it just might work. I cajoled my wife into authorizing a trip to Microcenter to pick up a Strix Halo Mini PC.

Setup, and a deliberately boring first target ¶

Had to rummage around for a USB stick to boot from, only to find two paltry 512 MB ones. I resorted to cannibalizing my GoPro's microSD card. Fedora 44 was the weapon of choice, since I needed a kernel with all the recent dma-buf goodies. Imaged. After a brief side quest benchmarking Kimi-Linear-48B-A3B-Instruct with a Q6_K quant using llama.cpp — and pointing Claude Code at the Anthropic-compatible messages API for funsies — it was time to get back to business.

First up: take a rados-nkv device and pass it to a guest via vfio-user. Working. Next: pass the iGPU into the guest. There are several tricks. The main one is that the host cannot touch any of the associated PCIe devices, so you blacklist the kernel modules that interact with the iGPU. You also extract the AMD driver and vBIOS ROM files to pass as PCIe devices to the guest. I set up a "no GPU" boot option as a fallback, because the moment the kernel loads with no video driver, you've got a headless node. Rebooted the host, connected remotely, and configured GPU passthrough with vfio-pci. I now had a guest with both a rados-nkv namespace and an 8060S GPU.

As much as I wanted to go straight for the prize — GPU-initiated IO to a key-value namespace — it was wiser to start boring: pass a conventional NVMe block namespace backed by RBD. GPU-initiated IO to a block namespace is already a known quantity, so it makes the perfect dress rehearsal: if it breaks, the bug is in my plumbing, not in anything novel.

How ROCm XIO drives a queue pair ¶

ROCm XIO is the piece that lets GPU code drive an NVMe queue pair. Its nvme-ep ("endpoint") runs a __device__ kernel that:

Writes an NVMe Submission Queue Entry (SQE) — a 64-byte command — into the SQ.
Rings the SQ-tail doorbell to tell the controller "go."
Spins on the Completion Queue, watching the phase bit flip.
Rings the CQ-head doorbell to acknowledge the completion.

A companion kernel module (/dev/rocm-xio) handles the privileged glue: pinning queue memory, intercepting the kernel NVMe driver's admin commands to inject the right physical addresses, and exporting GPU VRAM buffers for peer-to-peer DMA. Long term the upstream NVMe Linux driver might be updated to accommodate userspace requests for IO queues, sparing this interception dance — but that's a discussion for another day. Crucially, the queues, doorbells, and data buffers can each live in host RAM or GPU VRAM, selected by a 4-bit --memory-mode mask:

bit	`0x1`	`0x2`	`0x4`	`0x8`
structure	SQ	CQ	doorbell	data buffer

Mode 0x0 = everything in host RAM. Mode 0x8 = flip just the data buffer into VRAM, so the payload P2P-DMAs straight into GPU memory. Mode 0x8 is the whole point; mode 0x0 is the "make it work first" baseline.

On the host, SPDK's nvmf_tgt presents an emulated NVMe controller over the vfio-user protocol: a Unix socket that speaks the VFIO device model. QEMU connects with a vfio-user-pci device, and the guest sees a perfectly ordinary /dev/nvme0. The clever bit — which becomes central later — is that SPDK advertises the NVMe doorbell registers (BAR0 offset 0x1000+) as a sparse-mmap'd shared page by default. The client mmaps it; doorbell writes land in shared memory; SPDK's poll loop reads the new tail value. No socket round-trip per doorbell.

The spell fizzles ¶

Excitement builds. It's time to summon GPU-initiated IO. The spell is cast. Fizzle.

Turns out the GPU and the QEMU-emulated doorbell live in alternate dimensions: the GPU needs to peer-write to a physical BAR, and our NVMe device is stranded in the world of software-defined wonder. I needed a portal — a bridge between these two dimensions. A light bulb went off in my head. Or a doorbell rang. Something of the sort.

There was another SDC session Stephen had on the agenda:

This talk presents a different approach: a standalone userspace server that emulates a complete PCIe RDMA NIC via the VFIO-user protocol. Built on Nutanix's open-source libvfio-user library, the server exposes PCI BARs to an unmodified QEMU/KVM guest, which loads a companion kernel driver and a custom rdma-core provider. Applications in the guest see a standard InfiniBand device and use ordinary libibverbs calls — ibv_post_send, ibv_poll_cq, perftest, iperf3 — with no guest-side awareness that the device is emulated.

After some sleuthing, a QEMU branch Stephen had created was discovered, and in it was something wonderful: pcie-mmio-bridge support. The power to bind physical hardware to the software-defined world was in my hand, and I was ready to abuse the heck out of it. Rolled it into a binary, and it was game time. The moment of truth. Fizzle.

NVMe CQ poll timeout after 10000000 polls (cq_head=0 expected_phase=1)
:0:rocdevice.cpp :3587: Queue aborting with error :
HSA_STATUS_ERROR_EXCEPTION code: 0x1016 rocm-xio ERROR: nvme-ep:
gpuKernel sync failed for queue 8 (719)

The GPU built its command, rang the doorbell, and then spun forever waiting for a completion that never came. The 0x1016 GPU exception is a symptom: it's the __builtin_trap() the kernel hits after the CQ-poll loop gives up. The real question is why no completion was ever posted.

I had a half-dozen plausible theories: wrong memory mode, a doorbell-stride mismatch, hipHostRegister-of-MMIO not routing GPU writes correctly on an integrated GPU, SPDK not being P2P-capable into VRAM, CQ-phase logic, the SMU being flaky. Most of them were wrong. All of them were expensive to test. Brutal, even.

On this APU, the GPU's SMU (System Management Unit) is platform-owned. Passing the iGPU through and then having a guest fault on it leaves the SMU in a state the next guest cannot re-initialize:

amdgpu: PSP create ring failed!
amdgpu: hw_init of IP block <psp> failed -22

Only a full host reboot clears it. And — the part that really hurts — even a successful run dirties the SMU on guest teardown. So the budget was literally one GPU shot per host reboot.

The bug hunt: prove, don't try ¶

Guess-and-check was not an option. With a one-shot-per-reboot budget, the methodology had to change from "try things" to "prove things."

Step 1 — Trace all three codebases at the source level ¶

Before touching the GPU again, we read the actual code on all three sides of the completion path and confirmed it was correct by construction:

QEMU bridge: forwards via address_space_write to pci_get_bar_addr(target, bar) + offset — model-agnostic, reaches any BAR including a vfio-user one.
SPDK lib/nvmf/vfio_user.c: doorbells at 0x1000+ are sparse-mmap'd; the poller reads *sq_dbl_tailp from that shared page every tick; post_completion writes the CQE to guest memory and flips the phase bit with no interrupt required (a polling client sees it). Offsets matched: SQ tail at 0x1000 + 2·qid·4, exactly what rocm-xio computes.

Conclusion: on paper, the entire path works. So the bug was dynamic, not structural.

Step 2 — A reboot-free, GPU-free isolation test ¶

Here's the trick that broke the problem open. The GPU and the bridge↔SPDK plumbing are separable. So we wrote a tiny CPU program (cpu_bridge_probe) that maps the bridge's shadow ring and drives it from the CPU — no GPU, no passthrough, no reboot. It issued the same ring commands the GPU would, and read NVMe controller registers back through the bridge:

[1] READ NVMe VS   (BAR0 0x08) -> COMPLETE value=0x20000 (NVMe 2.0)
[2] READ NVMe CAP  (BAR0 0x00) -> COMPLETE value=0x80000201e0100ff
[3] READ NVMe CSTS (BAR0 0x1c) -> COMPLETE value=0x1 (RDY=1)
[4] WRITE SQ doorbell (0x1008) -> COMPLETE
VERDICT: bridge<->SPDK round-trip WORKS from CPU

Those VS/CAP/CSTS reads come from SPDK's non-mmap'd register region, so they did a full vfio-user socket round-trip to the controller and back. The bridge, vfio-user-pci, and SPDK were proven flawless — from the CPU. Which meant the bug was specifically in the GPU's interaction with the shadow ring. We had cut the problem in half without spending a single GPU boot.

Step 3 — Instrument, don't guess ¶

The QEMU bridge already had trace events. We enabled pci_mmio_bridge* tracing and took one instrumented GPU shot. The trace was damning — and surprising:

338 × pci_mmio_bridge_invalid_command command=0
  1 × pci_mmio_bridge_poll_processed   processed 338 commands
  0 × pci_mmio_bridge_write

The bridge processed 338 zeroed commands in a single poll — and, critically, this happened even on a boot where amdgpu failed to initialize and the GPU kernel never ran. That single observation exonerated the GPU entirely. The garbage wasn't coming from the GPU at all. Something else was writing a bogus producer_idx over an all-zero command region.

Bug #1 — a guest-physical address collision ¶

The shadow ring lived at shadow-gpa = 0x80000000. The guest console told the whole story:

pci_bus 0000:00: root bus resource [mem 0x80000000-0xdfffffff window]
pci 0000:00:01.0: [1234:1111] class 0x030000 BAR 0 [mem 0x80000000-0x80ffffff pref]

0000:00:01.0 is the QEMU standard VGA framebuffer (the boot display). With 2 GB of guest RAM, the 32-bit PCI MMIO window starts at 0x80000000 — and the VGA's 16 MB framebuffer BAR landed exactly on top of the shadow ring. The bridge was faithfully polling the VGA framebuffer and interpreting pixels as ring metadata. Every GPU doorbell write was also vanishing into the framebuffer. The GPU-free CPU test had worked only because its device layout left 0x80000000 clear.

The fix: move shadow-gpa out of the PCI window entirely, into the free gap above 4 GB:

- shadow-gpa=0x80000000
+ shadow-gpa=0x100000000   # above 2GB RAM, above the 32-bit PCI window,
                           # below the 64-bit PCI window (0x380000000000)

We validated this reboot-free with the CPU probe at the new address: clean ring (producer=0, no garbage), all reads round-tripping. None of the GPU-coherence theories had been right. It was an address-space overlap.

Bug #2 — 1024 > 256 ¶

With the collision gone, the next GPU shot produced the first clean doorbell:

pci_mmio_bridge_write BDF=0x0028 BAR=0 offset=0x1040 value=0x1 size=4

A real, valid SQ-tail doorbell for queue 8 (0x1000 + 2·8·4 = 0x1040), forwarded perfectly to the NVMe BAR. The GPU→shadow→bridge→SPDK path was working. But still no completion — and now SPDK's log said exactly why:

handle_create_io_q: *ERROR*: invalid I/O queue size 1024
handle_create_io_q: *ERROR*: invalid I/O queue size 1024
handle_del_io_q:    *ERROR*: I/O sqid:8 does not exist

Recall CAP = 0x80000201e0100ff from the CPU probe. The low 16 bits — MQES, Maximum Queue Entries Supported — are 0x00ff = 255, i.e. a max queue size of 256. But rocm-xio hard-codes 1024-entry queues. SPDK rejected both CREATE_IO_CQ and CREATE_IO_SQ, so queue 8 never existed — and the GPU's perfectly-delivered doorbell rang for a queue that wasn't there.

The fix (SPDK-side, no GPU rebuild): raise the controller's advertised queue depth.

nvmf_create_transport -t VFIOUSER -q 1024 -m 16

Again validated reboot-free with the CPU probe — CAP went ...0100ff → ...0103ff, MQES 255 → 1023. The controller would now accept 1024-entry queues.

Two bugs. A framebuffer overlap and a queue-size ceiling. Neither was CQ handling, memory mode, hipHostRegister, SMU, or GPU coherence — every exotic theory we'd been afraid of was wrong.

The payoff: GPU-initiated reads, from RADOS, into VRAM ¶

First GPU-initiated read ¶

With both fixes, the next boot:

=== *** FIRST GPU-INITIATED NVMe READ *** ===
Iterations: 50
Test completed successfully!

And the bridge trace showed a fully GPU-driven queue pair — SQ-tail doorbells at 0x1040 and CQ-head doorbells at 0x1044, counts marching up as the GPU submitted reads and reaped completions on its own.

Now from RADOS ¶

Swap the Malloc backend for bdev_rbd pointed at rbd/gpuimg, and the RADOS counters prove the bytes came from Ceph:

RD_OPS 65 → 249     RD 49 KiB → 2.1 MiB     (during the GPU run)

The gfx1151 iGPU drove NVMe block reads from __device__ code → the bridge → SPDK → librbd → a RADOS OSD. A GPU reading from a distributed storage cluster, no guest CPU in the datapath.

Latency: stop being the bottleneck ¶

The first runs clocked ~987 µs/IO — suspiciously close to the bridge's 1 ms poll interval. Dropping poll-interval-ns from 1000000 → 10000 (10 µs):

mode 0:  Min 105 µs · Avg 210 µs · Max 367 µs

The bridge is no longer the limiter; the storage is. That's exactly where you want to be.

Mode 8: the payload lands in VRAM ¶

The finale of the dress rehearsal. Flip the data buffer into GPU memory (--memory-mode 8) and the kernel log shows the peer-to-peer path lighting up:

rocm-xio: Passthrough NVMe — using P2PDMA IOVA
rocm-xio: ✅ P2PDMA address: 0x6e00000 (size: 2097152)
rocm-xio: Registered buffer ... (P2PDMA attachment kept alive)
Iterations: 50   Average: 136.24 us   Test completed successfully!

rocm-xio allocated the read buffer in GPU device memory, exported it via P2PDMA to a guest-physical IOVA that SPDK can reach through its vfio-user DMA map, injected that as the read command's PRP, and SPDK DMA'd the RADOS payload directly into GPU memory — no host bounce. It was even faster than mode 0 (136 µs), partly because there's no host copy and partly because the RBD object was warm.

The block-first proof — note the backend is bdev_rbd/librbd against an RBD image, not rados-nkv. This is the dress rehearsal, not the headline.

Block done. Now for the thing the post is named for.

GPU-initiated IO to the KV namespace ¶

The RBD run was always the warm-up. GPU-initiated IO to a block namespace is, as the BaM paper showed, already a known quantity — and block is the very interface this whole project argues is wrong for KV cache. The actual summit was getting that same GPU-initiated, CPU-free, payload-into-VRAM path to terminate at a key-value namespace backed by RADOS objects: content-addressed, no (device, offset, length) lookup table, no coordinator.

That meant teaching ROCm XIO a new dialect. Its nvme-ep kernel knew how to build exactly one kind of SQE — a block read: opcode 0x02, an LBA, a length. A KV Retrieve is a different 64-byte command: the key travels in the command itself, and the value buffer is described by the usual PRP/SGL the GPU already knows how to point at VRAM. So the kernel-side work was to stop emitting block opcodes and start emitting key-value ones.

Concretely, the new kvSqeSetup() packs a KV Retrieve into the 64-byte SQE like this. The opcode is 0x01 for Store / 0x02 for Retrieve — numerically the same as block write/read; the controller routes it as key-value not because of a distinct opcode space but because the namespace's Command Set Identifier is Key Value (0x1). The key — up to 16 bytes — rides inline in the command: bytes 0–7 in CDW2/CDW3 and bytes 8–15 in CDW14/CDW15, with the key length in CDW11[7:0] and the requested value size in CDW10. The value buffer is described by the ordinary data pointer (PRP1/PRP2 in the DPTR) — the exact same field a block read uses for its destination. That last part is the whole reuse dividend: the VRAM dma-buf the GPU already knows how to aim a block read at is described identically for a KV Retrieve, so the existing calculatePrps() path that resolves a VRAM P2PDMA IOVA under --memory-mode 8 needs no changes at all. The only genuinely new kernel-side code is the SQE field-packing; everything about where the value lands is shared with the block path. (The true stored length comes back in the completion's DW0 — a field we'll have a small war story about in a moment.)

On the target side, almost nothing changed — which is the whole reward for having proven the plumbing generic. The doorbell → shadow ring → pci-mmio-bridge → vfio-user → SPDK path is interface-agnostic; it forwards MMIO and posts completions without caring whether the command is a block read or a KV Retrieve. We simply put the rados-nkv kvdev behind the NVMe controller model instead of bdev_rbd. Now a Retrieve SQE the GPU rings becomes a kvdev_rados lookup → librados → a RADOS object named by the key.

First GPU-initiated KV Retrieve ¶

✅ GPU-initiated NVMe-KV → Ceph/RADOS — proven end-to-end

The one post-reboot shot landed first try (staging everything before the launch was the trick):

host:  GPU bd:00.0  Kernel driver in use: vfio-pci   (kernel 6.19.10)
guest: rocm-xio nvme-ep --kv-op store --memory-mode 8 --pci-mmio-bridge
        Detected PCI MMIO bridge at 0000:00:08.0
        Model: AMD Radeon Graphics
        NVMe /dev/nvme0 (PCI 0000:02:00.0) VID=0x4e58 -> PASSTHROUGH
        Test completed successfully!
Ceph:  rados ls kvpool/gpukv   -> 6770756b65793031   (= hex "gpukey01")
        rados stat              -> kvpool/6770756b65793031  size 4096

Chain: GPU __device__ code → doorbell MMIO → QEMU pci-mmio-bridge → SPDK vfio-user NVMe-KV → kvdev_rados → librados → a RADOS object in Ceph.

The bytes are real, not a zero-filled placeholder. Seed the GPU's value buffer with a known LFSR pattern (--lfsr-seed 0xC0FFEE) on the Store, and that exact pattern shows up as the RADOS object's data:

$ rados -p kvpool -N gpukv get $(printf gpukey01 | xxd -p) - | xxd | head -3
00000000: 543c f4ed 6793 fd9e 806b 6a17 7cc9 44e9  T<..g....kj.|.D.
00000010: 6b50 02fb e02b 1108 314d ca9a 30a8 f2e9  kP...+..1M..0...
00000020: 3de4 43e4 1340 1515 6b05 d5c1 4a86 ef61  =.C..@..k...J..a
# sha256 = ea1b6431…  (not the all-zeros hash) — the value the GPU stored, by key.

And the finale for real. With --memory-mode 8, the value the GPU asked for by key — fetched from a RADOS object by SPDK — P2P-DMAs straight into GPU memory:

guest: rocm-xio nvme-ep --kv-op retrieve --key gpukey01 --memory-mode 8 --pci-mmio-bridge
        NVMe /dev/nvme0 (PCI 0000:02:00.0) VID=0x4e58 -> PASSTHROUGH
        KV retrieve OK: returned value length = 4096 bytes
        Iterations: 50   Average: 140.21 µs/IO      (bridge poll-interval-ns = 10000)
kernel: rocm-xio: Passthrough NVMe (0000:02:00.0) — using P2PDMA IOVA
        rocm-xio: ✅ P2PDMA address: 0x11c200000 (size: 2097152)
        rocm-xio: Registered buffer ... (P2PDMA attachment kept alive)

That 140 µs/IO is the same story as the block dress rehearsal, one interface up. Run with the bridge's default 1 ms poll, the KV Retrieve sits at ~991 µs — poll-bound, exactly as block did. Drop poll-interval-ns to 10 µs and it collapses to 140 µs, landing right on top of the block path's 136 µs: at that point we're past the bridge and into the real RADOS read latency. The bridge is not the limiter; the cluster is — which is precisely where you want to be, and it's the same floor whether the command is a block read or a KV Retrieve.

That's the thing the whole post was named for, the doorbell finally tolling for the right interface:

A GPU, from __device__ code, issues an NVMe key-value Retrieve; the value — sourced from a RADOS object in a distributed Ceph cluster — lands directly in GPU memory. No guest CPU in the datapath. No block-to-object lookup table. Content-addressed all the way down.

Let me be precise about what's new. GPU-initiated storage over the key-value command set isn't conjured from nothing. On the CPU side, xNVMe has long unified NVMe IO across command sets — key-value included — and its maintainers explicitly name efficient GPU storage access as the road ahead, though that part is still aspiration, not shipped code. The KV-over-vfio-user plumbing has a lineage too: Simon Lund's vfio-user-kvssd emulates an NVMe KV SSD behind the very same vfio-user interface rados-nkv speaks. And GPU-initiated KV cache tiering already exists — in proprietary silicon: Pliops XDP LightningAI issues GPU-initiated key-value requests with the CPU out of the data path, and NVIDIA's CMX / DOCA Memos fronts a KV-cache context tier with key-value APIs on BlueField.

What I haven't seen before is this exact stack: a GPU-initiated NVMe key-value command driven from an AMD GPU — as far as I know, a first — landing not in a local KV SSD or a proprietary appliance but in a RADOS object in a distributed Ceph cluster, on entirely open-source software-defined storage, with no bespoke DPU or ASIC in the path. And it matters because key-value is the interface KV cache actually wants. The hash of a token sequence is the key; two GPUs computing the same prefix arrive at the same RADOS object with zero coordination, no central map to consult or keep coherent.

Same GPU-initiated path, KV all the way down: the value is addressed by key and the backend is rados-nkv → librados → a RADOS object.

Wavefront multi-key — and a torn read (Bug #3) ¶

A single Retrieve per doorbell is leaving the GPU on the table. ROCm XIO has a wavefront KV path: threads 1..B of a workgroup each pack one key's SQE into its own slot, and thread 0 rings the doorbell once for the whole batch, then reaps B completions. Point it at a manifest of keys:

rocm-xio nvme-ep --kv-op store --keys shard0 shard1 shard2 shard3 \
    --batch-size 4 --value-size 4096 --data-buffer-size 16384 --pci-mmio-bridge

Four keys, one ring — and four RADOS objects appear with the same mtime, the fingerprint of a single batched submission:

shard0 -> kvpool/736861726430  size 4096   (all four at the same mtime)
shard1 -> kvpool/736861726431  size 4096
shard2 -> kvpool/736861726432  size 4096
shard3 -> kvpool/736861726433  size 4096

The matching batched Retrieve, though, surfaced a real bug — and a satisfying one, because it was the kind we'd spent the whole post learning to hunt. The first key in every batch came back with a value length of zero, while the rest were correct:

KV retrieve OK: key_idx=0 returned value length = 0 bytes     ← wrong
KV retrieve OK: key_idx=1 returned value length = 4096 bytes
KV retrieve OK: key_idx=2 returned value length = 4096 bytes
KV retrieve OK: key_idx=3 returned value length = 4096 bytes

Reading the completion path: cqeRead copies the 16-byte CQE low dword first, and the phase bit lives in the last dword. The poll loop spins on a single cqeRead until the phase flips — but a single read can capture a stale cdw0 (the value length, still zero) together with the freshly-flipped phase bit. A torn read. It only ever bit key_idx=0 because that's the completion whose poll races the controller's write most tightly; by the time the loop reaches slots 1–3 the writes have long since landed. The fix is a one-liner of memory ordering — once the phase flip is observed, fence and re-read before trusting cdw0:

  if (NVME_CQE_STATUS_PHASE(cqe_new.status) == expected_phase) {
+   // The phase bit is the CQE's last dword, but cqeRead copies cdw0 first —
+   // one read can capture a stale cdw0 (0) with the freshly-flipped phase.
+   // Fence + re-read so DW0 (the KV value length) is coherent before use.
+   __threadfence_system();
+   cqe_new = nvme_ep::cqeRead(cqe_entry);
    break;
  }

Because xio-tester is plain userspace HIP, this was a live fix: edit, rebuild in the guest, re-run — no host reboot, the precious GPU session preserved. After the rebuild, all four slots (and the single-key path) report the true 4096. Same lesson as Bugs #1 and #2, one layer down: read the source, and the "weird" symptom has an unexciting, specific cause.

Lessons ¶

A few things generalized well beyond this specific stack:

Bisect across process/hardware boundaries, not just code. The single most valuable move was the CPU-driven shadow-ring probe. It cut the search space in half and cost zero GPU boots — in an environment where each GPU boot was the scarcest resource we had.
Validate fixes with the cheap path before spending the expensive one. Both fixes (the GPA relocation and the MQES bump) were confirmed reboot-free with the CPU probe before we committed a precious GPU boot to them. We never "tried" a fix on the GPU — by the time it ran there, we already knew it worked.
Read the source of every layer before theorizing. Tracing QEMU, SPDK, and rocm-xio at the line level proved the path was correct by construction, which reframed the hunt from "what's broken in the design" to "what's different at runtime."
Instrument over guess. One QEMU trace flag turned "0x1016, no idea" into "338 zeroed commands from a source that isn't the GPU" — and that single data point killed five theories at once.
The exotic explanation is usually wrong. We spent real worry on integrated-GPU P2P semantics, hipHostRegister-of-MMIO, SMU flakiness, and CQ-phase races. The actual bugs were a framebuffer at the wrong address, a number that was 4× too big, and a torn 16-byte read.

Doing the math ¶

The ROCm XIO kernel drives a single 1024-entry submission queue (--num-queues 1 --queue-length 1024) — rowdy, but this is serious AI business, and serious AI business needs to do lots of IO. That 1024-entry ceiling is exactly what the MQES 255 → 1023 fix in Bug #2 unlocked, and the wavefront path rings a batch of up to queue_length − 1 SQEs per doorbell, so the "one ring submits the whole batch" claim below is literal.

1024 SQEs × 256 tokens per block = 262,144 tokens. That's 2.7× the 96k median agentic context reported by SemiAnalysis:

We pulled data from 432k real coding-agent requests, and the median one isn't 32k, isn't 64k, but 96k input tokens — more than the entire text of The Great Gatsby shoved into the model before you've even typed your question. — SemiAnalysis

The 256-token block isn't an arbitrary chunk, either: it's the granularity LMCache and llm-d-kv-cache already aggregate vLLM's 16-token paged blocks into before handing them across the kv-connector. We content-address at the unit the ecosystem already speaks. If you take a model like IBM Granite 4.0 30B, a 256-token block works out to about 32MB (128K per token).

So a single SQ-tail doorbell ring submits the whole batch — meaning we can request an entire 262k-token prefix with one ring of the submission doorbell. SPDK dispatches the work across its async reactors, which drive librados asynchronously to RADOS, and P2P-DMAs each value into a VRAM-backed dma-buf.

Ride the lightning.

What's next ¶

The KV Retrieve path is the foundation; there's a lot stacked on top of it. The cls allowlist — execute-where-the-data-rests via the gRPC-controlled opcode map — turns this from a place to put bytes into a place to run code next to them. rados-nkv-weights lets vLLM stream model weights over the same namespace. And there are three deployment shapes worth their own post: co-resident on the hypervisor (what you saw here, useful today on commodity TCP/IP RADOS), on a DPU like BlueField-4 (provided it exposes a hardware doorbell abstraction to the host), and storage-host-side across an RDMA fabric (IB / RoCEv2 / Ultra Ethernet) with a GDS-class NIC. That's the next mountain. And to make any of this reproducible, there's work underway on a recipe that stitches the pieces together — SPDK with rados-nkv, ROCm XIO, the patched QEMU, and the guest plumbing — so you won't have to reassemble the factory from scratch.

A note on scope: everything above is a functional prototype. The latencies are single-box, warm-cache figures meant to show the bridge is no longer the bottleneck — not a benchmark. Real throughput/TTFT numbers, and the non---no-huge hugepage-backed configuration you'd actually measure on, are coming.

Have questions? Catch the Ceph Tech Talk on July 8th. Can't make it? Reach out — kbader@ibm.com, or @kbader on Ceph Slack #gpu-initiated.

Managing Multiple Lua Scripts with Ceph Object Storage

2026-06-10T00:00:00Z

Since the Pacific release, Lua scripting in Ceph's RADOS Gateway (RGW) has provided users the ability to interpolate a single script to upload operations per request context and tenant. This way of working might be completely fine for a storage deployment with limited scripting customizations. However, script management becomes increasingly difficult as more than one team wants to get involved in managing the Lua script within the same context and tenant.

For this reason, we've released a new feature in RGW allowing you to manage more than one Lua script at a time. In this blog, we'll go over a couple of examples to walk you through how this feature can help your team to reduce runtime errors and lower developer friction when managing Lua scripts in RGW. To begin, we will create a script to enforce a bucket security control and then add another script to demonstrate the new feature.

Script 1: Bucket security ¶

Let's suppose we are part of a security team and want to mitigate the risk of ransomware threats happening on our storage platform. One possible way to achieve this is to implement a Write Once, Read Many (WORM) strategy to add an object lock onto any newly created bucket. By using an object lock, we can add a constraint to write objects to disk only once, ensuring that an infected client cannot delete or override objects at a later time.

To set our WORM strategy, we can create a Lua script in the prerequest context that aborts when a create_bucket operation is made without the write-once (object lock) requirement.

objectlock.lua

-- enforcing object lock on bucket creation

if Request.RGWOp == "create_bucket" and
  Request.HTTP.Metadata["x-amz-bucket-object-lock-enabled"] ~= "true" then
  RGWDebugLog("object lock is missing on bucket: " .. Request.Bucket.Name)
  Request.Response.Message = "Bucket must have object lock enabled"
  return RGW_ABORT_REQUEST
end

You can create a new objectlock.lua file with the contents above and run the CLI commands below to upload the script to RGW.

Add the objectlock.lua script to the preRequest context:

$ radosgw-admin script put --infile=./objectlock.lua --context=prerequest

End Result: You'll know if this worked correctly if the aws s3 mb (make bucket) command below has failed.

$ aws s3 mb --endpoint-url http://127.0.0.1:8000 \
  s3://fish1 \
  --debug

Error:

Bucket must have object lock enabled

Instead, you'll need to use the aws s3api create-bucket command, which includes the --object-lock-enabled-for-bucket flag to set the write-once capability.

$ aws s3api create-bucket --endpoint-url http://127.0.0.1:8000 \
  --bucket fish2 \
  --object-lock-enabled-for-bucket

Once the script is uploaded, it's stored as a Ceph S3 object, so you will get the same benefits of data consistency and availability as you would get with any other object in the platform.

To delete the script, use radosgw-admin script rm --context={context}.

Script 2: Cost optimization ¶

Let's say that our engineering team now wants to implement object auto-tiering to manage storage efficiently. Specifically, the team wants to classify objects with a StorageClass attribute, one of STANDARD (<16KB), STANDARD_IA (>=16KB and <1MB) or ONE_ZONE_IA (>1MB) to provide them with a heuristic on where to store the data in a cost effective manner. Although we are using AWS Storage Class names here, please avoid using GLACIER with Ceph's RGW because some clients/SDKs expect specific behavior that is not currently supported.

Similar to the first script, the engineering team wants to run their new script in the prerequest context before the object request starts. However, whereas the earlier example targets the create_bucket operation, the latter focuses exclusively on the put_obj operation.

autotier.lua

-- Lua script to auto-tier S3 object PUT requests

-- exit script quickly if it is not a PUT request
if Request == nil or Request.RGWOp ~= "put_obj"
then
  return
end

-- apply StorageClass only if user hasn't already assigned a storage-class
if Request.HTTP.StorageClass == nil or Request.HTTP.StorageClass == '' then
  if Request.ContentLength < 16384 then
    Request.HTTP.StorageClass = "STANDARD"
  elseif Request.ContentLength < 1048576 then
    Request.HTTP.StorageClass = "STANDARD_IA"
  else
    Request.HTTP.StorageClass = "ONE_ZONE_IA"
  end
  RGWDebugLog("applied '" .. Request.HTTP.StorageClass .. "' to object '" .. Request.Object.Name .. "'")
end

You can create the autotier.lua file from the contents above, but you might quickly notice a new issue.

If we were to combine both scripts together, the engineering team's Lua script returns early if the request is not a put_obj operation. This means script order matters: if the engineering team's script runs first our initial script may never execute.

To avoid this merge conflict scenario and as mentioned earlier, we can use the multiple Lua script support in RGW by setting the --script-name parameter to add more than one script to the same request context.

Adding more than one Lua script to the same context ¶

Since it's important that both teams be able to run their own scripts in the prerequest context, you can add more than one script without affecting others by using the script put command with the --script-name flag.

For example, try adding both scripts with the following commands below:

$ radosgw-admin script put --infile=./objectlock.lua --script-name=worm-policy --context=prerequest
$ radosgw-admin script put --infile=./autotier.lua --script-name=smart-tiering --context=prerequest

One important caveat to remember when using multiple Lua scripts is that they have completely separate local scopes during execution. This means that a global variable in one script cannot be accessed in another script regardless of the context.

Apart from this, the Lua scripts will run sequentially and in lexicographical order, which follows the same ordering to how the scripts are listed in the admin command output below. The only exception to this is --script-name=default which is reserved for the unnamed script and will always be run first.

List all uploaded scripts with the script list command:

$ radosgw-admin script list --context=prerequest
smart-tiering
worm-policy

Do note that if you haven't deleted the unnamed objectlock.lua script yet from the earlier example, you might get an extra default script included in the script list output.

Finally, to delete a named script, use radosgw-admin script rm --context={context} --script-name={script-name}.

Wrapping Up ¶

This was a brief look at the new multiple Lua scripting support in RGW to show how you can manage more than one script in the same context across teams in an independent way. For more information about this feature, you can find the complete reference in The Ceph Documentation on Lua Scripting.

v19.2.4 Squid released

2026-06-01T00:00:00Z

This is the fourth backport release in the Squid series. We recommend all users update to this release.

Notable Changes ¶

OSD / BlueStore ¶

BlueFS: Increased the default WAL volume size to 1GB to prevent ENOSPC failures.
Volume Selection: Fixed the usage of bluestore_volume_selection_reserved_factor and updated row naming conventions in RocksDBBlueFSVolumeSelector.
Health: Added a time_added field to mon_info_t to prevent freshly added monitors from incorrectly triggering a MON_DOWN health status.
Tooling: Updated monmaptool to correctly respect set features when adding addresses.

RADOS / Manager (mgr) ¶

RADOS pybind: Fixed a parameter reversal bug in WriteOp.zero() where offset and length were swapped when calling the underlying C API.
Daemon Health: Modified the manager to clear health metrics for down or out OSDs instead of removing them entirely from the daemon state.
Autoscaler: Updated pg_autoscale_mode logic to accurately reflect the no_autoscale flag in both standard and JSON command outputs.

RADOS Gateway (RGW) ¶

Notifications: Resolved a reserved_size drift in the 2pc_queue causing ENOSPC errors and fixed a leak where overhead was not properly decremented during commit/abort.
Multipart Uploads: Fixed an issue where tags could not be retrieved from objects created via multipart uploads.
Kafka Integration: Updated the internal Kafka version support to 3.9.2.

CephFS / MDS ¶

Fragments: Simplified fragment printing and corrected frag_t endianness conversion for network and storage consistency.
FUSE Client: Updated the FUSE client to clarify that fallocate disk space reservation is only supported with specific flags like FALLOC_FL_KEEP_SIZE.
Metadata: Updated the MDS to dump frag_t as an object and included sysinfo in the status command output for better diagnostics.

Dashboard ¶

Services: Fixed a bug where changing MDS placement to a label incorrectly triggered the creation of an entirely new MDS service.
Access Control: Fixed role-based access control permissions and added missing types to the mgr-module list.

librbd & rbd-mirror ¶

RBD: Introduced a new RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT policy for rbd_lock_acquire(). This is a low-level interface intended to allow a peer to grab exclusive lock manually for short periods of time with other peers pausing their activity and waiting for the lock to be released rather than instantly aborting I/O and returning an error. It's possible to switch from RBD_LOCK_MODE_EXCLUSIVE to RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT policy and vice versa even if the lock is already held.

Changelog ¶

auth: msgr2 can return incorrect allowed_modes through AuthBadMethodFrame (pr#65335, Miki Patel)
backport build-with-container patches from main (pr#65186, John Mulligan, Dan Mick, Zack Cerza)
backports variants improvements and Dockerfile.build changes (pr#66011, John Mulligan, Zack Cerza)
blk/kernel: improve DiscardThread life cycle (pr#65214, Igor Fedotov)
bluestore/BlueFS: fix bytes_written_slow counter with aio_write (pr#66354, chungfengz)
build-with-container: add argument groups to organize options (pr#65629, John Mulligan)
build-with-container: build image variants (pr#65945, John Mulligan)
ceph-fuse: Improve fuse mount usage message (pr#61274, Kotresh HR)
ceph-volume: drop udevadm subprocess calls (pr#65302, Guillaume Abrioux)
ceph-volume: fix UdevData initialisation from empty /run/udev/data/* file (pr#65924, Matteo Paramatti)
ceph-volume: use udev data instead of LVM subprocess in get_devices() (pr#65922, Guillaume Abrioux)
cephadm: add ability to continue on failure when applying multiple specs (pr#64082, Adam King)
cephadm: rgw: allow specifying the ssl_certificate by filepath (pr#64084, Alexander Hussein-Kershaw)
cephadm: stop sidecar systemd units when restarting main units (pr#62593, Adam King, John Mulligan)
CephContext: acquire _fork_watchers_lock in notify_post_fork() (issue#63494, pr#62051, Venky Shankar)
cephfs-journal-tool: fix segfault during 'journal import' from invalid dump file (pr#62115, Jos Collin)
cephfs-journal-tool: Journal trimming issue (pr#65602, Kotresh HR)
cephfs-shell: add option to remove xattr (pr#62391, Neeraj Pratap Singh)
cephfs-top: exception when terminal size greater than PAD_WIDTH (pr#61774, Jos Collin)
cephfs: fix monclient not subscribed monmap/config (pr#66471, Shaohui Wang)
cephfs: MDCache request cleanup (pr#66472, Abhishek Lekshmanan)
cephfs_mirror: 'ceph fs snapshot mirror ls' command (pr#60177, Jos Collin)
client: account for mixed quotas in statfs (pr#66473, Christopher Hoffman)
client: cephfs user-space client fixes (issue#70726, issue#71510, pr#64090, Venky Shankar, Adam Emerson, Kotresh HR, Shachar Sharon)
client: fix unmount hang after lookups (pr#65289, Dhairya Parmar)
client: Handle empty pathnames for ceph\_chownat() and ceph\_statxat() (pr#61166, Anoop C S)
client: ll_walk will process absolute paths as relative (pr#62499, Patrick Donnelly)
client: prohibit unprivileged users from setting sgid/suid bits (pr#66039, Kefu Chai)
client: use path supplied in statfs (pr#65133, Christopher Hoffman)
cmake: remove _FORTIFY_SOURCE define (pr#65659, Casey Bodley)
common/ceph_context: use std::atomic<std::shared_ptr> (pr#66185, Kefu Chai, Max Kellermann)
common/frag: properly convert frag_t to net/store endianness (pr#66541, Patrick Donnelly, Max Kellermann)
common/options: fix typo in description (pr#64217, Lorenz Bausch)
common: Allow PerfCounters to return a provided service ID (pr#65588, Adam C. Emerson)
common: drop stack singleton object of temp messenger for foreground ceph daemons (pr#66897, dongdong tao)
deb/cephadm: add explicit --home for cephadm user (pr#64605, Casey Bodley, Shawn Edwards)
deb/mgr: remove deprecated distutils from ceph-mgr.requires (pr#64456, Casey Bodley)
deb: use glob match to support systemd unit dir changes (pr#64013, Kefu Chai)
debian/control: add iproute2 to build dependencies (pr#66739, Kefu Chai)
debian: radosgw: add media-types packages as alternative for mime-support (pr#63725, Thomas Lamprecht)
doc/cephadm: remove sections that do not not apply to Squid in rgw.rst (pr#66970, Ville Ojamo)
doc/cephfs: add a note about estimated replay completion time (issue#71629, pr#65057, Venky Shankar, Zac Dover)
doc/cephfs: edit ceph-dokan.rst (1 of x) (pr#64735, Zac Dover)
doc/cephfs: edit ceph-dokan.rst (2 of x) (pr#64759, Zac Dover)
doc/cephfs: edit ceph-dokan.rst (3 of x) (pr#64785, Zac Dover)
doc/cephfs: edit disaster-recovery.rst (pr#64644, Zac Dover)
doc/cephfs: edit disaster-recovery.rst (pr#64608, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65379, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65087, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65125, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65122, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65093, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65090, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65077, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65096, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65046, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65043, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65040, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65036, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65025, Zac Dover, Venky Shankar)
doc/cephfs: edit troubleshooting.rst (pr#64903, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#64900, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#64878, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#64871, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#64852, Zac Dover)
doc/cephfs: edit troubleshooting.rst (Slow MDS) (pr#65202, Zac Dover)
doc/cephfs: Improve mount-using-fuse.rst (pr#64472, Anthony D'Atri)
doc/cephfs: Update quota.rst (pr#65082, Jannis Speer, Zac Dover)
doc/dev/config: Document how to use :confval: directive for config op… (pr#64166, Kefu Chai)
doc/dev/crimson: Update docs (pr#65311, Matan Breizman)
doc/dev/release-process.rst: document new Jenkins job for containers (pr#62612, Dan Mick)
doc/dev:update blkin.rst doc for lttng trace (pr#65211, lizhipeng)
doc/man/8: Improve mount.ceph.rst (pr#65183, Anthony D'Atri)
doc/mgr/crash.rst: remove outdated module enabling instructions (pr#64284, Kefu Chai)
doc/mgr: edit telemetry.rst (pr#63809, Zac Dover)
doc/mgr: edit telemetry.rst (pr#64343, Zac Dover)
doc/rados/configuration: Mention show-with-defaults and ceph-conf (pr#65206, Niklas Hambüchen)
doc/rados/configuration: Small improvements in ceph-conf.rst (pr#64287, Ville Ojamo)
doc/rados/operations: Improve health-checks.rst (pr#65238, Anthony D'Atri)
doc/rados/ops: add caps restore command (pr#64321, Zac Dover)
doc/rados/ops: edit cache-tiering.rst (pr#64496, Zac Dover)
doc/rados: document section absent in release < T (pr#64867, Zac Dover)
doc/rados: remove clonedata command (pr#64393, Zac Dover)
doc/rados: repair short underline (pr#65137, Zac Dover)
doc/radosgw/s3: Document delete-if-unmodified-since (pr#64315, Anthony D'Atri)
doc/radosgw: add rgw_enable_lc_threads & rgw_enable_gc_threads (pr#64338, Zac Dover)
doc/radosgw: edit "Lifecycle Settings" (pr#64547, Zac Dover)
doc/radosgw: edit config-ref.rst (pr#64647, Zac Dover)
doc/radosgw: edit metrics.rst (pr#63812, Zac Dover)
doc/radosgw: Improve rgw-cache.rst (pr#64475, Ville Ojamo)
doc/radosgw: remove cloud-restore from squid (pr#65639, Zac Dover)
doc/rbd/rbd-config-ref: add clone settings section (pr#66174, Ilya Dryomov)
doc/src/common/options: mgr.yaml.in edit (pr#63764, Zac Dover)
doc: add note admonitions in two files (pr#64492, Zac Dover)
doc: Clarify the status of MS Windows client support (pr#64481, Anthony D'Atri)
doc: do not depend on typed-ast (pr#64399, Kefu Chai)
doc: Document ceph-mgr module configuration options (pr#64396, Kefu Chai)
doc: fetch releases from main branch (pr#67001, Patrick Donnelly)
doc: Fix links to mClock config reference (pr#64752, Pierre Riteau)
doc: Fix unterminated inline literal in ceph-conf.rst (pr#64170, Kefu Chai)
doc: Fixed a spelling error (pr#64147, Instelligence.io)
doc: Fixes a typo in balancer operations (pr#65550, Tyler Brekke)
doc: Pin pip to <25.3 for RTD as a workaround for pybind in admin/doc-read-the-docs.txt (pr#66117, Ville Ojamo)
doc: Remove sphinxcontrib-seqdiag Python package from RTD builds (pr#67501, Ville Ojamo)
doc: update mgr modules notify_types (pr#64530, Nitzan Mordechai)
github: pin GH Actions to SHA-1 commit (pr#65758, Ernesto Puerta)
Handle failures in metric parsing (pr#65228, Anmol Babu)
HealthMonitor: Add topology-aware netsplit detection and warning (pr#63024, Kamoltat Sirivadhna)
install-deps.sh: install proper compiler version on Debian/Ubuntu (pr#66013, Dan Mick)
install-deps: Replace apt-mirror (pr#66668, David Galloway)
librbd/cache/pwl: WriteLogOperationSet::cell can be garbage (pr#67704, Ilya Dryomov)
librbd/mirror: detect trashed snapshots in UnlinkPeerRequest (pr#67582, Ilya Dryomov)
librbd: don't complete ImageUpdateWatchers::shut_down() prematurely (pr#67580, Ilya Dryomov)
librbd: fix ExclusiveLock::accept_request() when !is_state_locked() (pr#66627, Ilya Dryomov)
librbd: images aren't closed in group_snap\_*_by_record() on error (pr#64621, Miki Patel)
librbd: introduce RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT (pr#67278, Ilya Dryomov)
mds,client: correct last_issue struct fields and consistently send last_issue in cap messages (pr#61300, Patrick Donnelly)
mds/MDSDaemon: unlock mds\_lock while shutting down Beacon and others (pr#64886, Max Kellermann)
mds: add MDS asok command for dumping stray directories (issue#56442, pr#62517, Igor Golikov)
mds: add more debug logs and log events (pr#65279, Xiubo Li)
mds: check for snapshots on parent snaprealms (pr#63019, Patrick Donnelly)
mds: do not schedule quiesce for non-head imported inodes (pr#61857, Patrick Donnelly)
mds: dump export_ephemeral_random_pin as double (pr#65162, Enrico Bocchi)
mds: dump next_snap when checking dentry corruption (pr#61977, Milind Changire)
mds: fix heap-use-after-free in C_Flush_Journal (pr#65141, Milind Changire)
mds: Fix invalid access of mdr->dn[0].back() (pr#61451, Anoop C S)
mds: fix snapdiff result fragmentation (pr#65363, Igor Fedotov, Md Mahamudur Rahaman Sajib)
mds: include auth credential in session dump (pr#65267, Patrick Donnelly)
mds: nudge log for unstable locks after early reply (pr#64539, Patrick Donnelly)
mds: session in the importing state cannot be cleared if an export subtree task is interrupted while the state of importer is acking (pr#62055, Zhansong Gao)
mds: skip charmap handler check for MDS requests (pr#64954, Patrick Donnelly)
memory lock issues causing hangs during connection shutdown (pr#65785, Nitzan Mordechai)
mgr/alerts: enforce ssl context to SMTP_SSL (pr#66141, Nizamudeen A)
mgr/cephadm: add "allow_set_io_flusher_fail = true;" to ganesha conf (pr#60360, Adam King)
mgr/cephadm: allow disabling rgw_run_sync_thread through spec (pr#61350, Adam King)
mgr/cephadm: fix typo with vrrp_interfaces in keepalive setup (pr#61902, Adam King)
mgr/cephadm: mgr orchestrator module raise exception if there is trailing tab in yaml file (pr#64086, Shweta Bhosale)
mgr/cephadm: open ceph-exporter when firewalld is enabled (pr#59932, Redouane Kachach)
mgr/cephadm: update grafana conf for disconnected environment (pr#66210, Nizamudeen A)
mgr/cephadm: Use a persistent volume to store Loki DB (pr#66024, Aashish Sharma)
mgr/DaemonServer: fixed mistype for mgr_osd_messages (pr#63344, Konstantin Shalygin)
mgr/DaemonState: Minimise time we hold the DaemonStateIndex lock (pr#65462, Brad Hubbard)
mgr/dashboard : Fixed mirrored image usage info bar (pr#65492, Abhishek Desai)
mgr/dashboard : Fixed usage bar for secondary site in rbd mirroing (pr#65928, Abhishek Desai)
mgr/dashboard: 72409 : Fixed parsing error in grafana for host overall performance iframe (pr#64884, Abhishek Desai)
mgr/dashboard: add .nvmrc so ci can pick the node version (pr#64665, Nizamudeen A)
mgr/dashboard: allow deletion of non-default zone and zonegroup (pr#66212, Aashish Sharma)
mgr/dashboard: Blank entry for Storage Capacity in dashboard under Cluster > Expand Cluster > Review (pr#65706, Naman Munet)
mgr/dashboard: Changing placement of a mds to label - creates a new mds-service, mds.label (pr#66884, Dnyaneshwari Talwekar)
mgr/dashboard: Dashboard nfs export editor rejects ipv6 addresses (pr#65242, Afreen Misbah)
mgr/dashboard: fix access control permissions for roles (pr#62454, Nizamudeen A)
mgr/dashboard: fix bucket replication disable (pr#64776, Naman Munet)
mgr/dashboard: fix health popover not showing (pr#66145, Nizamudeen A)
mgr/dashboard: Fix inline markup warning in API documentation (pr#64269, Kefu Chai)
mgr/dashboard: fix nvmeof mtls (pr#65167, Tomer Haskalovitch)
mgr/dashboard: fix zone update API forcing STANDARD storage class (pr#65620, Aashish Sharma)
mgr/dashboard: Fixed incorrect snapshot scheduled date for rbd block in GUI (pr#64880, Abhishek Desai)
mgr/dashboard: service creation fails if service name is same as service type (pr#66480, Naman Munet)
mgr/dashboard: update teuth_ref hash in api test (pr#66707, Nizamudeen A)
mgr/dashboard: use system packages when running tox (pr#64613, Nizamudeen A, Paulo E. Castro)
mgr/Mgr.cc: clear daemon health metrics instead of removing down/out osd from daemon state (pr#67527, Cory Snyder)
mgr/nfs: make sure sectype is a list when parsed (pr#63115, Adam King)
mgr/nfs: scrape nfs monitoring endpoint (pr#64081, avanthakkar)
mgr/orchestrator: allow passing group to apply/add nvmeof commands (pr#59880, Adam King)
mgr/orchestrator: fix nvmeof group parameter for backward compatibility (pr#67623, Kefu Chai, Alexander Indenbaum)
mgr/snap_schedule: correctly fetch mds_max_snaps_per_dir from mds (pr#59647, Milind Changire)
mgr/telemetry: add stretch cluster data (pr#66389, Naveen Naidu)
mgr/vol: add command to get snapshot path (pr#62918, Rishabh Dave)
mgr/vol: include group name in subvolume's pool namespace name (pr#64205, Rishabh Dave)
mgr/vol: keep and show clone source info (pr#64652, Rishabh Dave)
mgr/vol: make "snapshot getpath" cmd work with v1 and legacy (pr#63222, Rishabh Dave)
mgr: avoid explicit dropping of ref (pr#65006, Milind Changire)
mon,cephfs: require confirmation when changing max_mds on unhealthy cluster (pr#60398, Rishabh Dave)
mon/HealthMonitor: avoid MON_DOWN for freshly added Monitor (pr#67324, Patrick Donnelly)
mon/MgrMonitor: add a space before "is already disabled" (pr#64686, Zehua Qi)
mon/MonMap: Dump addr in backward compatible format (pr#68323, Anoop C S)
mon/OSDMonitor: call no_reply() on ignored osd alive (pr#64509, Nitzan Mordechai)
mon: Track and process pending pings after election (pr#62924, Kamoltat Sirivadhna)
monc: synchronize tick() of MonClient with shutdown() (pr#66915, Radoslaw Zarzynski)
monitor: Enhance historic ops command output and error handling (pr#64842, Nitzan Mordechai)
monitoring: add user-agent headers to the urllib (pr#65475, Nizamudeen A)
monitoring: fix "In" OSDs in Cluster-Advanced grafana panel. Also change units from decbytes to bytes wherever used in the panel (pr#65671, Aashish Sharma)
monitoring: fix CephPgImbalance alert rule expression (pr#66829, Aashish Sharma)
monitoring: fix MTU Mismatch alert rule and expr (pr#65709, Aashish Sharma)
monitoring: fix rgw_servers filtering in rgw sync overview grafana (pr#66990, Aashish Sharma)
monitoring: make cluster matcher backward compatible for pre-7.1 metrics (pr#66985, Aashish Sharma)
monitoring: upgrade grafana version to 12.3.1 (pr#66964, Aashish Sharma)
msg/async: Encode message once features are set (pr#65624, Aishwarya Mathuria)
msg: drain stack before stopping processors to avoid shutdown hang (pr#64924, Nitzan Mordechai)
os/bluestore: compact patch to fix extent map resharding (pr#66518, Igor Fedotov, Adam Kupczyk)
os/bluestore: Disable invoking unittest_deferred (pr#66357, Adam Kupczyk)
os/bluestore: Fix default base size for histogram (pr#67398, Adam Kupczyk)
os/bluestore: In BlueFS::truncate accept wierd alloc_unit (pr#63753, Adam Kupczyk)
os/bluestore:fix bluestore_volume_selection_reserved_factor usage (pr#66838, Igor Fedotov)
osd/PrimaryLogPG: encode an empty data_bl for empty sparse reads (pr#67356, Ilya Dryomov)
osd/scrub: avoid using moved-from auth_n_errs (pr#66247, Ronen Friedman)
osd/scrub: do not limit operator-initiated repairs (pr#64915, Ronen Friedman)
osd/scrub: do not reduce min chunk on preemption (pr#66236, Ronen Friedman)
osd: Access/Modify epoch maps under mutex in OSDSuperblock class (pr#64732, Mohit Agrawal)
osd: add clear_shards_repaired command (pr#60567, Daniel Radjenovic)
osd: stop scrub_purged_snaps() from ignoring osd_beacon_report_interval (pr#65584, Radoslaw Zarzynski)
OSDMonitor: Make sure pcm is initialised (pr#63804, Brad Hubbard)
pybind/mgr/dashboard: Use teuthology's actual requirements (pr#65417, David Galloway)
pybind/mgr: Fix missing empty lines in mgr_module.py (pr#64266, Ville Ojamo)
pybind/mgr: pin cheroot version in requirements-required.txt (pr#65636, Nizamudeen A, Adam King)
pybind/rados: fix the incorrect order of offset,length in WriteOp.zero (pr#61894, Wang Chao)
qa/cephadm: stop cephadm mgr module during cleanup (pr#61905, Adam King)
qa/cephfs: ignore warning that pg is stuck peering for upgrade jobs (pr#65449, Rishabh Dave)
qa/cephfs: ignore when specific OSD is reported down during upgrade (pr#60391, Rishabh Dave)
qa/cephfs: override testing kernel with -k option (pr#61301, Patrick Donnelly)
qa/cephfs: randomize configs in fs:thrash:workloads (pr#60839, Venky Shankar, Patrick Donnelly)
qa/cephfs: update ignorelist (pr#61417, Rishabh Dave)
qa/rbd/iscsi: ignore MON_DOWN warning in logs (pr#64597, Adam King)
qa/rgw: bucket notifications use pynose (pr#67450, Casey Bodley)
qa/rgw: remove hadoop-s3a subsuite (pr#64668, Casey Bodley)
qa/standalone/scrub: fix "scrubbed in 0ms" in osd-scrub-test.sh (pr#64455, Ronen Friedman)
qa/standalone: fix/improve bluefs tests (pr#67884, Igor Fedotov)
qa/suites/krbd: use a standard fixed-1 cluster in unmap subsuite (pr#64919, Ilya Dryomov)
qa/suites/upgrade: add "Replacing daemon mds" to ignorelist (issue#71615, issue#50279, pr#65298, Venky Shankar)
qa/tasks/backfill_toofull.py: Fix assert failures with & without compression (pr#68119, Sridhar Seshasayee)
qa/tasks/ceph_manager: population must be a sequence (pr#64747, Kyr Shatskyy)
qa/tasks/cephfs/mount: use 'ip route' instead 'route' (pr#63130, Kyr Shatskyy)
qa/tasks/mgr: test_module_selftest set influx hostname to avoid warnings (pr#67800, Nitzan Mordechai)
qa/tasks/rbd_mirror_thrash: don't use random.randrange() on floats (pr#67162, Ilya Dryomov)
qa/tasks/thrashosds-health: whitelist PG_BACKFILL_FULL (pr#66973, Naveen Naidu)
qa/tasks/workunit: fix no module named 'pipes' (pr#66251, Kyr Shatskyy)
qa/tasks: generalize stuck pg ignorelist entry (pr#64420, Laura Flores)
qa/tasks: make rbd_mirror_thrash inherit from ThrasherGreenlet (pr#67794, Ilya Dryomov)
qa/tests: added 19.2.3 to the mix (pr#65244, Yuri Weinstein)
qa/tests: added initial test for client-upgrade-squid-tentacle (pr#64897, Yuri Weinstein)
qa/valgrind.supp: make gcm_cipher_internal suppression more resilient (pr#67280, Ilya Dryomov)
qa/workunits/cephtool: add extra privileges to cephtool script (pr#63026, Laura Flores)
qa/workunits/fs/misc: remove data pool cleanup (pr#63018, Patrick Donnelly)
qa/workunits/rbd: adapt rbd_mirror.sh for trial nodes (pr#67151, Ilya Dryomov)
qa/workunits/rbd: reduce randomized sleeps in live import tests (pr#67153, Ilya Dryomov)
qa/workunits/rgw: drop netstat usage (pr#67185, Kyr Shatskyy)
qa: add missing .qa links (pr#67497, Patrick Donnelly)
qa: correct daemon for warning conf (pr#61304, Patrick Donnelly)
qa: Disable OSD benchmark from running for tests (pr#67066, Sridhar Seshasayee)
qa: do not fail cephfs QA tests for slow bluestore ops (pr#61302, Patrick Donnelly)
qa: do read checks with non-empty file (pr#61858, Patrick Donnelly)
qa: don't assume that /dev/sda or /dev/vda is present in unmap.t (pr#67076, Ilya Dryomov)
qa: fix multi-fs tests in test_mds_metrics.py (pr#64336, Jos Collin)
qa: ignore cluster warning (evicting unresponsive ...) with tasks/mgr-osd-full (issue#73278, pr#66126, Venky Shankar)
qa: ignore pg availability/degraded warnings (pr#61303, Patrick Donnelly)
qa: ignore variant of down fs (pr#62091, Patrick Donnelly)
qa: ignore warnings variations (pr#59617, Patrick Donnelly)
qa: increase the wait time to prevent check_counter failing (pr#64449, Jos Collin)
qa: krbd_blkroset.t: eliminate a race in the open_count test (pr#67074, Ilya Dryomov)
qa: krbd_rxbounce.sh: do more reads to generate more errors (pr#67454, Ilya Dryomov)
qa: make test_progress atomically capture OSD marked in/out events (pr#67761, Kamoltat (Junior) Sirivadhna)
qa: multiple fixes in test_mirroring.py (pr#59829, Jos Collin)
qa: rbd_mirror_fsx_compare.sh doesn't error out as expected (pr#67796, Ilya Dryomov)
qa: relocate subvol creation overrides and test (pr#59922, Milind Changire)
qa: some test set refuse\_client\_session, so the cluster log is expected (issue#66639, pr#59688, Venky Shankar)
qa: suppress OpenSSL valgrind leaks (pr#65661, Laura Flores)
qa: use a larger timeout for kernel_untar_build workunit (issue#68855, pr#65280, Venky Shankar)
rbd-mirror: add cluster fsid to remote meta cache key (pr#66296, Mykola Golub)
rbd-mirror: allow incomplete demote snapshot to sync after rbd-mirror daemon restart (pr#66165, VinayBhaskar-V)
rbd-mirror: prevent image deletion if remote image is not primary (pr#64739, VinayBhaskar-V)
Recent pipeline backports (pr#65249, Dan Mick)
Revert "doc: mgr/dashboard: add OAuth2 SSO documentation" (pr#66797, Ville Ojamo)
Revert "pybind/mgr: Hack around the 'ImportError: PyO3 modules may on… (pr#65959, Nizamudeen A)
rgw-test: fix bug kafka unexpected keyword argument 'expected_sizes' (pr#66033, Hoai-Thu Vuong)
rgw/account: bucket acls are not completely migrated once the user is migrated to an account (pr#65665, kchheda3)
rgw/admin: Add max-entries and marker to bucket list (pr#65486, Tobias Urdin)
rgw/lc: LCOpAction_CurrentExpiration checks mtime for delete markers (pr#65966, Casey Bodley)
rgw/lc: stop using merge_and_store_attrs in remove_bucket_config (pr#64741, Casey Bodley)
rgw/notification: Prevent reserved_size leak by decrementing overhead on commit/abort (pr#67575, Krunal Chheda, kchheda3)
rgw/s3: remove 'aws-chunked' from Content-Encoding response (pr#65219, Casey Bodley)
rgw: add metric when send message with kafka and ampq (pr#65903, Hoai-Thu Vuong)
rgw: allow specifying ssl certificate for radosgw-admin operations (pr#64357, Mark Kogan)
rgw: beast add ssl hot-reload (pr#66289, Anthony D'Atri, Yuval Lifshitz, Henry Richter)
rgw: check all JWKS for STS (pr#64936, Alex Wojno)
rgw: discard olh\_ attributes when copying object from a versioning-suspended bucket to a versioning-disabled bucket (pr#65556, Jane Zhu)
rgw: don't use merge_and_store_attrs() when recreating a bucket (pr#64387, Casey Bodley)
rgw: fix 'bucket rm --bypass-gc' for copied objects (pr#66003, Casey Bodley)
rgw: fix radosgw-admin object unlink ... (pr#66152, J. Eric Ivancich)
rgw: fix empty storage class on display of multipart uploads (pr#64313, J. Eric Ivancich)
rgw: fix v2 topics radosgw-admin output in test (pr#64485, Yuval Lifshitz)
rgw: java s3-tests change setting of JAVA_HOME (pr#68227, J. Eric Ivancich)
rgw: make keystone work without admin token(service ac requirement) (pr#64202, Deepika Upadhyay)
RGW: multi object delete op; skip olh update for all deletes but the last one (pr#65500, Oguzhan Ozmen)
rgw: trigger resharding of versioned buckets sooner (pr#63567, J. Eric Ivancich, Casey Bodley, Mykola Golub)
rgw: update keystone repo stable branch to 2024.2 (pr#66242, Kyr Shatskyy)
RGW:fix obj by multipart upload cant get tag (pr#66335, Raja Sharma)
Rocky 9/10 support backports (pr#64780, Zack Cerza, John Mulligan, David Galloway)
run-make-check.sh: handle sudo and command that may not run in container (pr#65838, John Mulligan)
src/ceph_osd, osd: Implement running benchmark during OSD creation - Phase 1 (pr#65523, Sridhar Seshasayee)
src/common: add guidance for mon_warn_pg_not_scrubbed (pr#62551, Zac Dover)
src/pybind/mgr/pg_autoscaler/module.py: fix 'pg_autoscale_mode' output (pr#59443, Kamoltat)
suites/rados/cephadm: typo in ignore list for still running message (pr#65444, Nitzan Mordechai)
sync build-with-container patches from main (pr#65844, John Mulligan, Dan Mick)
tasks/cbt_performance: Tolerate exceptions during performance data up… (pr#66104, Nitzan Mordechai)
tasks/cephfs/mount: use 192.168.144.0.0/20 for brxnet (pr#63133, Kyr Shatskyy)
test-rgw-multisite: create default realm in multisite test script (pr#65273, Shilpa Jagannath)
test/common: unittest_fault_injector omits unit-main target (pr#63981, Casey Bodley)
test/encoding/readable: Add backward incompat checks (pr#67392, Nitzan Mordechai)
test/libcephfs: copy DT_NEEDED entries from input libraries (pr#63720, Patrick Donnelly)
test/rbd: remove unit tests about cache tiering (pr#64589, Laura Flores)
test/rgw/kafka: fix kafka relase to more recent one (pr#67994, Yuval Lifshitz)
test: disable known flaky tests in run-rbd-unit-tests (pr#67558, Ilya Dryomov)
test: use json_extract instead of awkward json_tree (pr#67322, Patrick Donnelly)
The compilation of ISAL compress in the current code depends on the macro HAVE_NASM_X64_AVX2. However, the macro HAVE_NASM_X64_AVX2 has been removed, resulting in the compression not using ISAL even if the compressor_zlib_isal parameter is set to true (pr#64815, Yao guotao)
tools: handle get-attr as read-only ops in ceph_objectstore_tool (pr#66538, Jaya Prakash)
tools: respect set features when adding addresses (pr#62061, Radosław Zarzyński)
Wip trackers 50371 67352 67489 69639 squid (pr#62472, Brad Hubbard, Patrick Donnelly)

From Laminated Pages to Petabytes: Why I'm Joining the Ceph Foundation

2026-05-06T00:00:00Z

Hello, Ceph Community! I'm Emmanuel Ameh, and I'm thrilled to be joining as your new Technical Writer. My path into technical writing started during a university internship at a manufacturing plant, where I was handed a problem that, looking back, shaped how I think about documentation to this day. The first piece of documentation I wrote was a one-page troubleshooting guide. It was a simple list of common production-line issues and their fixes, taped to a workstation. But it worked. Operators stopped escalating the same problems, uptime held steady, and that single laminated page did the job of a dozen phone calls.

I think about that page a lot. It wasn't elegant. It wasn't comprehensive. It didn't win any awards. But it sat exactly where the people who needed it could find it, and it told them exactly what to do. That, to me, is what good documentation looks like, whether it's protecting a production line or a petabyte-scale storage cluster.

That experience taught me my core philosophy: good documentation isn't measured by how thorough it looks. It's measured by what it lets people do. A doc that keeps a cluster running is worth more than a hundred pages nobody reads.

That belief is what brings me to the Ceph community today.

My Approach: Test-Driven Documentation ¶

My background is in enterprise software and developer advocacy, where I've focused on making complex systems feel approachable. I treat usability as seriously as technical accuracy. To me, a "perfectly correct" doc that a user can't follow is a broken feature.

I advocate for test-driven documentation. In practice, this means:

Validation: Running every command and workflow in a clean environment to ensure it actually works. If a tutorial says "run this, and you'll see X," then somewhere, on a fresh install, I should be able to run it and see X. If I can't, neither can the reader.
Friction Logging: Identifying the exact moment a newcomer gets stuck and smoothing that path. The most valuable feedback I can get is "I gave up here, and this is why."
Modular Architecture: Organizing information so you find the "how-to" immediately, with the "why-it-works" just a click away. Operators in the middle of an incident shouldn't have to read three pages of theory to recover a degraded cluster.

This matters more for a project like Ceph than for most. When documentation fails for a CRUD app, someone files a support ticket. When documentation fails for a storage system, data is at stake.

Why Ceph? ¶

Ceph sits at an incredible intersection where small configuration decisions can have large operational consequences. Ceph is a deeply technical infrastructure that many people depend on. Research labs, cloud providers, hospitals, universities, and enterprises all run on Ceph, and most of their users have no idea. That's the kind of project I want to work on. Software that is so reliable it becomes invisible.

I'm also drawn to the Ceph project because it operates in the territory where documentation matters most. Distributed systems are hard. The mental model required to understand placement groups, CRUSH maps, and OSD recovery isn't something you absorb in an afternoon. The key difference between a confident operator and a terrified one is whether the documentation addressed their specific needs.

I'll be honest. Like any mature project, Ceph has a "gravity" of knowledge that can feel intimidating. My goal is to do the translation work: taking the nuanced engineering expertise living in maintainers' heads and shaping it into guidance that someone two time zones away can act on at 2 a.m. when an OSD is degraded.

What I'm Focused On ¶

In my first few months, I'll be doing more listening than writing. Specifically, I want to:

Work through the documentation issue backlog. There are a significant number of open documentation issues, and many of them represent real friction that real users have already flagged. Steadily closing them out is one of the highest-leverage things I can do early on.
Identify the "fall-off cliffs" where readers commonly give up, get confused, or start guessing.
Talk to contributors and operators about what's missing, what's outdated, and what they wish existed.
Build a roadmap for documentation improvements that's transparent, prioritized, and grounded in real user needs.

You won't see me rewriting things for the sake of rewriting them. Where the docs already work well, I'll leave them alone. Where they don't, I'd rather fix the right problem slowly than the wrong one quickly.

How You Can Help Me ¶

Good documentation is a collaborative effort. To help me hit the ground running, I'm looking for three things from you:

Share the "Why": When reviewing a doc change, the reasoning behind a design decision is often more valuable than the decision itself. If you've ever explained a feature in a Slack thread or a mailing list reply, that explanation is gold to me.
Highlight the Friction: Tell me where the docs let you down. If you had to dive into the source code because a section was confusing, that is where I need to spend my time. The same applies to sections you skip, doubt, or wish were present.
Give Blunt Feedback: I'd much rather get a "this is wrong/missing context" review than a polite one. Push back on my drafts. The faster I learn what doesn't work, the faster the docs get better for everyone.

In return, here's what you can expect from me. I'll show my work. I'll explain my reasoning. I'll ask questions before making big changes. And I'll treat the docs the same way I'd want code I depend on to be treated: with care, with tests, and with a clear changelog.

Let's Connect ¶

I'm not just here to write; I'm here to learn this project deeply alongside you. Whether you're spinning up your first cluster or operating at petabyte scale, I want to make sure the documentation supports you. Great infrastructure only reaches its potential when people can actually use it. My job is to close that gap, and I'm looking forward to doing that with the Ceph community.

Find me on Slack/IRC: @eameh
Flag an issue: Tag me in any documentation PRs or issues on GitHub.
Send me your war stories: If you've ever fought the docs and won, I want to know how you did it.

I joined the Ceph Foundation because I believe that great infrastructure deserves great documentation, and that the second is what lets the first reach the people who need it. I'm looking forward to building that with you.

v20.2.1 Tentacle released

2026-04-06T00:00:00Z

This is the first minor release in the Tentacle series. We recommend that all users update to this release.

Release Date ¶

April 06, 2026

Notable Changes ¶

OSD / BlueStore ¶

EC Recovery: Fixed a length calculation bug in erase_after_ro_offset() that caused empty shards to retain data, leading to shard_size >= tobj_size assertion failures when recovering small objects in EC pools.
BlueFS Volume Selector: Updated the BlueFS volume selector to properly account for file size changes when recovering the WAL in envelope mode.
BlueFS: Fixed a bug where stat() missed the actual file size update after indexing WAL envelope files.

Monitor (mon) ¶

Fast EC Restrictions: Denied the ability to enable EC optimizations ("fast EC") for non-4K-aligned chunk sizes. Unaligned chunk sizes handled by fast EC perform poorly and suffer from bugs, so attempts to force this configuration are now rejected.
Peering: Ensured ceph pg repeer proposes a correctly sized pg temp, as optimized EC cannot cope with mismatched sizes.
NVMeoF Gateway: Added a new nvme-gw listeners command to display all existing listeners (including auto-listeners) inside a pool/group.
NVMeoF Failover: Overhauled the NVMeoF Gateway fast-failover logic. Beacon timeouts are now evaluated within prepare_beacon to support shorter intervals, and the mechanism for detecting monitor slowness was improved.

librbd & rbd-mirror ¶

RBD: Introduced a new RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT policy for rbd_lock_acquire(). This is a low-level interface intended to allow a peer to grab exclusive lock manually for short periods of time with other peers pausing their activity and waiting for the lock to be released rather than instantly aborting I/O and returning an error. It's possible to switch from RBD_LOCK_MODE_EXCLUSIVE to RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT policy and vice versa even if the lock is already held.

Ceph Object Gateway (RGW) ¶

Multi-Part Operations: Fixed conditional validation handling in MultiWrite, Delete, and MultiDelete workflows.

mgr/dashboard ¶

UI Navigation: Redesigned the main landing page; the "Dashboard" navigation item was renamed to "Overview" and uses a new carbonized productive card layout.
NVMeoF Management: Added the nvmeof get_subsystems CLI command, fixed JSON output indentation for NVMeoF CLI commands, and reverted the server_addr API parameter back to traddr for consistency.
Hosts View: Fixed a bug causing the IP addresses of hosts to be hidden on the Hosts page due to an issue with fact merging.
Forms & Modals: Standardized forms onto the Carbon Design System, including the pools form, service form, multi-site realm token export modal, delete zone modal, and password change forms.
Form Validation: Generalized form error handling and validations using a new cdValidate directive.

mgr/cephadm ¶

Monitoring Stack: Bumped the default container image versions for the monitoring stack: Prometheus to v3.6.0, Node-exporter to v1.9.1, Alertmanager to v0.28.1, and Grafana to v12.2.0.

Security Changes ¶

Monitoring Stack Images: Updated Prometheus, Alertmanager, and Grafana container image versions, picking up upstream security and stability fixes.

Configuration Changes ¶

bluefs_check_volume_selector_on_mount: The previous bluefs_check_volume_selector_on_umount debug setting was renamed and repurposed. It now checks for volume selector inconsistencies on both mount and unmount phases.
mon_nvmeofgw_beacon_grace: The default grace period before marking a gateway as failed has been reduced from 10 seconds to 7 seconds for faster failover.
nvmeof_mon_client_tick_period: The default beacon tick interval has been lowered from 2 seconds to 1 second.

Changelog ¶

[rgw][tentacle] backport of cloud-restore related PRs (pr#65830, Soumya Koduri)
Add normalization and casesensitive options to the subvolume group creation command (pr#65564, Venky Shankar, Xavi Hernandez)
auth: msgr2 can return incorrect allowed_modes through AuthBadMethodFrame (pr#65336, Miki Patel)
backports variants improvements and Dockerfile.build changes (pr#66010, John Mulligan, Zack Cerza)
Beacon diff (pr#66958, Leonid Chernin, Samuel Just)
blk/kernel: bring "bdev_async_discard" config parameter back (pr#65609, Igor Fedotov)
blk/kernel: improve DiscardThread life cycle (pr#65213, Igor Fedotov)
bluestore/BlueFS: fix bytes_written_slow counter with aio_write (pr#66355, chungfengz)
build-with-container: add argument groups to organize options (pr#65628, John Mulligan)
build-with-container: build image variants (pr#65946, John Mulligan)
ceph-mixin: Update monitoring mixin (pr#65692, Aashish Sharma, SuperQ, Ankush Behl)
ceph-volume: fix UdevData initialisation from empty /run/udev/data/* file (pr#65923, Matteo Paramatti)
ceph-volume: lvm.Lvm.setup_metadata_devices refactor (pr#65925, Guillaume Abrioux)
ceph-volume: support additional dmcrypt params (pr#65544, Guillaume Abrioux)
ceph-volume: use udev data instead of LVM subprocess in get_devices() (pr#65921, Guillaume Abrioux)
ceph_release, doc/dev: update tentacle as stable release (pr#65988, Laura Flores)
cephadm, debian/rules: Use system packages for cephadm bundled dependencies (pr#66256, Kefu Chai)
cephadm: fix building rpm-sourced cephadm zippapp on el10 (pr#65292, John Mulligan)
cephadm: set default image for tentacle release (pr#65719, Adam King)
cephadm: support custom distros by falling back to ID_LIKE (pr#65696, bachmanity1)
cephfs-journal-tool: Journal trimming issue (pr#65601, Kotresh HR)
client: fix async/sync I/O stalling due to buffer list exceeding INT_MAX (pr#65256, Dhairya Parmar)
client: fix dump_mds_requests to valid json format (issue#73639, pr#66156, haoyixing)
client: fix unmount hang after lookups (pr#65254, Dhairya Parmar)
client: use path supplied in statfs (pr#65132, Christopher Hoffman)
common/frag: properly convert frag_t to net/store endianness (pr#66540, Patrick Donnelly, Max Kellermann)
common: Allow PerfCounters to return a provided service ID (pr#65587, Adam C. Emerson)
debian/control: add iproute2 to build dependencies (pr#66737, Kefu Chai)
debian/control: Add libxsimd-dev build dependency for vendored Arrow (pr#66248, Kefu Chai)
debian/control: record python3-packaging dependency for ceph-volume (pr#66590, Thomas Lamprecht, Max R. Carrara)
doc/cephfs: fix docs for pause_purging and pause_cloning (pr#66452, Rishabh Dave)
doc/mgr/smb: document the 'provider' option for smb share (pr#65617, Sachin Prabhu)
doc/radosgw: change all intra-docs links to use ref (1 of 6) (pr#67043, Ville Ojamo)
doc/radosgw: change all intra-docs links to use ref (2 of 6) (pr#67084, Ville Ojamo)
doc/radosgw: Cosmetic improvements and ref links in account.rst (pr#67064, Ville Ojamo)
doc/rbd/rbd-config-ref: add clone settings section (pr#66175, Ilya Dryomov)
doc: add Tentacle to os recommendations (pr#66464, Casey Bodley, Joseph Mundackal)
doc: fetch releases from main branch (pr#67002, Patrick Donnelly)
doc: Pin pip to <25.3 for RTD as a workaround for pybind in admin/doc-read-the-docs.txt (pr#66106, Ville Ojamo)
doc: Remove sphinxcontrib-seqdiag Python package from RTD builds (pr#67296, Ville Ojamo)
doc: Update dashboard pending release notes (pr#65984, Afreen Misbah)
encode: Fix bad use of DENC_DUMP_PRE (pr#66565, Adam Kupczyk)
Fast failover (pr#67150, leonidc, Leonid Chernin)
Fix multifs auth caps check (pr#65358, Kotresh HR)
Form retains old data when switching from edit to create (pr#65654, pujashahu)
Generalize error handling for angular forms (pr#66904, Afreen Misbah)
github: pin GH Actions to SHA-1 commit (pr#65761, Ernesto Puerta)
install-deps.sh: install proper compiler version on Debian/Ubuntu (pr#66015, Dan Mick)
install-deps: Replace apt-mirror (pr#66672, David Galloway)
libcephfs: New feature - add ceph_setlk and ceph_getlk functions (pr#65258, Giorgos Kappes)
librbd: fix ExclusiveLock::accept_request() when !is_state_locked() (pr#66628, Ilya Dryomov)
librbd: introduce RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT (pr#67279, Ilya Dryomov)
mds/FSMap: fix join_fscid being incorrectly reset for active MDS during filesystem removal (pr#65777, ethanwu)
mds/MDSDaemon: unlock mds\_lock while shutting down Beacon and others (pr#64885, Max Kellermann)
mds: dump export_ephemeral_random_pin as double (pr#65163, Enrico Bocchi)
mds: fix rank 0 marked damaged if stopping fails after Elid flush (pr#65778, ethanwu)
mds: Fix readdir when osd is full (pr#65346, Kotresh HR)
mds: fix snapdiff result fragmentation (pr#65362, Igor Fedotov, Md Mahamudur Rahaman Sajib)
mds: include auth credential in session dump (pr#65255, Patrick Donnelly)
mds: Return ceph.dir.subvolume vxattr (pr#65779, Edwin Rodriguez)
mds: skip charmap handler check for MDS requests (pr#64953, Patrick Donnelly)
mds: wrong snap check for directory with parent snaps (pr#65259, Patrick Donnelly)
mgr/alerts: enforce ssl context to SMTP_SSL (pr#66140, Nizamudeen A)
mgr/cephadm: Add some new fields to the cephadm NVMEoF spec file (pr#66987, Gil Bregman)
mgr/cephadm: bump monitoring stack versions (pr#65895, Nizamudeen A)
mgr/cephadm: Change the default of max hosts per namespace in NVMEoF to 16 (pr#66819, Gil Bregman)
mgr/cephadm: don't mark nvmeof daemons without pool and group in name as stray (pr#65594, Adam King)
mgr/cephadm: update grafana conf for disconnected environment (pr#66209, Nizamudeen A)
mgr/cephadm: Use a persistent volume to store Loki DB (pr#66023, Aashish Sharma)
mgr/DaemonServer: fixed mistype for mgr_osd_messages (pr#63345, Konstantin Shalygin)
mgr/DaemonState: Minimise time we hold the DaemonStateIndex lock (pr#65464, Brad Hubbard)
mgr/dasboard : Carbonize pools form (pr#66789, Abhishek Desai, Ankit Kumar)
mgr/dashboard : Fixed labels issue (pr#66603, Abhishek Desai)
mgr/dashboard : Carbonize -> Report an issue modal (pr#66048, Abhishek Desai)
mgr/dashboard : fix - about model tooltip issue (pr#66276, Devika Babrekar)
mgr/dashboard : fix - CephFS Authorize Modal Update issue (pr#66419, Devika Babrekar)
mgr/dashboard : fix css for carbon input fields (pr#65490, Abhishek Desai)
mgr/dashboard : Fix secure-monitoring-stack creds issue (pr#65943, Abhishek Desai)
mgr/dashboard : Fixed mirrored image usage info bar (pr#65491, Abhishek Desai)
mgr/dashboard : Fixed usage bar for secondary site in rbd mirroing (pr#65927, Abhishek Desai)
mgr/dashboard : Fixed warning icon colour issue with carbon colour (pr#66271, Abhishek Desai)
mgr/dashboard : Hide suppressed alert on landing page (pr#65737, Abhishek Desai)
mgr/dashboard : Remove subalerts details for multiple subalerts (pr#66295, Abhishek Desai)
mgr/dashboard : Skip calls until secure_monitoring_stack is enabled (pr#65673, Abhishek Desai)
mgr/dashboard: --no-group-append default value to False, aligned with old cli" (pr#65678, Tomer Haskalovitch)
mgr/dashboard: Add Archive zone configuration to the Dashboard (pr#67131, Aashish Sharma)
mgr/dashboard: add customizations to table-actions (pr#65956, Naman Munet)
mgr/dashboard: Add full page tearsheet component (pr#66892, Afreen Misbah)
mgr/dashboard: Add generic wizard component (pr#66893, Afreen Misbah)
mgr/dashboard: add get_subsystem nvme command (pr#66941, Tomer Haskalovitch)
mgr/dashboard: add indentation to the json output of nvmeof cli commands (pr#66940, Tomer Haskalovitch)
mgr/dashboard: add multiple ceph users deletion (pr#65658, Pedro Gonzalez Gomez)
mgr/dashboard: add nsid param to ns add command (pr#65677, Tomer Haskalovitch)
mgr/dashboard: add nsid param to ns list command (pr#65749, Tomer Haskalovitch)
mgr/dashboard: Add overview page and change 'Dashboard' to 'Overview' (pr#67118, Afreen Misbah)
mgr/dashboard: Add productive card component (pr#67147, Afreen Misbah)
mgr/dashboard: add text-label-list component (pr#66312, Pedro Gonzalez Gomez)
mgr/dashboard: Adding QAT Compression dropdown on RGW Service form (pr#66642, Devika Babrekar)
mgr/dashboard: allow deletion of non-default zone and zonegroup (pr#66211, Aashish Sharma)
mgr/dashboard: Allow FQDN in Connect Cluster form -> Cluster API URL (pr#65622, Aashish Sharma)
mgr/dashboard: Blank entry for Storage Capacity in dashboard under Cluster > Expand Cluster > Review (pr#65705, Naman Munet)
mgr/dashboard: bump validator package to address vulnerability (pr#66227, Naman Munet)
mgr/dashboard: Carbonize - Multisite Zone (pr#67117, Dnyaneshwari Talwekar)
mgr/dashboard: Carbonize Administration module > Create Realm/Zone group/zone (pr#66986, Dnyaneshwari Talwekar)
mgr/dashboard: Carbonize multisite sync policy forms (pr#66302, Naman Munet)
mgr/dashboard: carbonize service form (pr#66978, Pedro Gonzalez Gomez)
mgr/dashboard: Carbonize the Change Password Form (pr#66401, Afreen Misbah)
mgr/dashboard: carbonize-delete-zone-modal (pr#67100, Sagar Gopale)
mgr/dashboard: carbonize-delete-zonegroup-modal (pr#67014, Sagar Gopale)
mgr/dashboard: carbonized-multisite-export-realm-token-modal (pr#66649, Sagar Gopale)
mgr/dashboard: change the default max namespace from 4096 to None in subsystem add command (pr#65951, Tomer Haskalovitch)
mgr/dashboard: Edit user via UI throwing multiple server errors (pr#66081, Naman Munet)
mgr/dashboard: empty-data-message (pr#66902, Sagar Gopale)
mgr/dashboard: fetch all namespaces in a gateway group (pr#67140, Afreen Misbah)
mgr/dashboard: fix command alias help message (pr#65750, Tomer Haskalovitch)
mgr/dashboard: fix dashboard freeze on missing smb permissions (pr#65873, Pedro Gonzalez Gomez)
mgr/dashboard: fix data mismatch in Advance section in Tiering (pr#65672, Dnyaneshwari Talwekar)
mgr/dashboard: Fix display of IP address in host page (pr#67146, Afreen Misbah)
mgr/dashboard: fix icon alignment in navigation header (pr#66091, Naman Munet)
mgr/dashboard: fix misaligned text links on login page (pr#66052, prik73, Afreen Misbah)
mgr/dashboard: fix missing schedule interval in rbd API (pr#65560, Nizamudeen A)
mgr/dashboard: fix multi-cluster route reload logic (pr#66504, Aashish Sharma)
mgr/dashboard: fix multisite wizard realm configuration mode (pr#66017, Aashish Sharma)
mgr/dashboard: fix None force param handling in ns add_host so it won't raise exceptions (pr#65679, Tomer Haskalovitch)
mgr/dashboard: fix ns add and resize commands help (pr#66939, Tomer Haskalovitch)
mgr/dashboard: fix oauth2-service creation UI error (pr#66139, Nizamudeen A)
mgr/dashboard: fix prometheus API error when not configured (pr#65856, Nizamudeen A)
mgr/dashboard: fix rbd form mirroring toggle (pr#65874, Nizamudeen A)
mgr/dashboard: fix RBD mirror schedule inheritance in pool and image APIs (pr#67107, Imran Imtiaz)
mgr/dashboard: fix smb button and table column (pr#65657, Pedro Gonzalez Gomez)
mgr/dashboard: Fix table width expansion on manager module dropdown selection #74089 (pr#66647, Sagar Gopale)
mgr/dashboard: fix the separation between CLI and API only commands (pr#65781, Tomer Haskalovitch)
mgr/dashboard: Fix timestamps in APIs (pr#66029, Afreen Misbah)
mgr/dashboard: fix total capacity value in dashboard (pr#65647, Nizamudeen A)
mgr/dashboard: fix typo in error when gw does not exist (pr#66956, Tomer Haskalovitch)
mgr/dashboard: fix zone update API forcing STANDARD storage class (pr#65619, Aashish Sharma)
mgr/dashboard: fixes for quick-bootstrap script (pr#67040, Nizamudeen A)
mgr/dashboard: FS - Attach Command showing undefined for MountData (pr#65675, Dnyaneshwari Talwekar)
mgr/dashboard: Group similar alerts (pr#65493, Abhishek Desai)
mgr/dashboard: Handle pool creation in tiering local storage class creation (pr#65680, Dnyaneshwari, Naman Munet)
mgr/dashboard: Maintain sentence case consistency in side nav bar titles (pr#66050, Aashish Sharma)
mgr/dashboard: ns list now support not passing nqn param (pr#65897, Tomer Haskalovitch)
mgr/dashboard: raise exception if both size and rbd_image_size are being passed in ns add (pr#65816, Tomer Haskalovitch)
mgr/dashboard: rbd consistency group and snapshot APIs (pr#66935, Imran Imtiaz)
mgr/dashboard: Remove illegible texts from the dashboard (pr#66306, Afreen Misbah)
mgr/dashboard: remove not needed 'cli_version' field from gw info com… (pr#66942, Tomer Haskalovitch)
mgr/dashboard: Remove the time dropdown from grafana iframe (pr#65853, Abhishek Desai)
mgr/dashboard: removes nx folder (pr#67003, Afreen Misbah)
mgr/dashboard: rename 'Zone Group' labels to 'Zonegroup' (pr#66790, Sagar Gopale)
mgr/dashboard: Rename Alerts tab to All Alerts (pr#66532, Sagar Gopale)
mgr/dashboard: Rename side-nav panel items (pr#65846, Naman Munet)
mgr/dashboard: replace bootstrap badges with carbon tags (pr#66350, pujaoshahu)
mgr/dashboard: replace usage or progress bar with carbon meter chart (pr#66934, Naman Munet)
mgr/dashboard: rgw accounts form group mode disable option is not working (pr#66351, Naman Munet)
mgr/dashboard: server side table rendering improvements (pr#65828, Nizamudeen A)
mgr/dashboard: service creation fails if service name is same as sevice type (pr#66481, Naman Munet)
mgr/dashboard: Set max subsystem count to 512 rather than 4096 (pr#66284, Afreen Misbah)
mgr/dashboard: support gw get_stats and listener info (pr#65896, Tomer Haskalovitch)
mgr/dashboard: Tiering form - Placement Target in Advanced Section (pr#65653, Dnyaneshwari Talwekar)
mgr/dashboard: update teuth_ref hash in api test (pr#66706, Nizamudeen A)
mgr/dashboard:[NFS] add Subvolume Groups and Subvolumes in "Edit NFS Export form" (pr#65650, Dnyaneshwari Talwekar)
mgr/prometheus: Handle empty/invalid JSON from orch get-security-config (pr#65906, Sunnatillo)
mgr/telemetry: add 'ec_optimizations' flag to 'basic_pool_flags' collection (pr#65969, Laura Flores)
mgr/vol: handling the failed non-atomic operation (pr#65728, Neeraj Pratap Singh)
mgr/vol: keep and show clone source info (pr#64650, Rishabh Dave)
mgr/volumes: Keep mon caps if auth key has remaining mds/osd caps (pr#65262, Enrico Bocchi)
mgr/volumes: remove unnecessary log error lines from earmark handling (pr#66991, Avan Thakkar)
mgr: avoid explicit dropping of ref (pr#65005, Milind Changire)
mgr:python: avoid pyo3 errors by running certain cryptographic functions in a child process (pr#66794, Nizamudeen A, John Mulligan, Paulo E. Castro)
mon/FSCommands: avoid unreachable code triggering compiler warning (pr#65261, Patrick Donnelly)
mon/MgrMonitor: add a space before "is already disabled" (pr#64687, Zehua Qi)
mon/OSDMonitor.cc: optionally display availability status in json (pr#65794, Shraddha Agrawal)
mon: Add command "nvme-gw listeners" (pr#66584, Vallari Agrawal)
mon: ceph pg repeer should propose a correctly sized pg temp (pr#66324, Alex Ainscow)
mon: Deny EC optimizations (fast EC) for non-4k-aligned chunk-sizes (pr#67319, Alex Ainscow)
monc: synchronize tick() of MonClient with shutdown() (pr#66916, Radoslaw Zarzynski)
monitoring: fix "In" OSDs in Cluster-Advanced grafana panel. Also change units from decbytes to bytes wherever used in the panel (pr#65670, Aashish Sharma)
monitoring: fix "Total gateway" and "Ceph Health NVMeoF WARNING" grafana graphs (pr#66225, Vallari Agrawal)
monitoring: fix CephPgImbalance alert rule expression (pr#66828, Aashish Sharma)
monitoring: Fix Filesystem grafana dashboard units (pr#66018, Ankush Behl)
monitoring: fix MTU Mismatch alert rule and expr (pr#65708, Aashish Sharma)
monitoring: fix rgw_servers filtering in rgw sync overview grafana (pr#66989, Aashish Sharma)
monitoring: Fixes for smb overview (pr#66019, Ankush Behl)
monitoring: make cluster matcher backward compatible for pre-reef metrics (pr#66984, Aashish Sharma)
monitoring: update NVMeoFTooManyNamespaces to 4096 ns (pr#67039, Vallari Agrawal)
monitoring: upgrade grafana version to 12.3.1 (pr#66963, Aashish Sharma)
nvmeof: refactor beacon timer for exact frequency timing with drift correction (pr#66536, Alexander Indenbaum)
Objecter: respect higher epoch subscription in tick (pr#66972, Nitzan Mordechai)
os/bluestore: cumulative patch to fix extent map resharding and around (pr#65964, Igor Fedotov, Adam Kupczyk, Jaya Prakash)
os/bluestore: fix vselector update after enveloped WAL recovery (pr#67333, Igor Fedotov, Adam Kupczyk)
os/bluestore: introduce device type specific allocation policy (pr#66839, Igor Fedotov)
osd/ECUtil: Fix erase_after_ro_offset length calculation and add tests (pr#66825, Alex Ainscow)
osd/PeeringState: re-evaluate full OSDs while waiting for recovery re… (pr#65701, Nitzan Mordechai)
osd/scrub: do not reduce min chunk on preemption (pr#66214, Ronen Friedman)
osd/scrub: fix blocked scrub accounting (pr#66220, Ronen Friedman)
osd/scrub: new/modified perf counters for scrub preemption (pr#66234, Ronen Friedman)
osd: Do not remove objects with divergent logs if only partial writes (pr#66725, Alex Ainscow)
osd: Fix fast EC truncate to whole stripe (pr#66543, Alex Ainscow)
osd: Fix for num_bytes mismatch occurring from snapshot workloads with partial writes in fast_ec (pr#67137, Jon Bailey)
osd: Fix memory leak of ECDummyOp (pr#66977, Alex Ainscow)
osd: Fix stats mismatch cluster error seen during scrubbing occasionally (pr#65793, Jon Bailey)
osd: Relax missing entry assert for partial writes (pr#65860, Alex Ainscow)
osd: stop scrub_purged_snaps() from ignoring osd_beacon_report_interval (pr#65478, Radoslaw Zarzynski)
pickup object corpus 20.2.0 380 gdbcbbd3f281 (pr#66592, Nitzan Mordechai)
prometheus: Add Cephadm orch ps output metric to prometheus (pr#66760, Ankush Behl)
pybind/mgr/dashboard: dashboard/requirements-lint.txt: re-pin rsscheck (pr#66877, Ronen Friedman)
pybind/mgr/pg_autoscaler: Introduce dynamic threshold to improve scal… (pr#66871, Prashant D)
pybind/mgr: pin cheroot version in requirements-required.txt (pr#65635, Adam King)
pybind/rados: Add list_lockers() and break_lock() to Rados Python interface (pr#65098, Gil Bregman)
qa/multisite: switch to boto3 (pr#67318, Shilpa Jagannath, Adam C. Emerson)
qa/rgw: bucket notifications use pynose (pr#67449, Casey Bodley, Adam C. Emerson)
qa/standalone/availability.sh: retry after feature is turned on (pr#67226, Shraddha Agrawal)
qa/suites/nvmeof: add upgrade sub-suite (pr#65583, Vallari Agrawal)
qa/suites/rados/thrash-old-clients: Add OSD warnings to ignore list (pr#65369, Naveen Naidu)
qa/suites/rbd/valgrind: don't hardcode os_type in memcheck.yaml (pr#66196, Ilya Dryomov)
qa/suites/upgrade: add "Replacing daemon mds" to ignorelist (issue#71615, issue#50279, pr#64888, Venky Shankar)
qa/suites: wait longer before stopping OSDs with valgrind (pr#63716, Nitzan Mordechai)
qa/tasks/ceph_manager: population must be a sequence (pr#64746, Kyr Shatskyy)
qa/tasks/qemu: rocky 10 enablement (pr#67283, Ilya Dryomov)
qa/tasks/rbd_mirror_thrash: don't use random.randrange() on floats (pr#67163, Ilya Dryomov)
qa/tasks/workunit: fix no module named 'pipes' (pr#66250, Kyr Shatskyy)
qa/tests: added inital draft for tentacle-p2p (pr#67765, Patrick Donnelly, Yuri Weinstein)
qa/tests: added messages to the whitelist (pr#65645, Laura Flores, Yuri Weinstein)
qa/tests: wait for module to be available for connection (pr#67196, Nizamudeen A)
qa/valgrind.supp: make gcm_cipher_internal suppression more resilient (pr#67281, Ilya Dryomov)
qa/workunits/nvmeof/basic_tests: use nvme-cli 2.13 (pr#67285, Vallari Agrawal)
qa/workunits/rados: remove cache tier test (pr#65540, Nitzan Mordechai)
qa/workunits/rbd: adapt rbd_mirror.sh for trial nodes (pr#67152, Ilya Dryomov)
qa/workunits/rbd: reduce randomized sleeps in live import tests (pr#67154, Ilya Dryomov)
qa/workunits/rbd: use the same qemu-iotests version throughout (pr#67282, Ilya Dryomov)
qa/workunits/rgw: drop netstat usage (pr#67184, Kyr Shatskyy)
qa/workunits: add Rocky Linux support to librados tests (pr#67091, Nitzan Mordechai)
qa: Disable OSD benchmark from running for tests (pr#67068, Sridhar Seshasayee)
qa: don't assume that /dev/sda or /dev/vda is present in unmap.t (pr#67077, Ilya Dryomov)
qa: Fix test_with_health_warn_with_2_active_MDSs (pr#65260, Kotresh HR)
qa: ignore cluster warning (evicting unresponsive ...) with tasks/mgr-osd-full (issue#73278, pr#66125, Venky Shankar)
qa: Improve scalability test (pr#66224, Vallari Agrawal)
qa: krbd_blkroset.t: eliminate a race in the open_count test (pr#67075, Ilya Dryomov)
qa: Run RADOS suites with ec optimizations on and off (pr#65471, Jamie Pryde)
qa: suppress OpenSSL valgrind leaks (pr#65660, Laura Flores)
rbd-mirror: add cluster fsid to remote meta cache key (pr#66297, Mykola Golub)
rbd-mirror: allow incomplete demote snapshot to sync after rbd-mirror daemon restart (pr#66164, VinayBhaskar-V)
Relax scrub of shard sizes for upgraded EC pools (pr#66021, Alex Ainscow)
Revert "Merge pull request #66958 from Hezko/wip-74413-tentacle" (pr#67750, Patrick Donnelly)
Revert "PrimeryLogPG: don't accept ops with mixed balance_reads and rwordered flags" (pr#66611, Radoslaw Zarzynski)
RGW | fix conditional Delete, MultiDelete and Put (pr#65949, Ali Masarwa)
RGW | fix conditional MultiWrite (pr#67425, Ali Masarwa)
rgw/account: bucket acls are not completely migrated once the user is migrated to an account (pr#65666, kchheda3)
rgw/admin: Add max-entries and marker to bucket list (pr#65485, Tobias Urdin)
rgw/lc: LCOpAction_CurrentExpiration checks mtime for delete markers (pr#65965, Casey Bodley)
rgw/tentacle: clean up .rgw_op.cc.swn file (pr#66161, Soumya Koduri)
rgw: add metric when send message with kafka and ampq (pr#65904, Hoai-Thu Vuong)
rgw: fix 'bucket rm --bypass-gc' for copied objects (pr#66004, Casey Bodley)
rgw: fix radosgw-admin object unlink ... (pr#66151, J. Eric Ivancich)
RGW: multi object delete op; skip olh update for all deletes but the last one (pr#65488, Oguzhan Ozmen)
rgw: update keystone repo stable branch to 2024.2 (pr#66241, Kyr Shatskyy)
rpm: default to gcc-toolset-13, not just for crimson (pr#65752, John Mulligan, Casey Bodley)
scripts/build/ceph.spec.in: fix rhel version checks (pr#66865, Ronen Friedman)
src/ceph_osd, osd: Implement running benchmark during OSD creation - Phase 1 (pr#65522, Sridhar Seshasayee)
src: Move the decision to build the ISA plugin to the top level make file (pr#67894, Alex Ainscow)
sync build-with-container patches from main (pr#65843, John Mulligan, Dan Mick)
systemd services: fix installing ceph-volume@ (pr#66861, Thomas Lamprecht)
tasks/cbt_performance: Tolerate exceptions during performance data up… (pr#66102, Nitzan Mordechai)
test/ceph_assert.cc: Disable core files (pr#66334, Bob Ham)
test/neorados: Catch timeouts in Poll test (pr#65605, Adam C. Emerson)
test: disable known flaky tests in run-rbd-unit-tests (pr#67559, Ilya Dryomov)
tools: handle get-attr as read-only ops in ceph_objectstore_tool (pr#66537, Jaya Prakash)

Ceph Q1 2026 Newsletter

2026-03-31T00:00:00Z

During this quarter, the Ceph Foundation focused on strengthening its structure by establishing new governance charters, event strategies, and financial plans. To enhance transparency as our community evolves, this newsletter offers a look behind the scenes at these foundational details. Our aim is to update Ceph Community members on the latest developments within the Foundation and to clarify how they can contribute to our ongoing growth. There are several ways to get involved with the foundation. If you have a concept for a Ceph-related project, we encourage you to take the first step toward bringing your idea to the next level and submit a funding request whenever you are ready. Feedback and suggestions will be offered along your journey with Ceph.

In this issue ¶

CSC and Ceph Foundation Board Meeting
Ceph Foundation Charters Approved
Ceph Governing Board Hiring a Technical Writer
OVHcloud Spending Update
Ceph Tech Talks Are Back (Monthly Schedule)
Upcoming Ceph Days Events
Ceph Community Slack Upgraded to Pro

CSC and Ceph Foundation Board Meeting ¶

The Ceph Foundation Board recently hosted the Ceph Steering Committee (CSC) for a collaborative discussion on the current state of the Ceph project and how both sides can work together to support the Ceph community.

These quarterly meetings are designed to foster communication between the two committees working to build Ceph for the benefit of its users and contributors. The Ceph Board's goal is to help provide greater context to the CSC as they make decisions and to support their missions, thereby bridging the communication gap. The meeting's agenda is available here.

Key discussion areas ¶

Operational complexity
Friction in contributing and getting reviews
Fragmented communication
Unclear strategy in some areas
Unclear ownership across parts of the ecosystem

Major takeaways ¶

The Board will continue to work on closing the gap between developer experience and real-world operator needs, with help from the CSC around failure handling and upgrades.
Both committees agreed that contributor experience remains a key challenge, with CI complexity and limited reviewer bandwidth identified as the biggest sources of friction, not the contribution process itself.
The Board and the CSC recognize that perception matters. After hearing feedback that Ceph feels “unwelcoming,” this concern is being addressed and will guide improvements in onboarding and engagement.
The Ceph Foundation will keep focusing on areas like performance and efficiency. There's an opportunity to improve how project-wide priorities are communicated, helping to align contributors and ecosystem partners.

Action Items ¶

Ceph Community Manager: Document best practices for large contributions requiring early community engagement
Ceph Community Manager: Improve discoverability of contributor guidelines and ambassador resources
CSC: Prepare Q2 response on project priorities, pain points, and foundation delegation opportunities
Ceph Community Manager: Share contributor feedback data with CSC for deeper analysis

Road to Fully Onboarding into the Linux Foundation: Charters Approved ¶

Ceph now has a clearly defined governance model that separates technical decision-making from funding and community growth, making it easier to understand how decisions are made and how to get involved. This task began in September 2025, when the Ceph Foundation initiated its transition into The Linux Foundation, marking a significant step toward strengthening the long-term sustainability and neutrality of the Ceph project.

The outcome of this work was the formal approval of two foundational governance frameworks. The Ceph Foundation Charter was established to define how the Foundation raises, allocates, and manages resources in support of the project, while also creating a clear and transparent structure for decision-making, community outreach, and ecosystem development. At its core, the Charter exists to ensure that Ceph operates as a vendor-neutral, community-driven project with sustainable funding and broad industry participation.

In parallel, the Ceph Technical Charter was approved by the Ceph Steering Committee (CSC), reinforcing the independence of the project’s technical governance and clarifying how technical decisions are made in alignment with the needs of the community.

Together, these milestones establish a balanced governance model: the Foundation focuses on funding, outreach, and ecosystem growth, while the technical community retains authority over the project’s technical direction and innovation.

The third and final step of this process, which is currently in the works, involves transferring the Ceph trademark to the Linux Foundation after completing steps one and two.

Review the Foundation Charter: https://cdn.platform.linuxfoundation.org/agreements/cephfoundation.pdf

Review the Ceph Technical Charter:
https://github.com/ceph/ceph/blob/main/doc/technical-charter.rst

Opportunity: Technical Writer Role Open ¶

Ceph is investing directly in better documentation, making it easier for new users to adopt and experienced operators to scale, and helping those interested in contributing to Ceph get onboarded for development. The Ceph Technical Committee is actively interviewing a Technical Writer to help improve the quality and accessibility of Ceph documentation.

About the role ¶

Focus on clarity, consistency, and usability of technical content
Collaborate with developers and contributors across the community
Help lower the barrier to entry for new users and operators

How to get involved ¶

Review docs and flag gaps
Suggest onboarding pain points
Participate in doc sprints

OVHcloud Spending Update ¶

The Ceph Governing Board, with Mark Nelson (Clyso) and Patrick Donnelly (IBM), is working to reduce infrastructure costs. Reports from Mark and Joachim Kraftmayer (Clyso) on OVHcloud show potential for lower monthly hosting expenses. Negotiations with OVHcloud could yield savings, which would be reinvested to boost the Foundation's support for community programs, events, and infrastructure.

Current overview ¶

Monthly costs over the past year have ranged between $5K–$10K USD
Spending peaked around August and has since been reduced to approximately $7K/month
Costs remain higher than earlier in the year, prompting further review

Key cost drivers ¶

A significant portion of expenses comes from storage
Four 10TB volumes (supporting Chacra nodes), along with associated snapshots, accounted for over half of total monthly costs (~$4K USD)

Recent actions ¶

Unused snapshots and resources have been removed by David Galloway (IBM)
Ongoing collaboration with OVHcloud to identify additional optimization opportunities
Further cost savings are expected as usage is refined

What’s next ¶

Continued monitoring and cost optimization efforts
Improved visibility into infrastructure usage and spending
Updates will be shared in the next newsletter and monthly reports as new efficiencies are realized

Ceph Tech Talks Are Back ¶

Ceph Tech Talks have officially returned, giving the community a consistent way to learn directly from contributors and share real-world experience. Together, we have discussed the Running Teuthology Outside of the Sepia lab and How to Get Involved with the Ceph Ambassador Program.

Attend the Tech Talk on April 22, 2026, at 12 pm EST/9 am PDT. Our topic will be, MAAS as a Backend: Provisioning Infrastructure for Teuthology Suites.

Follow the Ceph Community Calendar for more information.

What to expect ¶

Deep dives into real-world Ceph use cases and features
Presentations led by community members and contributors
A wide variety of topics for users and developers
Interactive sessions with opportunities for Q&A

Get involved ¶

Attend upcoming sessions to stay current
Submit a proposal to present your work here

Upcoming Ceph Events and the State of Cephalocon ¶

The Ceph Foundation is evolving its approach to events in order to better serve the global community.

Instead of focusing all of our resources on a single large event like Cephalocon, the Foundation is now emphasizing local and regional engagement to expand Ceph’s reach into nearby communities and projects. This shift was prompted by new guidelines regarding spending within the foundation. These current fiduciary guidelines still require broader community support and sponsorship participation.

What’s changing ¶

Increased focus on Ceph Days, meetups, data-driven conferences, and community-led events
Strategic investment in opportunities that introduce Ceph to new audiences
A shift toward distributed, community-driven engagement

Important note on future large events ¶

The Foundation remains open to hosting large-scale events like Cephalocon
This ensures alignment with the Foundation’s financial responsibilities while enabling sustainable growth

How to get involved ¶

As part of the Foundation’s financial stewardship under the Linux Foundation, large-scale events require broader community sponsorship and participation
Organize or support a Ceph event in your region
Partner with related open source or data infrastructure communities

Need support? ¶

Community members can submit funding requests to attend or organize events, as well as to support Ceph-related activities
Requests are reviewed by the Board (approval is not guaranteed)

Upcoming Ceph Days – March 2026 ¶

Ceph Days continues to grow as the primary way the community connects locally. There were two community-driven events that happened this March:

Ceph Days India

Sponsors: IBM and Clyso
Attendees: 192

Ceph Days Raleigh

Sponsor: IBM
Attendees: 78

Why attend ¶

Connect with other Ceph users and contributors
Learn from real-world deployments and technical sessions
Grow your local Ceph network

Get involved ¶

Share your ideas for a Ceph Days: https://pad.ceph.com/p/ceph-days-2026
Help organize or promote events in your region

Ceph Community Slack Upgraded to Pro ¶

The community can now access Slack messages from the last 90 days! The Ceph Community Slack workspace has been upgraded to Slack Pro, courtesy of the Linux Foundation, as part of negotiations requested by the Ceph Board.

What this means ¶

No more losing conversations!
There is no additional cost to the Ceph community
A more robust platform for collaboration and communication

Why it matters ¶

Supports better knowledge sharing across contributors and users
Improves accessibility of past discussions and technical insights
Strengthens real-time collaboration within the community

Get involved ¶

Become a moderator for the Ceph Slack workspace
Email amiddleton@linuxfoundation.org for more information

v18.2.8 Reef released

2026-03-20T00:00:00Z

This is the eighth, and expected to be last, backport release in the Reef series. We recommend that all users update to this release.

Release Date ¶

March 20, 2026

Known Issues ¶

During QA for v18.2.8, it was found that there was a bug for upgrades from Pacific to Reef. Pacific OSDs (and other Ceph daemons) were still using a deprecated connection feature bit that was adopted to indicate a Reef OSD. This can cause a OSD_UPGRADE_FINISHED warning before all OSDs are actually upgraded to Reef. There are no known issues associated with Pacific and Reef OSDs interoperating where Pacific OSDs are "advertising" Reef compatibility; however, out of an abundance of caution, we no longer recommend upgrading from Pacific to Reef directly.

Security Fixes ¶

CephFS Client: A fix was merged to prohibit unprivileged users from modifying the sgid or suid bits on a file. Previously, unprivileged users were inadvertently permitted to set these bits if they were the sole bits being modified.
Mgr Alerts: The SMTP SSL context was enforced in the mgr/alerts module to resolve a security vulnerability (GHSA-xj9f-7g59-m4jx).

Notable Changes ¶

RGW (RADOS Gateway):
- Fixed an issue where bucket rm --bypass-gc was mistakenly removing head objects instead of tail objects, potentially causing data inconsistencies.
- Fixed rgw-restore-bucket-index to handle objects with leading hyphens and to process versioned buckets correctly.
- Addressed an issue in the msg/async protocol that caused memory locks and hangs during connection shutdown.
- RGW STS: Made JWKS URL verification configurable for AWS compliance via the rgw_enable_jwks_url_verification configuration.
CephFS / MDS:
- Prevented the MDS from stalling (up to 5 seconds) during rename/stat workloads by forcing the log to nudge for unstable locks after early replies.
- Fixed cephfs-journal-tool so it no longer incorrectly resets the journal trim position during disaster recovery, which was causing stale journal objects to linger forever in the metadata pool.
- Fixed a bug where ll_walk incorrectly processed absolute paths as relative paths.
- Prevented the ceph fs volume create command from accidentally deleting user-created pools if the command aborted during cleanup.
- MDS Batched Operations: Added a new mds_allow_batched_ops configuration option (default: true) to control whether the MDS can batch lookup or getattr RPCs.
- CephFS Subvolumes: Added the ceph fs subvolume snapshot getpath command to allow users to retrieve the absolute path of a snapshot of a subvolume.
BlueStore:
- Fixed a bug where the bytes_written_slow performance counter incorrectly reported 0 when using aio_write.

Changelog ¶

.github: Fix RTD build retrigger (pr#63616, David Galloway)
Ensure the ETag format is consistent with AWS S3 API (pr#62608, Casey Bodley, liubingrun)
[reef] os/bluestore: fix _extend_log seq advance (pr#61653, Pere Diaz Bou)
[reef] RGW backports (pr#63031, Soumya Koduri)
[reef] rgw/dbstore: Update bucket attrs as part of put_info() (pr#64488, Soumya Koduri)
auth: msgr2 can return incorrect allowed_modes through AuthBadMethodFrame (pr#65334, Miki Patel)
backport build-with-container patches from main (pr#65188, John Mulligan, Dan Mick, Zack Cerza)
Backport the hybrid_btree2 allocator and prereqs (pr#62539, Igor Fedotov, Jrchyang Yu)
backports variants improvements and Dockerfile.build changes (pr#66012, John Mulligan, Zack Cerza)
blk/kernel: improve DiscardThread life cycle (pr#65216, Igor Fedotov)
blk/KernelDevice: Introduce a cap on the number of pending discards (pr#62220, Joshua Baergen)
blk/kerneldevice: notify_all only required when discard_drain wait for condition (pr#62152, Yite Gu)
blk/kerneldevice: some fix for device discard (pr#62481, Igor Fedotov, Yite Gu)
bluestore/BlueFS: fix bytes_written_slow counter with aio_write (pr#66353, chungfengz)
build backports (pr#65066, John Mulligan, Zack Cerza)
build-with-container: add argument groups to organize options (pr#65630, John Mulligan)
build-with-container: build image variants (pr#65944, John Mulligan)
build-with-container: two small fixes (pr#62339, John Mulligan)
ceph-fuse: Improve fuse mount usage message (pr#61275, Kotresh HR)
ceph-volume: allow zapping partitions on multipath devices (pr#62178, Guillaume Abrioux)
ceph-volume: do not convert LVs's symlink to real path (pr#59989, Guillaume Abrioux)
ceph-volume: fix regex usage in set\_dmcrypt\_no\_workqueue (pr#62791, Guillaume Abrioux)
ceph.spec.in: add man/rgw-gap-list (pr#63999, Matan Breizman)
ceph.spec.in: Remove rgw-restore-bucket-index.8* from packaging (pr#64130, Kefu Chai)
cephfs,mon: fs rename must require FS to be offline and refuse_client_session to be set (issue#66088, pr#61410, Rishabh Dave, Venky Shankar)
cephfs-journal-tool: fix segfault during 'journal import' from invalid dump file (pr#62114, Jos Collin)
cephfs-journal-tool: Journal trimming issue (pr#65603, Kotresh HR)
cephfs-shell: add option to remove xattr (pr#62409, Neeraj Pratap Singh)
cephfs-top, qa: Remove unnecessary global statements in tests (pr#62606, Kefu Chai)
cephfs-top: exception when terminal size greater than PAD_WIDTH (pr#61773, Jos Collin)
cephfs: session tracker accounts for killing sessions (pr#65253, Abhishek Lekshmanan)
client: fix d_reclen for readdir (pr#61519, Xavi Hernandez)
client: fixed a bug that read operation hung (pr#60695, Tod Chen)
client: Handle empty pathnames for ceph\_chownat() and ceph\_statxat() (pr#61165, Anoop C S)
client: ll_walk will process absolute paths as relative (pr#62500, Patrick Donnelly)
client: prohibit unprivileged users from setting sgid/suid bits (pr#66040, Kefu Chai)
client: return EOPNOTSUPP for fallocate with mode 0 (pr#60657, Milind Changire)
cls/rbd: write image mirror status if state is CREATING (pr#63236, N Balachandran)
cls/rgw: non-versioned listings skip past version suffix (pr#62591, Casey Bodley)
common/options: fix the description of osd_max_scrubs (pr#62378, Satoru Takeuchi)
common/options: fix typo in description (pr#64218, Lorenz Bausch)
common/pick_address: Add IPv6 support to is_addr_in_subnet (pr#62814, Nitzan Mordechai)
container: small container image improvements (pr#62345, John Mulligan)
crush: use std::vector instead of variable length arrays (pr#62014, Kefu Chai)
debian/control: add iproute2 to build dependencies (pr#66738, Kefu Chai)
debian: package mgr/rgw in ceph-mgr-modules-core (pr#57874, Kefu Chai)
doc/architecture: remove sentence (pr#61615, Zac Dover)
doc/cephadm/services: Add mention of --zap for OSD removal (pr#62444, Anthony D'Atri)
doc/cephadm/services: Correct indentation in osd.rst (pr#62428, Anthony D'Atri)
doc/cephadm/services: Fix formatting in osd.rst (pr#62811, Anthony D'Atri)
doc/cephadm/services: improve rgw.rst and snmp-gateway.rst (pr#62695, Anthony D'Atri)
doc/cephadm: Add admonition re restarting an OSD service (pr#62797, Anthony D'Atri)
doc/cephadm: Add PG autoscaler advice to upgrade.rst (pr#62380, Anthony D'Atri)
doc/cephadm: clarify "Monitoring OSD State" (pr#61665, Zac Dover)
doc/cephadm: Correct formatting in upgrade.rst (pr#63148, Anthony D'Atri)
doc/cephadm: correct markup in rgw.rst (pr#63074, Zac Dover)
doc/cephadm: improve "Maintenance Mode" (pr#63496, Zac Dover)
doc/cephadm: s/confg/config/ (pr#62645, Zac Dover)
doc/cephfs: add a note about estimated replay completion time (issue#71629, pr#65058, Venky Shankar, Zac Dover)
doc/cephfs: correct ill-formatted command (pr#63502, Zac Dover)
doc/cephfs: correct reference structure in fs-volumes.rst (pr#63545, Zac Dover)
doc/cephfs: Cosmetic changes and small fixes in cephfs-mirroring.rst (pr#63468, Ville Ojamo)
doc/cephfs: document first-damage.py (pr#63978, Zac Dover)
doc/cephfs: edit ceph-dokan.rst (1 of x) (pr#64736, Zac Dover)
doc/cephfs: edit ceph-dokan.rst (2 of x) (pr#64760, Zac Dover)
doc/cephfs: edit ceph-dokan.rst (3 of x) (pr#64786, Zac Dover)
doc/cephfs: edit disaster-recovery.rst (pr#64645, Zac Dover)
doc/cephfs: edit disaster-recovery.rst (pr#64609, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65380, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65094, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65091, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65126, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65123, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65097, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65078, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65088, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65047, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65044, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65041, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65037, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#65026, Zac Dover, Venky Shankar)
doc/cephfs: edit troubleshooting.rst (pr#64904, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#64901, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#64879, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#64872, Zac Dover)
doc/cephfs: edit troubleshooting.rst (pr#64853, Zac Dover)
doc/cephfs: edit troubleshooting.rst (Slow MDS) (pr#65201, Zac Dover)
doc/cephfs: Improve mount-using-fuse.rst (pr#64473, Anthony D'Atri)
doc/cephfs: link section for pausing async threads in section for (pr#62875, Rishabh Dave)
doc/cephfs: Update deprecation notice in experimental-features.rst (pr#63949, Ville Ojamo)
doc/cephfs: Update quota.rst (pr#65083, Jannis Speer, Zac Dover)
doc/dev/cephfs-mirroring: edit file 1 of x (pr#63299, Zac Dover)
doc/dev/cephfs-mirroring: edit file 2 of x (pr#63274, Zac Dover)
doc/dev/cephfs-mirroring: edit file 3 of x (pr#63548, Zac Dover)
doc/dev/cephfs-mirroring: edit file 4 of x (pr#63661, Zac Dover)
doc/dev/config: Document how to use :confval: directive for config op… (pr#64167, Kefu Chai)
doc/dev/release-process.rst: document new Jenkins job for containers (pr#62613, Dan Mick)
doc/dev/release-process.rst: release builds cannot build containers (pr#61818, Dan Mick, Zac Dover)
doc/dev: Debuggging with gdb (pr#63994, Matan Breizman)
doc/dev: update link to backporter manual (pr#63991, Zac Dover)
doc/dev:update blkin.rst doc for lttng trace (pr#65212, lizhipeng)
doc/glossary: s/OMAP/omap/ (pr#63738, Zac Dover)
doc/man/8: Improve mount.ceph.rst (pr#65184, Anthony D'Atri)
doc/mgr/ceph_api: edit index.rst (pr#63198, Zac Dover)
doc/mgr/crash.rst: remove outdated module enabling instructions (pr#64285, Kefu Chai)
doc/mgr/dashboard_plugins: edit feature_toggles.inc.rst (pr#63705, Zac Dover)
doc/mgr: edit administrator.rst (pr#63208, Zac Dover)
doc/mgr: edit alerts.rst (pr#63201, Zac Dover)
doc/mgr: edit cli_api (pr#63744, Zac Dover)
doc/mgr: edit cli_api.rst (pr#63690, Zac Dover)
doc/mgr: edit crash.rst (pr#63539, Zac Dover)
doc/mgr: edit dashboard.rst (pr#63316, Zac Dover)
doc/mgr: edit debug.inc.rst (pr#63394, Zac Dover)
doc/mgr: edit diskpredictor.rst (pr#63424, Zac Dover)
doc/mgr: edit feature_toggles.inc.rst (pr#63397, Zac Dover)
doc/mgr: edit hello.rst (pr#63508, Zac Dover)
doc/mgr: edit influx.rst (pr#63455, Zac Dover)
doc/mgr: edit insights.rst (pr#63511, Zac Dover)
doc/mgr: edit iostat.rst (pr#63681, Zac Dover)
doc/mgr: edit iostat.rst (pr#63514, Zac Dover)
doc/mgr: edit localpool.rst (pr#63670, Zac Dover)
doc/mgr: edit localpool.rst (pr#63551, Zac Dover)
doc/mgr: edit mds_autoscaler.rst (pr#63493, Zac Dover)
doc/mgr: edit modules.rst (pr#63667, Zac Dover)
doc/mgr: edit modules.rst (pr#63578, Zac Dover)
doc/mgr: edit motd.inc.rst (pr#63403, Zac Dover)
doc/mgr: edit nfs.rst (pr#63664, Zac Dover)
doc/mgr: edit nfs.rst (pr#63581, Zac Dover)
doc/mgr: edit orchestrator.rst (pr#63584, Zac Dover)
doc/mgr: edit progress.rst (pr#63658, Zac Dover)
doc/mgr: edit progress.rst (pr#63587, Zac Dover)
doc/mgr: edit prometheus.rst (pr#63590, Zac Dover)
doc/mgr: edit rgw.rst (pr#63593, Zac Dover)
doc/mgr: edit telegraf.rst (pr#63612, Zac Dover)
doc/mgr: edit telemetry (1 of x) (pr#63769, Zac Dover)
doc/mgr: edit telemetry (2 of x) (pr#63772, Zac Dover)
doc/mgr: edit telemetry (3 of x) (pr#63775, Zac Dover)
doc/mgr: edit telemetry (4 of x) (pr#63778, Zac Dover)
doc/mgr: edit telemetry.rst (pr#64344, Zac Dover)
doc/mgr: edit telemetry.rst (pr#63810, Zac Dover)
doc/mgr: edit telemetry.rst (pr#63906, Zac Dover)
doc/mgr: edit telemetry.rst (pr#63865, Zac Dover)
doc/mgr: edit telemetry.rst (pr#63693, Zac Dover)
doc/mgr: edit telemetry.rst (lines 300-400) (pr#63868, Zac Dover)
doc/mgr: Improve prometheus.rst (pr#62931, Zac Dover, Anthony D'Atri)
doc/mgr: Small improvements in rgw.rst (pr#63626, Ville Ojamo)
doc/monitoring: correct list formatting (pr#63542, Zac Dover)
doc/rados/configuration/bluestore-config-ref: Fix lowcase typo (pr#62261, Adam Kupczyk)
doc/rados/configuration/bluestore-config-ref: Fix lowercase typos (pr#62291, Dan van der Ster)
doc/rados/configuration: Correct admonition in ceph-conf.rst (pr#62621, Anthony D'Atri)
doc/rados/configuration: Improve ceph-conf.rst (pr#63943, Anthony D'Atri)
doc/rados/configuration: Mention show-with-defaults and ceph-conf (pr#65207, Niklas Hambüchen)
doc/rados/configuration: Small improvements in ceph-conf.rst (pr#64288, Ville Ojamo)
doc/rados/operations/stretch-mode: Improve doc (pr#61654, Kamoltat Sirivadhna)
doc/rados/operations: Actually mention upmap\_max\_deviation setting … (pr#64119, Niklas Hambüchen)
doc/rados/operations: add kernel client procedure to read balancer documentation (pr#65440, Laura Flores)
doc/rados/operations: Add settings advice to balancer.rst (pr#63536, Anthony D'Atri)
doc/rados/operations: Additional improvements to placement-groups.rst (pr#63650, Anthony D'Atri)
doc/rados/operations: Address suggestions for stretch-mode.rst (pr#63850, Zac Dover)
doc/rados/operations: edit cache-tiering.rst (pr#63696, Zac Dover)
doc/rados/operations: Improve erasure-code.rst (pr#62574, Anthony D'Atri)
doc/rados/operations: Improve health-checks.rst (pr#65239, Anthony D'Atri)
doc/rados/operations: Improve placement-groups.rst (pr#63647, Anthony D'Atri)
doc/rados/operations: Improve stretch-mode.rst (pr#63816, Anthony D'Atri)
doc/rados/ops: add caps restore command (pr#64322, Zac Dover)
doc/rados/ops: edit cache-tiering.rst (pr#64497, Zac Dover)
doc/rados/ops: edit cache-tiering.rst (pr#63831, Zac Dover)
doc/rados: document section absent in release < T (pr#64868, Zac Dover)
doc/rados: edit balancer.rst (pr#63684, Zac Dover)
doc/rados: edit ops/user-management.rst (pr#63893, Zac Dover)
doc/rados: enhance "pools.rst" (pr#63862, Zac Dover)
doc/rados: improve markup in cache-tiering.rst (pr#63505, Zac Dover)
doc/rados: remove clonedata command (pr#64394, Zac Dover)
doc/rados: repair short underline (pr#65138, Zac Dover)
doc/rados: s/enpty/empty/ in pgcalc doc (pr#63499, Zac Dover)
doc/rados: Update mClock doc on steps to override OSD IOPS capacity config (pr#63072, Sridhar Seshasayee)
doc/radosgw /notifications: fix topic details (pr#62405, Laimis Juzeliunas)
doc/radosgw/admin.rst: explain bucket and uid flags for bucket quota (pr#64022, Hyun Jin Kim)
doc/radosgw/cloud-transition: fix details (pr#62835, Laimis Juzeliunas)
doc/radosgw/s3: Document delete-if-unmodified-since (pr#64316, Anthony D'Atri)
doc/radosgw: add "persistent_topic_size" (pr#64140, Zac Dover)
doc/radosgw: add rgw_enable_lc_threads & rgw_enable_gc_threads (pr#64339, Zac Dover)
doc/radosgw: Cosmetic and formatting improvements in vault.rst (pr#63230, Ville Ojamo)
doc/radosgw: Cosmetic improvements in cloud-transition.rst (pr#63449, Ville Ojamo)
doc/radosgw: Cosmetic improvements in dynamicresharding.rst (pr#64059, Ville Ojamo)
doc/radosgw: edit "Lifecycle Settings" (pr#64548, Zac Dover)
doc/radosgw: edit cloud-transition (1 of x) (pr#64025, Zac Dover)
doc/radosgw: edit config-ref.rst (pr#64648, Zac Dover)
doc/radosgw: edit metrics.rst (pr#63813, Zac Dover)
doc/radosgw: edit sentence in metrics.rst (pr#63701, Zac Dover)
doc/radosgw: Fix RST syntax rendeded as text in oidc.rst (pr#62990, Ville Ojamo)
doc/radosgw: improve "pubsub_push_pending" info (pr#64114, Zac Dover)
doc/radosgw: Improve and more consistent formatting (pr#62910, Ville Ojamo)
doc/radosgw: Improve cloud-restore and cloud-transition (pr#62667, Anthony D'Atri)
doc/radosgw: Improve formatting in layout.rst (pr#63000, Anthony D'Atri)
doc/radosgw: Improve layout.rst (pr#62450, Anthony D'Atri)
doc/radosgw: Improve rgw-cache.rst (pr#64476, Ville Ojamo)
doc/radosgw: Promptify CLI commands and fix formatting in layout.rst (pr#63916, Ville Ojamo)
doc/radosgw: Promptify CLI, cosmetic fixes (pr#62857, Ville Ojamo)
doc/radosgw: remove "pubsub_event_lost" (pr#64127, Zac Dover)
doc/radosgw: remove "pubsub_event_triggered" (pr#64156, Zac Dover)
doc/radosgw: remove cloud-restore from reef (pr#65638, Zac Dover)
doc/radosgw: update aws specification link (pr#64096, Zac Dover)
doc/radosgw: Use ref for hyperlinking to multisite (pr#63312, Ville Ojamo)
doc/rbd/rbd-config-ref: add clone settings section (pr#66173, Ilya Dryomov)
doc/rbd: add mirroring troubleshooting info (pr#63847, Zac Dover)
doc/rgw: add man documentation for the rgw-gap-list tool (pr#63997, J. Eric Ivancich)
doc/rgw: clarify path-style vs virtual-hosted-style access (pr#61987, Casey Bodley)
doc/rgw: document Admin and System Users (pr#62882, Casey Bodley)
doc/rgw: remove metrics.rst which did not apply to reef (pr#66320, Casey Bodley)
doc/rgw: use 'confval' directive to render sts config options (pr#63442, Casey Bodley)
doc/src/common/options: mgr.yaml.in edit (pr#63765, Zac Dover)
doc/src: edit osd.yaml.in (osd_deep_scrub_interval_cv) (pr#63956, Zac Dover)
doc/start: edit documenting-ceph.rst (pr#63653, Zac Dover)
doc/start: edit documenting-ceph.rst (pr#63708, Zac Dover)
doc: add note admonitions in two files (pr#64493, Zac Dover)
doc: Clarify the status of MS Windows client support (pr#64482, Anthony D'Atri)
doc: do not depend on typed-ast (pr#64400, Kefu Chai)
doc: Document ceph-mgr module configuration options (pr#64397, Kefu Chai)
doc: fix formatting in cephfs_mirror dev doc (pr#63251, Jos Collin)
doc: Fix links to mClock config reference (pr#64798, Pierre Riteau)
doc: Fix missing blank line Sphinx warnings (pr#63338, Ville Ojamo)
doc: Fix unterminated inline literal in ceph-conf.rst (pr#64171, Kefu Chai)
doc: Fixed a spelling error (pr#64148, Instelligence.io)
doc: Fixes a typo in balancer operations (pr#65740, Tyler Brekke)
doc: mgr/dashboard: add OAuth2 SSO documentation (pr#64034, Pedro Gonzalez Gomez, Zac Dover)
doc: Pin pip to <25.3 for RTD as a workaround for pybind in admin/doc-read-the-docs.txt (pr#66118, Ville Ojamo)
doc: Remove sphinxcontrib-seqdiag Python package from RTD builds (pr#67528, Ville Ojamo)
doc: Revert "doc/radosgw: add "persistent_topic_size"" (pr#64179, Zac Dover)
doc: Revert "doc: mgr/dashboard: add OAuth2 SSO documentation" (pr#66796, Ville Ojamo)
doc: Revert doc/cephadm: correct markup in rgw.rst (pr#66971, Ville Ojamo)
doc: src/pybind/mgr/dashboard: edit HACKING.rst (pr#63697, Zac Dover)
doc: update cephfs-journal-tool docs (pr#63109, Jos Collin)
doc: update mgr modules notify_types (pr#64531, Nitzan Mordechai)
fix: the RGW crash caused by special characters (pr#64052, mertsunacoglu, Emin)
github: pin GH Actions to SHA-1 commit (pr#65759, Ernesto Puerta)
Handle failures in metric parsing (pr#65595, Anmol Babu)
install-deps.sh: install proper compiler version on Debian/Ubuntu (pr#66014, Dan Mick)
install-deps: Replace apt-mirror (pr#66669, David Galloway)
librbd/cache/pwl: fix memory leak in SyncPoint persist context cleanup (pr#64093, Kefu Chai)
librbd/migration/QCOWFormat: don't complete read_clusters() inline (pr#64195, Ilya Dryomov)
librbd: disallow "rbd trash mv" if image is in a group (pr#62967, Ilya Dryomov)
librbd: images aren't closed in group_snap\_*_by_record() on error (pr#64620, Miki Patel)
librbd: respect rbd_default_snapshot_quiesce_mode in group_snap_create() (pr#62962, Ilya Dryomov)
LogMonitor: set no_reply for forward MLog commands (pr#62212, Nitzan Mordechai)
mds/Beacon: wake up the thread in shutdown() (pr#61513, Max Kellermann)
mds: add an asok command to dump export states (pr#61512, Zhansong Gao)
mds: add more debug logs and log events (pr#61518, Xiubo Li)
mds: do not process client metrics message with fast dispatch (issue#68865, pr#61339, Venky Shankar)
mds: drop client metrics during recovery (pr#61299, Patrick Donnelly)
mds: dump next_snap when checking dentry corruption (pr#61978, Milind Changire)
mds: Fix invalid access of mdr->dn[0].back() (pr#61516, Anoop C S)
mds: Fix invalid access of mdr->dn[0].back() (pr#61450, Anoop C S)
mds: Fix readdir when osd is full (pr#65348, Kotresh HR)
mds: fix snapdiff result fragmentation (pr#65364, Igor Fedotov, Md Mahamudur Rahaman Sajib)
mds: nudge log for unstable locks after early reply (pr#64540, Patrick Donnelly)
mds: prevent duplicate wrlock acquisition for a single request (pr#61839, Xiubo Li, Sunnatillo)
mds: session in the importing state cannot be cleared if an export subtree task is interrupted while the state of importer is acking (pr#61514, Zhansong Gao)
mds: use SimpleLock::WAIT_ALL for wait mask (pr#67495, Patrick Donnelly)
memory lock issues causing hangs during connection shutdown (pr#65786, Nitzan Mordechai)
mgr/alerts: enforce ssl context to SMTP_SSL (pr#66142, Nizamudeen A)
mgr/cephadm: Fix unfound progress events (pr#58450, Prashant D)
mgr/DaemonState: Minimise time we hold the DaemonStateIndex lock (pr#65463, Brad Hubbard)
mgr/dashboard: adapt service creation form to support nvmeof creation (pr#63304, Afreen Misbah)
mgr/dashboard: add .nvmrc so ci can pick the node version (pr#64666, Nizamudeen A)
mgr/dashboard: Add ceph_daemon filter to rgw overview grafana panel queries (pr#62268, Aashish Sharma)
mgr/dashboard: add prometheus read permission to cluster_mgr role (pr#62651, Nizamudeen A)
mgr/dashboard: Dashboard not showing Object/Overview correctly (pr#62664, Aashish Sharma)
mgr/dashboard: fix access control permissions for roles (pr#62455, Nizamudeen A)
mgr/dashboard: Fix empty ceph version in GET api/hosts (pr#62730, Afreen Misbah)
mgr/dashboard: Fix inline markup warning in API documentation (pr#64270, Kefu Chai)
mgr/dashboard: fix make check tests (pr#63186, Afreen Misbah)
mgr/dashboard: fix zone update API forcing STANDARD storage class (pr#65621, Aashish Sharma)
mgr/dashboard: show non default realm sync status in rgw overview page (pr#65002, Aashish Sharma)
mgr/dashboard: use system packages when running tox (pr#64612, Nizamudeen A)
mgr/nfs: validate path when modifying cephfs export (pr#62278, Dhairya Parmar)
mgr/rbd_support: always parse interval and start_time in Schedules::remove() (pr#62964, Ilya Dryomov)
mgr/snap_schedule: fix typo in error message during retention add (pr#65295, Milind Changire)
mgr/snap_schedule: handle volume delete (pr#61187, Milind Changire)
mgr/vol: add command to get snapshot path (pr#62917, Rishabh Dave)
mgr/vol: don't delete user-created pool in "volume create" command (pr#63069, Rishabh Dave)
mgr/vol: print proper message when subvolume metadata filename is too long (pr#62050, Rishabh Dave)
mgr/volumes: allow disabling async job threads (pr#62436, Rishabh Dave)
mgr/volumes: fix dangling symlink in clone index (pr#62109, Neeraj Pratap Singh)
mgr/volumes: Keep mon caps if auth key has remaining mds/osd caps (pr#65297, Enrico Bocchi)
mgr/volumes: periodically check for async work (issue#61867, pr#61230, Venky Shankar)
mgr: add status command (pr#62505, Patrick Donnelly)
mgr: allow disabling always-on modules (pr#60563, Rishabh Dave)
mgr: process map before notifying clients (pr#57065, Patrick Donnelly)
mon [stretch mode]: support disable_stretch_mode & qa/workunits/mon: ensure election strategy is "connectivity" for stretch mode (pr#60630, Laura Flores, Kamoltat Sirivadhna)
mon/AuthMonitor: provide command to rotate the key for a user credential (pr#58236, Patrick Donnelly)
mon/test_mon_osdmap_prune: Use first_pinned instead of first_committed (pr#63343, Aishwarya Mathuria)
mon: Track and process pending pings after election (pr#62925, Kamoltat Sirivadhna)
monitor: Enhance historic ops command output and error handling (pr#64843, Nitzan Mordechai)
monitoring: add user-agent headers to the urllib (pr#65473, Nizamudeen A)
monitoring: fix MTU Mismatch alert rule and expr (pr#65710, Aashish Sharma)
objclass: deprecate cls_cxx_gather (pr#60195, Nitzan Mordechai)
os/bluestore: Disable invoking unittest_deferred (pr#66359, Adam Kupczyk)
os/bluestore: do cache locally compressor engines ever used (pr#62145, Igor Fedotov, Adam Kupczyk)
os/bluestore: fix bdev expansion and more (pr#62216, Igor Fedotov)
os/bluestore: Fix ExtentDecoderPartial::_consume_new_blob (pr#62054, Adam Kupczyk)
os/bluestore: Fix race in BlueFS truncate / remove (pr#62840, Adam Kupczyk)
os/bluestore: In BlueFS::truncate accept wierd alloc_unit (pr#66056, Adam Kupczyk)
os/bluestore: make BlueFS an exclusive selector for volume reserved (pr#62721, Igor Fedotov)
osd/scheduler/OpSchedulerItem: Fix calculation of recovery latency counters (pr#62801, Sridhar Seshasayee)
osd/scrub: allow longer waits for replicas to respond (pr#63940, Ronen Friedman)
osd/scrub: discard repair_oinfo_oid() (pr#62569, Ronen Friedman)
osd: add clear_shards_repaired command (pr#60566, Daniel Radjenovic)
osd: don't send stale hb msgr's addresses in MOSDBoot (pr#56520, Radosław Zarzyński)
osd: fix osd mclock queue item leak (pr#62364, Samuel Just)
OSD: Split osd_recovery_sleep into settings applied to degraded or clean PGs (pr#62399, Md Mahamudur Rahaman Sajib)
osd_types: Restore new_object marking for delete missing entries (pr#63152, Nitzan Mordechai)
OSDMonitor: exclude destroyed OSDs from "ceph node ls" output (pr#62326, Nitzan Mordechai)
OSDMonitor: Make sure pcm is initialised (pr#63805, Brad Hubbard)
PendingReleaseNotes; doc/rados/operations: document "rm-pg-upmap-primary-{all}" commands (pr#62468, Laura Flores)
PGMap: remove pool max_avail scale factor (pr#61320, Michael J. Kidd)
pybind/mgr/dashboard: Use teuthology's actual requirements (pr#65418, David Galloway)
pybind/mgr: attempt to fix mypy importing from python-common (pr#63313, John Mulligan)
pybind/mgr: Fix missing empty lines in mgr_module.py (pr#64267, Ville Ojamo)
pybind/mgr: pin cheroot version in requirements-required.txt (pr#65637, Nizamudeen A, Adam King)
qa/cephfs: ignore warning that pg is stuck peering for upgrade jobs (pr#65448, Rishabh Dave)
qa/cephfs: randomize configs in fs:thrash:workloads (pr#61341, Venky Shankar)
qa/cephfs: switch to ubuntu 22.04 for stock kernel testing (pr#62492, Venky Shankar)
qa/cephfs: update ignorelist (pr#61383, Rishabh Dave)
qa/multisite: add extra checkpoints in datalog_autotrim testcase (pr#61508, Shilpa Jagannath)
qa/rbd/iscsi: ignore MON_DOWN warning in logs (pr#64596, Adam King)
qa/rgw: bump maven version in hadoop task to resolve 404 Not Found (pr#63927, Casey Bodley)
qa/rgw: fix perl tests missing Amazon::S3 module (pr#64281, Mark Kogan)
qa/rgw: remove hadoop-s3a subsuite (pr#64669, Casey Bodley)
qa/rgw: run verify tests with garbage collection disabled (pr#62953, Casey Bodley)
qa/suites/krbd: use a standard fixed-1 cluster in unmap subsuite (pr#64918, Ilya Dryomov)
qa/suites/orch/cephadm: add PG_DEGRADED to ignorelist (pr#63055, Shraddha Agrawal)
qa/suites: wait longer before stopping OSDs with valgrind (pr#63717, Nitzan Mordechai)
qa/tasks/ceph_manager: population must be a sequence (pr#64748, Kyr Shatskyy)
qa/tasks/cephfs/mount: use 'ip route' instead 'route' (pr#63129, Kyr Shatskyy)
qa/tasks/workunit: fix no module named 'pipes' (pr#66252, Kyr Shatskyy)
qa/tests: added initial test for client-upgrade-reef-tentacle (pr#64761, Yuri Weinstein)
qa/workunits/fs/misc: remove data pool cleanup (pr#63017, Patrick Donnelly)
qa: add missing .qa links (pr#67529, Patrick Donnelly)
qa: Disable OSD benchmark from running for tests (pr#67067, Sridhar Seshasayee)
qa: enable debug mds/client for fs/nfs suite (issue#63482, pr#65251, Venky Shankar)
qa: fix multi-fs tests in test_mds_metrics.py (pr#64340, Jos Collin)
qa: fix test_cephfs_mirror_stats failure (pr#62116, Jos Collin)
qa: ignore pg availability/degraded warnings (pr#61297, Patrick Donnelly)
qa: ignore variant of down fs (pr#62092, Patrick Donnelly)
qa: increase the http.maxRequestBuffer to 100MB and enable the git debug logs (pr#61279, Xiubo Li)
qa: suppress OpenSSL valgrind leaks (pr#65663, Laura Flores)
qa: use a larger timeout for kernel_untar_build workunit (issue#68855, pr#61340, Venky Shankar)
rados/test_crash.sh: add PG_DEGRADED to ignorelist (pr#62396, Shraddha Agrawal)
rbd-mirror: add cluster fsid to remote meta cache key (pr#66272, Mykola Golub)
rbd-mirror: allow incomplete demote snapshot to sync after rbd-mirror daemon restart (pr#66163, VinayBhaskar-V)
rbd-mirror: prevent image deletion if remote image is not primary (pr#64738, VinayBhaskar-V)
rbd-mirror: release lock before calling m_async_op_tracker.finish_op() (pr#64091, VinayBhaskar-V)
rbd: display mirror state creating (pr#62939, N Balachandran)
Recent pipeline backports (pr#65250, Dan Mick)
resolve pacific/quincy upgrade failures (pr#67657, Patrick Donnelly)
rgw/iam: add policy evaluation for Arn-based Conditions (pr#62434, Casey Bodley)
rgw/rados: enable object deletion at rados pool quota (pr#62094, Casey Bodley, Samuel Just)
rgw/sts: Implementation of validating JWT using modulus and exponent (pr#63053, Pritha Srivastava)
rgw: Try to handle unwatch errors sensibly (pr#62403, Adam C. Emerson)
rgw: add force option to radosgw-admin object rm ... (pr#64311, J. Eric Ivancich)
rgw: add missing last_modified field to swift API (pr#61553, Andrei Ivashchenko)
rgw: allow bucket notification send message to kafka with multiple br… (pr#61825, Hoai-Thu Vuong)
rgw: bring rgw-restore-bucket-index up to current version (pr#64514, J. Eric Ivancich, Michael J. Kidd)
rgw: Changed discard buffer size (pr#63711, Artem Vasilev)
rgw: check all JWKS for STS (pr#64937, Alex Wojno)
rgw: correctly set worker thread names (pr#63095, Milind Changire)
rgw: don't use merge_and_store_attrs() when recreating a bucket (pr#64411, Casey Bodley)
rgw: fix 'bucket rm --bypass-gc' for copied objects (pr#66002, Casey Bodley)
rgw: fix bug with rgw-gap-list (pr#62723, J. Eric Ivancich, Michael J. Kidd)
rgw: fix empty storage class on display of multipart uploads (pr#64312, J. Eric Ivancich)
rgw: fix to correctly store updated attrs in backend store after erasing an attr/attrs for delete ops on a bucket (pr#61996, Pritha Srivastava, Wei Wang)
rgw: Head/GetObject support partNumber (pr#62544, Casey Bodley)
rgw: keep the tails when copying object to itself (pr#62656, Jane Zhu)
rgw: make incomplete multipart upload part of bucket check efficient (pr#64464, J. Eric Ivancich)
rgw: make keystone work without admin token(service ac requirement) (pr#64200, Deepika Upadhyay)
rgw: make rgw-restore-bucket-index more robust (pr#64622, J. Eric Ivancich)
rgw: optimize bucket listing to skip past regions of namespaced entries (pr#62234, J. Eric Ivancich)
rgw: prevent crash in radosgw-admin bucket object shard ... (pr#62885, J. Eric Ivancich)
rgw: PutObjectLockConfiguration can enable object lock on existing buckets (pr#62063, Casey Bodley)
rgw: radoslist improvements primarily to better support gap list tool (pr#62418, J. Eric Ivancich)
rgw: trigger resharding of versioned buckets sooner (pr#63598, J. Eric Ivancich)
rgw: update keystone repo stable branch to 2024.2 (pr#66243, Kyr Shatskyy)
Rocky 9/10 support backports (pr#64658, Zack Cerza, John Mulligan, David Galloway, Alexander Indenbaum)
run-make-check.sh backports (pr#65837, John Mulligan, luo rixin)
run-make.sh: Typo in argument addition (pr#66690, David Galloway)
scrub: use a generic interface for scheduling timer based events (pr#63558, Samuel Just, Ronen Friedman)
src/common/options: Clarify scope of scrub intervals in osd.yaml.in (pr#63490, Anthony D'Atri)
src/common: add guidance for deep-scrubbing ratio warning (pr#62503, Zac Dover)
src/common: add guidance for mon_warn_pg_not_scrubbed (pr#62552, Zac Dover)
src/mon/OSDMonitor.cc: [Stretch Mode] WRN non-existent CRUSH location assigned to MON (pr#62040, Kamoltat Sirivadhna)
src: modernize sample.ceph.conf (pr#61642, Anthony D'Atri)
suites/rados: cache tier deprecated, no need to keep the tests for it (pr#62210, Nitzan Mordechai)
sync build-with-container patches from main (pr#65845, John Mulligan, Dan Mick)
tasks/cephfs/mount: use 192.168.144.0.0/20 for brxnet (pr#63134, Kyr Shatskyy)
test/common: unittest_fault_injector omits unit-main target (pr#63979, Casey Bodley)
test/librbd/test_notify.py: conditionally ignore some errors (pr#62688, Ilya Dryomov)
test/librbd/test_notify.py: force line-buffered output (pr#62751, Ilya Dryomov)
test/rbd: remove unit tests about cache tiering (pr#64588, Laura Flores)
TEST_backfill_grow fails after finding "num_bytes mismatch" in osd log (pr#60901, Mohit Agrawal)
tools/ceph-objectstore-tool: tricks to tolerate disk errors for "pg export" command (pr#62122, Igor Fedotov)
Wip trackers 50371 67352 67489 69639 reef (pr#62473, Brad Hubbard, Patrick Donnelly)

Assessing the performance of the CLAY Erasure Code Plugin

2026-02-11T00:00:00Z

CBT Performance Benchmarking - Part 4. What can we say about CLAY?

Outline of the Blog Series ¶

Part 1 - How to start a Ceph cluster for a performance benchmark with CBT
Part 2 - Defining YAML contents
Part 3 - How to start a CBT performance benchmark
Part 4 - Assessing the performance of the CLAY erasure code plugin

Contents:

Client IO results for CLAY
Client IO with an OSD down
What is CLAY good at?
Problems with using CLAY
How does CLAY read data from the drive?
CLAY is broken in tentacle
Summary

Client IO results for CLAY ¶

As a refresher lets quickly look back on the client IO results of CLAY compared to JErasure:

If we look back to Step 3 in Part 3 of the blog (Generating a comparison report), we saw that reads had practically identical curves between CLAY & JErasure for both 4K random reads and 1024K sequential reads.

However, when we compared writes we saw that the performance hit to CLAY was substantially larger, particularly for higher bandwidths. The 1024k Sequential Writes diagram represents this:

Click to see Part 3 diagrams

So why was this?

This is because of CLAY's encoding process, it is significantly more complex. While JErasure performs a single encoding pass, CLAY uses three phases:

50% of data is encoded using PRT (Product Recovery Transform), 50% of the data is copied to form an intermediate set of buffers
All the intermediate data is encoded using RS (Reed-Solomon) to form a second set of intermediate buffers
50% of the result is encoded using PFT (Parity Fractional Transform), 50% of the data is copied to form the output buffers

Essentially, CLAY performs 2x the encoding plus an additional memcpy (memory copy) compared to JErasure's 1x encoding. This overhead therefore directly translates to lower write throughput for CLAY, as shown by the diagrams above. The performance impact increases for larger IO sizes because more data is being encoded.

Referenced the following: 'Clay Codes: Moulding MDS Codes to Yield an MSR Code' above for information on CLAY's encoding process.

Client IO with an OSD down ¶

Click to see Part 3 diagram

We then moved onto Step 4 in Part 3 of the blog (Running a test with an OSD down), and we saw that performance had got even worse for CLAY here. The curves are no longer near identical for the reads (as shown by the above diagram). CLAY is clearly performing worse in this scenario, which we did not initially expect.

This latency increase is due to the specific implementation of CLAY within Ceph. For degraded read IOs (when a client requests data from a missing shard), the system is configured to read and decode all the data to reconstruct the missing information. Just as the encode process (for write IOs) has a higher overheads when using CLAY, the decode process (for degraded read IOs) has similarly higher overheads. This is an implementation choice - when recovering objects (see next section) CLAY uses a more efficient method for recovering the data. This method could also have been used for degraded reads.

What is CLAY good at? ¶

Now you may be thinking, if CLAY is slower for writes and degraded reads, why use it? The answer is for Network Bandwidth Optimisations during recovery processes like backfill and recovery that use the erasure code to reconstruct and repair the missing parts of objects.

While JErasure requires k (data shards) to reconstruct one missing shard, CLAY uses coupled layers to reconstruct data using a significantly smaller amount of data from the remaining shards. In a standard 4+2 setup, JErasure would need to pull 100% of the data from the other 4 shards to rebuild the 5th.

This is what it would look like if we were to use JErasure and simulate a recovery of data when shard 0 is missing:

Now we will compare this to how CLAY would recover data if shard 0 was missing. CLAY reduces this traffic by approximately 50% as you can see in the below example:

It's important to note that the above configuration has a chosen non-default stripe unit of 32K which, with a 4+2 CLAY code, results in a sub chunk size of 4K and matches both the NVMe block size and the Bluestore allocation unit. See the Ceph documentation here for how to calculate the sub chunk size for your configuration.

We can see that with the CLAY example above more data shards are read, however overall less data is read. We can therefore see that for our configuration, CLAY will be more efficient in recovering data when a shard is missing.

With this erasure code profile CLAY will always read 50% of each other shard to recover a missing shard, however the subchunks that are read will vary depending on which shard is missing. The next diagram shows which subchunks will be read for each missing shard:

Note: While the diagram shows a 50% saving in network traffic, this comes at the cost of IOPS. We can see how shards 4 and 5 must perform four individual reads per stripe to gather those specific sub-chunks, so we can see here how it can be dependant on which shard is missing.

In summary CLAY reads much less data than JErasure during recovery/backfill saving approximately 50% network bandwidth which in systems that are limited by the network should improve the performance of recovery.

Problems with using CLAY ¶

Choosing your stripe unit is critical:

If stripe unit is 4K: Sub-chunks become tiny (512 bytes) and reads of less than 4K are rounded up to 4K.

This leads to extra data reads because the NVMe block size is 4K. This means that recovery reads 1x to 4x the amount of data from drives but transmits 50% less data across the network, there is still many more IOPs and CPU usage in this scenario.

Let's break this down a step further using examples of shards 0, 1 and 2 missing. Here we can see in blue the desired amount of data that we want to read, however the orange is the actual amount of data that is read due to these allignment issues.

If stripe unit is 32K: This fixes the fragmentation issue that we see above (sub-chunks align better with 4K drive blocks), but introduces some classic and fast EC problems:

In a classic EC pool, any overwrite requires reading the entire stripe, even if you only changed one byte. At 32K, small writes become incredibly expensive because of the Read-Modify-Write overhead. In classic EC objects are padded to a multiple of the stripe width, so a larger stripe unit increases wasted capacity. In fast EC objects are not padded but a larger stripe unit still results in more coding parity data and less storage efficiency. So there are still negatives to bare in mind if you are to pick a stripe unit of 32K.

How does CLAY read data from the drive? ¶

Fragmented Reads ¶

As shown above, CLAY issues fragmented reads. If the stripe unit gets smaller, for example 4K, the sub-chunk size drops to 512 bytes. This is because NVMe and HDD drives have a minimum block size of 4K, therefore any 512 byte read is rounded up to this minimum of 4K. This can result in CLAY reading the same 4K block multiple times to extract different 512 byte sub-chunks, and discarding the rest of the data. This therefore wastes CPU and drive IOPs, so if either of these are your performance bottlenecks this is not a good scenario.

Squid recovery also always tries to read 2MB from each stripe and expects the read to be truncated if the object is smaller than 2MB * number of stripes. With CLAY this results in a lot of small reads being issued beyond the end of the object. While these as quickly fail and do not stop CLAY recovering the data, this does waste additional CPU resources.

Referring to the same paper as before: Results have been shown that encoding data can take up to 70% longer in terms of CPU usage, if your cluster isn't CPU limited then you won't notice this. These results also showed dramatic savings in backfill and recovery time - but they were done on a system that was network limited and used much wider erasure codes (26 node cluster) than most people would typically use.

There is scope to improve the implementation of CLAY - currently the reads are issued serially, which will add a lot of latency to the recovery. A more efficient approach would be to issue a single read in parallel using readv or to read the entire stripe into memory once, then transmit the required data for the network. The latter would be the better method. This would trade drive bandwidth for a considerable saving in CPU utilisation and drive IOPs.

More in depth: ¶

We went over the 3 phases of how CLAY encodes data earlier. Decoding is also done in 3 phases, but on half the quantity of data:

25% of the data is decoded using PRT, 25% of the data is copied to form an intermediate set of buffers
All (50%) of the intermediate data is decoded using RS to form a 2nd set of intermediate buffers
25% of the data is decoded using PFT, 25% of the data is copied to form the output data

Therefore, CLAY has an additional 0.5x memcpy of the data and the same decoding costs, as JErasure. Hence there is slightly more overhead for CLAY (memcpy's + slight inefficiencies from performing several smaller decodes rather than one large decode). CLAY requires less data to perform the recovery so we can save on network bandwidth (and if implemented correctly, drive bandwidth)

To round off:

CLAY has higher encoding costs and the same decoding cost
CLAY has some memcpy's that JErasure does not have
CLAY has multiple encode/decode steps and there will be some small overheads/inefficiencies - for example, encoding 12K of data in 3 batches of 4K (CLAY) versus encoding 12K of data in 1 batch (JErasure)

CLAY is broken in Tentacle ¶

When performing benchmarking on the Tentacle release, a significant issue was discovered: The recovery benefit was non-existent for Tentacle.

In the tests, recovery in Tentacle transmitted the full amount of data, behaving like standard JErasure but with a higher CPU overhead of CLAY. This isn't the case for Squid however, which is what was used for the updated performance benchmarking used throughout this blog.

Summary ¶

CLAY is a fascinating project and definitely has potential, but for the average user, remains niche.

I'd recommend CLAY if: Your cluster is strictly Network Bottlenecked and you use wide erasure codes (eg 20+ nodes) where the 50% saving is a very considerable amount.

I'd recommend you avoid CLAY if: You are CPU or IOPs limited, or if you primarily use HDDs, as the fragmented serial reads will cripple recovery performance.

For most production environments, the simplicity and predictable performance of JErasure remains the better choice I believe.

Please note that there is a plan to end support for CLAY from the V release. Please see here for more details.

Link to connect with Ceph on slack

Link to previous parts of the blog series

RGW Bucket Resharding Without Pausing

2026-02-01T00:00:00Z

Introduction: The Foundation of Scalable Object Storage ¶

In the modern data landscape, object storage has evolved from a simple file repository into the foundational layer for AI/ML pipelines, data lakehouses, real-time analytics, and massive-scale archival systems. At the heart of this evolution is a deceptively simple question: How do you efficiently locate and access billions of objects stored in a single bucket?

The answer lies in one of Ceph's most critical performance mechanisms: bucket index sharding. This architectural pattern divides a bucket's index into multiple parallel structures, enabling concurrent operations across thousands of objects while maintaining the consistency and reliability that enterprise workloads demand.

But there's always been a catch. As workloads grow and evolve, buckets need to be resharded. Historically, when the buckets to be resharded had a vast number of objects, this operation came with a painful trade-off: blocking client writes from seconds to minutes, with a chance of causing application disruptions, 504 Gateway errors, and operational headaches.

With Ceph Tentacle, we're eliminating this trade-off. The new near-zero impact bucket resharding architecture transforms what was once a maintenance window event into a seamless background operation that your applications will never notice.

Note: As of 2026/02/05, the functionality described in this article is expected in an upcoming Tentacle update.

Executive Summary ¶

The Challenge: In Ceph Squid, resharding a 20-million-object bucket blocked writes for 4+ minutes, returning 504 errors. Even larger buckets (500M objects) required 94 minutes of complete write unavailability.

The Solution: Ceph Tentacle's two-phase architecture moves the heavy lifting to a non-blocking background phase, eliminating the impact on clients IO.

The Results:

(note: in this graphic 8.1 refers to Squid and 9.0 to Tentacle)

In this deep dive, we'll explore:

Why bucket sharding is essential for modern workloads
The challenges of resharding in Ceph Squid and earlier versions
The enhanced two-phase architecture in Ceph Tentacle
Before/after performance comparison from production testing
The future of bucket indexing with in-order sharding

The Scalability Enabler: Understanding Bucket Index Sharding ¶

The Bucket Index and omaps ¶

In Ceph's Object Gateway(RGW), the ability to list bucket contents is fundamental to object storage operations. The Object Gateway(RGW) implements this using a dedicated structure called the bucket index, which maintains an inventory of all objects in a bucket. This index is stored using a special RADOS feature called the Object Map (omap) - essentially a key-value store associated with a RADOS object, physically residing in the RocksDB database on each OSD's DB partition.

Without sharding, a bucket's entire index is stored in a single RADOS object. While elegant in its simplicity, this creates a fundamental performance problem:

The Single-Index Bottleneck: Since only one operation can modify this index at a time, you're looking at complete serialization. Write operations must queue and wait their turn to update the index. As your bucket grows to millions of objects with thousands of concurrent write operations, this serialization becomes a severe bottleneck.

Think of it like a busy airport with only one runway. No matter how many planes are waiting to land, only one can touch down at a time.

Sharding: Parallelism Through Distribution ¶

Bucket index sharding solves this bottleneck by dividing the index into multiple parts (shards), with each shard stored as a separate RADOS object within the .rgw.buckets.index pool. When an object is written, the Ceph Object Gateway (RGW) calculates a hash of the object's name to determine which shard should receive the index update. This enables multiple operations to run concurrently across multiple Placement Groups (PGs), distributing requests among the the OSDs that host the index pool.

Returning to our airport analogy: you now have multiple runways, each handling different aircraft simultaneously. The more runways (shards) you have, the more parallel operations you can support.

The sharding mechanism uses the rgw_max_objs_per_shard tunable (default: 100,000 objects per shard) to determine optimal distribution.

We recommend maintaining no more than 102,400 objects per shard for optimal performance.

Why Single-Bucket Scale is Mission-Critical in Modern Object Workloads ¶

Here's where bucket sharding becomes even more critical: modern analytics architectures are converging on single-bucket designs.

The Data Lakehouse Pattern

Apache Iceberg, Apache Hudi, and Delta Lake (the table formats revolutionizing data architecture) organize petabytes of data within a single bucket using hierarchical prefixes:

s3://analytics-lakehouse/
├── warehouse/
│   ├── sales_db/
│   │   └── transactions/
│   │       ├── data/
│   │       │   ├── year=2025/month=11/
│   │       │   │   ├── 00045-23-a1b2c3d4.parquet
│   │       │   │   └── 00046-24-e5f6g7h8.parquet
│   │       │   └── year=2025/month=10/
│   │       └── metadata/
│   │           ├── v1.metadata.json
│   │           ├── v2.metadata.json
│   │           └── snap-1234567890.avro
│   └── customer_db/
│       └── profiles/
├── staging/
└── archive/

The implication? Modern data platforms need buckets that can:

Scale to billions of objects distributed across thousands of prefixes
Handle mixed workloads: batch ETL, interactive queries, real-time streaming
Adapt dynamically to growth and contraction without downtime
Maintain sub-second listing performance across massive object counts

This is precisely where seamless resharding becomes absolutely critical.

The Challenge: Resharding in Ceph Squid and Earlier ¶

Understanding the improvements in Ceph Tentacle requires understanding the challenges of the previous approach.

The Blocking Resharding Process ¶

In Ceph Squid and earlier versions, bucket resharding followed this process:

Resharding operation initiates (manually or via dynamic resharding)
All client write operations are blocked to the bucket
Index entries are copied from source shards to destination shards
Applications receive 504 Gateway Timeout errors
Operations teams monitor progress

In buckets with small object counts, resharding is almost unnoticeable, but as object counts grow, the write pause can last from minutes to hours, depending on bucket size. Read operations continued, but write unavailability required careful planning for production workloads.

The Operational Impact ¶

This blocking behavior created several operational constraints:

Maintenance Windows: Resharding typically requires scheduling during off-peak hours with advance notification to application teams.

Capacity Planning Tradeoffs: Teams set high initial shard counts based on pre-sharding usage estimates for the bucket, but these are hard to calculate up front.

Dynamic Resharding Concerns: Automatic reshards could trigger during peak business hours, potentially causing disruptions. Some organizations disabled dynamic resharding entirely and managed sharding manually.

Ceph Tentacle addresses these challenges with a fundamentally different approach.

The Solution: Non-Pausing Resharding in Ceph Tentacle ¶

Now let's explore what changes in Ceph Tentacle - and why it's transformational.

Reshard Two-Phase Architecture ¶

The Ceph Object Gateway(RGW) engineering team fundamentally redesigned the resharding process from the ground up. Instead of blocking all writes while copying index entries, Ceph Tentacle introduces an intelligent two-phase incremental approach that keeps your applications running:

Phase 1: Log Record Phase (Non-Blocking)

During this phase, which comprises the bulk of the resharding operation:

Client writes continue normally - no blocking whatsoever
Index operations are logged to source shards alongside regular write operations
Background migration begins - existing index entries start copying to destination shards
Change tracking - a sophisticated logging mechanism captures all modifications

Phase 2: Progress Phase (Minimal Pause, zero client impact)

Only after the bulk of entries have been migrated does Phase 2 begin:

Brief write pause - With zero client impact (milliseconds to low seconds)
Log synchronization - recent changes recorded during Phase 1 are applied to destination shards
Conflict resolution - entries modified during migration are reconciled
Bucket stats recalculation - metadata is updated to reflect the new shard layout
Cutover - bucket switches to the new index layout
Normal operations resume

The Key Innovation: By recording changes as lightweight logs during Phase 1, the system only needs to synchronize recent modifications during the brief Phase 2 pause. The bulk of the work - migrating millions of existing entries - happens entirely in the background while your applications continue writing uninterrupted.

Backward Compatibility: Ceph Tentacle's resharding maintains compatibility as a superset of the previous implementation. If some OSD nodes haven't yet upgraded, resharding safely fails rather than risking data loss, and the system checks version compatibility before proceeding.

What This Means For Your Operations ¶

The practical implications extend far beyond eliminating 504 errors:

1. Eliminate Maintenance Windows. No more scheduling resharding operations for 2 AM on Sunday. Trigger reshards during peak business hours if needed - your applications won't notice.

2. Enable True Dynamic Scaling
Dynamic bucket resharding can now be fully trusted. The automation you've wanted - automatic scaling up and down with minimal client interruption.

3. Production Confidence. Deploy resharding changes without coordination, without warning application teams, without anxiety. It just works.

4. Faster Response to Demand. Workload explodes? Trigger an immediate upshard. No more waiting for a maintenance window.

5. Simplified Operations. One less thing requiring complex runbooks, escalation procedures, and off-hours coordination. Focus on value-add activities instead.

Performance Comparison: Before and After ¶

To validate the architectural improvements, we conducted extensive testing comparing Ceph Squid and Tentacle under identical conditions. The results demonstrate the transformational impact of near-zero impact resharding resharding.

Test Scenario: Small-Scale Bucket with 20 Million Objects ¶

Configuration:

Environment: Single-site deployment using s3cmd
Bucket size: ~20 million objects
Resharding operation: Manual upshard (401 → 10,001 shards for 8.1, 307 → 10,001 for 9.0)
Test action: Upload a 300MB object during active reshard

Results:

The Impact: Uploads that previously required 4+ minutes due to complete blocking now complete in 17 seconds for 300MB objects, with zero errors. That's a 93% reduction in client-perceived latency - or more accurately, the elimination of the problem entirely.

From an application perspective, resharding is now completely transparent. Your applications continue serving requests without any indication that a major infrastructure operation is happening beneath them.

Test Scenario: Medium-Scale Bucket with 500 Million Objects ¶

For larger buckets, the improvements are even more dramatic.

Test Methodology Note: This test was deliberately conducted as a stress scenario to evaluate behavior under extreme conditions. The cluster was pushed to near-saturation with concurrent large-object uploads during resharding operations. This aggressive test configuration amplifies resharding times significantly beyond typical production scenarios, allowing us to validate the improvements under worst-case conditions.

Configuration:

Environment: Single-site deployment using s3cmd
Test: Upload 300MB and 1GB objects during downshard operation
Resharding operation: Downshard from 10,001 → 1,999 shards
Load: Concurrent large uploads pushing cluster toward capacity limits

The Results:

The Impact: While typical production resharding in Ceph Squid would complete faster than the 94 minutes shown here, this stress test reveals critical behavior differences. Under load, Ceph Squid's blocking architecture creates cascading issues - the longer the reshard takes, the longer applications are blocked, potentially triggering timeouts and retry storms. Ceph Tentacle's non-blocking architecture eliminates this entire failure mode. Whether resharding takes 10 minutes or 90 minutes, applications continue operating normally.

At a Glance: The Transformation ¶

Aspect	Ceph Squid	Ceph Tentacle
Client Impact	Complete write blocking	Zero write blocking
Error Rate	504 Gateway errors	No errors
20M Object Upshard	4m23s blocked	17s upload (no pause)
500M Object Downshard	94 minutes blocked	5-17s uploads (no pause)
Maintenance Window	Required	Not required
Dynamic Resharding	Often disabled	Enabled

Looking Forward: The Future of Bucket Indexing ¶

The near-zero-impact bucket resharding feature in Ceph Tentacle is transformational, but it's part of a broader evolution in how Ceph handles bucket indexing at scale.

In-Order Sharding: The Next Frontier ¶

Currently, RGW's hashed sharding optimizes for write distribution but presents challenges for alphabetical listing operations. To fulfill a paginated list request, RGW must perform a "scatter-gather" operation: querying every shard and sorting the combined results. For buckets with thousands of shards, this becomes a bottleneck.

In-order sharding (ordered bucket listing) is in active development and will revolutionize listing performance:

The Change: Instead of using a hash function, objects will be placed into shards based on lexicographical name ordering.

The Impact:

List requests can target specific shard ranges instead of querying all shards.
Paginated listing becomes dramatically faster (query 1-2 shards instead of thousands).
Prefix-based queries (critical for data lakehouses) become highly efficient.
Iterating through object keys becomes significantly more performant.

Why This Matters for Data Lakehouses:

Apache Iceberg, Hudi, and Delta Lake all rely heavily on prefix-based object discovery:

s3://lakehouse/warehouse/sales_db/transactions/data/year=2025/month=11/

With in-order sharding, a query for this prefix would hit only the specific shards containing objects in that lexicographical range - not all 10,000 shards in the bucket.

Combined with non-pausing resharding, Ceph is building toward virtually unlimited, performant scalability within a single bucket - exactly what modern data platforms demand.

For a detailed slide deck on the topic, check out Eric Ivancich's excellent Cephalocon talk:

Video

Slides

Conclusion: A New Era of Operational Excellence ¶

Ceph Tentacle's near-zero impact bucket resharding represents a fundamental shift in production object storage operations, eliminating one of the most significant pain points in large-scale deployments.

As Ceph continues evolving with features like in-order sharding, the vision becomes clear: single-bucket architectures that scale infinitely without operational complexity.

For data lakehouse architects building on Apache Iceberg, for AI/ML engineers managing billions of training artifacts, and for enterprise architects demanding the highest availability without operational friction, Ceph Tentacle delivers the operational maturity that production workloads require.

*All test configurations were performed on HDD production-equivalent hardware. Results may vary based on hardware specifications, network topology, and workload characteristics. Consult the official documentation for detailed configuration guidance and best practices.

We would like to thank IBM for the time to author these articles.

Mastering IAM in Ceph: Multi-Tenancy, Access Control, and Why ACLs Must Die

2026-01-24T00:00:00Z

Introduction ¶

Introduction: When Security Theater Becomes a Real Disaster ¶

In March 2017, a misconfigured S3 bucket at Verizon exposed the personal information of 14 million customers. The root cause wasn't a sophisticated attack; it was a simple oversight in access permissions. The bucket was set to be publicly accessible due to S3 permission misconfiguration, and no one noticed because ACLs were managed separately from the company's centralized IAM policies. The security team had implemented careful, identity-based access controls, but a resource-level ACL silently bypassed them by granting access to "All Users."

This scenario repeats constantly across the industry: ACLs creating invisible access paths that security teams don't know exist, buckets accidentally exposed to the public internet, and contractors uploading data that the bucket owner cannot reliably read or administer, while still consuming capacity.

Between 2017 and 2019, major companies exposed hundreds of millions of records via misconfigured S3 permissions (ACLs and/or bucket policies):

Verizon (2017): 14 million customers - An AWS S3 bucket configured for public access exposed names, addresses, account PINs
Facebook (2019): 540 million records - Third-party apps stored user data in publicly accessible S3 buckets
Instagram (2019): 49 million records - Marketing firm left influencer database unprotected in AWS S3

The AWS response was clear: since April 2023, all new S3 buckets default to "ACLs disabled" (BucketOwnerEnforced) and Block Public Access enabled. AWS strongly recommends disabling ACLs on existing buckets and migrating to a pure policy-based model with IAM Accounts architecture.

If you're running the Ceph Object Gateway (RGW), you have access to the same IAM Accounts model introduced in Ceph Squid 19.2.0. This post explains why ACLs must be disabled immediately and how to implement modern, secure access control with IAM policies.

Do This First (Quick Security Wins)

Before reading further, take these two actions on all production buckets:

Enable Block Public Access - Prevents public exposure via ACLs or bucket policies

Deny ACL operations - Add explicit deny for s3:PutObjectAcl and s3:PutBucketAcl as defense-in-depth

These changes prevent the attack patterns described in this post. Continue reading to understand why and how.

Why ACLs Failed? ¶

Access Control Lists (ACLs) were S3's original permission system. They failed for several critical reasons that made them fundamentally unsafe for production use.

Public Access Disasters ¶

The most dangerous ACL failure was a silent public exposure. A single misconfigured ACL could grant the entire internet access to your data, and your security team would never know because ACLs weren't visible in centralized IAM policies.

How it happened:

$ export RGW_ENDPOINT="https://rgw.example.com"

# Developer accidentally makes object public during testing
$ aws --profile developer --endpoint-url "$RGW_ENDPOINT" s3api put-object-acl \
  --bucket bucketacl \
  --key hosts \
  --grant-read uri=http://acs.amazonaws.com/groups/global/AllUsers
$ aws --profile developer --endpoint-url "$RGW_ENDPOINT" s3api get-object-acl \
  --bucket bucketacl \ 
  --key hosts
{
    "Owner": {
        "DisplayName": "developer",
        "ID": "developer"
    },
    "Grants": [
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"
        }
    ]
}

# Security team checks IAM policies - looks fine (against the same RGW endpoint)

$ aws --profile account-root --endpoint-url "$RGW_ENDPOINT" iam get-user-policy \
  --user-name developer \
  --policy-name S3Access

# ✓ Least privilege, no issues detected

# Meanwhile, the object is public to anyone who can reach the RGW endpoint:

$ curl "$RGW_ENDPOINT/bucketacl/hosts" 
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.2XX.0.X   ceph01 

# Full access, no authentication required
# The same risk exists at bucket scope; a public bucket ACL enables unauthenticated listing
# which can leak keys and metadata

$ aws --profile developer --endpoint-url "$RGW_ENDPOINT" s3api put-bucket-acl \
--bucket bucketacl --acl public-read

# Unauthenticated Access to list bucket contents

$ curl -s "$RGW_ENDPOINT/bucketacl" | xmllint --format -
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>bucketacl</Name>
 ...
  <Contents>
    <Key>hosts</Key>
    <LastModified>2025-12-31T08:58:21.346Z</LastModified>
    <ETag>"71ae31ad9b6e7fda9cb5a8628b2e152a"</ETag>
    <Size>415</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>developer</ID>
      <DisplayName>developer</DisplayName>
 ...
</ListBucketResult>

Why was it catastrophic?

Decentralized control: ACLs could be set per-bucket and per-object, creating millions of potential exposure points
No visibility: ACLs didn't appear in the IAM console - security teams had no way to audit them centrally
Silent bypasses: Even perfect IAM policies couldn't prevent an ACL from granting public access
Object-level chaos: With millions of objects, each having its own ACL, comprehensive auditing was impossible

Real-world impact: The three breaches in our introduction (Verizon, Facebook, Instagram) all involved publicly accessible S3 data caused by permission misconfiguration (ACLs, bucket policies, or both), combined with weak central visibility and auditing; exactly the problems that policy-based access control solves.

The Object Ownership Problem ¶

Beyond public access, ACLs created an ownership nightmare. When external accounts uploaded objects to your bucket, they owned those objects, not you.

# Contractor uploads data to your bucket
$ aws --endpoint-url "$RGW_ENDPOINT" s3 cp sensitive.pdf s3://company-bucket/contractor-data/ --profile contractor
upload: ./sensitive.pdf to s3://company-bucket/contractor-data/sensitive.pdf

# Who owns this object?
$ aws --endpoint-url "$RGW_ENDPOINT" s3api get-object-acl \
  --bucket company-bucket \
  --key contractor-data/sensitive.pdf \
  --profile contractor
{
    "Owner": {
        "DisplayName": "Contractor Account",
        "ID": "contractor"  ← Contractor owns it, not you!
    }
}

# You (bucket owner) can't READ the object
$ aws --endpoint-url "$RGW_ENDPOINT" s3 cp \
  s3://company-bucket/contractor-data/sensitive.pdf \
  ./test.pdf --profile company-admin
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

# You can't even GET the ACL to see permissions
$ aws --endpoint-url "$RGW_ENDPOINT" s3api get-object-acl \
  --bucket company-bucket --key contractor-data/sensitive.pdf \
  --profile company-admin
fatal error: An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied


# You can't MODIFY the ACL
$ aws --endpoint-url "$RGW_ENDPOINT" s3api put-object-acl \
  --bucket company-bucket --key contractor-data/sensitive.pdf \
  --acl private --profile company-admin
fatal error: An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied

For on-premises Ceph deployments, while there's no per-GB billing surprise, the operational and compliance problems are identical: you can't read, audit, or manage data in your own infrastructure.

In Ceph RGW, bucket owners CAN delete objects they don't own. However, they still can't read, view ACLs, or manage those objects, creating operational blind spots and compliance risks.

ACLs include grants that appear safer than "public" but remain dangerously broad. In S3, authenticated-read grants read access to the AuthenticatedUsers group; in Ceph RGW terms, that can translate to "any identity that can authenticate to this RGW endpoint/cluster," not "only my team." On a shared on-premises platform (multiple accounts, tenants, service accounts, CI users, integrations), this can lead to accidental cross-team or cross-tenant data exposure.

# Finance team uploads "internal" data with authenticated-read
# (thinking it's safer than public)
$ aws --endpoint-url "$RGW_ENDPOINT" s3 cp finance-report.pdf \
  s3://company-bucket/finance-report.pdf \
  --acl authenticated-read --profile finance-team
 upload: ./finance-report.pdf to s3://company-bucket/finance-report.pdf

# Check the ACL - looks reasonable?
$ aws --endpoint-url "$RGW_ENDPOINT" s3api get-object-acl \
  --bucket company-bucket \
  --key finance-report.pdf --profile finance-team
{
    "Owner": {
        "DisplayName": "Finance Team",
        "ID": "finance-team"
    },
    "Grants": [
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AuthenticatedUsers"
            },
            "Permission": "READ"  ← ANY authenticated user on the cluster!
        }
    ]
}

# DevOps team (completely different department) can read it!
$ aws --profile devops --endpoint-url "$RGW_ENDPOINT" s3 cp \
  s3://company-bucket/finance-report.pdf ./leaked.pdf
download: s3://company-bucket/finance-report.pdf to ./leaked.pdf

# Contractor user (or any other authenticated user) can also access it
$ aws --profile contractor --endpoint-url "$RGW_ENDPOINT" s3 cp \
  s3://company-bucket/finance-report.pdf ./contractor-copy.pdf
download: s3://company-bucket/finance-report.pdf to ./contractor-copy.pdf

# Anonymous users are still blocked
$ aws s3 cp s3://company-bucket/finance-report.pdf ./anon.pdf \
  --endpoint-url "$RGW_ENDPOINT" --no-sign-request
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Public write is an integrity disaster, not just a leak ¶

ACL errors are not solely about "read" exposure. With bucket ACLs, public-read-write (or broad write grants) can enable untrusted PUT requests to a bucket. That turns into an integrity incident: poisoned datasets, overwritten "golden" artifacts, malware hosting, or backup tampering. Even on-prem "internal-only" does not save you; it just changes the attacker's vector, but the threat still exists.

WRITE_ACP is the "permission to rewrite permissions." ¶

ACLs don’t just control data-plane actions; they can delegate control-plane authority over the ACL itself. In Ceph RGW S3 semantics, WRITE_ACP the permission that allows changing a bucket's ACL (required WRITE_ACP for PUT Bucket ACL). If the wrong principal has it, they can escalate later by granting broader access (including public exposure), and this delegation is distributed across buckets and objects. This is a governance anti-pattern because the system contains a hidden "permission to change permissions."

# Step 1: Bucket owner grants contractor WRITE + WRITE_ACP
$ aws --endpoint-url "$RGW_ENDPOINT" s3api put-bucket-acl \
  --bucket company-bucket \
  --grant-write id=contractor \
  --grant-write-acp id=contractor \
  --profile developer

# Verify the ACL
$ aws --endpoint-url "$RGW_ENDPOINT" s3api get-bucket-acl \
  --bucket company-bucket --profile developer
{
    "Owner": {
        "DisplayName": "developer",
        "ID": "developer"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "Contractor Account",
                "ID": "contractor",
                "Type": "CanonicalUser"
            },
            "Permission": "WRITE"
        },
        {
            "Grantee": {
                "DisplayName": "Contractor Account",
                "ID": "contractor",
                "Type": "CanonicalUser"
            },
            "Permission": "WRITE_ACP"  ← Contractor can modify ACLs!
        },
        {
            "Grantee": {
                "DisplayName": "developer",
                "ID": "developer",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

# Step 2: Contractor abuses WRITE_ACP to make bucket PUBLIC
$ aws --endpoint-url "$RGW_ENDPOINT" s3api put-bucket-acl \
  --bucket company-bucket \
  --acl public-read --profile contractor
# Success! Contractor just made the bucket public

# Step 3: Verify the escalation
$ aws --endpoint-url "$RGW_ENDPOINT" s3api get-bucket-acl \
  --bucket company-bucket --profile developer
{
    "Owner": {
        "DisplayName": "developer",
        "ID": "developer"
    },
    "Grants": [
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"  ← NOW PUBLIC! Anyone can list contents
        },
        {
            "Grantee": {
                "DisplayName": "developer",
                "ID": "developer",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

# Step 4: Anonymous users can now list the bucket
$ aws s3 ls s3://company-bucket/ \
  --endpoint-url "$RGW_ENDPOINT" --no-sign-request
2025-12-31 05:00:00         27 finance-report.pdf
# Public exposure complete

The Solution: Stop using ACLs immediately ¶

AWS and the Ceph Object Gateway (RGW) provide controls to disable ACLs entirely. This should be your first action on any production bucket.

Step 1: Block Public Access ¶

Enforce public access blocks to prevent bucket ACLs from granting public access.

Ceph AWS CLI Configuration Note

All aws CLI commands in this guide assume your AWS CLI profile is configured: See the Ceph documentation on AWS CLI configuration and AWS CLI endpoint configuration for details.

Bucket-level (Granularity per individual bucket):

# Anon access is enabled on bucket from previous example

$ aws s3 ls s3://company-bucket/ \
  --endpoint-url "$RGW_ENDPOINT" --no-sign-request
                           PRE contractor-data/
2025-12-31 07:13:55         26 finance-report.pdf

# We use public-access-block on our bucket

$ aws s3api put-public-access-block \
  --bucket company-bucket \
  --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true \
  --endpoint-url "$RGW_ENDPOINT" \
  --profile developer

# Public access has been removed from the bucket,
# a non-authorized request fails after the put-public-access-block

$ aws s3 ls s3://company-bucket/ --endpoint-url "$RGW_ENDPOINT" --no-sign-request
fatal error: An error occurred (AccessDenied) when\
  calling the ListObjectsV2 operation: Access Denied
# Some AWS CLI versions surface certain error responses
# poorly; if you see a Python exception, re-run with
# --debug to confirm the underlying HTTP 403/AccessDenied.

What each setting does:

BlockPublicAcls: Prevents new public ACLs from being applied (redundant if BucketOwnerEnforced, but adds defense in depth)
IgnorePublicAcls: Ignores existing public ACLs (treats them as private)
BlockPublicPolicy: Prevents bucket policies that grant public access
RestrictPublicBuckets: Blocks public access to buckets even if policies exist

Step 2: Deny ACL Operations via IAM Policy ¶

As the root account administrator, you should establish a security baseline that prevents ACL usage by default for all users and groups. This way, even if a developer tries to use ACLs in the future, they'll get an immediate Access Denied error, preventing accidents before they happen.

The governance pattern creates a standard "DenyACLs" policy that you attach to every new user or group you create. This establishes ACL blocking as your organization's security baseline.

Create the standard policy:

$ cat > deny-acl-operations.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyACLOperations",
      "Effect": "Deny",
      "Action": [
        "s3:PutObjectAcl",
        "s3:PutObjectVersionAcl",
        "s3:PutBucketAcl"
      ],
      "Resource": [
        "arn:aws:s3:::*",
        "arn:aws:s3:::*/*"
      ]
    }
  ]
}
EOF

Here is an example of how to apply the policy to new users as you create them:

# Create a new developer
$ aws iam create-user --user-name alice
{
    "User": {
        "Path": "/",
        "UserName": "alice",
        "UserId": "4abb3a59-7991-4644-8863-347b02adc48f",
        "Arn": "arn:aws:iam::RGW89761398048153XXX:user/alice",
        "CreateDate": "2025-01-03T15:44:06.920034Z"
    }
}
$ aws iam create-access-key --user-name alice

# Immediately apply the ACL deny policy (before giving any other permissions)
$ aws iam put-user-policy \
  --user-name alice \
  --policy-name DenyACLs \
  --policy-document file://deny-acl-operations.json

# Now grant the user their actual S3 permissions
$ aws iam attach-user-policy --user-name alice --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

If Alice later tries to configure ACLs on any bucket, she will get Access Denied:

# Create a bucket as Alice, upload an Object and try to apply a public ACL on the Object
$ aws --profile alice --endpoint-url="$RGW_ENDPOINT" \
  s3 mb s3://alicebucket
$ aws --profile alice --endpoint-url "$RGW_ENDPOINT" \
  s3 cp finance-report.pdf s3://alicebucket
$ aws --profile alice --endpoint-url "$RGW_ENDPOINT" \
  s3api put-object-acl --bucket alicebucket --key \
  finance-report.pdf --acl public-read
#  Error: Access Denied
fatal error: An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied
# Some AWS CLI versions surface certain error responses poorly; if you see a Python exception, re-run with --debug to confirm the underlying HTTP 403/AccessDenied.

With ACLs disabled, you might be wondering: How do I grant cross-account access to share my datasets?

Previously, you might have used ACLs to grant a contractor account read access to specific objects or allowed a partner account to upload files. With ACLs gone, how do you securely share data between accounts?

Two modern approaches exist:

Approach	How It Works	Access Pattern	Best For
Bucket policies	Resource owner adds bucket policy; requesting account adds identity policy	Direct, always-on access	Static, permanent sharing
IAM Role assumption	Resource owner creates an assumable role; requesting account assumes it	Temporary session (1-12h)	Dynamic, auditable access

We'll focus on IAM role assumption because it provides:

Temporary credentials that auto-expire (vs. permanent keys)
Detailed audit trails showing who assumed what role and when (vs. static access logs)
Instant revocation by deleting the role (vs. updating multiple policies)
Least privilege with time-bound access (vs. always-on permissions)

This is also AWS's recommended pattern and follows zero-trust principles. Let's see how.

IAM Accounts: The Modern Solution ¶

The Ceph Object Gateway (RGW) implements AWS-compatible IAM Accounts, introduced in Squid/19.2.0. This provides proper multi-tenancy with policy-based access control instead of ACLs.

What is an IAM Account? ¶

An IAM Account provides isolation for identities and access control:

Account: finance-team (ID: RGW12345678901234567)
├── Users & Groups (isolated per account)
├── Roles (isolated per account)  
├── Policies (fine-grained permissions)
└── S3 Buckets (owned by account)

S3 bucket names are globally unique across ALL accounts in a flat namespace (just like AWS S3). If Finance creates a bucket called financial-reports no other account can use that name. However, bucket ownership and access control are account-specific, only Finance can manage their financial-reports bucket.

Ceph accounts can optionally belong to a tenant for namespace isolation. Within a tenant, bucket names are unique to that tenant; they are not globally unique across all tenants.

Key distinction:

Account Root User: Emergency admin access only, created with --account-root flag
IAM Users: Day-to-day access, follows the least privilege principle

For this post, we'll assume you have two accounts already set up:

Finance Account (ID: RGW00893359550361292)
DevOps Account (ID: RGW89761398048153888)

For a complete guide on creating IAM Accounts, users, and basic configuration, see our previous post: Enhancing Ceph Multitenancy with IAM Accounts.

Scenario: Finance needs to give DevOps read-only access to backup data for disaster recovery testing. Previously, this might have been done with ACLs. Now, we use cross-account role assumption.

Requirements:

DevOps can read backups, but cannot modify or delete them
Access uses temporary credentials (not long-term keys)
Finance can revoke access instantly
Fully auditable (who accessed what, when)

How It Works ¶

The key insight: Create a role in the Finance account (same account as the bucket). When DevOps assumes this role, they temporarily "become" a Finance account principal with Finance credentials.

This is the same STS pattern we covered in our previous post on temporary credentials, but now applied to cross-account access.

Implementation ¶

1. Finance Creates a Cross-Account Role for the Devops Team ¶

Finance creates a role called devops-backup-reader in their account with two policies:

The Trust Policy (who can assume this role):

$ cat > trust-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::RGW89761398048153888:user/dave-backup-ops"
    },
    "Action": "sts:AssumeRole"
  }]
}
EOF

This says: "DevOps account user ‘dave’ can assume this role."

You can use in the trust policy the RGWXXXX:root formatting for the Principal. This gives access to all users in the devops account to assume the role. Then we could configure in the devops account to allow users from a specific IAM group to be able to assume the finance devops-backup-reader role.

And the Permission Policy (what the role can do):

$ cat > role-permissions.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": [
      "arn:aws:s3:::finance-backups",
      "arn:aws:s3:::finance-backups/*"
    ]
  }]
}
EOF

This says: "This role can list & read the finance-backups bucket."

Once we have the policy files created, we can go ahead and create the IAM role devops-backup-reader :

$ aws --profile finance-admin s3 mb s3://finance-backups 
$ aws iam create-role \
  --profile finance-admin \
  --role-name devops-backup-reader \
  --assume-role-policy-document file://trust-policy.json

$ aws iam put-role-policy \
  --profile finance-admin \
  --role-name devops-backup-reader \
  --policy-name ReadBackups \
  --policy-document file://role-permissions.json

2. DevOps User Accesses the Finance Account Dataset ¶

Dave from the DevOps team assumes the role and gets temporary Finance credentials:

# Assume Finance role
$ aws --endpoint-url "$RGW_ENDPOINT" sts assume-role \
  --profile dave-backup-ops \
  --role-arn "arn:aws:iam::RGW00893359550361292:role/devops-backup-reader" \
  --role-session-name david-devops-backup-finance \
  --region default

{
    "Credentials": {
        "AccessKeyId": "ASIA****************",
        "SecretAccessKey": "REDACTED",
        "SessionToken": "REDACTED",
        "Expiration": "2025-0X-15TXX:00:00Z"
    }
}

# Use temporary credentials
$ export AWS_ACCESS_KEY_ID=ASIA****************
$ export AWS_SECRET_ACCESS_KEY=REDACTED
$ export AWS_SESSION_TOKEN=REDACTED

# Access Finance backups (using Finance account credentials!)
$ aws --endpoint-url "$RGW_ENDPOINT" s3 ls s3://finance-backups/
2025-01-14 02:00:00  daily-backup-2025-01-14.tar.gz

$ aws --endpoint-url "$RGW_ENDPOINT" s3 cp s3://finance-backups/daily-backup-2025-01-14.tar.gz .
download: s3://finance-backups/daily-backup-2025-01-14.tar.gz to ./daily-backup-2025-01-14.tar.gz

Why This Works (And Why No Bucket Policy Is Needed) ¶

The role devops-backup-reader is in the Finance account (same account as the bucket). When Dave assumes this role, he receives temporary Finance account credentials. From the bucket's perspective, this is same-account access: only the role's policy is required; no bucket policy is needed.

The cross-account part: Only the AssumeRole action crosses accounts. The actual bucket access is the same account (role and bucket), both in Finance.

Security Benefits of This Approach ¶

Temporary credentials: Expire after 1 hour (configurable up to 12 hours)
No shared secrets: DevOps never sees Finance's long-term keys
Instant revocation: Finance deletes the role → all access stops immediately
Audit trail: Logs show role name, session name, and requesting account
Least privilege: Role has only read permissions, nothing more
Better than ACLs: Centralized control, no object-level chaos

What the Audit Logs Show ¶

The Ceph Object Gateway (RGW) audit logs capture the complete cross-account access pattern. Here's what you will see:

Note: Ensure RGW audit logging is enabled. See the Ceph documentation on bucket and object audit logging (OPS logs) for configuration details.

Example audit log extract when DevOps assumes the Finance role:

$ tail -f /var/log/ceph/ops-log-ceph-client.rgw.default.ceph02.fvqogr.log | jq .
{
...
  "time": "2025-01-04T17:34:07.711570Z",
  "time_local": "2025-01-04T17:34:07.711570+0000",
  "remote_addr": "10.251.0.21",
  "user": "98b5e284-bd74-4a54-922e-cf1ee1d460c2",
  "operation": "assume_role",
  "uri": "POST / HTTP/1.1",
  "http_status": "200",
  "bytes_sent": 999,
  "user_agent": "aws-cli/1.38.34 md/Botocore#1.37.34 ua/2.1 os/linux#5.14.0-496.el9.x86_64 md/arch#x86_64 lang/python#3.9.19 md/pyimpl#CPython m/N cfg/retry-mode#legacy botocore/1.37.34",
  "referrer": "",
  "trans_id": "tx000001bb92497c13eba06-00695aa48f-494246-default",
  "access_key_id": "MPUWRVKZFH9XXXXXXX",
  "temp_url": false
}

# We can then get any specific details on this user
$ radosgw-admin user info --access-key=MPUWRVKZFH9XXXXXXX
{
    "user_id": "98b5e284-bd74-4a54-922e-cf1ee1d460c2",
    "display_name": "dave-backup-ops",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    ...
}

Example audit log extract when Dave from the DevOps Account accesses the Finance bucket:

{
  "bucket": "finance-backups",
  "object": "daily-backup-2025-01-14.tar.gz",
  "time": "2026-01-04T17:42:35.956711Z",
  "time_local": "2026-01-04T17:42:35.956711+0000",
  "remote_addr": "10.251.0.21",
  "object_owner": "RGW00893359550361292",
  "user": "98b5e284-bd74-4a54-922e-cf1ee1d460c2",
  "operation": "get_obj",
  "uri": "GET /finance-backups/daily-backup-2025-01-14.tar.gz HTTP/1.1",
  "http_status": "200",
  "bytes_sent": 26,
  "bytes_received": 0,
  "object_size": 26,
  "total_time": 3,
  "user_agent": "aws-cli/1.38.34 md/Botocore#1.37.34 ua/2.1 os/linux#5.14.0-496.el9.x86_64 md/arch#x86_64 lang/python#3.9.19 md/pyimpl#CPython m/N cfg/retry-mode#legacy botocore/1.37.34",
  "trans_id": "tx00000a13eeac4ce551ce2-00695aa68b-494246-default",
  "authentication_type": "STS",
  "sts_info": {
    "role_name": "$devops-backup-reader",
    "role_session": "david-devops-backup-finance"
  },
  "temp_url": false
}

What this tells you:

Who: Dave from DevOps (identified by role session name and the user uid)
When: 2026-01-04T17:42:35.956711Z (exact UTC timestamp)
What: Downloaded daily-backup-2025-01-14.tar.gz from finance-backups bucket
How: Via STS temporary credentials (authentication_type: "STS")
- Assumed role: devops-backup-reader
- Session: david-devops-backup-finance
From where: IP address 10.251.0.21
Bucket owner: Finance account RGW00893359550361292
Status: Success (http_status: 200, 26 bytes transferred)

Key security insights from this log:

Authentication type is explicitly marked as "STS" - You can easily filter all temporary credential access
User who assumed the role is identified - (98b5e284-bd74-4a54-922e-cf1ee1d460c2)
Role name is captured - You know which role was used (devops-backup-reader)
Session name is captured - You can trace back to who initiated the session (Dave via david-devops-backup-finance)
Object owner is logged - Confirms the bucket belongs to the Finance account, not the accessor
Full HTTP details - User agent shows it was AWS CLI, complete with version

Compared to ACLs: With ACLs, you had no audit trail showing who from which account accessed what. The logs only showed "someone accessed the object" with no attribution to the originating account or session context.

Comparison of IAM Roles Versus ACLs:

ACLs: Decentralized, object-level, permanent, no audit trail of cross-account access
IAM Roles: Centralized, temporary, revocable, full audit trail with account attribution

Understanding Policy Evaluation ¶

To use IAM effectively, you need to understand how permissions are evaluated.

The Basic Rule ¶

When a user requests access to an S3 resource, it follows the following workflow, taking into account that any DENY always wins over ALLOW

Explicit DENY always wins, even if there are multiple ALLOW statements.

Same-Account vs Cross-Account ¶

Same-Account Access (user and bucket in the same account):

Permission needed in either the bucket policy or the identity policy
One ALLOW is sufficient

Cross-Account Access (using role assumption):

Permission needed for AssumeRole (on both sides - trust policy + identity policy)
Role's identity policy grants bucket access (same-account from bucket's perspective)
No bucket policy needed

The Security Roadmap: Enterprise S3 Security Coming to Ceph ¶

The Ceph community is making a significant investment in enterprise S3 security. Several critical features are under active development to bring Ceph RGW to full feature parity with AWS S3's modern security model. Here's what's coming and why it matters.

BucketOwnerEnforced: Disabling ACLs (Coming in a Tentacle update) ¶

Status: Merged into Ceph v20.3.0 (Tentacle) (Issue #63323)

What it does: The PutBucketOwnershipControls API with BucketOwnerEnforced setting disables ACLs entirely and forces all objects to be owned by the bucket owner regardless of who uploaded them.

The problem it solves:

Before (with ACLs):

Contractor uploads → contractor owns object → you, as the owner of the bucket, can't delete it
Developer sets ACL to public → bucket exposed to the internet
Objects disappear from inventory (owned by other accounts)

After (BucketOwnerEnforced):

Anyone uploads → you own the object → you control it completely
ACLs are ignored → impossible to make the bucket public accidentally via ACLs
All objects visible in your inventory reports

How it will work:

# Enable BucketOwnerEnforced on a bucket
$ aws s3api put-bucket-ownership-controls \
  --bucket company-data \
  --ownership-controls 'Rules=[{ObjectOwnership=BucketOwnerEnforced}]'

Once enabled, any requests that include ACL headers (e.g., --acl public-read) will fail. Applications must be audited before enabling this feature on their buckets because if the application is using ACLs in their workflow the application requests using the ACL headers will start failing.

S3Control API Block Public Access (Coming Soon) ¶

Status: Active development, PR #64293 under review

You've disabled ACLs in your Finance account. You've enabled Block Public Access. Your security team is confident the Finance buckets are locked down. Then someone in the Marketing account creates a new IAM user, spins up a bucket, and accidentally makes it public during a website deployment test. Your Finance settings didn't apply to Marketing's account because each account manages its own configuration independently.

This is where account-level controls become critical. While individual buckets can have their own Block Public Access settings, managing hundreds or thousands of buckets individually is error-prone. The S3Control API allows you to set account-level defaults that apply automatically to all buckets in that account, both existing and any new bucket created in the future.

Account-level enforcement prevents all public access:

# Block all public access for entire account
$ aws s3control put-public-access-block \
  --account-id RGW11111111111111111 \
  --public-access-block-configuration \
    BlockPublicAcls=true,\
    IgnorePublicAcls=true,\
    BlockPublicPolicy=true,\
    RestrictPublicBuckets=true

Once the account administrator sets this policy using S3Control, regular account users cannot override it. If a user later tries to disable Block Public Access on a specific bucket, make a bucket public via ACL, or add a public bucket policy, all those attempts will fail with "Access Denied." The account-level setting takes precedence and cannot be bypassed by bucket-level operations. This creates a secure-by-default environment in which enabling public access using ACLs at the bucket-level is impossible.

What each setting will do:

BlockPublicAcls: Prevents new public ACLs from being applied to buckets/objects
IgnorePublicAcls: Ignores existing public ACLs (treats them as private)
BlockPublicPolicy: Prevents bucket policies that grant public access
RestrictPublicBuckets: Blocks public access even if policies exist

Account-level Block Public Access is enforced by the account administrator on regular users within that account, but the account administrator themselves can still modify or disable it. For enforcement from a higher authority, you need organization-level controls. See the next section on Organizational Units and SCPs, which enable Ceph/RGW cluster administrators to enforce immutable policies across all accounts.

Organizational Units and Service Control Policies (Future) ¶

Status: Roadmap item for future Ceph releases

What it will do: Enable cluster administrators to enforce immutable security policies across multiple accounts—policies that even account administrators cannot disable or modify.

The problem it solves: Account-level controls rely on administrator discipline. A determined (or compromised) account administrator can disable Block Public Access or re-enable ACLs. Organization-level controls provide actual enforcement from a higher authority that cannot be bypassed.

Example use cases (when available):

Immutable Block Public Access: Cluster admin sets organization-wide "no public buckets" policy: account admins cannot disable it
Required encryption: Force all objects to use encryption → accounts cannot opt out
Cross-account access policies: Restrict which accounts can share data with external accounts
Audit requirements: Enforce logging and monitoring so that accounts cannot be disabled

This will provide enterprise multi-tenant governance that scales to thousands of accounts with immutable top-down policy enforcement.

Conclusion: Ceph's Enterprise Security Transformation ¶

The migration from ACLs to IAM represents a fundamental shift in S3 security philosophy: from decentralized, object-level chaos to centralized, policy-based control.

Available today in Ceph Squid and later:

IAM Accounts: Multi-tenant isolation with proper account boundaries
Cross-account role assumption: Secure data sharing with temporary credentials
Comprehensive audit logging: Full visibility into who accessed what, when, and how

Coming soon (active development):

BucketOwnerEnforced (Upcoming Tentacle update): Disable ACLs, fix ownership chaos
S3Control Block Public Access (Tentacle/Umbrella): Account-level public access prevention
Organizational Units & SCPs (future): Immutable cluster-wide security policies

The Ceph community is making a substantial investment to bring Ceph Object Gateway (RGW) to full feature parity with AWS S3's modern security model. The roadmap is clear, and the commitment is real.

The modern S3 security model is simpler, safer, and more auditable than ACLs ever were. ACLs created invisible access paths that security teams couldn't see. IAM policies are explicit, centralized, and visible in one place.

Disable ACLs today. Your future self will thank you.

Daniel would like to thank IBM for supporting the community with his time to create these posts.

Breaking the Static Key Habit: Modernizing Ceph RGW S3 Security with STS

2025-12-18T00:00:00Z

Introduction: The USD 148 Million Lesson ¶

In late 2016, Uber learned that intruders had accessed a trove of personal data stored in an Amazon S3 bucket. The entry point was painfully mundane: attackers accessed Uber's source code on GitHub using stolen credentials, found an AWS credential, and used it to access Uber’s data. That single, long-lived credential exposed data on roughly 57 million users and 600,000 drivers.

The breach was bad; the duration risk was worse. Static access keys do not expire. Once leaked, they remain active until someone notices, locates every instance in use, and rotates them. That makes credential theft uniquely dangerous in cloud and S3-style storage, because an attacker can repeatedly return, automate access, and quietly expand their footprint.

Uber ultimately agreed to a $148 million multistate settlement related to how the incident was handled and disclosed. The exact dollar figure is not the main lesson, though. The lesson is this: a single static key can turn a small mistake into a durable breach.

If you are running the Ceph Object Gateway (RGW), you face the same dynamic: S3 credentials in an application configuration file config.yaml, embedded in scripts, or stored in CI/CD variables. Each one is a long-lived credential that, once copied, can be used from anywhere the S3 endpoint is reachable.

This post shows you how to eliminate static credentials using Security Token Service (STS) with temporary credentials that expire automatically. By the end, you'll understand how to implement the same security model that prevented these breaches from being even worse, and how to adapt it for Ceph RGW.

The Static Credential Problem ¶

Let's take a look at some examples of how most applications access S3 storage today:

# app-config.yaml (application config file)
s3:
  endpoint: https://s3.example.com
  access_key: AKIA1234567890ABCDEF
  secret_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  bucket: production-data

Or with the credentials embedded directly in code:

# backup.py
import boto3

s3 = boto3.client(
    's3',
    endpoint_url='https://s3.example.com',
    aws_access_key_id='AKIA1234567890ABCDEF',
    aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
)

Or in environment variables (slightly better, but not by much):

export AWS_ACCESS_KEY_ID=AKIA1234567890ABCDEF
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Why This Is Dangerous? ¶

The Permanence Problem ¶

The fundamental issue with static credentials is that they never expire. Once created, these keys authenticate requests indefinitely, working the same on day one as they do five years later. This creates a dangerous organizational memory gap. Keys made in 2020 still work in 2025, but no one remembers which application uses them, what permissions they have, or whether they're even still needed. When rotation finally becomes necessary, it requires coordinated updates across all applications simultaneously, often in the middle of an incident when coordination is most difficult.

Key Proliferation ¶

Static credentials spread like a virus through an organization's infrastructure. They start in a configuration file for a single application, then get copied into container images where they're baked into immutable layers. They're added to CI/CD pipelines where they're shared across multiple projects. Developers copy them to their laptops for testing, where they sync to cloud backup services. They end up in documentation and internal wikis, pasted as "helpful examples" for other teams. Each copy represents another attack vector, another place where the credentials might leak.

The Revocation Nightmare ¶

When credentials are eventually stolen, and with this level of exposure, it's when" not if. The response options are replete with shortcomings. The credentials work from anywhere where the S3 endpoint is accessible, so there's no easy way to distinguish legitimate requests from attacker activity. Revoking them immediately breaks every application that depends on those keys, forcing an emergency deployment across potentially dozens of services. The alternative is to leave them active while attackers maintain access, then race to update applications before further damage occurs. Organizations need to coordinate emergency updates during an active security incident, precisely when coordination is hardest.

The Permission Accumulation Problem ¶

Static keys tend to accumulate permissions over time. They start with minimal access, but as requirements evolve, it's easier to grant permissions than to audit what's truly necessary carefully. This key needs to read and write, just to be safe. Let's give it access to all buckets; we might expand to new ones later. No one wants to risk disrupting production by restricting access, mainly when credentials are spread across so many systems that tracking down every usage point seems impossible.

The Real Cost ¶

The Uber incident shows the real cost of a leaked static key. A single exposed AWS access key pai9r exposed sensitive data to roughly 57 million users and 600,000 drivers, and Uber later agreed to a USD 148 million multistate settlement related to the incident and its handling.

The uncomfortable truth is that static keys turn small mistakes into persistent breaches because credentials do not naturally "die”. Without expiration, containment depends entirely on detection and coordinated rotation across every place that the key has spread.

The Solution: Temporary Credentials via STS ¶

Security Token Service (STS) fundamentally reimagines how applications authenticate with S3. Instead of using permanent credentials that live forever, applications request temporary credentials that expire automatically after a defined window, typically between fifteen minutes and twelve hours. This simple shift transforms the entire security model.

The mechanics work like this: Applications maintain a minimal service account that is authorized to assume a role. When the application needs to access S3, it calls the STS service using those service account credentials to request temporary credentials for a specific role. STS validates that the service account is authorized to assume that role, then issues time-limited credentials. The application uses these temporary credentials for actual S3 operations. When they expire, the application requests fresh credentials. The entire process is transparent to the application's business logic.

![](images/sequence.png align="center")

The Security Transformation ¶

With static keys, credentials remain valid indefinitely. Once stolen, they persist indefinitely. STS eliminates these problems through automatic expiration. When an application calls AssumeRole, it specifies a DurationSeconds parameter that defaults to 3600 seconds (one hour). The temporary credentials returned include an expiration timestamp that cannot be modified or extended. If an attacker steals temporary credentials from a compromised server or intercepts them in transit, those credentials become worthless the moment they expire.

The audit trail improves dramatically as well. Instead of seeing generic access key IDs that could be used by any application anywhere, the RGW logs now show which specific role was assumed (role_name) and the session name provided when the role was assumed (role_session_name). When applications use descriptive session names that include the application name and a timestamp, security teams can immediately identify which application and which specific execution generated each request. This attribution becomes critical during incident response, when distinguishing legitimate traffic from attacker activity can mean the difference between containing a breach and suffering a complete data exfiltration.

Consider the compromise scenario: An attacker gains access to a production server and dumps memory, capturing the application's current S3 credentials. With static keys, this can represent full, ongoing access to your data, potentially for months before detection. With STS, the attacker has at most one hour before those credentials expire and become useless. STS is not a silver bullet: it will not stop an attacker already on the host. It does put every stolen credential on a timer, which sharply limits persistence and reduces the “evergreen access key” problem. The application continues to operate normally, automatically refreshing its credentials; incident response can focus on evicting the attacker and preventing further refreshes rather than racing to replace long-lived keys everywhere.

"Wait, aren't we still using a static key to assume the role?" ¶

Yes, but with a critical difference. The service account (e.g., backup-service) possesses static Access and Secret Keys, but this user has zero permissions to access S3 data. It cannot list buckets, read objects, or delete data.

Its only capability is to call the STS API to assume a specific Role. If these credentials are leaked, an attacker cannot directly steal data. They would have to know which Role to assume and how to use it, which would add significant friction. Furthermore, you have traces in the audit logs, and you can rotate these service keys without disrupting the application's active S3 sessions.

Quick Primer: Understanding Roles (Just What You Need for STS) ¶

Roles are part of the IAM (Identity and Access Management) API, which the Ceph Object Gateway (RGW) implements to provide AWS-compatible identity management. In this post, we focus on how roles enable STS-based authentication. We'll dive deeper into the full IAM capabilities, including users, groups, policies, and account-level governance, in a specific IAM security post coming soon.

The Role Structure ¶

Every role has two policies:

Trust Policy - Defines who can assume the role
Permission Policy - Defines what the role can do once assumed

Here's the complete flow: Your application holds a minimal service account that is authorized to assume a role (via the role trust policy, an identity policy, or both). When it needs to work (e.g., access S3 resources), it calls STS to assume a role (e.g., backup-reader). STS checks the role's trust policy, validates the request, and issues temporary credentials (access key, secret key, session token) that inherit the role's permissions. Those credentials expire after one hour. The application uses them for S3 operations and automatically requests new credentials as needed.

Here is an example Trust Policy (who can assume the role) allowing the user backup-service to assume the role:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::123456:user/backup-service"},
    "Action": "sts:AssumeRole"
  }]
}

Here is an example Permission Policy (what the role can do), allowing read-only access to the bucket backups:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": [
      "arn:aws:s3:::backups",
      "arn:aws:s3:::backups/*"
    ]
  }]
}

In this post, we'll use inline policies (policies embedded directly in the role). There are other canned policy types available in the IAM API, which we'll cover in a future IAM post.

Beyond Service Accounts: Single Sign-on Authentication with OIDC Integration ¶

The pattern we'll implement uses a service account with static credentials to assume roles. However, RGW also supports AssumeRoleWithWebIdentity, which allows applications to assume roles using tokens from an enterprise identity provider (such as RHSSO (Keycloak), IBM Security Verify, etc.) via OpenID Connect (OIDC). This eliminates the need for static credentials: applications authenticate via your existing SSO system to obtain a JWT, which they then use to request a temporary credential directly from the STS API. This is the most secure option for organizations with mature identity infrastructure, though it requires additional OIDC provider configuration in RGW. We'll cover this advanced pattern in a future post on identity federation.

Implementing STS in Ceph RGW: Step by Step ¶

This implementation builds on the IAM foundation covered in Enhancing Ceph Multitenancy with IAM Accounts. If you're new to Ceph IAM accounts, that post covers account creation, user management, and policy basics. Here, we focus specifically on enabling STS and using roles for temporary credentials.

Let's build on an example use case. We'll create a role for a backup service that needs read-only access to a specific bucket.

To follow this guide, you will need:

Admin access to the Ceph cluster: SSH access to a node where you can run ceph and radosgw-admin commands.

AWS CLI: Installed on your workstation to interact with the RGW S3 endpoint.

Python 3 and Boto3: For running the automation scripts (pip install boto3).

Ceph Squid or later: While basic STS works on older versions, the IAM Accounts feature used in this guide requires Ceph Squid (19.2.0) or newer.

Step 1: Enable STS in RGW Configuration ¶

STS must be explicitly enabled in your RGW configuration. The configuration uses the Ceph config database and requires two settings.

Generate a secure STS key (must be exactly 16 characters):

# Generate a 16-character random key
$ openssl rand -hex 8
# Example output: 0a1b2c3d4e5f6789

Configure RGW to use STS:

Most deployments use client.rgw.default as the RGW client identifier. If your deployment uses a custom service name, replace default with your service name.

# Set the STS encryption key (MUST be exactly 16 characters)
$ ceph config set client.rgw.default rgw_sts_key 0a1b2c3d4e5f6789

# Enable STS authentication
$ ceph config set client.rgw.default rgw_s3_auth_use_sts true

Ceph-Specific Configuration Note

Unlike AWS, where STS is a global service enabled by default, Ceph requires you to explicitly configure the encryption key used to sign the session tokens.

Critical Requirement: The rgw_sts_key must be exactly 16 characters long. If it is 15 or 17 characters, the STS handshake will fail silently or with opaque 500 errors.

Restart all RGW instances to apply changes:

# For default service
$ ceph orch restart client.rgw

Verify the configuration:

$ ceph config get client.rgw.default rgw_s3_auth_use_sts
$ ceph config get client.rgw.default rgw_sts_key

Step 2: Create IAM Account, Root User, and Service User ¶

IAM accounts provide multi-tenancy and resource organization. We'll create an account, a root user for administrative tasks, and a restricted service user for applications.

Create the IAM account: ¶

$ radosgw-admin account create  --account-name=backup-team 
{
    "id": "RGW89761398048153888",
    "tenant": "",
    "name": "backup-team",
    "email": "",
    "quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "max_users": 1000,
    "max_roles": 1000,
    "max_groups": 1000,
    "max_buckets": 1000,
    "max_access_keys": 4
}

Create the Account Root User (for administrative tasks) ¶

The account root user has full permissions on all resources within the account by default, including the ability to use the IAM API to create roles and manage policies. This is built into the account system; no additional capabilities are needed.

Create the root user for the account: ¶

$ radosgw-admin user create   --account-id=RGW89761398048153888 \
  --uid=backup-admin   --display-name="Backup-Team-Admin" \
  --account-root   --gen-access-key   --gen-secret

The --account-root flag is critical: it designates this user as the account's root user, granting full administrative permissions within the account's scope.

The Ceph documentation stats that: Account owners are encouraged to use this account root user for management only, and create users and roles with fine-grained permissions for specific applications.

For this tutorial, we'll use the root user for setup tasks to keep things simple. In production, you would typically use the root user to set up IAM users with specific permissions, then remove or restrict the root user's credentials.

Create the Backup Service User (for applications) ¶

This user will have minimal permissions, only the ability to assume roles. No direct access to S3 resources.

$ radosgw-admin user create \
  --account-id=RGW89761398048153888 \
  --uid=backup-service \
  --display-name="backup-service" \
  --gen-access-key \
  --gen-secret

The service account has no S3 permissions and no IAM capabilities. It can only assume roles that explicitly trust it.

Configure AWS CLI Profiles ¶

Configure two AWS CLI profiles, one for each user. Each profile contains the user's credentials and the RGW/STS endpoint URL, so we don’t need to specify the endpoint on each AWS CLI command. See the AWS CLI configuration documentation for details.

AWS Profile summary for this setup: ¶

backup-admin profile: Uses root user credentials, S3/IAM/STS endpoint https://s3.cephlabs.com
backup-service profile: Uses service account credentials, S3/IAM/STS endpoint https://s3.cephlabs.com

Here is an example .aws/config file:

[profile backup-admin]
region = default
output = json
services = ceph-rgw

[profile backup-service]
region = default
output = json
services = ceph-rgw

[services ceph-rgw]
s3 =
  endpoint_url = https://s3.cephlabs.com
s3api =
  endpoint_url = https://s3.cephlabs.com
iam =
  endpoint_url = https://s3.cephlabs.com
sts =
  endpoint_url = https://s3.cephlabs.com

Verify both profiles:

# Test root user (should work - has full permissions)
$ aws s3 ls --profile backup-admin

# Test service user (should fail - has no S3 permissions yet)
$ aws s3 ls --profile backup-service
# Expect: AccessDenied in RGW logs
argument of type 'NoneType' is not iterable

Identity Summary ¶

At this point, you have two users in the IAM account:

||User||Type||Permissions||Used For|| |backup-admin|Account root user (--account-root)|Full permissions on all account resources + IAM API access Creating buckets, creating/managing roles via AWS CLI| |backup-service|Regular user|None (can only assume roles)|Running backup applications with temporary credentials|

Step 3: Create the Backup Bucket ¶

Run this as the backup admin user (who has S3 permissions):

$ aws s3 mb s3://backups --profile backup-admin

Why the admin user? The service account (backup-service) has no S3 permissions yet; it can only assume roles. The admin user creates the infrastructure (buckets), then creates roles that grant specific permissions to those buckets.

Verify the bucket exists:

$ aws s3 ls --profile backup-admin
2025-12-12 17:09:25 backups

Step 4: Create the Role ¶

Run these commands as the account root user (backup-admin), who has full IAM API permissions.

Create a role trust policy (who can assume this role):

cat > trust-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::RGW89761398048153888:user/backup-service"},
    "Action": "sts:AssumeRole"
  }]
}
EOF

_ARNs in IAM Accounts (Ceph Object Gateway): In the IAM Accounts model, the user ARN is built from the account ID plus the user name; in Ceph this “name” corresponds to the user’s display-name (not the --uid). If your --uid and --display-name differ, ensure that your trust policy Principal ARN uses the display-name value, or the AssumeRole request will not match.

Authorization to assume a role can be granted in two ways. In this tutorial we grant it via the role trust policy by naming the service user as the Principal. In same-account setups, this is sufficient; no user policy is required. If you instead trust the whole account or you are doing cross-account access, attach an identity policy to the user or group allowing sts:AssumeRole on the specific role ARN._

Create the role:

$ aws iam create-role \
  --profile backup-admin \
  --role-name backup-reader \
  --assume-role-policy-document file://trust-policy.json

Create permission policy (what the role can do):

cat > permissions-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:ListBucket"
    ],
    "Resource": [
      "arn:aws:s3:::backups",
      "arn:aws:s3:::backups/*"
    ]
  }]
}
EOF

Attach permissions to role:

$ aws iam put-role-policy \
  --profile backup-admin \
  --role-name backup-reader \
  --policy-name backup-read-policy \
  --policy-document file://permissions-policy.json

Verify that the role was created:

$ aws iam get-role \
  --profile backup-admin \
  --role-name backup-reader
{
    "Role": {
        "Path": "/",
        "RoleName": "backup-reader",
        "RoleId": "8c8eec8c-c647-42bb-8a53-36c6d2fc747a",
        "Arn": "arn:aws:iam::RGW89761398048153888:role/backup-reader",
        "CreateDate": "2025-12-12T22:10:18.644Z",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "arn:aws:iam::RGW89761398048153888:user/backup-service"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        },
        "Description": "",
        "MaxSessionDuration": 3600
    }
}

$ aws --profile backup-service sts assume-role --role-arn "arn:aws:iam::RGW89761398048153888:role/backup-reader" --output json --role-session-name testbr
{
    "Credentials": {
        "AccessKeyId": "reUwxxxxxxn",
        "SecretAccessKey": "CQGxxxxxxx",
        "SessionToken": "nADwRdQ5xxxx90qMZlDPl4ozBjcQKF1tceytgNVGD5D4h2FpoMvjybl31cXI9uh/nUrQePW+Ob3TmpMa4QXdXfml/gQYSYeQLJEzNncQPUQB9+QUl5TShDy4RYYziRulTMWrkYokL6kI0uN0LksQ56/qOyd59A1qbWtsBNYBdvxUUi7r3lhrifn4MNWQbErJKCVNdVOBSzN1L34JDMvjEqN2QyKWLQI16D+XhCq8V05OnQFMHsf128BealrX+KkWS6+74G960WzoHzWDwHF1uO08VlFYCdHO0A==",
        "Expiration": "2025-12-12T23:34:00.247844317Z"
    },
    "AssumedRoleUser": {
        "Arn": "arn:aws:sts::RGW89761398048153888:assumed-role/backup-reader/testbr"
    },
    "PackedPolicySize": 0
}

Step 5: Write Application Code (Python Example) ¶

This code runs with the service account credentials (backup-service), which have no direct S3 access. The application calls STS to assume the backup-reader role and receives temporary credentials for S3 operations.

Identity flow:

The application starts with backup-service credentials (long-term, minimal permissions)
Calls AssumeRole using those credentials to request the backup-reader role
Receives temporary credentials (access key + secret + session token)
Uses temporary credentials for all S3 operations
Temporary credentials expire after 1 hour (or configured duration)
Application manually checks expiration before each operation and refreshes if needed

Upload test file (as admin user who has write permissions):

$ echo "test backup data" > test-backup.txt
$ aws s3 cp test-backup.txt s3://backups/ --profile backup-admin

Running the script: download the script from GitHub Gist, and export the variables of the backup-service user:

# Download script
$ wget -O backup_service.py https://gist.githubusercontent.com/likid0/f7c40c4851bf32c595c7a5e63cf21f35/raw/137bfeea46c20d46d37fa026e29f1b5193c3e281/gistfile1.txt
# Make it executable
$ chmod +x backup_service.py
# Export Vars
$ export AWS_ACCESS_KEY_ID='AKIAIOSFODNN7EXAMPLE'
$ export AWS_SECRET_ACCESS_KEY='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
$ export S3_ENDPOINT_URL='https://s3.example.com'

Run the example backup_service.py cript:

$ python backup_service.py

================================================================================
 Backup Service - STS Temporary Credentials Demo
================================================================================

Configuration:
  Endpoint: https://s3.cephlabs.com
  User:     backup-service
  Key:      VQQPNR4XOW...

Note: These are the service account's PERMANENT credentials
      They have NO direct S3 permissions (can only assume roles)

================================================================================
Calling AssumeRole API to get temporary credentials...
================================================================================

AssumeRole Parameters:
  RoleArn:         arn:aws:iam::RGW89761398048153888:role/backup-reader
  RoleSessionName: backup-job-1765579042
  DurationSeconds: 3600 (1 hour)

Authentication:  Using service account credentials (backup-service)
                 AccessKey: VQQPNR4XOW...

Calling STS endpoint: https://s3.cephlabs.com

SUCCESS! Received temporary credentials:
  AccessKeyId:     YAhacPIIT4BcUWiyPC0M...
  SecretAccessKey: S4VSWFTM2U... (redacted)
  SessionToken:    Yn3A4Mt4VGQoIvloer2ByH3aecQAeP... (redacted)
  Expiration:      2025-12-12 23:37:22.878813+00:00
================================================================================


 Listing backups in bucket 'backups'...
   Found 1 object(s):

    test-backup.txt
      Size: 0.00 MB (17 bytes)
      Modified: 2025-12-12 22:11:23.474000+00:00


================================================================================
Demo completed successfully!
================================================================================

This example script uses manual credential checking: the _check_credentials() method checks expiration time before each operation and calls _refresh_credentials() when needed. This is simple and works well for most use cases.

For long-running jobs (hours or days), see the "Handling Long-Running Jobs: Credential Refresh Strategies" section later in this post, which covers automatic credential refresh using Boto3's RefreshableCredentials. With automatic refresh, Boto3 handles the timing and renewal for you so you never have to think about expiration.

Handling Long-Running Jobs: Credential Refresh Strategies ¶

A critical consideration for production deployments is handling jobs that run longer than the credential lifetime.

The Challenge ¶

The DurationSeconds Parameter controls how long the temporary credentials remain valid:

Minimum: 900 seconds (15 minutes) configurable via rgw_sts_min_session_duration Default: 3600 seconds (1 hour) Maximum: Limited by the role's max_session_duration attribute (defaults to 3600)

When a role is created, it has a max_session_duration of 3600 seconds by default. This means even if you request DurationSeconds=7200 (2 hours), the request will be limited to the role's maximum. To allow longer sessions, you would need to modify the role's max_session_duration when creating it (though for security, shorter durations are recommended).

Here, we share three example strategies for handling this.

Strategy 1: Increase Token Duration (Up to 12 Hours) ¶

The most straightforward approach is to request longer-lived credentials and configure the role to allow them.

Configure maximum session duration on the role: ¶

When creating the role, you can set a maximum session duration:

$ aws iam create-role \
  --profile backup-admin \
  --role-name backup-reader \
  --assume-role-policy-document file://trust-policy.json \
  --max-session-duration 43200  # 12 hours

Or modify an existing role:

$ aws iam update-role \
  --profile backup-admin \
  --role-name backup-reader \
  --max-session-duration 43200  # 12 hours

Verify the setting:

$ aws iam get-role \
  --profile backup-admin \
  --role-name backup-reader \
  --query 'Role.MaxSessionDuration'

RGW Configuration: the following config option controls the global maximum:

ceph config set client.rgw.default rgw_sts_max_session_duration 43200

Limitations:

Maximum duration in Ceph RGW: 12 hours (43,200 seconds)
Not an ideal solution, as it extends the duration of the tokens to twelve hours
Suitable for jobs that can be completed within 12 hours

Strategy 2: Automatic Credential Refresh with RefreshableCredentials ¶

For jobs longer than 12 hours, or to avoid managing token duration, implement automatic refresh using botocore's RefreshableCredentials. This pattern continuously calls AssumeRole to get fresh credentials before expiration.

An enhanced BackupService example script with STS token Auto-Refresh is available here.

How it works:

RefreshableCredentials wraps your credential fetching logic
Before each AWS API call, Boto3 checks if credentials are expired or expiring soon
If needed, boto3 automatically calls _refresh_credentials() to get fresh credentials
Your application never sees authentication errors due to expiration
Each refresh calls AssumeRole using the original service account credentials

Key Advantages:

Works for jobs of any length (days, weeks)
No manual credential management needed
Boto3 handles refresh timing automatically
Original service account credentials remain secure (never exposed to S3 operations)

Important Notes:

The service account's long-term credentials must remain valid for the entire job
Each refresh makes a new AssumeRole call to STS
Credentials are cached in memory only (not written to disk)

Strategy 3: Use Third-Party Libraries ¶

If you prefer not to work with botocore internals, use a well-maintained library:

Install the library aws-assume-role-lib:

$ pip install aws-assume-role-lib

Reference the library in code:

import boto3
import aws_assume_role_lib

# Create session with automatic refresh
parent_session = boto3.Session(
    aws_access_key_id='BACKUP_SERVICE_KEY',
    aws_secret_access_key='your-secret-key'
)

# This session automatically refreshes expired credentials
assumed_role_session = aws_assume_role_lib.assume_role(
    parent_session, 
    'arn:aws:iam::RGW12345678901234567:role/backup-reader'
)

# Use it like any boto3 session
s3 = assumed_role_session.client('s3', endpoint_url='https://s3.example.com')
s3.list_buckets()  # Credentials auto-refresh as needed

Static Key Rotation: Completing the Security Model ¶

You've now implemented STS for temporary credentials, but there's one final layer to complete the security architecture: rotating the service account's static keys.

Background: The create_date Field ¶

Starting with Tentacle, Ceph RGW now includes a create_date timestamp for each access key in the user metadata. This addition enables programmatic key age tracking and automated rotation: a critical capability for eliminating static credential risk.

Example output from radosgw-admin user info:

{
    "user_id": "backup-service",
    "keys": [{
        "user": "backup-service",
        "access_key": "XXXXXXXX",
        "secret_key": "XtDhTWsb6vkNOsAnWBXSIhDhqdRBYXXXXXXX",
        "active": true,
        "create_date": "2025-12-12T22:02:16.628205Z"  ← Key creation timestamp
    }]
}

Recommended Approach: Use a Secrets Manager ¶

The best way to implement key rotation is with a secrets manager such as HashiCorp Vault, IBM GKLM, AWS Secrets Manager, Google Secret Manager, or Azure Key Vault. This approach enables zero-downtime rotation without code changes.

How it works:

Application queries secrets manager (no hardcoded credentials):
- Application starts up and queries Vault/secrets manager for credentials
- Gets current access_key and secret_key
- Uses these to call AssumeRole and get temporary STS credentials
When keys rotate (automated monthly rotation using the create_date field time stamp):
- Generate a new Ceph access key with radosgw-admin key create
- Update the secret in Vault/secrets manager with new credentials
- Keep both old and new keys active in Ceph for a 7-day transition period
- After 7 days, remove the old key from Ceph
Application automatically gets new keys:
- The next time the application restarts or refreshes credentials, it queries the secrets manager
- Gets the new credentials automatically
- No code changes required: the application doesn't know rotation happened
- No downtime: the old key still works during the transition

Migration Strategy: From Static to Temporary ¶

You can't flip a switch and convert all applications overnight. The transition requires methodical planning, careful testing, and phased rollout. Organizations that rush this process end up with broken applications, emergency rollbacks, and frustrated teams. The ones that succeed treat it as a deliberate migration project with clear phases and success criteria.

The challenge isn't technical; the STS implementation is straightforward once one understands roles. The challenge is organizational: identifying where static credentials exist, understanding what each application actually needs, and coordinating updates across teams that may not even realize they're using S3. This is why the first phase isn't about changing anything; it's about understanding what you have.

Coming in the Next Post ¶

You now have STS working in your Ceph environment. Your applications use temporary credentials that expire automatically, dramatically reducing the blast radius of credential theft. The permanent credentials your applications hold can't access S3 directly; they can only assume specific roles with limited permissions. Each role follows least privilege. Every access is logged with full attribution.

We kept the IAM explanation minimal in this post, just enough to implement STS. In the next post, we'll dive into IAM architecture and access control patterns. We'll cover the new IAM Accounts model introduced in Ceph Squid, how it creates proper multi-tenancy, and why the distinction between root account and IAM users matters for security. We'll explore advanced least privilege patterns, trust policy design for cross-account access, and how to test policies before deployment. We'll also examine organizational mandates, such as blocking ACLs entirely and using the new S3Control API for account-level governance.

The authors would like to thank IBM for supporting the community with our time to create these posts.

RocksDB Compression in Ceph: Space Savings with No Performance Cost

2025-12-17T00:00:00Z

Introduction ¶

In the world of data storage, engineers and architects constantly face a fundamental dilemma: the trade-off between performance and efficiency. It’s a balancing act. When you want to save space, you typically enable features like compression, but the common assumption is that this will cost you performance, a CPU cycle tax that slows throughput.

But what if you could significantly reduce your metadata storage footprint without slowing things down?

This search for an answer to this question started with research work from Mark Nelson, who published a blog post on ceph.io that covers RocksDB tuning in depth, exploring RocksDB compression with positive results. These promising results sparked a conversation on the upstream GitHub about enabling compression by default; a link to the PR is available here.

To build on the previous investigation, the Ceph performance team ran tests on a robust hardware configuration running IBM Storage Ceph 7.1 (Reef). The cluster used the BlueStore OSDs for an erasure-coded (EC 4+2) pool, with a hybrid OSD storage setup: HDDs for object data and fast NVMe drives for the BlueStore WAL+DB.

To understand the test, it's helpful to know what the WAL+DB is. In modern Ceph, the BlueStore storage engine manages all data on the OSDs (physical devices). To do this, it must maintain a vast catalog of internal metadata: think of it as a high-speed index that quickly locates every piece of data.

RocksDB, a high-performance key-value database, manages this critical index. In our hybrid cluster, the RocksDB database runs on the fast NVMe deviceses, while the actual object data resides on the slower HDDs.

Because this metadata can grow very large, RocksDB's efficiency, how much space it consumes on those expensive NVMe drives, is a critical factor in the cluster's overall cost and performance. Our test, therefore, focuses on a simple, high-stakes question:

Can we compress this metadata to save space without paying a performance penalty?

Executive Overview ¶

The results were not just positive; they were counterintuitive, revealing a powerful opportunity for optimization that comes with virtually no downside.

The results confirm that using RocksDB compression has no detrimental effect on either throughput or resource consumption in Ceph, while providing significant savings in DB space (compression ratio), especially for smaller objects. As a result of the tests, RocksDB encryption is now enabled by default beginning with the Squid release.

Test Environment and Details ¶

All tests were run against the Ceph Gateway (RGW) to simulate a typical Object Storage workload.

Two different sets of object sizes were used in testing. Each workload leveraged five clients and a range of fixed sizes (one object size per bucket, repeated across the total bucket count), as listed below.

Testing Configuration ¶

Smaller
- 1 KiB, 4 KiB, 8 KiB, 64 KiB, 256 KiB
- 100K objects, 300 buckets, five clients (150M total objectss)
- Fill Workload (~8%) - 3hr
- Hybrid workload (45% reads, 35% writes, 15% stats, 5% deletes)
Larger
- 1 MiB, 4 MiB, 8 MiB, 64 MiB, 256 MiB
- 300 objects, 100 buckets, five clients (150K total objects)
- Fill Workload (~7%) - 40m
- Hybrid workload (45% reads, 35% writes, 15% stats, 5% deletes)

Hardware Used ¶

Two identical clusters, each with

3x Monitor / Manager nodes
- Dell R630
- 2x E5-2683 v3 (28 total cores, 56 threads)
- 128 GB RAM
8x OSD / RGW nodes
- Supermicro 6048R
- 2x Intel E5-2660 v4 (28 total cores, 56 threads)
- 256 GB RAM
192x OSDs (BluesStore): 24 2TB HDD and 2x 800GB NVMe SSD for WAL/DB per node
Pool: site{1,2}.rgw.buckets.data EC 4+2, pg_num=4096

The "Free Lunch" is Real: Significant Space Savings at Zero Performance Cost ¶

The primary and most surprising finding from our tests is that enabling RocksDB compression had no negative impact on performance. The specific algorithm used was LZ4, a lightweight solution known for its high speed. Our analysis suggests that modern CPUs are so efficient at processing algorithms like LZ4 that the overhead is negligible, particularly when compression operations occur on the high-speed NVMe devices where the RocksDB database resides.

Across a variety of hybrid workloads (45% reads, 35% writes, 15% stats, and 5% deletes), we observed no detrimental effect on throughput or CPU resource consumption compared to running the same workloads without compression. This effectively eliminates the traditional trade-off.

Graph 1. CPU Consumption for Small Objects

Graph 2. CPU Consumption for Large Objects

Graph 3. Throughput for Small Object Writes

Graph 4. Throughput for Small Object Reads

The results confirm that using RocksDB compression has no detrimental effect on either throughput or resource consumption in Ceph, while providing significant savings in DB space (compression ratio), especially for smaller objects. This allows a smaller WAL+DB offload partition for each OSD, or conversely helps avoid spillover of RocksDB level data onto the BlueStore slow device.

Small Objects, Massive Gains: A Game-Changer for Object Storage Workloads. ¶

While compression proved beneficial across the board, its impact was most dramatic on small-object workloads. Our tests, which used object sizes ranging from 1 KiB to 256 KiB, showed a remarkable reduction in the storage required for RocksDB metadata. In a BlueStore configuration, Ceph's internal metadata is managed by a RocksDB database running on top of the BlueFS file system on a fast storage device, in our case, an NVMe SSD.

The single most impactful data point we recorded was this: with compression enabled, the bluefs db_used_bytes workload for small objects required 2.68 times less storage during the cluster fill. This is a massive efficiency gain. For any organization whose workload involves storing millions or even billions of tiny objects, the metadata overhead can become a significant storage burden. This feature directly and powerfully addresses that specific pain point by compressing the metadata database on the fast offload device, not object data on HDDs.

This is particularly critical for Object Storage (RGW) workloads. When using the Ceph Object Gateway (RGW), all rich metadata associated with an object, such as its name, size, ACLs, and custom user tags, is stored in RocksDB instances spread across the OSDs that comprise the index pool. Furthermore, the bucket index, which lists all objects within a bucket, is maintained as omap entries in this same database.

For clusters with millions or billions of small objects, this metadata and index data can swell to consume terabytes of space, often becoming the primary capacity bottleneck on the expensive, high-speed NVMe drives. Compressing RocksDB directly compresses this RGW metadata, providing massive and immediate relief on that fast tier.

It's Not Just for Small Objects: Large Objects Also See a Clear Benefit ¶

The positive effects were not limited to small objects. Our tests on large-object workloads, ranging from 1 MiB to 256 MiB, also showed clear benefits. While the source report highlights the most dramatic space savings for small objects, it explicitly notes that the positive effect across both sets of object sizes is evident, making compression a clear win for large-object workloads as well.

Furthermore, our test plan included stressful OSD failure scenarios to measure behavior under duress. The overall conclusion of "no detrimental effect" on performance or resource consumption held even during these fault and recovery operations. This implies that RocksDB compression is not just efficient but also a stable and robust feature under pressure.

Graph 5. Throughput for Small Object Reads During Failure

Graph 6. Throughput for Small Object Writes During Failure

Conclusion: A Feature That Should Be Enabled By Default ¶

Based on this comprehensive testing, RocksDB compression in a Ceph environment is a low-risk, high-reward feature. It breaks the old rule that says efficiency must come at the expense of performance. The evidence points to a clear win: substantial storage savings on the metadata layer, with no measurable trade-off in throughput or CPU usage.

This led to a simple conclusion: given the potential for substantial space savings with no performance downside, the decision was to enable RocksDB LZ4 compression by default in the Squid release.

# ceph version
ceph version 19.2.1-222.el9cp (f2cd71cc2f7b46709c2351134ac89ea3e9f609b6) squid (stable)

# ceph config get osd bluestore_rocksdb_options
compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0

The authors would like to thank IBM for supporting the community with our time to create these posts.

Ceph RGW Rate Limiting

2025-12-16T00:00:00Z

Introduction ¶

The Tentacle release introduces significant enhancements to Object Gateway (RGW) rate limiting, addressing a critical gap that has long challenged administrators managing multi-tenant object storage environments. With the addition of rate limiting for LIST and DELETE operations, along with improved STS integration, administrators now have more granular control over resource consumption across their storage infrastructure.

Understanding Rate Limiting in the Ceph Object Gatway (RGW) ¶

Rate limiting in the Ceph Object Gateway has been a powerful tool for controlling resource consumption and preventing individual users or applications from monopolizing cluster resources. Before this enhancement, RGW supported rate limiting for:

Read operations (max-read-ops): Controlling GET request frequency
Write operations (max-write-ops): Limiting PUT request rates
Read bandwidth (max-read-bytes): Throttling data egress
Write bandwidth (max-write-bytes): Controlling data ingress

These limits operate within configurable time windows (controlled by the rgw_ratelimit_interval option), traditionally defaulting to 60 seconds. The system uses a token bucket algorithm to track resource consumption, and when limits are exceeded, RGW returns HTTP 503 responses to throttle clients.

The Scope of Rate Limiting ¶

Rate limits can be applied at multiple scopes:

User scope: Limits apply to a specific user across all buckets
Bucket scope: Limits apply to operations in a particular bucket
Global scope: Limits apply cluster-wide across all users and buckets
Anonymous scope: Limits for unauthenticated requests

Important Architectural Considerations ¶

The Ceph Object Gateway's rate-limiting feature is not a complete QoS system. Key points:

Per-RGW enforcement: Limits are enforced per RGW instance, not cluster-wide. With 2 RGWs and a desired 10 ops/minute limit, configure each RGW for 5 ops/minute. If the client request load isn't evenly distributed across the endpoints, the required limits may be lower than expected.
Limit intersection: Both user-level AND bucket-level limits must be satisfied. Requests are rejected if either limit is exceeded.
No traffic shaping: Throttled requests are immediately rejected (503) rather than queued.
No mid-request throttling: Bandwidth is counted after a request completes, not during. Users who exceed limits go into "debt" (max: 2x the limit) and are blocked from new requests until the next interval(s) repay the debt.

The Problem: Missing Control for List and Delete Operations ¶

While read and write operation limits provided good coverage for data transfer operations, two critical operation types remained uncontrolled:

List Operations ¶

Bucket listing operations, particularly against buckets with millions of objects, can place a significant load on the cluster. These operations:

Scan bucket indexes extensively
Consume RADOS read IOPS on index pools
Can impact overall cluster performance when executed at high frequency
Are costly when using prefixes and delimiters that require filtering

Previous limitation: LIST operations (which use GET/HEAD HTTP methods) were counted as read operations under the max-read-ops limit, making it impossible to control listing separately from regular GET operations. This meant administrators couldn't prevent list-heavy workloads from consuming the entire read operation budget while still allowing standard data retrieval.

Consider a workload performing checkpoint validation by repeatedly listing with prefixes like:

$ aws s3api list-objects-v2 --bucket data --prefix checkpoint-flag --max-items 1

Even though this returns minimal data, each request triggers index scanning operations that consume cluster resources.

As an example, Apache Iceberg tables in data lakehouse environments have been particularly challenging to maintain metadata for Iceberg's deleteOrphanFiles procedure, which cleans up unreferenced data files, requiring complete table listings that can overwhelm object storage systems.

![](images/f06a38ed-1eb5-4457-a13e-45a0eba48684.png align="center")

Delete Operations ¶

Single-object and multi-object delete operations were also uncontrolled, creating challenges for:

Preventing abuse during bulk deletion scenarios
Managing garbage collection workload
Controlling the rate at which storage capacity is reclaimed
Protecting against accidental or malicious mass deletion events

Previous limitation: DELETE operations were classified as write operations (non-GET/HEAD HTTP methods) and counted against max-write-ops, making it impossible to limit deletion rates from PUT operations separately. Workloads that combined uploads and deletions had to balance their write-ops budget across both operation types.

Without dedicated controls for these operations, administrators had limited options for managing workloads that mixed listing, reading, writing, and deleting operations in different proportions.

The Solution: Enhanced Rate Limiting in Tentacle ¶

![](images/a0ea6524-e006-496b-b2bf-855834372d56.jpeg align="center")

Tentacle introduces two new rate-limiting parameters that address these gaps.

New Configuration Options ¶

Max-list-ops: Specifies the maximum number of bucket listing requests per accumulation interval. A value of 0 (default) disables this limit, maintaining backward compatibility.
Max-delete-ops: Specifies the maximum number of delete operations per accumulation interval. A value of 0 (default) disables this limit.

Critical: Backward Compatibility Behavior ¶

Important: The new limits work in conjunction with existing read/write operation limits:

LIST operations: Count against both max-read-ops AND max-list-ops
DELETE operations: Count against both max-write-ops AND max-delete-ops

Both limits must be satisfied for a request to proceed. Administrators upgrading from earlier versions will see no behavior change unless they explicitly configure the new parameters.

![](images/36e1a637-6316-40de-90d6-df76a8fcb97f.png align="center")

Configurable Time Windows ¶

The rgw_ratelimit_interval configuration option allows administrators to adjust the interval for rate limit accumulation. This is particularly important for workloads that exhibit bursty behavior:

$ ceph config set client.rgw.rgw.1 rgw_ratelimit_interval 10

The default 60-second interval may not be optimal for all workloads. Bursty workloads, such as Apache Iceberg's metadata maintenance operations (snapshot expiration, orphan file cleanup), can exhaust their LIST operation budget in the first few seconds of a time window. Since Iceberg's deleteOrphanFiles procedure performs complete table listings across potentially thousands of partitions in rapid succession, the accumulated operations can quickly exceed the rate limit, resulting in extended throttling periods during which subsequent maintenance tasks are blocked. Shorter intervals (1-10 seconds) can provide more consistent behavior by allowing the operation budget to replenish more frequently, preventing long stalls in critical table maintenance workflows.

STS Integration ¶

A new enhancement to the STS/IAM feature ensures that rate limits now apply correctly when users authenticate with temporary credentials obtained via the Security Token Service (STS):

User rate limits configured on an account continue to be enforced when that user assumes an IAM role and operates with temporary credentials.
Bucket rate limits are enforced adequately for operations performed using STS credentials, regardless of how the user authenticated.
Global rate limits now work seamlessly with federated authentication flows, such as AssumeRoleWithWebIdentity.

This closes a previous gap where rate limiting enforcement may not have worked correctly with STS sessions, ensuring consistent rate limit policies across all authentication methods.

Rate Limiting Configuration Examples ¶

Example 1: Configuring LIST Operation Rate Limits ¶

Set up a user with list operation limits to control the frequency of bucket listings.

Create a test user:

$ radosgw-admin user create --uid=testuser --display-name="Test User"
{
    "user_id": "testuser",
    "display_name": "Test User",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "testuser",
            "access_key": "TESTUSER_ACCESS_KEY",
            "secret_key": "TESTUSER_SECRET_KEY"
        }
    ],
    "caps": [],
    "op_mask": "read, write, delete",
    "type": "rgw"
}

Set rate limits for list operations. We have two RGW services deployed in our cluster, so if we want to limit the operations to 10, we need to divide the ops limit by the number of RGWs running in the cluster:

$ radosgw-admin ratelimit set --ratelimit-scope=user --uid=testuser \
    --max-list-ops=5 \
    --max-read-ops=100

Enable rate limiting for this user:

$ radosgw-admin ratelimit enable --ratelimit-scope=user --uid=testuser

Verify the configuration:

$ radosgw-admin ratelimit get --ratelimit-scope=user --uid=testuser
{
    "user_ratelimit": {
        "max_read_ops": 100,
        "max_write_ops": 0,
        "max_list_ops": 5,
        "max_delete_ops": 0,
        "max_read_bytes": 0,
        "max_write_bytes": 0,
        "enabled": true
    }
}

Example 2: Configuring DELETE Operation Rate Limits ¶

Set up delete operation limits to control the rate of deletions. Set rate limits for delete operations on the same user:

$ radosgw-admin ratelimit set --ratelimit-scope=user --uid=testuser \
    --max-delete-ops=10 \
    --max-write-ops=100

Verify the updated configuration:

$ radosgw-admin ratelimit get --ratelimit-scope=user --uid=testuser
{
    "user_ratelimit": {
        "max_read_ops": 100,
        "max_write_ops": 100,
        "max_list_ops": 5,
        "max_delete_ops": 10,
        "max_read_bytes": 0,
        "max_write_bytes": 0,
        "enabled": true
    }
}

Observing Rate Limiting in Action ¶

Let's see what happens when a user exceeds their configured limits.

Test Scenario: Exceeding List Operation Limits ¶

With the configuration from Example 2 (5 list ops per RGW, 10 list ops total per minute), configure AWS CLI with test credentials:

$ aws configure set aws_access_key_id TESTUSER_ACCESS_KEY
$ aws configure set aws_secret_access_key TESTUSER_SECRET_KEY

Create a test bucket:

$ aws --endpoint-url http://rgw.example.com s3 mb s3://test-bucket
make_bucket: test-bucket

Populate the bucket with test objects:

$ for i in {1..100}; do
    echo "Test object $i" | aws --endpoint-url http://rgw.example.com s3 cp - s3://test-bucket/object-$i
done

Rapidly execute list operations to exceed the limit. I will use a script that uses curl to list the contents of the bucket test-bucket repeatedly:

bash script.sh
Testing Rate Limit with list-objects-v2...
------------------------------------------------
Attempt 1: ✅ SUCCESS (200)
Attempt 2: ✅ SUCCESS (200)
Attempt 3: ✅ SUCCESS (200)
...
Attempt 10: ✅ SUCCESS (200)
Attempt 11: 🛑 BLOCKED (503) - Limit Reached
Attempt 12: 🛑 BLOCKED (503) - Limit Reached
Attempt 13: 🛑 BLOCKED (503) - Limit Reached

Testing Delete Rate Limits ¶

Attempt to delete objects beyond the configured limit (20 deletes per minute) with the AWS CLI client:

$ for i in {1..25}; do
    echo "Delete attempt $i"
    aws --endpoint-url http://rgw.example.com s3 rm s3://test-bucket/object-$i 2>&1 | grep -E "delete:|error"
done
Delete attempt 1
delete: s3://test-bucket/object-1
Delete attempt 2
delete: s3://test-bucket/object-2
Delete attempt 3
delete: s3://test-bucket/object-3
...
Delete attempt 19
delete: s3://test-bucket/object-19
Delete attempt 20
delete: s3://test-bucket/object-20
Delete attempt 21
delete failed: s3://limits-bucket/object-21 argument of type 'NoneType' is not iterable
Delete attempt 22
delete failed: s3://limits-bucket/object-22 argument of type 'NoneType' is not iterable

Known Limitations and Future Enhancements ¶

Current Limitations ¶

Backward Compatibility Constraint*: LIST operations still count against max-read-ops, and DELETE operations count against max-write-ops. The new max-list-ops and max-delete-ops limits provide additional constraints but do not replace the legacy limits. Both limits must be satisfied for a request to proceed. This design choice maintains backward compatibility but means you cannot completely isolate LIST/DELETE operations from general read/write operation budgets.
Multi-Object Delete: The S3 DeleteObjects API (bulk delete) is not currently rate-limited but is tracked for future enhancement: RFE
IAM Account Limitation: Rate limits on IAM accounts (as opposed to users) do not currently work. This is tracked as an RFE for a future release. RFE
Multipart Upload Accounting: During multipart uploads with limited write ops, the CreateMultipartUpload, UploadPart, and CompleteMultipartUpload operations each count against the write-ops limit. For large files split into many parts, this can quickly consume the operation budget. RFE
Improved Logging Output. Currently, when hitting a rate limit, we see only the following opaque errors in the RGW log, which don’t specify which rate limit we have reached. RFE

2025-11-20T16:39:40.030+0000 7f9e6423a640  2 req 15365199512736087891 0.001000024s s3:delete_obj check rate limiting
2025-11-20T16:39:40.030+0000 7f9e6423a640 20 req 15365199512736087891 0.001000024s op->ERRORHANDLER: err_no=-2218 new_err_no=-2218
2025-11-20T16:39:40.030+0000 7f9e6423a640  2 req 15365199512736087891 0.001000024s s3:delete_obj http status=503

Conclusion ¶

The addition of LIST and DELETE operation rate limiting in Tentacle represents a significant maturity improvement for the Object Gateway. Combined with the new STS integration and configurable time intervals, administrators now have comprehensive tools for managing multi-tenant object storage workloads.

These enhancements are particularly valuable for:

Enterprises implementing department-level chargebacks and resource governance
Cloud-native applications using federated identity with OIDC
Data analytics platforms with mixed read-heavy and metadata-intensive operations

While some limitations remain (particularly around multi-object delete and IAM accounts), the current implementation provides production-ready capabilities that have been extensively tested with workloads ranging from small-object writes to multi-million object listings.

Get Involved ¶

We encourage you to test these new capabilities in your environment and share your experiences with the community.

The authors would like to thank IBM for supporting the community with our time to create these posts.

Special thanks to the Ceph community and the IBM Storage Ceph QE team for their extensive testing and validation of these features, covering functional, scale, and regression scenarios with millions of objects and hundreds of gigabytes of test data.

Ending Support for some Erasure Code Plugins

2025-12-16T00:00:00Z

A plan to end support for some erasure code plugins and techniques in the Ceph V release.

The Erasure Code Plugin Interface ¶

Ceph uses a plugin interface for erasure coded pools. These plugins are external code libraries that are used to do the encoding and decoding of data. Ceph pass chunks of data to the plugin. The plugin uses an encoding algorithm to produce additional chunks called parity (or coding) chunks.

When an erasure coded pool is created, an erasure code profile must be selected. Among other things, the profile includes the plugin and the technique that will be used for the pool. The technique defines the algorithm that the plugin will use for encoding and decoding, and some plugins support multiple different techniques. Because the parity chunks generated are different for each combination of plugin and technique, there is no way to change the plugin and technique after the pool has been created.

Ceph currently supports five erasure code plugins, some of which support multiple techniques:

Jerasure
- reed_sol_van
- reed_sol_r6_op
- cauchy_orig
- cauchy_good
- liberation
- blaum_roth
- liber8tion
ISA-L (Intel Intelligent Storage Acceleration Library)
- reed_sol_van
- cauchy
SHEC (Shingled Erasure Code)
CLAY (Coupled Layer)
LRC (Locally Repairable Erasure Code)

Why are there so many options?

In the distant past, before CPUs supported SIMD instructions (Single Instruction, Multiple Data) and could encode and decode lots of data in parallel, Jerasure's XOR-optimized techniques (cauchy, liberation, liber8tion and blaum_roth) offered a performance improvement when encoding and decoding data. Now, with SSE, AVX (and other) instructions, the need for techniques that rely on XOR operations has been greatly reduced, and reed_sol_van is very close or in some cases better, than the XOR-optimized techniques. See the comparison chart later in this post for the data!

SHEC and CLAY both focus on trying to improve the recovery efficiency (by optimizing network and disk usage when decoding data) when an OSD or server fails and data must be rebuilt. Both of these plugins build on top of Jerasure, with additional logic that aims to speed up recovery.

LRC also builds on top of Jerasure and intends to improve recovery efficiency by using locally available data (e.g. data in the same data centre or same rack) to minimise transfers between racks or sites.

Ending Support For Plugins ¶

In the Ceph Tentacle release we introduced a new version of erasure coding that has become known as Fast EC. Fast EC offers significant performance improvements and some capacity savings when using erasure coding, particularly for block and file workloads. There are even more improvements to Fast EC coming in future releases. See Lee Sanders' blog post https://ceph.io/en/news/blog/2025/tentacle-fastec-performance-updates/ for more details about Fast EC.

Fast EC changes the interface between Ceph and the erasure code plugins. In Tentacle, only ISA-L (using reed_sol_van or cauchy) and Jerasure (using reed_sol_van) support Fast EC. The old EC code has been kept as a separate code path in Ceph, and the other plugins (and other Jerasure techniques) continue to use old EC.

Our proposal is that we should end support for the least used (and least useful) plugins and techniques in the V release. Ceph clusters using these plugins and techniques will not be able to upgrade to the V release unless data is first migrated to a pool that uses a supported plugin and technique.

Why not continue to support all of these plugins using the old EC code path?

The Fast EC work exposed the amount of development effort required to continue to support such a big list of plugins and techniques. Even though only the most important plugins and techniques support Fast EC, code changes were still required to ensure that the other plugins continue working correctly. We now have two separate erasure code paths to maintain. Along with extra development work, supporting a big list of plugins also means lots more testing needs to be done to ensure nothing gets broken.

We don't think this effort is justified given the small number of users using some plugins, and the lack of benefits that these plugins and techniques provide according to performance benchmarks. Developer focus would be better spent on improving other parts of Ceph.

The proposed list of plugins and techniques that we will support in the V release are:

Jerasure
- reed_sol_van
ISA-L
- reed_sol_van
- cauchy
LRC (Although LRC doesn't currently support Fast EC and we wouldn't recommend using it yet, we think we will be able to use LRC in future to improve support for erasure coded pools in stretched clusters.)

Telemetry Data ¶

So let's look at some data. How many people are actually using each plugin? Not every Ceph cluster has opted in to upload usage data to Telemetry, but enough have to give us a good idea about the plugins and techniques that people are using. Here is a recent snapshot of the clusters using erasure coded pools:

Performance Data ¶

My talk at Cephalocon 2024 (https://www.youtube.com/watch?v=aM8sJgDD-x4) discussed why we've made ISA-L the default plugin for new EC pools. The talk included performance data captured using Ceph's EC benchmarking tool, and I've included that here. These charts demonstrate how advancements in SIMD instructions have brought the performance of the reed_sol_van technique to a point where reed_sol_van is almost as good as or better than other techniques. Note that the ISA-L vandermonde and cauchy lines are overlapping in the encode graph:

As mentioned earlier, the goal of both SHEC and of CLAY is to improve recovery efficiency when an OSD is down. A recent blog post written by Jake Squelch uses the Ceph Benchmarking Tool (CBT) to compare performance of the Jerasure and CLAY plugins. His results show that there is a trade-off when using CLAY. Although CLAY can reduce network bandwidth usage during recovery by around 50%, there is a performance penalty for client I/O during normal operation, and when an OSD is down and the cluster needs to recover data. Data is being read in a very inefficient way, particularly when using the default stripe_unit value. See https://ceph.io/en/news/blog/2025/cbt-performance-benchmarking-part3/ for more detail.

Pool Migration ¶

What if you're the owner of the single cluster using Jerasure's blaum_roth technique in the telemetry data? As we end support for the above list of plugins and techniques, we will need a way to move data from those pools into new pools that use supported plugins. In the Umbrella release we plan to add such a pool migration feature. This new feature will provide a way to non-disruptively move data from one pool to another. The migration will run as a background task, similar to backfill and recovery, with no downtime where data is inaccessible. This will allow you to migrate all the objects from a pool that uses an unsupported plugin to a new pool that uses a supported plugin and technique, and then upgrade the cluster to the V release. See https://github.com/ceph/ceph/pull/65703 for the pool migration design document.

Conclusion ¶

As we've developed Fast EC, it's become clear that continuing to support such a big list of plugins and techniques is too much effort for the value that some of the plugins and techniques provide.

In the Umbrella release we will deprecate the plugins and techniques not included in the supported list mentioned above. In the V release we will end support for those plugins and techniques. You will not be able to upgrade to the V release if your cluster has any pools that use those plugins and techniques. You will be able to use the new pool migration feature in Umbrella to migrate data from a pool to a new pool that uses one of the supported plugins and techniques.

Reducing our list of supported plugins and techniques will allow us to focus our development efforts and continue to improve Fast EC, without the risk of breaking lesser-used plugins.