Ceph Troubleshooting: Debugging CRUSH maps

The last days I’ve been busy trying to find a “error” on my CRUSH map.
I found that some of my OSD’s where underused or unused at all… I didn’t know why, cause I built a CRUSH map from scratch with the common architecture based on datacenter, rack & cluster. And It was correct from the ceph point of view (It was running on the cluster).

I decided to simplify the map to a much simpler one.
Something like this:

Output form ceph osd crush tree:

But still it didn’t allocate PG’s on pair nodes!

I kept reading about CRUSH parameters, then i found a good explanation of chooseleaf here:

step chooseleaf firstn: What I’ve found is that this value is the DEPTH of your CRUSH tree:

• 0 Single Node Cluster
• 1 for a multi node cluster in a single rack
• 2 for a multi node, multi chassis cluster with multiple hosts in a chassis
• 3 for a multi node cluster with hosts across racks, etc.

But this definition was not accurate… You can try to understand the official documentaion, read about:
step choose firstn {num} type {bucket-type}
What I understand is that the {num} is the “osd pool size” at this stage on the CRUSH mapping algorithm.
And on the next step:
step chooseleaf firstn {num} type {bucket-type}
The “osd pool size” on the “leaf” nodes.

So in my case, I’ll have 4 copies at “datacenter” layer and 2 on the “osd” layer, so CRUSH will put 2 copies of the leaf copies on each datacenter.

What I’ve seen is you want to have a fix number on “size” and “min_size” on your pool. The best option will be setup a placement rule forcing that options.
For example:

The checks give me a “good” result: