"Judging from the Application Size, Baseline Size and the high Primary Block Count - I suspect that you have a high change rate and do not have have extended retention turned on your Storage Policy."
You are right.
"Assuming that you are not doing DDB backups at the same time as when the jobs are running, the best thing to help is to use the DDB v4 Gen2 which does a daily mark and sweep instead of having to maintain a Primary Block counter."
That is correct assumption. With support engagement we've been able to Enable Garbage Collection and pruning logs for reconstruction today. Fortunatelly it turned out that time consuming compactfile secondary wasn't neccessary as this DDB was created on SP14. I hope we will see some results soon.
"That said, how large was the file used for IOMeter (if RAW isn't an option, I try to create as large a file as I can)?"
It was tested on 500GB file with should be sufficient given that current partition size is around 450GB.
"Also, is this a recent phenomenon - have the DDB graphs done a hockey-stick and trended up? Is it a recent Data Aging Job that has aged off a lot off jobs triggered it (the process of moving Primary & Scondary into the Zero-Reference table can be a massive task)? Do you see much in the way of pending deletes?"
Not exactly. Q&I was high for some time, but it was closer to 1300-1400us and lately it started to rise. This happened before but it usually went up to 1600s us and go back to it's usual 1300-1400us. This is the first time it went as high as 1800-1900us.
"Check if you have some clients that are deduping poorly and then create a Storage Policy that just writes compressed backups instead to reduce the load on the DDB.
You could also trial reducing the number of connections to the DDB (Control Panel/Media Management = Maximum number of parallel data transfer operations for deduplication engine)"
These are valid points, but as we made somewhat big change today, I will hold on from additional changes until I see the results. Thanks for the hint. I think I will go back and look into this in 2 weeks when I'll get fresh 3 & 14 days statistics for upgraded DDB.
This is all very helpful. Thank you.