Dedup DB reconstruction is very slow on Prune Records

Last post 02-07-2020, 9:49 AM by DoctorB. 1 replies.
Sort Posts: Previous Next
  • Dedup DB reconstruction is very slow on Prune Records
    Posted: 02-07-2020, 4:52 AM

    We have had some performance problems with backups (very high Q&I time) and when contacting Commvault support, they suggested that we should do a compact om the GDDB database so we could enable garbage collection. The compact however failed which had as result that both our partitions in the GDDB got corrupt and went offline.

    This happened on 23 january and between then and now, we managed to recover one of the partitions. The other one however is very slow on the Prune Records state. It has been running since 29/01/2020 12:45 and is now at 87 replayed records and has 13 pending records. After multiple contact with Commvault support, the only response we get is: you need to wait untill the job is finished.

    We have made some additional workarounds (disabled dedupe when we still hadn't any partition back and after we got the first partition back, we have allowed jobs to run to the copy with at least 1 partition available) but right now we are running out of storage and it seems that the pruning job is going to take 4 days until it is ready (if it ever gets there because I'm not that sure).

    Multiple escalations have not helped, changing the priority to critical has not helped because the only response we get is: you just have to wait until it is done. I understand that may be the case, but there should be some kind of workaround so we can still have our backups running? We can't free any more storage and sealing the db isn't an option anymore since we don't have enough space. It used to be an option but support thought it better to wait until the job is complete

    Does anyone here have any ideas to survive the weekend? Would be an option to disable the MA which has still the pruning records job running so that all jobs go to the online MA?

    Incident number @Commvault is 200115-309

    Kind regards,

    Timothy

  • Re: Dedup DB reconstruction is very slow on Prune Records
    Posted: 02-07-2020, 9:49 AM

    Hi Timothy,

    High Q&I time can simply be due to low IOPS of the partition where the DDB is hosted, aka hardware not meeting the requirements. Have you tested the number of IOPS using our IO Meter tool?

    https://documentation.commvault.com/commvault/v11/article?p=8825_1.htm

    Also, if you host multiple DDBs on the same partition you can have poor performance. In saying that what is the service pack?

    It appears at this stage, you just need to wait for the reconstruction to finish. From what you said I gather one DDB partition is online, which means you should be able to run your data protection operations as per normal. Using non-dedupe backup will cause your storage to run out of space very quickly.

    If data aging is also fallen behind, then you can increase number of pruning threads on the MA that is online:

    http://documentation.commvault.com/additionalsetting/details?name=DedupPrunerThreadPoolSizeDisk

     

    Regards,

    Bob

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy