De-duplication database Q & I time reached to 100%

Last post 10-14-2019, 12:39 AM by Panks20. 5 replies.
Sort Posts: Previous Next
  • De-duplication database Q & I time reached to 100%
    Posted: 10-12-2019, 10:39 PM

    Hello,

    we have one Media agent in our environment fob backup complete environment with one partition. SSD disk is provisioned from Storage array and tested with IO meter.

    since last week we are seeing very hight Q & I time and it is now reached to more than 100 %. 

    increased the DAT_MEMORY and IDX_MEMORY to 8 GB in ctsrvr.cfg to improve the DDB performance.

    total number of unique block count is 712,318,536 and secondary block 1,169,412,898 in past 24hrs

     

    Here is ddb statistics as of now:

    Total application size 379 tb

    total data size on disk 48

    baseline size 18tb (application size 142tb)

    However, backups are still running on moderate throughput. we just brought one big Oracle database for backup up whose size is approx 11TB and application size is 20tb. The above figure drastically changed within a week. please assist shall we configue the ddb in partition mode on same Media agent or any other solution to get back things normal as I have seen commvault article that if this Q&I will continue with same stats for 14 days it addition of new subclient would not be possbile with this policy.

  • Re: De-duplication database Q & I time reached to 100%
    Posted: 10-13-2019, 4:22 AM

    Hi Panks20

    First off can we please confirm whether you are using Transactional DDB (or traditional DDB)

    A recommendation when seeing high Q&I times for DDB is to move to dedicated SSD (preferably local to the MediaAgent Server), as you have mentioned that you are using volume provisioned from Storage.

    The main reason for potentially high Q&I times is response time or latency when communicating to the DDB volume. 

    Here are various bottleneck in this type of configuration:

    • High Storage Process Utilization
    • High Queue length on SFP/Network at Storage Layer 
    • High utilization of Storage Cache - in most cases Storage Processor will cache the incoming request to Storage Process Cache before committing to disk, and if there are high-load at Storage Processor level this could cause IO commit to be slow, causing spike in Q&I times. 
    • Slowness on Network/Fibre Channel Fabric (depending on whether you are using iSCSI or FC)
    • Slowness at HBA level on MediaAgent Server. 
    Because of the above possible bottleneck, Q&I times could be impacted.
     
    However if you configure local SSD this will bypass any potential environmental bottlenecks.
     
    Regards
     
    Winston 
  • Re: De-duplication database Q & I time reached to 100%
    Posted: 10-13-2019, 6:40 PM

    Hi Panks20,

    During your backup window, if you open Windows Resource Monitor and click on the Disk tab and filter on SIDB2 process. Under Disk Activity you can see the Respone Time (milliseconds) to check whether there is latency from SAN.

    Are you able to share the results from the IOMeter test?  With the IOMeter test you will need to ensure the test is larger than any cache on the storage as the cache may skew results.  

     

  • Re: De-duplication database Q & I time reached to 100%
    Posted: 10-13-2019, 11:36 PM
    Hi Watson, I am attaching both io meter stats and diskperf report. Everything seems fine.
  • Re: De-duplication database Q & I time reached to 100%
    Posted: 10-13-2019, 11:53 PM
    From the test you got 1842 IOPS on 1 worker thread.  The test wrote 6GB data.
    We recommend running IOMeter with X+500GB (X is the amount of front end cache in your storage array).
    Another thing we can check is the number of connections to the DDB when you are seeing poor Query and Insert times and possibly implementing a pruning window during your backup window.
    Could you please create a support ticket and we will check this in further detail.
  • Re: De-duplication database Q & I time reached to 100%
    Posted: 10-14-2019, 12:39 AM
    Hi Watson, Number of connect is around 90. Yes, raised a support incident to look further. Rgds, Pankaj
The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy