DDB Average Q&I Time after v11 upgrade

Last post 11-15-2017, 9:03 AM by AKupi. 22 replies.
Sort Posts: Previous Next
  • DDB Average Q&I Time after v11 upgrade
    Posted: 09-06-2016, 8:44 AM

    Hi,

    I was wondering if anyone has noticed a change in their DDB avergae Q&I time after upgrading to v11.

    We have 4 DDBs and all of them have suffered a major increase in Average Q&I Time (100% to 400% increase). I attached a screen capture showing a graph for the last 180 days.

    Thanks!


  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 10-03-2016, 3:25 PM

    Same here.  And our backups are taking significantly longer.  I opened a support case today.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 10-03-2016, 3:45 PM

    Hi John,

    We're still having the issue, although we were able to lower the average Q&I times by spreading application backups across multiple storage policies, and schedules... 

    I noticed something last week about a time difference (one hour) between "Last DDB Backup Time" and the actual backup time of that DDB's backup. The actual time is 8AM but it's reported 7AM in the partition information of the deduplication engines pane. See screen captures.

    I was wondering if it's the case in your environment... Maybe it's the root cause for the DDB average Q&I times increase.

    Thank you.


  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 10-04-2016, 8:34 AM

    For us, it looks like the reported time and actual are the same.

    Did upgrading to v11 affect how long your backups run?

    Do you use SSD drives for your DDB?

    Here's what's going on:  This is the second time this has happened to us.  The last time, it was a service pack.  After the service pack, our average Q&I times went above the acceptable threshold.  And at the time, that SP included a new feature where, if you did go above a certain threshold, you couldn't add more subclients.  They must of got a lot of flack for that, because in the next sp they reveresed that.  And it was also at that time, that CV started telling their customers that they need to get SSD for their DDB.

    And that's just what the tech said yesterday when I called this in.  What makes this so annoying for us is;  how can you plan, and therefore budget, for a particular type of drives, when the bar randomly gets raised, once again, with a particular SP or version upgrade.  Pre-upgrade, our average Q&I was about 13,000.  After the upgrade, it's 23,400.  That's an 80% increase.  An unexcepted increase.  Our weekly Fulls have gone from taking 24 to 28 hours, to 5-7 days.  We have a serious problem.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 10-05-2016, 5:11 AM

    You do realize that Q&I times of 13000µs (or 23400µs) is just off the charts?

    There used to be a time in Simpana 10 when an avg Q&I time of 1000µs was the threshold for a automatic DDB seal! (luckily that got disabled, quick, it meant auto-full backups, more DDB space, more temp disklibrary storage)

    To put things in perspective. 13000µs is 13ms , so you are doing 1000 ms/13ms =  77   block inserts per second into the DDB.  I don't know what your dedupe block size is , but assuming it's the recommended 128 kB...

    That would mean your DDB in best case is allowing to ingest new data at a rate of  77 * 128 kB = 9.5 MB/sec ...      You said your backups were slow?

     

    I'm running multiple 600+ GB DDB's with over 2 billion primary records on FusionIO SSD PCIe boards. The Q&I inserts have not increased since my Simpana 11 upgrade in June.  Average Q&I insert times are 130µs and constant since I created the DDB's 2 years ago!   (per same calculation, that theoretically allows me to ingest 961 MB/sec of new data).

     

    Renaat

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 10-05-2016, 2:07 PM

    We're local government.  Thank you.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 10-13-2016, 2:58 PM

    What was supports answer? I'm very curious about this as we're upgrading from V10 to V11 in the near future.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 10-13-2016, 3:04 PM

    brianv:

    What was supports answer? I'm very curious about this as we're upgrading from V10 to V11 in the near future.

    For us it's a 2-part issue.  We need SSD drives for our DDB Q&I times.

    The incredibly slow backup times turned out to be something different.  I can't find the link right now, but in the BOL, it mentions that is you typcially always to Synthetic Fulls, you have to periodially do a regular Full, as a "best practice".  Our CommVault Support Engineer brought this up and suggested it after the upgrade.  We've done the Fulls.  Tomorrow night we'll do the SynFulls.  I'm expecting them to be back to normal, or at least very close.  I'll update this Monday when I know more.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 10-13-2016, 3:21 PM

    Support said that we should spread our workload across storage policies, and stagger schedules throughout the weekend to lessen the load on DDBs. Which we did, and it helped, but we're still above the threshold. I think of this as more of a workaround... It's not like we added that much workload in our environnement at the same time as we upgraded to v11...

    We currently have five issues escalated to DEV (or in process of) and four of them are post upgrade issues, so I would wait a bit before upgrading...

    Anyways, I'll keep you informed if ever we find a real solution for the DDB Q&I time issue...

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 10-24-2016, 10:53 AM

    Our Q&I value changed from 150 microseconds to 32000. That's what i call upgrade.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 10-28-2016, 2:09 PM

    I have seen synthetic fulls take longer over time if they're very big or if there's a very large count of files, and running an actual full fixed that a couple times for me.  

    Also, moving the DDB to an SSD drive will definitely help for all backups too if you're doing de-duplication.  The DDB is not needed for restores or for writing aux-copies to tape, but it is used for all backups--fulls as well as synthetic fulls and incrementals.

    For my environment, rather than purchase an expensive SSD drive or FusionIO card, I just bought a consumer grade PCI based SSD and it has worked great.  The first one lasted fine for the 2 years before I upgraded the server and my new one has been going great for the last few months.

    My DDB is only about 70GB.  Your mileage may vary, but depending on the size you need you may not need to spend thousands of dollars on this.   Mine is a 240GB model I got for around $150 this year.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 11-29-2016, 3:23 PM

    Totopolis:

    Our Q&I value changed from 150 microseconds to 32000. That's what i call upgrade.

    Do you have SSD drives for your DDB volume, or spinning drives?

    We were around 7000 before the v11 upgrade, and aroundd 23,000 after.

    We installed SSD drives for the DDB last week.  Our Q&I time is now 300 microseconds.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 11-29-2016, 3:27 PM

    Totopolis:

    Our Q&I value changed from 150 microseconds to 32000. That's what i call upgrade.

    Do you have SSD drives for your DDB volume, or spinning drives?

    We were around 7000 before the v11 upgrade, and aroundd 23,000 after.

    We installed SSD drives for the DDB last week.  Our Q&I time is now 300 microseconds.

    But they deny that the upgrade caused those crazy Q&I time increases.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 11-29-2016, 3:27 PM

    Totopolis:

    Our Q&I value changed from 150 microseconds to 32000. That's what i call upgrade.

    Do you have SSD drives for your DDB volume, or spinning drives?

    We were around 7000 before the v11 upgrade, and aroundd 23,000 after.

    We installed SSD drives for the DDB last week.  Our Q&I time is now 300 microseconds.

    But they deny that the upgrade caused those crazy Q&I time increases.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 12-07-2016, 8:36 AM

    We had to replace our SAS spinning drives with SSD to reduce our Q&I. It's about 200-400 microseconds now.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 12-16-2016, 10:21 AM

    We were using 15K SAS Raid with v10. The Q&I jumped a little after v11 upgrade. We migrated to low cost "consumer" grade sata ssd's in a raid. They worked great for about a year and then we started seeing severe degredation in Q&I. We found that the drives only did housekeeping when idle and they were almost always busy so they never did garbage collection. We transfered the ddb's to Intel PCIe cards and the Q&I have returned to 80-200us. We are seeing change blocks in the high hundred millions. 

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 12-20-2016, 5:37 AM

    I didnt saw that issue on my upgrades (it could be since we always recommend using SSD for the DDB, regarding the size of the customer and I might just missed it?) but I do wonder - Did anyone tried to upgrade to the new transactional DDB?

    http://documentation.commvault.com/commvault/v11/article?p=features/deduplication/t_transactional_ddb.htm

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 04-14-2017, 1:18 PM

    Has anyone had any updates to this? 

    We updated an environment from v10SP13 To v11SP6 a couple weeks ago. Since then the DDB's have been getting CRUSHED!

    DDB's are still v10, I need to solve this issue before adding more changes.

    All DDB's are on enterprise grade SSD internal to the MA's. 

    We have 5 DDB's each with 2 partitions, spread across 6 MA's. Pre v11 we were  in the 400 µs - 650 µs range on most of them. Since v11 we are hitting 3,569 µs on them regularly! We haven't increased the number of clients or the data protected, just the Commvault version. 

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 04-25-2017, 6:06 PM

    MarcC:

    Has anyone had any updates to this? 

    We updated an environment from v10SP13 To v11SP6 a couple weeks ago. Since then the DDB's have been getting CRUSHED!

    DDB's are still v10, I need to solve this issue before adding more changes.

    All DDB's are on enterprise grade SSD internal to the MA's. 

    We have 5 DDB's each with 2 partitions, spread across 6 MA's. Pre v11 we were  in the 400 µs - 650 µs range on most of them. Since v11 we are hitting 3,569 µs on them regularly! We haven't increased the number of clients or the data protected, just the Commvault version. 

    In my case, after upgrade DDB to V11 Q&I jumps to 40k. During pruning procces clinet connection to DDB simply timeout. Weekly synthetic fulls take 40hrs. 

    Tom Watson:

    We were using 15K SAS Raid with v10. The Q&I jumped a little after v11 upgrade. We migrated to low cost "consumer" grade sata ssd's in a raid. They worked great for about a year and then we started seeing severe degredation in Q&I. We found that the drives only did housekeeping when idle and they were almost always busy so they never did garbage collection. We transfered the ddb's to Intel PCIe cards and the Q&I have returned to 80-200us. We are seeing change blocks in the high hundred millions. 

    Same story here. Can you tell more about hausekeepoing/idle/garbage collection? 

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 07-17-2017, 11:19 AM

    Hi,

    While working on another DDB related issue with our synfulls, for a test, we tried skipping Exchange Mailbox (classic) backups, and the result was that our DDB Q&I time drasticaly dropped. We haven't tried the new agent yet.

    I just wanted to let you know of our findings.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 07-24-2017, 11:00 AM

    I also had this issue after the upgrade, but didnt accept the answers I received from support.. After many calls I was able to work with the development crew to create a new hotfix for this issue.

     

     Notes for the case:

    Hotfix: 1935

     

    High QI Time Reporting Issues

    Post created on Jul 19, 2017

    Customers may experience what looks to be high Q&I (Query & Insert) time due to recent changes in Version 11 regarding sample count. In Version 10 and 11 prior to SP6, the sample count collection was set at 100. Post Service Pack 6 the sample count had changed to 1000. 

     

     

    Applies to:

    Upgrading to V11 

    Installation of Service Pack 6, 7 or 8

     

    Validation:

    Utilize the Q&I reporting in the GUI from the point where either the customer upgraded to V11 or prior to service pack 6 installation to current time. A recognizable spikes in Q&I should be identified

     

     

    Resolution:

    A Hotfix has been provided for the following service packs to change the sample count back to 100 


    V9, +4000 clients, +100 MA
  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 07-29-2017, 4:07 PM

    Well said, 

     

    We had the same issue and we struggled for morethan 3 months, FInally it solved with latest V11 SP8 and all latest hostfixes.

  • Re: DDB Average Q&I Time after v11 upgrade
    Posted: 11-15-2017, 9:03 AM

    We upgraded last week from V10 SP15 to V11 SP9+. Same issue where backups after the upgrade slowed down significantly.

    Reached out to CV support, applied all hotfixes, and made some config changes - nothing helped.

    Working for CV Development, we rolled back one of the DDB from transactional db to in memory db (as part of upgrade to V11, CV Professional Services upgraded all DDB to transactional db). After changing the DDB to V10 format, backup speeds returned to normal.

    We are working with Development to roll back all the remaining CVMA ddb.

     

    Hope this helps.

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy