Significantly slower after v11 upgrade

Last post 04-11-2017, 2:24 PM by seabird. 17 replies.
Sort Posts: Previous Next
  • Significantly slower after v11 upgrade
    Posted: 10-03-2016, 10:38 AM

    Several weeks ago we upgraded from v10 to v11.  Today we're on v11 sp5.

    Our backups run significantly longer after the upgrade.  We have always run our weekly synthetic fulls on Friday night, around midnight.  And 90% of them were done in 24 to 28 hours.  Now many clients are taking several days, with a handful taking close to a week.  

    Looking at the forums, it doesn't look like anyone else is experiencing this same problem.  Any idea on what's going on here?

  • Re: Significantly slower after v11 upgrade
    Posted: 10-03-2016, 10:44 AM

    John,

    can you please open a ticket with support we would look into what is slowing down with the synthetic fulls.

    Regards,

    Prasad.

  • Re: Significantly slower after v11 upgrade
    Posted: 10-03-2016, 10:52 AM

    solr:

    John,

    can you please open a ticket with support we would look into what is slowing down with the synthetic fulls.

    Regards,

    Prasad.

    Done.

  • Re: Significantly slower after v11 upgrade
    Posted: 10-06-2016, 3:22 PM

    John,

    This happened to us as well. We went ahead and sealed in order to get on the latest version of DDB, however this is probably something that should be resolved without a seal.

  • Re: Significantly slower after v11 upgrade
    Posted: 10-08-2016, 12:37 AM

    I seem to have similar issue.

    check if there is a strange activity in /var/log/simpana/LogFiles/cvlaunch.log

    i have open ticket with CV

  • Re: Significantly slower after v11 upgrade
    Posted: 10-17-2016, 11:16 AM

    I still have the case open.  They didn't have an explanation for this.

    Our SE suggested we run one cycle of regular Fulls, and then return to the Syn-Fulls.  We've now completed that, and no change.  The Syn-Fulls on our file servers, that used to finish in approximately 36 hours are taking a very long time.  I have fulls that were started at midnight Friday (10/14) that are only 26% done as of late morning 10/17.  The Fulls that I ran the previous weekend would complete in about 36-40 hours.  So I'm going to have to change all my file server backups to Fulls.  I can't afford to let a Syn-Full run for 1 week. 

    I don't know what the explanation is but, there is a significant difference in the "average throughput" when I look at the backup job details.  The syn-fulls run around 20-25 gb/hr, and the regulart fulls run at 50 to 80 gb/hr.

  • Re: Significantly slower after v11 upgrade
    Posted: 10-17-2016, 11:40 AM

    Hi John:

    Can you send us the CV ticket number?

     

    Thanks,

    Prakash

  • Re: Significantly slower after v11 upgrade
    Posted: 10-17-2016, 11:40 AM

    May I know the ticket #, we will take a look.  

  • Re: Significantly slower after v11 upgrade
    Posted: 10-17-2016, 11:46 AM

    praksah:

    Hi John:

    Can you send us the CV ticket number?

     

    Thanks,

    Prakash

    Incident: 161003-227

  • Re: Significantly slower after v11 upgrade
    Posted: 10-17-2016, 11:48 AM

    Pavan Bedadala:

    May I know the ticket #, we will take a look.  

    Incident: 161003-227

  • Re: Significantly slower after v11 upgrade
    Posted: 10-21-2016, 5:41 PM

    Did you get a resolution on this? I just upgraded today and ran a test incremental backup which did seem to take longer than I expected.  It was a small one though so that may not be a problem.  The synthetic full I ran after that was not slower than normal, though.

  • Re: Significantly slower after v11 upgrade
    Posted: 10-28-2016, 8:40 AM

    It's still an ongoing support case.  It was noticed that the CVD service on the MA was starting and stopping frequently during backups.  We do our Fulls (or Syn-Fulls) over the weekend, and I'm going to try some of their suggestions.  Will update this thread next week.

  • Re: Significantly slower after v11 upgrade
    Posted: 10-31-2016, 4:42 PM

    also 161019-478

  • Re: Significantly slower after v11 upgrade
    Posted: 11-29-2016, 3:35 PM

    This turned out to be the DDB volume.  Not enough IOPs.

    We replaced the 15k SAS drives with SSD drives.

    For one particular file server, this was the backup time results, for a Synthetic Full:

    - prior to v11 upgrade:  ~19 hours

    - after v11 upgrade:  110 hours

    - after installing SSD drives:  3.5 hours

  • Re: Significantly slower after v11 upgrade
    Posted: 03-20-2017, 2:48 PM

    We are experiencing extremely slow synth fulls after upgrading to v11 as well but it is not a DDB IO bottleneck. When the synth fulls are running (several in parallel just as when on v10) the IO wait is 0, the load on the MA is 0, there is simply no blocks moving anywhere, or very little, just trickling here and there. I do have a case open for over a month now but it seems to be stuck. I find it hard to believe that things changed that much that our SSDs are not adequate for the identical sized workload that existed prior to the upgrade, but I will double check this weekend when they are running.

     

    Also - for those of you who have upgraded to v11, if you have statistics or job info from v10 still accessible, I would advise comparing those durations to post v11 upgrades so if there are more customers out there with this problem dev can be made aware of it.

  • Re: Significantly slower after v11 upgrade
    Posted: 03-28-2017, 5:53 PM

    Just checking if you have latest updates. CV release patch just for us sometime in December, i think SP6 is now wavailable, make sure you have that.

  • Re: Significantly slower after v11 upgrade
    Posted: 03-28-2017, 5:53 PM

    Just checking if you have latest updates. CV release patch just for us sometime in December, i think SP6 is now available, make sure you have that.

  • Re: Significantly slower after v11 upgrade
    Posted: 04-11-2017, 2:24 PM

    It had nothing to do with the latest updates not being applied. This is a setting which was changed at some point (around SP6 presumably) which greatly impaired throughput.

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2017 Commvault | All Rights Reserved. | Legal | Privacy Policy