Best strategy to deal with numerous CommVault failures

Last post 06-28-2019, 1:34 PM by Ali. 21 replies.
Sort Posts: Previous Next
  • Best strategy to deal with numerous CommVault failures
    Posted: 06-21-2019, 1:23 PM

    A quick question for everyone in the forum:  What do you think is the best way to deal with numerous job failues within CommVault?  I'm talking Aux Copys, VMware backups, Windows file system backups, Email content indexing... not every job is failing every time but there are failures throughout my CommVault installation.  I've got two tickets open already (one for the VMware backups failing and one for the Aux Copy failures) but I wonder if there's a better way to deal with this.

    Any suggestions?

    Ken

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-21-2019, 1:32 PM
    • Aplynx is not online. Last active: 07-17-2019, 2:54 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,642

    What's happening that is causing so many failures? CommVault Support has different focus teams depending on the issue which is why mulitple tickets may be needed. Is there an overarching problem? 

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-21-2019, 3:51 PM

    Re:  Is there an overarching problem?

    Excellent question.  I don't know the answer at this time so I guess I'll keep opening tickets one-by-one.  I was just wondering if there's a general technical support team that would be able to look at my environment wholeistically to see if there's a root cause between all these different issues.

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-22-2019, 9:38 PM
    • Ali is not online. Last active: 07-03-2019, 12:32 PM Ali
    • Top 10 Contributor
    • Joined on 08-05-2010

    Hi Ken, could you PM your incident numbers if you don't mind, we can follow up internally and come up with an approach to look at the enviornment holistically. 

    As you're saying there could be some central underlying issue where many of symptoms (if thats what they are) could be stemming from.

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-24-2019, 12:51 PM

    The tickets I have open now are:

    • 190618-707VMware backups fail
    • 190621-390Aux Copy jobs fail
  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-26-2019, 4:56 AM

    I wish I only had 2 outstanding tickets @ commvault :)

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-26-2019, 10:18 AM

    I've found Commvault is good at finding issues in the environment and often gets blamed for problems that aren't its fault.

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-26-2019, 10:25 AM
    • Aplynx is not online. Last active: 07-17-2019, 2:54 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,642

    I see the Aux Copy call is closed and was a communication issue with the DR side.

    The VSA call is still open, aside from the request to create a stand alone proxy, these KBs should be helpful. 

     

    https://ma.commvault.com/Article/Details/54513

     

    https://ma.commvault.com/Article/Details/52966

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-26-2019, 11:20 AM

    Thanks for the links.  I closed the first Aux Copy ticket when I learned that my sysadmin had applied Microsoft patches and rebooted the servers during the day.  A reboot of the media agent at the DR site cleared the problem.

    Unfortunately the Aux Copy problem has reoccurred - this time without any obvious cause.  I've opened a new ticket for this issue:  190626-479.

    Ken

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-26-2019, 11:24 AM

    LOL.  Thanks, I'm glad I'm not the olnly one.

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-26-2019, 11:24 AM

    LOL.  Thanks, I'm glad I'm not the only one.

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-26-2019, 11:39 AM

    KRobnett:

    I've found Commvault is good at finding issues in the environment and often gets blamed for problems that aren't its fault.

    Agreed.  Although it is frustrating that backups can run cleanly for months, or even years, and then encounter multiple failures.  Right now I'm dealing with an issue where VMware backups, backups that have been running for years, have started failing with a <servername>.vmdk Failed to open disk error.  Incremental backups during the week that used to take less than 6 hours are now taking 20 but I'm hoping that if we find the cause of the Failed to open disk issue, that this performance problem will disappear as well.

    Also, my Aux Copies have started breaking with disturbing messages like:

        Error occurred while processing chunk [8590150] in media [V_433406]...

    <Edit>

    Plus some GDDB_Backup_IndexServer and DDBBackup jobs hang and never finish.

    Plus Email content indexing jobs hang and never finish.

    Plus all backups of servers at my DR site with the CommVault client installed failed.

    Plus backups of the VMware host at the DR site failed so not a single virtual server was backed up last night.

    Plus multiple jobs with the agent "Big Data Apps / classicIndexInstance" (whatever that is) never finish.

    Given all these problems I thouight it might be better to have someone look at everything instead of taking them one-by-one.  Hence this forum post.

    <End edit>

    It's all just so weird.

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-26-2019, 4:36 PM

    ...and one more issue:  My request to send log files to CommVault support has been running for 5.5 hours.  The properties of the job show it fails to collect logs from some servers and the Events list shows "Failed to upload log files to web server".

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-27-2019, 7:31 AM

    In case you are running send logs job on many clients & commserver, you may try in chunks. Since one big send logs bundle is created and tried to upload to HTTP server. Even check the connectivity to the HTTP server.

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-27-2019, 7:32 AM
    In case you are running send logs job on many clients & commserver, you may try in chunks. Since one big send logs bundle is created and tried to upload to HTTP server. Even check the connectivity to the HTTP server.
  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-27-2019, 9:14 AM

    Take a look at the sendlogfiles.log file.

     

    It should give details at the bottom as to why it had trouble.

     

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-27-2019, 3:50 PM
    • Ali is not online. Last active: 07-03-2019, 12:32 PM Ali
    • Top 10 Contributor
    • Joined on 08-05-2010

    Hi Ken, our Support team is aware of the multiple issues you mentioned and we're going to look at this more holistically as well.

    Any issues please communicate it directly with the Support Engineer or  you can reach me directly as well if you need any additional direction or help...

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-27-2019, 4:54 PM

    Summary of the resolution for historical reference...

    Within CommVault, I have multiple subclients for my VMware host backups.  Each subclient backs up all servers within multiple, specific VMware Resource Groups and is configured to back up multiple virtual machines simultaneously (some subclients do two virtual servers at a time, some do four, and some do six).

    Previously, all subclients were configured to use Transport Mode for VMware=Auto and all backups were using the NBD mode.  After applying the most recent Microsoft and VMware patches three weeks ago, CommVault started trying to do backups using the Hot Add mode.  My CommServer is also my proxy server and is configured with a single (VMware paravirtual) SCSI controller which allows for a maximum of 16 disks.  The Hot Add backups of multiple subclients with multiple servers per subclient and multiple disks per server resulted in many more that 16 disks which resulted in VMware backup failures.

    The solution was to modify each of the subclients to Transport Mode for VMware=NBD.  Since making this change, backups of my virtual server have run both cleanly and faster than when it was attempting to use Hot Add.

    Oddly, all the other problems I was experiencing (Aux Copy running slow or failing, content indexing running slow or failing, etc) have been resolved.  I'm not sure how or why the transport mode for VMware backups would have such far reaching effects nonetheless, changing the transport mode on five subclients has made all the other problems evaporate.

    In answer to my post:  I'm still not sure what the best method is to deal with multiple failures.  I think I got lucky that the first ticket I opened fixed all the issues I was experiencing. 

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-28-2019, 10:52 AM
    • Ali is not online. Last active: 07-03-2019, 12:32 PM Ali
    • Top 10 Contributor
    • Joined on 08-05-2010

    Hi Ken, in future if you need immediate attention on Support based incidents the best place to start is to notify Support Management via this Quick Update form within Maintenance Advantage.

    As soon as you do this for any service concerns, incident updates etc, you should get a call back directly from the Management team within 30 minutes.

    Anything else let us know...

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-28-2019, 11:01 AM

    Ali:

    Hi Ken, in future if you need immediate attention on Support based incidents the best place to start is to notify Support Management via this Quick Update form within Maintenance Advantage.

    As soon as you do this for any service concerns, incident updates etc, you should get a call back directly from the Management team within 30 minutes.

    Anything else let us know...

     

    sorry, but I need to laugh with that 'quick update' statement.

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-28-2019, 11:14 AM

    Ali:

    Hi Ken, in future if you need immediate attention on Support based incidents the best place to start is to notify Support Management via this Quick Update form within Maintenance Advantage.

    As soon as you do this for any service concerns, incident updates etc, you should get a call back directly from the Management team within 30 minutes.

    Anything else let us know...

    Sounds good, thanks Ali.

  • Re: Best strategy to deal with numerous CommVault failures
    Posted: 06-28-2019, 1:34 PM
    • Ali is not online. Last active: 07-03-2019, 12:32 PM Ali
    • Top 10 Contributor
    • Joined on 08-05-2010

    BTW, this is 24x7 round-the-clock coverage no matter which geographic location you may be in.

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy