Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA

Last post 02-10-2014, 9:19 AM by Willie_F. 9 replies.
Sort Posts: Previous Next
  • Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA
    Posted: 01-30-2014, 5:35 AM

    Hello everybody!

    I noticed that some certain virtual machines fail to backup if vCenter instance is used. The problem VMs are hosted by different hosts and all other virtual machines on these hosts are backed up without any errors. Datastore connectivity is also not a reason. Datastore/Host vMotion of problem VMs did not help. There are following errors in vsbkp.log:

    <!-- VM1 -->

    DOWNLOADING VM Config File https://vCenter.domain/folder/VM1/VM1.vmx?dcPath=App-Cluster%231&dsName=Datastore0123%20(RAID5) to VSA Folder\VM1.vmx

    FAILED to download config file https://vCenter.domain/folder/VM1/VM1.vmx?dcPath=App-Cluster%231&dsName=Datastore0123%20(RAID5)->VSA Folder\VM1.vmx, <1><URL Not Found>

    _Downloading VM config file from ESX server [https://host1.domain/folder/VM1/VM1.vmx?dcPath=ha-datacenter&dsName=Datastore0123 (RAID5)] to [VSA Folder\VM1.vmx]

    Creating Snapshot of VM [VM1] Guid […]

    <!-- /VM1 -->

     

    <!-- VM2 -->

    DOWNLOADING VM Config File https://vCenter.domain/folder/VM2/VM2.vmx?dcPath=App-Cluster%232&dsName=Datastore1106%20(RAID5) to C:\VSA Folder\VM2.vmx

    FAILED to download config file https://vCenter.domain/folder/VM2/VM2.vmx?dcPath=App-Cluster%232&dsName=Datastore1106%20(RAID5)->C:\VSA Folder\VM2.vmx, <1><URL Not Found>

    _Downloading VM config file from ESX server [https://host2.domain/folder/VM2/VM2.vmx?dcPath=ha-datacenter&dsName=Datastore1106 (RAID5)] to [C:\VSA Folder\VM2.vmx]

    _Failed to download config file from ESX server [https://host2.domain/folder/VM2/VM2.vmx?dcPath=ha-datacenter&dsName=Datastore1106 (RAID5)]->[C:\VSA Folder\VM2.vmx], <1><Authentication Failed>

    *Downloading renamed VM config file https://vCenter.domain/folder/VM2/VM2.vmx.gxtmp?dcPath=App-Cluster#2&dsName=Datastore1106 (RAID5) to C:\VSA Folder\VM2.vmx

    Creating Snapshot of VM [VM2] Guid […]

    <!-- /VM2 -->

    Host1 has user identical to vCenter backup user, which is used by Simpana. Host2 has not.

    I did not change datastore and VMware cluster names.

    This issue is pretty similar with vmx file locked issue, but there is no ESX 4 in our production and VSA is working on Windows 2008 R2 server. It seems to be an obstacle on the way to Simpana 10, since ESXi instances are deprecated there. Backup of such a VM in 10 fails anyway:

    <!-- VM2 -->

    Downloading VM Config File VM2.vmx to C:\VSA Folder\VM2.vmx

    Failed to submit request https://vCenter.domain/folder/VM2/VM2.vmx?dcPath=App-Cluster%232&dsName=Datastore1106 (RAID5), The remote server returned an error: (404) Not Found.

    Failed to download file C:\VSA Folder\VM2.vmx from https://vCenter.domain/folder/VM2/VM2.vmx?dcPath=App-Cluster%232&dsName=Datastore1106%20(RAID5)

    Failed to Download VMX file [Datastore1106 (RAID5)] VM2/VM2.vmx

    <!-- /VM2 -->

    Any ideas to make it work in 10?

  • Re: Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA
    Posted: 01-30-2014, 11:11 AM

    @Totopolis - Could you share the entire VSBKP.log file with us? if not you can email to me directly at wfarinella@commvault.com


    Client Team Supervisor - Focus on Virtualization
    Follow me on Twitter - @wfarinella
  • Re: Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA
    Posted: 01-30-2014, 11:51 AM

    Hello Willie_F!

    Logs are with Commvault, the incident number is 140128-197. Are you able to view them? If not i'll send it to you.

     

    Is there any way to force simpana 10 to download renamed files in case of fail?

  • Re: Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA
    Posted: 02-03-2014, 4:22 AM

    Restore such a VM fails with Error Code: [91:41] Description: Unable to upload the VMX file for restore of Virtual Machine.

  • Re: Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA
    Posted: 02-03-2014, 10:33 AM

    @Totopolis - Could you please try and refresh the datacenters in the Commvault GUI?

     

    Expand the machine that has the Virtual Server iDA installed > Right click on the Virtual Server > All Tasks > Refresh Datacenters.

     

    Please make sure the role that you are using for the backups has rights at the vCenter Level. not just the datacenter level. VMware changed the premissions requirements with the 5.1 and 5.5 VDDK.

     

    Also make sure that the user if part of a domain please use the domain\user in the crendentials area.


    Client Team Supervisor - Focus on Virtualization
    Follow me on Twitter - @wfarinella
  • Re: Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA
    Posted: 02-03-2014, 11:45 AM

    I tried to backup these machines with two different commserves, 9 and 10, so it is not VM discovery issue.

    All the roles are configured corresponding to commvault documentation, so the backup user has rights at the vCenter level.

    Willie, could you tell me please if there are any undocumented specific requirements/restrictions for the following VMware objects:

    1. Datastore name
    2. Cluster name
    3. Virtual machine name
    For example, there is restriction for + or @ symbol in datastore name.
  • Re: Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA
    Posted: 02-03-2014, 12:11 PM

    That KB means for the environment so that would include all 3.

    1. Datastore name
    2. Cluster name
    3. Virtual machine name

    I will make sure that BOL gets updated with this in the what gets backed up and what does not. that link is located below.

    http://docs.commvault.com/commvault/v10/article?p=products/vs_vmware/backup_basic.htm

     

     


    Client Team Supervisor - Focus on Virtualization
    Follow me on Twitter - @wfarinella
  • Re: Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA
    Posted: 02-07-2014, 1:37 AM

    That was symbols () in datastore name and # in datacenter name what prevents these virtual machines to backup. Only machines with short name were failing to backup, for example "vTA", "st2000", "st1", "st2", "bd5" etc. Weird...

    I think it should be added to troubleshooting, because support was not able to resolve this issue.

  • Re: Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA
    Posted: 02-09-2014, 6:02 AM

    Hi Totopolis,

    Appreciate the update and feedback. Correct - special characters in the datastore or VM names may be interpreted as special HTTP escape codes and cause errors.

    The funny thing is, VMware 4.1 and below actually had the smarts to fix the URL provided and prevent this from occuring. For whatever reason, VMware 5.0+ removes that safety and passed the literal URL which causes all sort of havoc.

    I thought V10 had some code to work around this anomonly. I'll chase that up.

  • Re: Certain VMs failed to backup with 91:29 error vCenter 5.0.0 / ESXi 5.0.0 / CV9 / 2008 R2 VSA
    Posted: 02-10-2014, 9:19 AM

    Totopolis:

    That was symbols () in datastore name and # in datacenter name what prevents these virtual machines to backup. Only machines with short name were failing to backup, for example "vTA", "st2000", "st1", "st2", "bd5" etc. Weird...

    I think it should be added to troubleshooting, because support was not able to resolve this issue.

     

    I confirmed this with Development and the only other character was the # sign that would provide a problem and is being added to the BOL content of what will not get backed up.

     

    The () will not be an issue and was tested by development.


    Client Team Supervisor - Focus on Virtualization
    Follow me on Twitter - @wfarinella
The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy