VMware - Unable to quiesce guest file system during snapshot creation

Last post 12-22-2016, 11:04 AM by Bartlett_Nate. 20 replies.
Sort Posts: Previous Next
  • VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 11-13-2015, 10:00 AM

    Hi all. I recently pushed out SP12 to remote sites and began receiving an error stating that CommVault was unable to quiesce guest file system during snapshot creation. Everything was working fine on SP10 and SP11. I also noticed that the Virtual Disk Service in Windows 2008 R2 and Windows 2012 R2 is stopped when this happens.

    Has anyone else experienced this? I have one site where all of their VM's are failing  where another site only 1 VM is failing.

    vSphere error report:

    Type: Warning

    Description: Warning message on <VM> on <Host> in <Datacenter>: The guest OS has reported an error during quiescing.

    The error code was: 5
    The error message was: 'VssSyncStart' operation failed: IDispatch error #8449 (0x80042301)

    I have a ticket open with support.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 11-13-2015, 1:57 PM

    This is a change in SP12 (line 290: List of Updates in Service Pack (SP12)).

    To disable these warnings, you can configure an additional setting:

    ignoreQuiesceGuestErrors
    Description (VMware only) To ignore quiesce failure errors for virtual machine backups, create this additional setting on the Virtual Server Agent proxy and set the value to 1.
    Category VirtualServer
    Type Integer
    Windows Packages Default,Virtual Server Agent
    Min Value 0
    Max Value 1
    Default Value 0
    Supported Version 10.0
  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 11-16-2015, 3:30 PM

    Thank you! We will see if this fixes my issue with this.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 11-17-2015, 9:37 AM

    Is this something we necessarily want to ignore?  Won't there be issues restoring say a SQL server?  I did not get these errors in vCenter prior to SP12.  Something must have changed.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 11-17-2015, 9:47 AM

    I don't think ignoring an error like this should be the fix. I don't recall seeing these error messages in vSphere prior to upgrading to SP12. SP12 may not be the overall cause of what's happening, but now I can't even take a snapshot in VMware with the queisce option turned on.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 11-17-2015, 11:40 AM

    You are correct. If the backup is not able to quiesce the VM, the data may be backed up in a crash consistent state instead of an application consistent state. Although many applications will be able to recover successfully if the VM is restored (just as they would if the VM crashed and rebooted), this may affect the ability to recover certain application data in a usable form (particularly for applications which do not use transaction logs) or up to the point of the most recent transaction. Thus, the proper way to resolve this error is to fix the issue with quiescing the VM.

    The additional setting only needs to be used if you wish to suppress these errors. Some customers are not able or do not wish to perform this in depth troubleshooting. Thus, in lieu of or until you can fix the underlying VM quiescing errors, the additional setting is available to use at your discretion.

    The change in SP12 was to report these quiescing errors (previously they were not reported and the failure to quiesce was a silent error unless you dug in to the logs or looked in vSphere). The additional setting returns Commvault to its prior (pre-SP12) behavior and no error is reported. There should be no change when taking a snapshot outside of Commvault after applying SP12.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 11-17-2015, 11:50 AM

    Thanks jjbiv.  I looked back and can see these were happening in the past.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 11-17-2015, 12:11 PM

    My pleasure. I hope this information helps. Let us know if you would like assistance with the VM quiesce failures. Usually updating VMware Tools (to match the version of ESXi you are using) and fixing any guest VM VSS errors will take care of this (sometimes rebooting the VM is needed to fix VSS). With VMs that are particularly stubborn, very busy, very large, or very important, installing an agent inside the VM and running streaming application backups is always an option, too.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 02-14-2016, 6:53 AM

    Hi

    Please verify the type of disk mounted to the virtual machines as certain disk types are not supported.

    What Does Not Get Backed Up

    The following entities will not be included in the default subclient:

    Virtual machines containing SCSI bus sharing set to virtual
    Virtual machines configured with Fault Tolerance
    Virtual machines with no disks attached
    Physical RDMs
    Independent Disks
    http://documentation.commvault.com/commvault/v10/article?p=products/vs_vmware/backup_basic.htm
    Also, VSS is not able to quiesce Dynamic Disk as they are not supported by VSS.
     
    Kind Regards
    Nick Morkot
     
  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 04-13-2016, 4:59 AM

    We are having similair messages about "Failed Unable to quiesce guest file system during snapshot creation" strange thing is that we see that the VMware tools on the particulair VM are running, created a backup with Commvault it's changed to not running. Within VMware there is a error "An error occurred while saving the snapshot msg.snapshot.eror-QUIESCINGERROR". We logged on to the VM and stopped the VMware tools which will hang, killed process and started VMware tools. VMware tools is displayed as running again. Then we performed a snapshot within VMware with quiesce option which is able to complete successful without stopping the VMware tools! Is any one else having these problems?
    (VM has been rebooted, VSS writers status is OK, even deinstalled/installed VMware tools).

    We are running Simpana 10 SP13+ and VMWARE/ESX 5.5 with the latest version of VMWARE tools. Strange thing is some VM's are not having these problems or having the same error message but not causing the crash on the VMware tools.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 05-19-2016, 3:26 PM

    dhg, Have you found a solution?

    We are seeing this exact issue. It has been pretty widespread and random.

    Windows 2008 R2 VMs behave exactly as you describe. Windows 2012 R2 VMs are more severely affected.

    It has been impactful enough that we had to globally disable "Quiesce Guest File System and Applications". Absolutely 0 issues since that was disabled, but backups aren't as usable with that turned off.

    We opened tickets with VMware, CommVault, and Microsoft and haven't made much progress. With Windows Server 2012 R2 VMs, they would actually hang in a state that was unresponsive to user requests, but may respond to a ping. We started monitoring for VMware Tools service state, and every morning some VMs would be reported as having VMware Tools as stopped (crashed). Since we disabled Quiesce this has not occurred once.

    We've used multiple versions of VMware Tools (version 10246), ESXi (now on ESXi 6.0.0, 3620759), and Simpana 10 (now SP13+). I'm fairly certain most of the VMs affected are VM Version 8.

    VMware seemed pretty confident that the issue affecting 2008 R2 is different than the issue affecting 2012 R2. They worked with us to try to highlight the 2012 R2 crash for Microsoft to try to debug / troubleshoot but we haven't found the magic combination yet. Any info you could add would be helpful.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 05-20-2016, 10:46 AM

    Hi nj@cjb, just a quick response thanks for the reply. We had a session with our Commvault parnter today but they need further investigation and created extra logging.
    (In our case it's only Windows2012R2).

    If we have any solution or workaround we will inform you in this forum, also if you found a solution please share this with us.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 06-01-2016, 2:41 AM

    Hi dhg, we have also the same issue with vmware esx 5.5 and Server 2008 R2. It's started with the lastet version from vmware tools 10.0.8. When i downgrade vmware tools to 9.4.10 then works everthing fine.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 07-11-2016, 6:32 AM

    NoBody1:

    Hi dhg, we have also the same issue with vmware esx 5.5 and Server 2008 R2. It's started with the lastet version from vmware tools 10.0.8. When i downgrade vmware tools to 9.4.10 then works everthing fine.

     

    there is a new version out from vmware tools with bug fixed.

    Version 10.0.9 is released.
    https://pubs.vmware.com/Release_Notes/en/vmwaretools/1009/vmware-tools-1009-release-notes.html

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 07-14-2016, 9:14 AM

    Hi NoBody1,

    Thanks for your reply, our Commvault partner informed me also about the new VMware tools version. We are planning to deploy the new vmware tools version soon and we hope this indeed solves are problem. Once done we will inform the forum.

    kind regards

    dhg

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 08-23-2016, 10:30 AM

    Hi NoBody1,

    Updated VMWARE tools for one site and all backups completed successful without the errors. We will proceed with updating the other site's soon.

    Greetings

    dhg

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 08-23-2016, 10:34 AM

    VMware resolved for us!

    VMware ESXi, 6.0.0, 3825889
    CommVault 10r2
    VMWare Tools (version:10246)

    Error reads: "Failed Unable to quiesce guest file system during snapshot creation."


    Per the support log:
    1. Opened a console for the VM and found out that virtual disk service and the volume shadow copy service was not started.
    2. Changed the startup type to automatic for virtual disk service.
    3. Started the virtual disk service.
    4. Started the vmware tools and volume shadow copy service.
    5. Post that we were able to successfully perform the quiesce snapshot of the VM.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 10-26-2016, 8:17 PM

    These instructions about setting the services to Automatic seem to be at odds with the VMWare docs below.  Have you had any re-occurrence of the quiesce issues?

    FYI:

    https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1007696

     

    Troubleshooting Steps ...

    • While Idle:

      • Ensure that the COM+ System Application is listed as Started and that the startup type is listed as Manual.
      • Ensure that the COM+ Event System service is listed as Started and that the startup type is listed as Automatic.
      • Ensure that the Volume Shadow Copy service is not running and the startup type is listed as Manual.
      • The Microsoft Software Shadow Copy Provider service may or may not be started. Ensure that the startup type is listed as Manual.
      • Ensure that the VMware Snapshot Provider is not running and that the startup type is listed as Manual.

    • During the backup:

      • Ensure that the COM+ System Application is listed as Started and that the startup type is listed as Manual.
      • Ensure that the COM+ Event System service is listed as Started and that the startup type is listed as Automatic.
      • The Volume Shadow Copy service may or may not be started. Ensure that the startup type is listed as Manual.
      • Ensure that the Microsoft Software Shadow Copy Provider service startup type is listed as Manual.
      • Ensure that the VMware Snapshot Provider is listed as Started and that the startup type is listed as Manual.
      • Ensure that the Virtual Disk is listed as Started and that the startup type is listed as Manual.
    Cheers.
  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 10-27-2016, 1:50 PM

    Frown after upgrade to ESXi6.0U2 and tools upgrade on the VM's, problem back again after 2 weeks...

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 12-22-2016, 10:54 AM

    Has anyone else been experiencing this issue? I found an interesting post on VMware's Forum (https://communities.vmware.com/thread/533736?start=15&tstart=0) that seems to indicate they know of this issue and have put out a fix with version 10.0.9 of VMware Tools but that version seems to have the same issues. I have been running VMware Tools 10.0.x and I am getting the same issue randomly across all my Windows servers 2008 R2 and 2012 R2. If I down grade to an earlier version of VMware Tools 9.3.59 for instance the issue of quiesce of the quest goes away. This does not have anything to do with CommVault because if you manually try to snapshot a guest with the quiesce the VM selected it will error out as well. I have also un-installed and re-installed 10.0.240 and that seems to have fix one of my servers. More are popping up though so I am not sure this is the fix.

  • Re: VMware - Unable to quiesce guest file system during snapshot creation
    Posted: 12-22-2016, 11:04 AM

    We've had this same issue on some of our VM's.   Like you, it just started one day.

    Our's ended up being a combination of a few things:

    1.  If you have any floppy drives on the servers, disable them using Device Manager.   For some reason, they are a problem on the VM's now.

    2. We upgraded our VMware Tools from v 10.0.6 (known bugs) to 10.0.9 per a VMware support agent.   

    3.  If you have any older VSS agents installed, (we had old Backup Exec agents on a couple), you will need to remove them.

    Something did appear to change.   My suspesions lie with a Windows Update, but I don't have anything to prove it.   All I know is when we did the things above, we stopped having the issues.    Battled it for about 3 months though.

     

    Hope this helps

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy