VADP Metadata collection problems

Last post 01-04-2011, 5:04 AM by JayST. 9 replies.
Sort Posts: Previous Next
  • VADP Metadata collection problems
    Posted: 09-17-2010, 9:48 AM

    Hi,

    im having trouble backing up a v7 (vSphere) VM using the disk level option and with the "Metadata collection" turned on.

    The VM is just a simple Windows 2k3 machine. It backs up fine (full and incremental) when the metadata collection is turned off, but when it's turned on, it backs up the vmdks and stops after bringing them over to the proxy disklibrary.

    The vsbkp.log shows "service time...." entries forever. It seems some processes are waiting for eachother, but i cant figure out how to troubleshoot this further.

    I'm using the latest patches i think, including 18759.

    any thoughts?

  • Re: VADP Metadata collection problems
    Posted: 09-17-2010, 10:43 AM

    Can you share any further details on the machine?  Is the data size or structure on the machine extremely deep or hold an extreme amount of files (millions), in which the metadata collection phase may take some additional time to complete?  Are all the drives local VMDKs or are they physica'/virtual RDMs or on NFS datastores?

  • Re: VADP Metadata collection problems
    Posted: 09-20-2010, 9:57 AM

    The VM is just your avarage sized VM. No deep fs structure.

    All drives are local vmdk's.

    It looks like when the metadata collection process is initiated (as seen in the logs), the process does not actually start. It just hangs there.

    Again, full and incremental back-ups without the meta-data collection finish just fine.

  • Re: VADP Metadata collection problems
    Posted: 09-20-2010, 10:40 AM

    does the metadata collection process require the old vcb and/or converter software maybe?

  • Re: VADP Metadata collection problems
    Posted: 09-20-2010, 12:20 PM

    No, VCBmounter nor converter are needed for metadata collection. 

  • Re: VADP Metadata collection problems
    Posted: 09-20-2010, 12:22 PM

    JayST:

    The VM is just your avarage sized VM. No deep fs structure.

    All drives are local vmdk's.

    It looks like when the metadata collection process is initiated (as seen in the logs), the process does not actually start. It just hangs there.

    Again, full and incremental back-ups without the meta-data collection finish just fine.

    Does the VM's system event logs show any errors?

     

  • Re: VADP Metadata collection problems
    Posted: 09-21-2010, 5:28 AM

    no errors in the eventviewer (application or system). Only an informational event in the system eventlog with the messages that the Virtual Disk Service has been started (eventid 3). This was displayed at 11:24.

    The log below is from the vsbkp.log. As you can see it backsup the vmdk perfectly, and then starts the metadata collection. It just hangs there, no activity, just "service time" lines being added.

    =============

    After some time, the cvd.log shows


    5128 1ad8 09/21 11:25:33 3386 SdtTailServer::MainLoop: Client [SDTPipe_cvserver01_cvserver01_3386_1285060364_6196_2104], Id [1664] has not recvd network message in [384] seconds. Selected [true], Last message was recvd at [Tue Sep 21 11:19:09 2010] RCId [6365]

    ==============

    6196 fac  09/21 11:19:09 3386 vsbkp::backup() - Done Backing up disk/volume [SERVER01.vmdk]
    6196 fac  09/21 11:19:09 3386 vsbkp::backup() - Saving change Id [52 b0 2f 71 4f c5 fe 19-49 d0 01 28 03 cf 7d 08/187]
    6196 fac  09/21 11:19:09 3386 vsbkp::WriteChangeIDFile() - Creating change id file C:\Program Files\CommVault\Simpana\iDataAgent\JobResults\CV_JobResults\iDataAgent\VirtualServerAgent\2\39\vsidaCHGID_3386_(4218bc60-2696-2cec-5b1e-04cf4d08e926)_(SERVER01.vmdk)
    6196 fac  09/21 11:19:09 3386 vsbkp::backup() - Metadata collection completed...
    6196 fac  09/21 11:19:09 3386 vsBkpDiskConfig::addDiskForLVMProcessing() - Product Type   [1]
    6196 fac  09/21 11:19:09 3386 vsBkpDiskConfig::addDiskForLVMProcessing() - Disk type      [18]
    6196 fac  09/21 11:19:09 3386 vsBkpDiskConfig::addDiskForLVMProcessing() - file disk path [SERVER01.vmdk]
    6196 fac  09/21 11:19:09 3386 vsBkpDiskConfig::addDiskForLVMProcessing() - index dir name [SERVER01.vmdk]
    6196 fac  09/21 11:19:09 3386 vsBkpDiskConfig::addDiskForLVMProcessing() - extent size    [128]
    6196 fac  09/21 11:19:09 3386 vsbkp::backup() - Backup done for volume/disk [SERVER01.vmdk], number of extents backed up [124399]
    6196 fac  09/21 11:19:09 3386 ID=readCRC, Bytes Read = 0, Total time = 0.000000, Average = 0.000000, Samples = 0
    6196 fac  09/21 11:19:09 3386 ID=writePLBuffer, Bytes Read = 7976452096, Total time = 5.030684, Average = 1512.108154, Samples = 243422
    6196 fac  09/21 11:19:09 3386 ID=allocPLBuffer, Bytes Read = 15431007424, Total time = 15.732807, Average = 935.380127, Samples = 243422
    6196 fac  09/21 11:19:09 3386 ID=writeCRC, Bytes Read = 0, Total time = 0.000000, Average = 0.000000, Samples = 0
    6196 fac  09/21 11:19:09 3386 ID=calcCRC, Bytes Read = 0, Total time = 0.000000, Average = 0.000000, Samples = 0
    6196 fac  09/21 11:19:09 3386 ID=readdisk, Bytes Read = 8152678813, Total time = 169.602356, Average = 45.842525, Samples = 124401
    6196 fac  09/21 11:19:09 3386 vsbkp::run() - Metadata Collection enabled for VM[SERVER01]
    6196 fac  09/21 11:19:10 3386 There are 4 entries in the allocated extent table
    6196 fac  09/21 11:19:10 3386 vsBkpDiskConfig::doLVMProcessing() - LVM disk metadata processing completed succesfully. Now doing Volume metadata processing
    6196 fac  09/21 11:19:10 3386 vsBkpDiskConfig::doLVMProcessing() - Dumping volume information for [Volume1]
    6196 fac  09/21 11:19:17 3386 vsBkpConfig::collectVolumeMetadata() - Dumping NTFS metadata...
    6196 fac  09/21 11:19:17 3386 clImageTableOpen() - Table was created and opened. (C:\Program Files\CommVault\Simpana\iDataAgent\JobResults\CV_JobResults\iDataAgent\VirtualServerAgent\2\39\imageLevelTree(4218bc60-2696-2cec-5b1e-04cf4d08e926_Volume1))
    6196 fac  09/21 11:19:17 3386 clImageTableOpen() - Indexing was suppressed for the table (C:\Program Files\CommVault\Simpana\iDataAgent\JobResults\CV_JobResults\iDataAgent\VirtualServerAgent\2\39\imageLevelTree(4218bc60-2696-2cec-5b1e-04cf4d08e926_Volume1))
    6196 fac  09/21 11:19:17 3386 clImageTableOpen() - Returning Table id = 30637D0 (C:\Program Files\CommVault\Simpana\iDataAgent\JobResults\CV_JobResults\iDataAgent\VirtualServerAgent\2\39\imageLevelTree(4218bc60-2696-2cec-5b1e-04cf4d08e926_Volume1))
    6196 5 09/21 11:23:04 ### ### Timer Callback --- Service Time 9/21/2010 9:23:04 AM
    6196 5 09/21 11:28:04 ### ### Timer Callback --- Service Time 9/21/2010 9:28:04 AM

  • Re: VADP Metadata collection problems
    Posted: 09-21-2010, 8:26 AM

    Can you check the following directory to see if any of the files are updating?

    C:\Program Files\CommVault\Simpana\iDataAgent\JobResults\CV_JobResults\iDataAgent\VirtualServerAgent\2\39

  • Re: VADP Metadata collection problems
    Posted: 10-19-2010, 5:04 AM

    sorry for not replying sooner.

    Those directories get created and are filled up with files during the process. When it starts the meta data collection is stalls at some point.
    The latest files being created are:

    VMtoContentMap.dat
    VMtoContentMap.idx
    vsidaCHGID_4062_(4218372b-fae0-4d39-0729-6945591c5325)_(VM01_1.vmdk)
    DiskVolumes(4218372b-fae0-4d39-0729-6945591c5325).imf
    imageLevelTree(4218372b-fae0-4d39-0729-6945591c5325_Volume1).idx
    VolInfo(4218372b-fae0-4d39-0729-6945591c5325_Volume1).imf
    AFlist(4218372b-fae0-4d39-0729-6945591c5325_Volume1).imf
    imageLevelTree(4218372b-fae0-4d39-0729-6945591c5325_Volume1).dat

    The files are not being updated after the process stalls

     

    When looking at those folders, i notice there are some old files in there as well. Files like:

    vsidaCHGID_4019_(4218372b-fae0-4d39-0729-6945591c5325)_(VM01_1.vmdk)

    and

    vsidaEXTSZ_4019_(4218372b-fae0-4d39-0729-6945591c5325)_(VM01_1.vmdk)

    They go a month back in creation/update time. What are those files for?

     

  • Re: VADP Metadata collection problems
    Posted: 01-04-2011, 5:04 AM

    i got this figured out.

    We use EMC PowerPath as multipathing software.

    When i put 3 of the 4 available paths to each VMFS LUN on standby instead of all 4 on active, it all worked flawless. No delays in the metadata collection. So it looks like it's important to have only one active path to each VMFS LUN (at least, in our case).

    Seems like a vddk/vadp issue in multipath environments (maybe restricted to powerpath, but i'm not sure about that).

     

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy