Seeding Storage Policy troubles

Last post 03-04-2015, 2:49 PM by JamesMacDonald. 20 replies.
Sort Posts: Previous Next
  • Seeding Storage Policy troubles
    Posted: 01-20-2015, 3:11 PM

    Hi all,

    I am attempting to set up a storage policy seeding process in prepartion for DASH AUX Copies back here. I have been really struggling with this whole Aux copy business with Commvault and i am not sure how i am going to accomplish this one. All of the issues i am having are on the remote site configuration

    So here is the environment - Simpanan 10 SP8

    Remote site - ESX 5 farm loaded on 4 Cisco UCS Blades

    Remote Site - Media agent is installed on an guest VM. All Vm's and physical machines back up to it.

    I have a USB harddrive, however it cannot be hooked up to one of the Cisco hosts due to the fact the the USB drive does not have an external power adaptor, it relies on power from the USB port to run it, which is a problem because the Cisco blades had a connector that does not allow this to occur.

    So i have the USB drive connected to a separate physical host at the remote site, and i was able to set up a shared library to it from the Remote Site media agent and the main Datacenter site media agent using a UNC path to the drive.

    The next step is to create the secondary copy to USB. So i go to the remote site storage policy and click on create new copy. it lets me select the remote site media agent (a VM) and the USB drive, but when i click on enable deduplication, and go to the deduplication tab, under DDB Access path, i clcik on Configure, reduce the paths to 1 and clcik on Choose path, i cannot select the USB drive at all.

    I have tried mapping a drive on the Remote media agent to the USB drive on the physcial host, it did not work.

    I dont have any other options here, i cannot get a AC powered USB drive to the site. i dont dare just start an aux copy without seeding, the pipe is not that big, and i do not know what kind of stress it will put on the the line/environment.

    any suggestions would be appreciated. if any other info is needed , i can supply.

     

    thanks

  • Re: Seeding Storage Policy troubles
    Posted: 01-21-2015, 10:18 AM

    Hey James,

    I hear ya with the initial seeding and DASH copies thereafter can be a little confusing at first. Wait till you get into copying the CLDB databse around to ensure subsequent backups only transfer the deltas.

    Anyways, for your issue. The DDB you need to identify can be placed anywhere on the Source MA. Select a temp drive or use the Index Cache drive if you have it temporarily. It is only for generating the new signatures for this "Seeding" process. This DDB will be manually deleted later when you delete the SP Copy. (Configuration > Step 2d. and Step 13)

    http://documentation.commvault.com/hds/v10/article?p=features/deduplication/manualseedingprocess.htm

    I have worked this process to the bones, and have worked with Support to really understand what goes on here. I hope this gets you started.

    Sonny

  • Re: Seeding Storage Policy troubles
    Posted: 01-21-2015, 10:26 AM
    • Ali is not online. Last active: 07-03-2019, 12:32 PM Ali
    • Top 10 Contributor
    • Joined on 08-05-2010
  • Re: Seeding Storage Policy troubles
    Posted: 01-21-2015, 10:38 AM

    Interesting, so i have been doing this possibly all wrong

    I have been following the "manually seeding a deduplicated storage policy document:

    I set up the secondary copy, point the library to the USB drive on the Remote Media Agent, select the USB library as the location fot he DDB as i thought the DDB had to come back to the Data Center.

    From reading this, it appears i can put the DDB anywhere on the Remote Media Agent host, and it does not need to be copied.

    For Step 3, i use the Global Dedupe Storage Policy on the DataCenter Media agent instead of specifying another DDB.

     

    As for the CLDB copy, that is done from DataCenter Media agent. I navigate to D:\CVJR\CV_CLDB (where commvault is installed) and copy the CV_CLDB_AUX_xxx from DataCenter Media Agent to Remote Site Media Agent to C:\Program Files\Commvault\Simpana\iDataAgent\JobResults\CV_CLDB. I drop that CV_CLDB_AUX_xxx folder in there and delete it from the DataCenter Media agent.

    At least that is what i read from the documentation....

  • Re: Seeding Storage Policy troubles
    Posted: 01-21-2015, 10:40 AM

    Ali,

    That link you attached is the exact process i follow...

  • Re: Seeding Storage Policy troubles
    Posted: 01-26-2015, 3:41 PM
    • Ali is not online. Last active: 07-03-2019, 12:32 PM Ali
    • Top 10 Contributor
    • Joined on 08-05-2010

    JamesMacDonald:

    Ali,

    That link you attached is the exact process i follow...

    Hi James, if you're still having trouble may we ask if you call into the Hotline to log an incident, and send us the incident number offline, we can look into this with you.

  • Re: Seeding Storage Policy troubles
    Posted: 02-17-2015, 7:38 PM

    So i have been plowing ahead and getting remote sites seeded back to the datacenter, but i am still struggling big time with getting the synthetic full backups aux copied back.

    THe incrementals seems to go, mind you they take a long time, but the syn fulls are just not happening, some say they will take up to 30 days or more... not acceptable

    I have had calls opened with commvault, we troubleshot on one site, put every registry entry and tweak in that was available and still i am not able to get a syn full back here. These are Aux copies by the way, not the primary backup.

    So i am wondering if anyone else has this issues and what you have done to resolve it.

    Running Simpana v10 SP8.

    remote sites are VSA backups to disk of usually only 1 VM

    Aux copies run over night and full bore on weekends

    most all sites having the issue, from link speeds of 2MB to 20MB (different sites, different link speeds back)

     

     

  • Re: Seeding Storage Policy troubles
    Posted: 02-19-2015, 11:41 AM

    James,

    Sorry to hear you're still having issues here. But in general, sounds like everything is working as intended other than the Synthetic fulls. I am however confused as synth fulls occur on the MA, not at the source. This avoids transferring the FULLs across the wire.

    Where are you seeing the completion ETAs for these jobs? I have been catching up on one of our sites that generates 1T of backups a day, deduped down to 35GB-40GB over a 10mbps link. Its torture, but if I can keep that up, I'm sure you should be fine.

    If you need any help double-checking your configs, let me know. I may be able to shed some insight on whats going on...

    Sonny.

  • Re: Seeding Storage Policy troubles
    Posted: 02-19-2015, 6:00 PM

    I am not sure how you want to do this, but i am gonna give a high level overview of how i have things set up. The remote site config is the same for all remote sites backing up to disk.

    Remote Site - I backup up the remote site to itself. The server onsite is multi-purpose.

    Roles installed on this single server are:

    Hyper-v Host - hosts 3 VM's, only 1 is backed up using VSA

    CV Media Agent - has the media agent installed on the physical box.

    Backup Target - Physical box has a pile of disk. A raid-6 array for the disk library

    Dedupe DDB = A Raid 1 array that hosts the Global Dedupe Database for that site.

    So in commvault, i have a Global Dedupe Database set up for the site. the backup strategy is Daily incrementals and weekly synthetic fulls. THis is true for all remote sites that backup to local disk at the site.

     

    Data center site - I have a large server with a pile of backend SAN storage for  Disk library target. Global Dedupe Database is hosted on SSD drives.  This Media Agent is the target for all Auxilliary copies from out in the field

     

    Now when it came to setting up the auxiliary copy of the remote site backups to copy to the Datacenter, i followed the document from commvault on seeding a deduplicated storage policy.

     

    So basically i took a USB drive, shipped it to the remote site, had it plugged in, created the secondary USB copy and did the copy. Shipped it back, plugged it into the Datacenter media agent, run the tertiary copy. then did the aux copy from remote site back to datacenter. I half wonder if i did this process properly.

     

    So one thing you mentioned you were confused on was the synfulls happen on the MA not at the source. I am not sure what you mean.

    The synfulls do happen at the remote site, as do the incrementals. All i am trying to do is get those incrementals and synfulls copied back to here as an aux/secondary copy. The backup has already happened at the remote site. THe purpose of making a copy of those primary backups is in case the remote site server blows up, we have a copy of the backed up data in the datacenter.

     

    I can supply more information, but let me know how you want to take a look at things, i can take screen shots and what not of what i am seeing

     

    thanks

  • Re: Seeding Storage Policy troubles
    Posted: 02-19-2015, 10:48 PM

    Honestly, you're right. The synthetics do happen locally. I don't know what I was thinking there. But if your initial seeding included any full at all, it should be included in the cache and only the incrementals pass over the wire.

    If you like, we can look at what's going on together to see where this setup is failing. As I understand it, you have already exhausted the support option so I would hope not to step on anyone's toes.

    Let me know if you're interested, and if your permitted this kind of assistance,  then we could setup a remote session to really see what's going on.

    Sonny

  • Re: Seeding Storage Policy troubles
    Posted: 02-20-2015, 2:48 PM

    I dont think i would be able to do a remote session unfortunatly. 

    I am trying to think of the best way to get the info needed without a remote session.  I will see if i can figure out a way to do this without writing a book on what setup i have in place

     

    thanks

  • Re: Seeding Storage Policy troubles
    Posted: 02-20-2015, 8:01 PM

    Let get some of the break points out of the way.

    Can you tell me:

    Does your destination MA DASH copy data to another MA using "Network optimzed"?

    Can you tell me the file/folders details contained in the following:

    • [Source MA]C:\Program Files\CommVault\Simpana\iDataAgent\JobResults\CV_CLDB\
    • [Destination MA]C:\Program Files\CommVault\Simpana\iDataAgent\JobResults\CV_CLDB\

    Can you check the Storage policy used for this copy:

    • How many "Available" jobs are in the DASH copy? (Secondary copy)
    • How many are "Partial" or "to be copied"?

    Whats the size of the link this copy will be transferring over?

    Lets start here, we should begin to be able to see where things are breaking down.

     Sonny.

  • Re: Seeding Storage Policy troubles
    Posted: 02-23-2015, 12:42 PM

    well crap, i replied to this, but it didnt like my screen shots. so here is it is again in a word document for the screen shots i took

     

    For the first question though, the copy is set to Network Optimized.

    I have the screenshots attached.

     

    As for checking the available jobs, i am not sure where i can get this info for.

     

    The link from remote site to here is 2mbps (in reality like 1.5)

    We did have a Sr enterprise architect from Commvault run some sort of formula to give us the minimum bandwidtch required, and this site was 1mbps min.

     

    Attachment: Aux Copy Issues.docx
  • Re: Seeding Storage Policy troubles
    Posted: 02-23-2015, 1:17 PM

    So to review, from what I'm seeing:

    You will be trying to copy at MAX about 16GB of data over the wire daily.

    You do not do any AUX copies from the destination MA.

    Can you tell me the size of the files within the Source\..\JobResults\CV_CLDB\CV_CLDB_AUX_355\ ?(You should see 3 files)

    To check for the jobs currently in the copies, Right click the NAMIOTH5_Teriary_Copy and select View>Jobs. This will let you see what has been transferred and what needs to be transferred. (And what s in progress)

    If you perform this on the Primary, you would see all the jobs backups currently on disk that have not aged.

    The primary jobs will also give you an idea of how much data you will need to transfer over the wire each day.

    According to Support, the size limitations were supposedly removed from DDBCaches, but I would still try to keep the structure clean. You have a \CV_CLDB_AUX_355 folder in the JobResults directory I would move somewhere else.

     

    This is a good start.

  • Re: Seeding Storage Policy troubles
    Posted: 03-04-2015, 9:18 AM

    I am having the exact same problem.  Our incrementals will copy over no problem but when it has to copy a synth full it takes forever even though the synth full only wrote a few gb to the primary copy.  I have made sure the DDBs are synced but still no luck.  I am starting to think CommVault is not the answer to off site backups and we should have stuck with  physically taking tapes off site.  Let me know if you find a solution. 

  • Re: Seeding Storage Policy troubles
    Posted: 03-04-2015, 9:41 AM

    Hey, sorry to hear you are having issues.

    The issue here was that the FULL backup, whether a synthetic FULL or actual FULL, had aged out of the destination copy after the initial seeding. This occured in this situation due to the retention policy of retaining only 1 cycle. This leaves the door open to age out full backups after the time period elapses. Setting the cycles to a minimum of 2 ensures that you get a full to the destination before the previous one is aged.

    I would verify in the destination copy jobs for a full backup that has not been aged out. If there are no fulls, you will have to seed the source copy again to avoid moving the data over the WAN.

    Understand, if the destination does not include a successful FULL, the DASH copy will be forced to recopy the FULL in its entirety to the destination. "Synthetic" is a fancy term for restoring a FULL backup on the MA, and applying the INC backups to it, producing the new FULL backup.

    I hope this helps.

    -Sonny

  • Re: Seeding Storage Policy troubles
    Posted: 03-04-2015, 10:23 AM

    What Sonny stated is what i had going on, among other things. I never had a successful full in the destination, so everytime the synfull would run at the remote site, when the aux copy happened, it would try to send everything back here.

    I also learned to not mess with the type of transfer, Network optimized or disk optimized, everytime i changed those, it would mess things up so i have left them all at network optimized.

     

    I had to do the reseed process again for this site as it was completely pooched, but most of the rest of my sites have been somewhat ok.

    I have another site where the cache DB got corrupted this weekend, in the CV_CLDB folder, the files there are showing no size anymore compared to what they where,  not sure why, but i will need to reseed this one too.

     

    James

  • Re: Seeding Storage Policy troubles
    Posted: 03-04-2015, 10:48 AM

    Remember James, if you have all the data in the destination, you can recreate a new cache db using the destination data and copy the db back to the source server. This will avoid sending the drive and seeding the whole process all over again.

    Let me know if you need a hand setting it up.

    -Sonny

  • Re: Seeding Storage Policy troubles
    Posted: 03-04-2015, 10:55 AM

    Actually, if you could just detail the steps again, that would be helpful

    I did try it on the trouble site, didnt work right, i suspect i missed a step.

    as for the other site that the cache db disappeared, i know that one does not have all of the data here, so it will be a do it from scratch site.

    but this procedure would be useful for future occurances,i copied down the high level steps when we talked, but missed a few things i think

     

    James

  • Re: Seeding Storage Policy troubles
    Posted: 03-04-2015, 2:22 PM

    I hope this will be clear:

    Each Storage policy(SP) that uses a DASH copy will have a minimum of 2 copies. (Primary and Destination). We can limit our task to these 2 datastores.

    Setup:

    • Ensure that the destination MA has the "UseCacheDb" addtional setting enabled.
    • You have a local storage available for data copy. (It can be co-located alonside the production storage already created as this data will be deleted later.)
    • Ensure you have, at a minimum, 1xFULL backup in the destination copy (secondary copy) of the data you want to DASH across the WAN..

    We will be focusing on the second copy, (Destination).

    • Create a new copy, (TEMP_Copy_LocalDisk), in the storage policy you need to recreate the CacheDB.
    • Create a Name.
    • Select the library directly attached to the destination MA.
    • Enable deduplication.
    • Select the deduplication tab and configure a single partition in a TEMP_DDB location local to the Destination MA. This TEMP_DDB folder can be located alongside the PRODUCTION DDB as it will be deleted later.
    • Goto the advanced tab and select the "Network Optimized copy" option.
    • On the Copy policy tab, select the Secondary copy, (Destination) as the source for this Aux copy.
    • Set a date far enough back to ensure you have FULLs from all production systems assosiated with this policy.
    • On the retention tab, clear the "Enable data aging" option.
    • Run an AUX copy, selecting this new copy to run.

    You should now see a new folder get created in the ..\Program Files\CommVault\Simpana\iDataAgent\JobResults\CV_CLDB folder. This will be your new CacheDB for this Storage Policy.

    Once the copy has completed, Copy the CacheTable.dat and CacheTable.idx back to the Source MA, overwriting the current files located in the same path.

    Run the DASH on the second copy ensuring only new data is being sent over the wire. Lets be realistic, there may be alot of new data if the DASH copy had been suspended for awhile. If it has only been a few days, then it should be fine.

    If everything is running as intended, you can now delete the TEMP_Copy_LocalDisk copy from the Storage Policy, and delete the TEMP_DDB folder and contents from the drive leaving the original Primary copy, and secodary copy in the Storage policy.

    I have used this method succesfully to recreate corrupt cachedbs in our environment. This is a tricky procedure and should be used by experienced CV Admins only. I do not take responsibility for results in your environments. As always, try the procedure in a lab if you can before implementing on production resources.

    Good Luck! -Sonny

  • Re: Seeding Storage Policy troubles
    Posted: 03-04-2015, 2:49 PM

    Perfect Thanks!

    i completely had it messed up! go figure!

     

    i will set something up as a lab test before trying out in production for sure

     

    thanks again!

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy