CDR Replication Issues

Last post 11-10-2010, 12:38 PM by Ted P. 5 replies.
Sort Posts: Previous Next
  • CDR Replication Issues
    Posted: 07-12-2010, 10:19 AM

    I am curious to know if anyone out there is having issues with CDR not being able to keep files in sync.  We have had CDR for about 1 1/2 years and we have had difficulty keeping data current with CDR.  It is easy for me to point fingers at the Network group since we have either a T1 or 2-T1's at all but one of ours sites.  We have seen a site with a 40MB connection also stop replicating and have worked with support with very little success.  Has anyone else run into issues or been able to resolve CDR issues like this?

  • Re: CDR Replication Issues
    Posted: 07-12-2010, 10:26 AM

    RF,

    You mentioned two seperate issues;

    1.  Keeping files in sync

    2.  CDR stops replicating.

    For issue 1, do you see any rhyme/reason or pattern to the data not properly getting replicated?  Could folder/file permissions be preventing replication from moving the data, such as the removal of "local system" account? 

    For issue 2, Typically when the pair is failed or aborted (system abort) the failure reason is typically clear, such as running out of space in the log folder or broken links.  Do you see any such details on those pair details?

  • Re: CDR Replication Issues
    Posted: 07-15-2010, 11:31 AM

    RF1908,

    My company is one of the early CDR adopters, and over the years we've had various issues with replication depending on a MULTITUDE of factors - network blips, memory issues on source and destination servers, etc.  The list goes on.

    In general, it seems that once something interferes with log transmission when a pair is in Replicating state you will start seeing discrepancies.  The worst case is when the source has tracked the changes, dumped them into logs, and then the logs don't end up being properly replayed - at that point even a SmartSync may not catch the change since it wasn't flagged as needing to be transmitted.

    Due to audit requirements we now have 2 ways of dealing with this.  First we converted our CDR replication sets to Remote Backup mode.  Due to WAN constraints saying replication was "near real time" was simply not true, and we did not need THAT tight a replication window anyways.  We simply schedule CDR Remote Backup "jobs" every 3 hours during the day.  This completely eliminates the whole log creation/replay thing.  CDR does essentially a SmartSync "plus" and we have had little to no issues with source/destination parity in this mode.  Second, working with CV support we are using a utility they are working on that compares source and destination and will flag any discrepancies to be corrected by the next SmartSync.  With any luck they will add this functionality to Simpana 9 but who knows.

    Either way, my 1st recommendation is to evaluate whether or not you truly need that near real time RPO for replication.  In our case CDR replicates branch office data to our central datacenter for DR and our long-term retention requirements.  If, however, you need a true hot or warm site you will need to stick with Continuous mode, give it lots of bandwidth, and work with CV support on how to periodically run source/destination compares to ensure things are in sync.

  • Re: CDR Replication Issues
    Posted: 07-16-2010, 1:41 PM

    CDR disruption can be OS or network related. The CDR, CDRControl and CVReplication logs along with the OS logs (Windows System and Application events) should indicate the problem. In general you should ensure both source and destination nodes are on same CVLT update level for best reliability and ensure the source and destination volume along with the RepLog volume all have a minimum of 15% free space at all times. CDR will stop replication if the spacecheck detects low space, this can be seen in the logs.

    Good points about Backup Mode (DDR) as apposed to Continous (CDR), not only does DDR not require logs BUT in some cases you may end up using less bandwidth using a scheduled replication rather than near real time. If you dont have that aggressive RPO and the same data files get updates multiple times a day then DDR can be more bandwidth effecient.

    If you have an aggressive RPO then perhaps you need to do an hourly RP or CRP although you may only backup the recovery point once a day. If you create a batch file to run CDRXHelp and read the USN prior to each recovery point you can use that USN value if you need to roll back to a point in time and initiate a SmartSync from there.


    Calippo
  • Re: CDR Replication Issues
    Posted: 08-30-2010, 11:15 PM

    RF,

    I have seen this issue in a number of clients that have tested/used and continue to use this technology and like yourself they have had very little joy in locating the actual cause of a stop replication event.

    Just as everyone else has outlined here, Network, OS, Processor, Memory and any number of non CommVault related issues can cause CDR to stop replicating.

    Just recently CommVault have introduced the ability to run DeDuplicated Aux Copies of your data to another MA/data center. fro the testing and real world monitoring I have been doing with my customers in this area we are seeing that this method is far more reliable.

    The largest caveat with this technology is it cannot be rate limted lik CDR can be, which means it will use all it can eat.

    A nice thing about this technology is that even if it fails to copy the data, just like a normal Aux copy it will attempt to catch up and the system will also alert you if it falls behind.

    AK

  • Re: CDR Replication Issues
    Posted: 11-10-2010, 12:38 PM

    To expand upon Andreas' post above, there are two new features in Simpana 9.0 which can help out these situtations with replication:

    DASH copy is a new method to Aux Copy de-duplicated data which in alot of cases will be used instead of MediaAgent Replication (SDR) with the CDR agent.  Info on this feature can be found here:

    http://documentation.commvault.com/commvault/release_9_0_0/books_online_1/english_us/features/dash_copy/dash_copy_basic.htm

    CDR in 9.0 has a new "Optimized Sync" method which can re-check all the files on both sides and automatically correct any differnences between them.  Previous versions of CDR use the NTFS Change Journal to keep track of which files have changed so if data was not in sync for any reason, there was no easy way to go back and re-check the files with doing a Full Sync.  Optimized Sync gets around this and will get both sides in sync by doing a SmartSync on all files in the pair content.  This is a selectable option so you can choose to use this method in the event files fall out of sync.  More info about this here:

    http://documentation.commvault.com/commvault/release_9_0_0/books_online_1/english_us/features/data_replication/data_replication.htm

    Ted

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy