Strange and slow Aux Copy Network Behavior

Last post 04-03-2015, 10:43 AM by Ali. 8 replies.
Sort Posts: Previous Next
  • Strange and slow Aux Copy Network Behavior
    Posted: 03-28-2015, 2:49 PM

    I’ve been tracking an issue why our Aux Copy network transfers are taking so long. I opened a case and support was able to give me some tuning suggestions regarding storage policy setting and I also worked with my network support to bypass our wan optimizers. I have seen some improvement but the latency still persists.  In most cases I was seeing less than 25% bandwidth utilization. After some tweaking it has slightly improved. I’m not happy with the performance so I Friday night starting running performance monitor on a media server tracking bytes sent and bytes received after the synthetic full completed. I can see where the aux copy kicks off and start sending data and hovers around 3,035,867 bytes per second for about an hour and a half then the data transfer drops to an average of 50k for 45 minutes then picks up again. No other transfers were on going and given the hour of the night non are expected. I can’t explain the down time.


    @thefunfactory.com
  • Re: Strange and slow Aux Copy Network Behavior
    Posted: 03-31-2015, 11:09 AM

    Hello,

    Do you know if the copy is a DASH copy? The purpose of the DASH copy is to decrease WAN utilization. It compares blocks between deduplication engines on the source and destination side of the transfer and only sends the unique blocks over the network.

  • Re: Strange and slow Aux Copy Network Behavior
    Posted: 03-31-2015, 11:27 AM

    Yes we have dash copies enabled. That's extremely interesting. Thanks for the information. How does that play into throughput GB per hour?


    @thefunfactory.com
  • Re: Strange and slow Aux Copy Network Behavior
    Posted: 03-31-2015, 11:34 AM

    Hello,

    With a DASH copy you will see the Data Processed and the Data Transfered over Network within the details of the job. It is best to look at the Data Processed as not all data will be transfered over the network. Throughput is calculated via many factors and may at times have some inconsistency. We have logging that shows read and write speeds as well as DDB performance values. These all factor into your throughput as well as other variables.

  • Re: Strange and slow Aux Copy Network Behavior
    Posted: 04-02-2015, 12:17 PM
    • Ali is not online. Last active: 07-03-2019, 12:32 PM Ali
    • Top 10 Contributor
    • Joined on 08-05-2010
    Hi Sluggo, Here's more to consider below, also if its getting really troublesome you may want to reach your account team and ask for some local support to have your site review, seems like there may still be some other factors thats chocking down the WAN transfer. http://documentation.commvault.com/commvault/v10/article?p=features/performance_tuning/tunable_parameters.htm#Auxiliary_Copy
  • Re: Strange and slow Aux Copy Network Behavior
    Posted: 04-02-2015, 9:13 PM

    Attached is what our network performance looks like over an 8 hour aux copy period. Data was moved for about an hour while the job ran for 8 hours and was killed by the schedule (19:1111) unfinished. Seem like Aux Copy is doing something other than moving data. The network has no bottlenecks or policies. From time to time when I see this as it's happing, I can manually copy data to the source at expected rates.

    We haven't scheduled the on site visit yet. I'm hoping that will be done sooner than later.

    To date support has suggested these 4 items be changed.

    1. Distribute data evenly among multiple streams for offline read operation
    2. bypass wan optimization devices.
    3. Increasing the Speed of the Read Operation
    4. Enable Dash Copy/Network Optimized copy

    We applied the final pieces today. We should know the results shortly.


    @thefunfactory.com
  • Re: Strange and slow Aux Copy Network Behavior
    Posted: 04-03-2015, 7:57 AM

    Not sure if this was established in an earlier thread, what version of CV is this? V9 or V10?

    Thanks

    SG1

  • Re: Strange and slow Aux Copy Network Behavior
    Posted: 04-03-2015, 8:15 AM

    This is CV10 SP7.

    We had a breakthrough last night. After adding the changes to "Increasing the Speed of the Read Operation" and "Enable Dash Copy/Network Optimized" 10 of the 15 sites that were way behind completed hours before the schedule limit. Of the remainder 5, they have made real significant progress where they should be caught up in the next day or so.


    @thefunfactory.com
  • Re: Strange and slow Aux Copy Network Behavior
    Posted: 04-03-2015, 10:43 AM
    • Ali is not online. Last active: 07-03-2019, 12:32 PM Ali
    • Top 10 Contributor
    • Joined on 08-05-2010

    Thanks for the update, guesing no changes in the infrastructure as well that you know of?

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy