Disk to disk aux copy - very slow throughput

Last post 12-13-2013, 11:48 AM by Nevaduh. 9 replies.
Sort Posts: Previous Next
  • Disk to disk aux copy - very slow throughput
    Posted: 12-10-2013, 12:46 PM

    On average, my Windows file system storage policy aux copy job starts out with about 15 readers and maintains a steady pace of about 80 GB/hr. As each source copy completes, I notice the throughput of the job starts to decrease. I expect this, but when it goes down to one source copy being left, the throughput drops below 1 GB/hr. The source copy only contains ~5GB so I would expect the job to finish in under one hour.

    The source copies reside on an HP D2700 array using 900GB drives and replicate to an EMC VNX with the same drive type.

    Does anyone have any recommendation for performance tuning aux copy jobs on disk libraries? I've been through the documentation and tried a few options but no luck.

    On the MediaAgents, I have UseCacheDB and DataMoverUseLookAheadLinkReader options enabled.

    The mount paths are configured to use 1024 blocks for fragmentation, and number of writers are set to 15 (will boost them to 25 to test).

    The Aux Copy has DASH enabled and network optimized selected.

    Running out of decent options here. Anything else to check?

  • Re: Disk to disk aux copy - very slow throughput
    Posted: 12-10-2013, 1:10 PM

    Update:

    Disabled IPv4 Checksum Offload and TCP Checksum Offload (IPv4) in the HP NIC config utility. I have two 10GbE NIC's teamed. Disabling these settings shows some improvement.

    Any other suggestions on the NIC/OS side as well?

  • Re: Disk to disk aux copy - very slow throughput
    Posted: 12-10-2013, 1:56 PM

    Hi Frank,

    Is this on v10? if so, which service pack level are you on for the source and destination mediaagents.?

     

    Thanks

     Jaidev.

  • Re: Disk to disk aux copy - very slow throughput
    Posted: 12-11-2013, 9:30 AM
    Yes, V10 SP4a.
  • Re: Disk to disk aux copy - very slow throughput
    Posted: 12-11-2013, 9:35 AM

    Here's an example where 3 source copies completed with decent throughput, but the final copy drops below 1 GB/hr:

     

  • Re: Disk to disk aux copy - very slow throughput
    Posted: 12-11-2013, 5:49 PM

    frank_grimes,

    Would be good if you can call this into support.  There was a known issue addressed in sp3a that sounds similar to this, but you mentioned that you have sp4a on both source and destination Media Agents for the auxiliary copy.

    Thanks

     Jaidev.

  • Re: Disk to disk aux copy - very slow throughput
    Posted: 12-12-2013, 10:22 AM

    Thanks, Jaidev. I have a case open now and will update the thread once I know more.

  • Re: Disk to disk aux copy - very slow throughput
    Posted: 12-12-2013, 11:02 PM

    You said your MA has 2 * 10gb cards. What is the actual link speed and latency between the 2 media agents? In most cases, you will most likely want to have disk optimized instead of network optimized. Latency of the link and the type of disk you use for your destination DDB will definitely have a big impact too.

    With v10 on a 1gb link with less than 5ms latency, I copy my weekly full with peaks around 8TB/hr and average job once done is usually a couple of TB/hr.

    Phil

  • Re: Disk to disk aux copy - very slow throughput
    Posted: 12-12-2013, 11:04 PM

    Oh and if you're using a 10gb link then you might want to check with your network folks but you will probably want to make sure jumbo frames are enabled with MTU at 9000 if I remember correctly.

  • Re: Disk to disk aux copy - very slow throughput
    Posted: 12-13-2013, 11:48 AM

    frank_grimes:

    On average, my Windows file system storage policy aux copy job starts out with about 15 readers and maintains a steady pace of about 80 GB/hr. As each source copy completes, I notice the throughput of the job starts to decrease. I expect this, but when it goes down to one source copy being left, the throughput drops below 1 GB/hr. The source copy only contains ~5GB so I would expect the job to finish in under one hour.

    The source copies reside on an HP D2700 array using 900GB drives and replicate to an EMC VNX with the same drive type.

    Does anyone have any recommendation for performance tuning aux copy jobs on disk libraries? I've been through the documentation and tried a few options but no luck.

    On the MediaAgents, I have UseCacheDB and DataMoverUseLookAheadLinkReader options enabled.

    The mount paths are configured to use 1024 blocks for fragmentation, and number of writers are set to 15 (will boost them to 25 to test).

    The Aux Copy has DASH enabled and network optimized selected.

    Running out of decent options here. Anything else to check?

     

    i did some screen shots.   i played with all kinds of thing for over a year.   i now have aux copies like ofter our nightly incremental backups of @ 6 TB and the aux copy can do 6 Tb in under 2 hours.  If you watch the through put with nothing else running it can exceed 5000 Gb not your little 80 Gb.

     

    Defrag your media agent mount paths often.  If you do it every day it can complete in about an hour.  i do it right after the aux copys complete and before the after hour backups.

    My media agents have 8 X 300 10K drives done as one big raid 10   carved up as local disk c:\ for the system f:\  as the mount point volume   G:\ as the DDB    raid 10 is so fast that the DDB daily backup runs at about 700 Gb  so it is done in 20 minutes or so if it is 200 gig in size.

    I use Dell MD1200 disk arrays  12 disk X 3 TB 7.2k rpm drives.   You end up with about 30.7 Tb raid 5   carved up into 16 mount pounts of 2000 Gb and the small left over 733 Gb

    The big thing is to limit the number of readers and streams.   Use disk optimization not lan.   I have 10 gig lan now, but it was still really high when it was just 1 gig ethernet

    aux copy of 8 readers.   which uses 16 streams because of 8 on primary and 8 on secondary.

     

     

     

     

     

     

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy