Tips to improve Aux copy job speeds

Last post 09-21-2010, 2:09 AM by sssstew. 17 replies.
Sort Posts: Previous Next
  • Tips to improve Aux copy job speeds
    Posted: 07-31-2010, 1:24 PM

    We are having issues due to the slow speed of our Aux copy jobs. We backup and de-dupe to disk, then run an Aux copy of that data to tape.

    The media agent is dual quad core with 32GB of RAM, the de-dupe data is on fast disk, and the tape library has two LTO4 drives. However we only manage around 32GB/Hour and the Aux copy is taking so long that it doesn't complete during the week.

    I have made sure that virus scanning of the CommVault drives has been disabled, and tried disabling virus scanning altogether, but it made no difference. 

    The server CPU, Disk IO, RAM and network seem fine and the tape library is directly connected using SCSI.

    Any ideas how to speed up the Aux copy process?

     

    Natmam

  • Re: Tips to improve Aux copy job speeds
    Posted: 08-02-2010, 12:52 AM

    As a first step run a tape drive validation. This should give you an idea of what speed you can write to tape at. If this is the speed you are seeing already then thats where your trouble shooting needs to start.

    If that seems to work fine would be interesting to xcopy a couple of hundred mb of data off the maglib somewhere to understand what the achievable disk read throughput is.

    Once you have some "raw" performance numbers for each of those then it will be possible to direct faultfinding in the right direction.

  • Re: Tips to improve Aux copy job speeds
    Posted: 08-02-2010, 5:22 AM

    Did you enable combine streams and multiplexing on the SP?

  • Re: Tips to improve Aux copy job speeds
    Posted: 08-02-2010, 5:34 AM

    Hi Tapeman,

    Thanks for replying.

    If I run a standard backup to tape I get a throughput of between 70/90GB/Hr depending on what I back up (file server/Exchange).

    The Auxcopy speeds have dropped off, I used to get about 60GB/Hr and now I get about 30GB/Hr.

    However the amount of data written and being read from the disks has increased too.

    Looking at performance monitor the two disks (they are both 3TB LUNs provided from a SAN its SATA disk 12 spindles in RAID5) have a read queue of between 4-5 and reads/sec of about 160/170. There aren't any writes.

    Is the disk up to the job? Or am I barking up the wrong tree?

  • Re: Tips to improve Aux copy job speeds
    Posted: 08-02-2010, 5:39 AM

    Hi Telmont,

     

    Yes I have streams combined to 2 and multiplexing set at 4.

  • Re: Tips to improve Aux copy job speeds
    Posted: 08-02-2010, 6:06 AM

    Hi Natman,

    Is the LTO 4 tapelibrary connected via FC, what about the scsi reservation settings?

  • Re: Tips to improve Aux copy job speeds
    Posted: 08-02-2010, 8:32 AM

    The tape drive is connected by standard SCSI, it isn't fibre channel.

     

    Mark

  • Re: Tips to improve Aux copy job speeds
    Posted: 08-02-2010, 4:46 PM

    Are you really sure everything goes through SAN/FC/SCSI during AuxCopy?

    Looking at your basic throughput of max 90 GB/h it seems to me that either network or storage is basically overloaded. Everything below 150 GB/h (except brick level DB backups) is worth for detailed investigation on network or storage level issues, independent from server's CPU and RAM capacity.

     

     

     

     


    Regards,
    MBA
  • Re: Tips to improve Aux copy job speeds
    Posted: 08-03-2010, 3:59 PM

    Yes, I also suspect that you may be sending traffic across the network and not FC. 

    Run perfmon/resource monitor and see what your network bandwidth is when running ONLY aux copies.

    If you see heavy traffic on the MA NIC's, then you will know.  What you will need to do is attach both libraries on the same media agent. OR better yet, get gridstor and shared library license if needed, and present both libraries to all media agents.

  • Re: Tips to improve Aux copy job speeds
    Posted: 08-31-2010, 6:04 PM

    Hi Natman,

    Not sure if you have had a resolution to this yet. We had an almost an identical problem. After months of investigation CommVault was able to provide us with some potential changes, which when we made along with some other little chages has made huge improvements in speed on a continual bassis.

    Issue one: You LUN size is too big you need to reduce them from 3 to sub 2 TB, in our design we changed down to 1TB LUNS.

    Issue Two: DeDuplication - this fragments the crap out of your 3TB LUN which effectively kills any level of performance on your SAN back to CommVault, again reduce LUN size and run an alternating Defrag process on your backup LUNS, we don't use the in built windows one as it takes too long, spend a little $$ its worth it as you can configure schedules and individual LUN policies for defraging.

    Issue Three: Given your back end SAN are there times when reads and Writes are occuring, given they are SATA disks, any occurance like this is VERY VERY VERY bad and should be avoided like the plague.

    Bascially DeDupe, hammers the crap out of the disk and leaves it an absolute mess so these little steps should help to ensure your performance stays good, it certainly helped us.

    AK

     

  • Re: Tips to improve Aux copy job speeds
    Posted: 09-07-2010, 12:14 AM

    I had similar issues.  We use a 13Tb NetApp box for our disk and have a direct scsi attached library on the MA.  As the NetApp box filled up it slowed down... once it was around 80% full it was near uselsss.  Dedupe made the problem even worse.  It just didn't age off the disk very well.  It also didn't help that our disk was not really big enough to handle all of our data at once either ;-)

     

    Eventually we were lacking the space to even keep copy on disk as our data expanded so until i got new storage (i'm still waiting actually) i had to dump the dedupe on a lot of primary copies and set them to spool instead.  This led to the disk cleaning itself up a lot more and increasing that free space and tat in turn made the NetApp box speed up as well. Add in a small amount of multiplexing and the speed to tape increased a lot.

     

    I don't plan to reimplement dedupe much until V9 or when i get more storage for my disk copies. 

  • Re: Tips to improve Aux copy job speeds
    Posted: 09-08-2010, 8:05 PM

    very inetresting thread, we too have aux copy speed issues, we run our main de-dupe storage policy which gets really poor aux copy speeds to LTO4 tape of about 60gb per hour.

    We are using fibre attached Quantum i500 with 2 x LTO4 drives.

    A recent test of straight to tape seemed to get good results... hmm

    Does anyone else agree that regular defrags of the drives is beneficial?  Im thinking getting diskkeeper 2010 ($350) if you say that the windows one isnt any good.

    will be watching this thread closely.

  • Re: Tips to improve Aux copy job speeds
    Posted: 09-09-2010, 10:48 PM

    sssstew,

    This thread has alot of good information that will help you out.

    1st thing you can do is to increase the mux factor. Dedup works by backing up the data one and then referencing the same data many times. This is regardless of vendor. In an aux copy multiple reads occur on the disk before the actual data is found and then rehydrated.  While this happens the writers can just sit idle. by increasign the Mux factor multple readers will keep the writer active.

    2nd. Defragging the Lun will help increase aux performance.

    3rd. SILO: depending on retention and frequency of the aux copy it may be better to move the data to tape in dedup form.

     

    AaronA.

     

     

  • Re: Tips to improve Aux copy job speeds
    Posted: 09-13-2010, 10:05 PM

    1) so are you saying to push it out to more than 4 for Multiplex Source Stream setting on the tape copy?  reckon i should double it and see?

    2) hmm ok will need to look at this

    3) Retention for tape copy is 15 days 2 cycles, frequency is every 2 hours.  I didnt think you could de-dupe to tape with V8, thats a V9 feature i beleive?

  • Re: Tips to improve Aux copy job speeds
    Posted: 09-13-2010, 10:23 PM

    Hi sssstew,

    1) I would not double your current Multiplex value, if you think that you should be able to obtain faster throughput then you should increase by 1 or 2 threads and then monitor. Basically there will be a point where the number of streams gives you the best performance, so its something you should implement in an incremental manner, so you can document the values then choose the best mix.

    3) Yes you can de-Dupe to tape it is called a Silo copy, but a Silo copy is not an incremental data move, basically a SILO copy takes the entire Storage Policy you have selected and moves it to tape in de-duplicated form. It is an excat point in time copy.

    AK

  • Re: Tips to improve Aux copy job speeds
    Posted: 09-13-2010, 10:33 PM

    Hmm ok, dont think SILO is for us then.  cheers

  • Re: Tips to improve Aux copy job speeds
    Posted: 09-20-2010, 12:40 AM

    Do you defrag via commvault or some 3rd party app?

  • Re: Tips to improve Aux copy job speeds
    Posted: 09-21-2010, 2:09 AM

    just getting diskeeper for it now.

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy