Aux Copy performance

Last post 07-06-2011, 12:21 AM by jkj. 16 replies.
Sort Posts: Previous Next
  • Aux Copy performance
    Posted: 04-27-2011, 9:10 AM

    We are seeing slower than expected throughput on our Aux Copy jobs. Environment is 3 MA's in a gridstore, each attached to a 14TB, and 2 5TB volumes via iSCSI. The MA's are attached to the Quantum i2k library with 4GB FC to 8 LTO3 FC drives. The maglib is not being taxed at all and the throughput to the LTO's is in the 20-30GB/hr range. We are deduping the maglib during backup. Is there anything we can do to increase the throughput from disk to tape?

  • Re: Aux Copy performance
    Posted: 04-27-2011, 3:37 PM

    I think when you aux copy dedup data it has to "un-dedup" it before being written to tape which can cause a performance hit.  Silo can be used to move the dedup in native form to tape which may produce better results.

  • Re: Aux Copy performance
    Posted: 04-27-2011, 4:00 PM

    Tom Watson,

    Do you have stream randomization turned on?  You may want to disable it to see if that makes a difference:

    http://documentation.commvault.com/commvault/release_8_0_0/books_online_1/english_us/features/storage_policies/storage_policies_how_to.htm#storpol_enable_stream_random

    Do you have the combine to stream option turned on?  If so, what value do you have it set to?

    JW

     

  • Re: Aux Copy performance
    Posted: 04-27-2011, 4:48 PM

    Stream Randomization is off. We are combining source to 5 streams. Odd thing is that I am seeing periodically the throughput on 1 drive as high as 100-150GB/hr while the others are anywhere from 19-60GB/hr. I was hoping to see the 100+GB/hr across all the streams.

  • Re: Aux Copy performance
    Posted: 04-28-2011, 10:43 AM

    Tom Watson,

    Have you had a chance to run CVDiskPerf to see what output you receive?  Here is a link on the utility:

    http://documentation.commvault.com/commvault/release_8_0_0/books_online_1/english_us/features/single_instance/single_instance_how_to.htm#Measure_Disk_Performance

    JW

  • Re: Aux Copy performance
    Posted: 04-28-2011, 5:15 PM

    Might be worth trying the Look-Ahead Reader registry key.

    There is another thread discussing the key in this forum: (http://forum.commvault.com/forums/thread/8957.aspx)

    Phil

     

  • Re: Aux Copy performance
    Posted: 04-28-2011, 8:36 PM

    Tom Watson:

    Odd thing is that I am seeing periodically the throughput on 1 drive as high as 100-150GB/hr while the others are anywhere from 19-60GB/hr. I was hoping to see the 100+GB/hr across all the streams.

    I believe what you might be seeing is a bug, sort of.  I noticed when I had multiple streams running they would each total the average thoroughput I expected (in my case I had two streams going about 100GB/hr each for a total of 200GB).  However, one of the streams would finish before the other and it's final thoroughput would remain, however the unfinished stream would then shoot up to the total bandwidth.  Consequently, depending on when you check you might be seeing different results.  The stream info should include an indication whether it's completed or not.  Doesn't really answer why you're seeing slower-than-expected results, but hopefully it'll help explain the inconsistent stream thoroughput.


    Just my 1.4 cents (after taxes)
  • Re: Aux Copy performance
    Posted: 05-03-2011, 6:58 AM

    I spent quite some time with a customer last year looking at this exact problem (or so it looks...)

    Long story short, it appears to be a limitation of the software in the way it threads data to tape across the aux copies. In our environment, we had 10 LTO4 drives and never really saw throughput increase above 600GB/h across all jobs. 

    One thing to test to see if this is like the same issue, is to run an Aux copy across all 8 drives, from only one media agent. When we did that our throughput went up to 1TB/h+.

    If you see the throughput go up, then it's likey the same issue. 

    Another behaviour we were seeing was that if you had a number of streams set for the aux copy (say 10) all 10 would have to finish before a new set of 10 would start. This would mean that when you got down to the last 1 or 2, the overall throughput would tank.

    I believe stream randomization is meant to resolve this, but it didn't make any difference for us. We are on CV 8 SP5 btw. 

  • Re: Aux Copy performance
    Posted: 05-05-2011, 3:03 AM

    I know this doesn't help too much but CV doesn't really seem to be written for anything to do with tape (performance particularly). 

    One reg key that did help us a bit in v8 (granted this was tape to tape aux copies rather than Disk to tape) was nAuxCopyNetworkAgents; also nNumPipelineBuffers and SDTMaxThreads may be worth a look.

    Would be nice if CV released a list, description, and impacts of their reg keys.

  • Re: Aux Copy performance
    Posted: 05-09-2011, 4:04 PM

    Well I've tried every recommendation I can find and nothing has improved performance. I am due to upgrade to v9 starting next week and am praying that it is written better than 7 and 8 were, and gives more performance going from disk to tape. This seems to be a major problem with CV from everything I have seen in the last 2 versions. Keeping my fingers crossed......

  • Re: Aux Copy performance
    Posted: 05-09-2011, 4:20 PM

    Vincenzo_Basolino:
    I think when you aux copy dedup data it has to "un-dedup" it before being written to tape which can cause a performance hit.  Silo can be used to move the dedup in native form to tape which may produce better results.

    Just noticed this as I was re-reading this thread after the last response and wanted to mention something.  I did a lot of testing prior to our v9 rollout and one of my plans was to use SILO to tape for long-term data storage instead of rehydrated aux copies (the ratio of media usage between the two for a normal file server is probably 10-1, or better).  This works great unless your media agent takes a dive and you lose the library and DDB.  I went round and round with tech support on this and the end result was I could retrieve from SILO right after the recovery of the media agent, but as soon as I sealed the store I lost all access to that SILO'd data due to an index mismatch.  As a result we're using SILO for point-in-time recovery but we still do selective aux copies to retain for historical recovery in case we lose the media agent.


    Just my 1.4 cents (after taxes)
  • Re: Aux Copy performance
    Posted: 05-10-2011, 12:33 PM

    Tom Watson:

    Well I've tried every recommendation I can find and nothing has improved performance. I am due to upgrade to v9 starting next week and am praying that it is written better than 7 and 8 were, and gives more performance going from disk to tape. This seems to be a major problem with CV from everything I have seen in the last 2 versions. Keeping my fingers crossed......

    Can't comment on how it'll impact performance with that many libraries, but FWIW 9.0 SP2 with the look ahead registry key does seem to give very good aux copy performance.  We only have a single drive but I'm seeing between 150gb/hr to 300gb/hr depending on the job.

    Possibly teaching granny here but have you got the tape block size set to 256kb and a higher than normal chunk size such as 16gb or 32gb?

  • Re: Aux Copy performance
    Posted: 05-12-2011, 4:00 PM

    Were using "Use Application Setting" for the Chunk size and "Use Media Type" setting for the Block size. We also have hardware compression on.

  • Re: Aux Copy performance
    Posted: 05-12-2011, 4:42 PM

    Something I just noticed. The aux copies are all using the same path for their operations (thus the same media agent). We are set up in a gridstor and I have 3 media agents connected to the mag library and the tape library. The option to use round robin is greyed out. Also we are not using stream randomization on the aux copies to tape as I have read thats a no-no when you deduplicate? But would that matter if the deduplicated data is coming off the maglib and being rehydrated befor going to tape?

  • Re: Aux Copy performance
    Posted: 06-28-2011, 8:35 AM

    Hi Tom,

    what is your combine and multiplexing factor ?

    how many stream have you got on your Storage Policy ?

    Really thanks !!!

     

  • Re: Aux Copy performance
    Posted: 07-05-2011, 6:35 PM
  • Re: Aux Copy performance
    Posted: 07-06-2011, 12:21 AM

    Hi Andy,

    No, I'm not after the list from booksonline.  It's a very basic list without all the "good" reg keys.  I'm after the full list of reg keys.  There's plenty to choose from, but for example nBUFSIZE is set in the registry on vanilla Unix FS iDA install.  That's not in the books online.  Then there's the issue of keys that development keep to themselves, but that's not a battle I'll win :)  CV should at least release documentation on all the keys that are set by default on agent install though.

    Thanks anyway.

     

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy