Any downsides to using the Look-Ahead Reader?

Last post 10-07-2011, 9:48 AM by Tom Watson. 14 replies.
Sort Posts: Previous Next
  • Any downsides to using the Look-Ahead Reader?
    Posted: 04-25-2011, 9:51 AM

    I'm always looking for ways to improve aux copy performance from dedupe on disk copies to non-dedupe tape copies (rehydrating the data), so when I ran across this registry key I was curious about trying it.  But it's fairly hidden and not documented very well, which usually means we shouldn't just turn it on without understanding it better.  So that's why I am here... asking what this registry key actually does, and if there are any downsides to using it.  (If there are no downsides, one might wonder why it is not turned on by default?!).  This is the registry key in question:

     

    Look-Ahead Reader

    http://documentation.commvault.com/commvault/release_9_0_0/books_online_1/english_us/features/dedup_disk/advanced.htm#Look-Ahead_Logic

     

    Location  
    Windows HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance<xxx>\
    Unix /etc/CommVaultRegistry/Galaxy/Instance<xxx>/MediaAgent/.properties
    NetWare Not Applicable
    Key MediaAgent
    Value DataMoverUseLookAheadLinkReader (optional)
    Value Type (Windows only) DWORD
    Valid Range 0 or 1 (0 = disable, 1 = enable)
    Created in User-created on the MediaAgent computer
    Description Reads data using look-ahead logic.
    Additional Information You may find additional information in Deduplication - Other Features - Look-Ahead Reader.
    Applies To Deduplication enabled restore and auxiliary copy operations run on MediaAgents installed with this registry key.
  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 04-25-2011, 10:44 AM

    There are not really any downsides to adding the registry key. This will add additional reads request to the read side while awaiting write side to come back with completion of write.

    This key is typically used on DASH copy functionality.


    Thanks

    db2004

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 04-25-2011, 10:48 AM

    Will it improve performance of aux copy to tape (rehydrating from dedupe'd copy to non-dedupe'd copy)?

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 04-25-2011, 11:10 AM

    Hard to say... really depends on where the bottle neck is on the Aux copy. If issue is on read side it could increase performance as it will change the way data is read from the disk copy. If this is a write side issue it will be of no effect.

    This is really used for DASH operations and was not intended for traditional aux copy.

     

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 04-26-2011, 9:54 PM

    It's probably worth while ensuring that none of your Storage Policies are using Single Instance DeDuplication.

    I enabled this key on my CS/MA which still uses Single Instance DeDuplication, and whilst the throughput increased significantly, I failed to notice that the AuxCopy.exe process was leaking, in turn consuming all available memory.

    During the day, this was fine as no other jobs were running, however, during the nightly backup window, simpana processes were failing everywhere.

    Once the key was removed, everything returned to normal, including poor throughput.

     

     

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 04-28-2011, 5:11 PM

    db2004:

    Hard to say... really depends on where the bottle neck is on the Aux copy. If issue is on read side it could increase performance as it will change the way data is read from the disk copy. If this is a write side issue it will be of no effect.

    This is really used for DASH operations and was not intended for traditional aux copy.

     

    Hi,

    I just wanted to add something about the registry key. Even though it's been created for DASH operations, I think that any read operations from a dedup store should benefit from it. Obviously, this is not the only factor and there could be many other bottlenecks but just to give you an exemple, here is a test we did in our lab which show the benefit of the registry key:

    The test was a restore of 155gb of data (250 000 files) to a Windows volume. We first restored the files with the antivirus enabled, reg key disabled. Ended up taking 3 hours and 30 minutes. Then, we did the same test by disabling the antivirus. It took 56 minutes. Then, we ran the same test with the antivirus disabled and the reg key enabled. The restore took 29 minutes.

    As you can see, during the restore, the job ran almost twice as fast. If restoring many TBs, this could really make a huge difference and even though this isn't a DASH copy, the job still benefited from the key. I'm assuming aux copying to tape would benefit from it too.

    If you try it, it would be interesting if you could share your results.

    Phil

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 04-29-2011, 9:44 AM

    I am a little confused with this... Does the key only go on the MA attached to the dedup database (I have 2) or does it go on all MA's that are attached to the MagLib (I have 3 in a Grid configuration, each has several iSCSI LUN's)?

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 04-29-2011, 10:29 AM

    I'm not 100% sure but I would think you need it on the MAs that execute the jobs. The MA hosting the DDB will send whatever information queried by the other MAs. So logically, it should be the MA doing the aux copy that will ask for DDB information in advance and thus should have the key.

    Personally, it is set on all my MAs as I didn't see any issue having the key enabled if the MA doesn't use the feature.

    Phil

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 04-30-2011, 5:12 AM

    We've made quite a lot of hardware changes and Commvault upgrades in our environment, but right now, using that registry key and 9.0 SP2 I'm seeing around 200gb/hr on an aux copy of our full VMware backups.

    That's pulling deduplicated data from an Equallogic PS6000E iSCSI array in (currently) RAID50 connected directly to our MA and sending it to a SAS attached HH LTO4 library.

    I've not yet migrated our existing tape storage policies across to d2d but assuming I can get that kind of throughput on aux copy, whilist it's a bit slower than I see backing up directly to tape, it doesn't seem at all bad.

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 10-04-2011, 5:27 PM

    Bill,

    You obviously know our environment since you upgraded us, I just thought I'd mention that I tested this setting for aux copies and found that it cuts our performance in half.

    The best performance tweak for us is defragmenting the LUNs, so I've scheduled those to occur every Monday and Friday.

    After upgrading to Simpana 9 and implementing dedupe in August, our aux copy speed started off at 47 - 53 GB/hour. Previously, in Simpana 8 with no DDB, we ranged from 75 - 115 GB/hour.

    After defragging the LUNs, performance increased to 104 GB/hr, essentially doubling our speed back to what we obtained before deduplication. The difference in speed means finishing our aux copy in ~12 hours vs. over 24 hours.

    I then enabled the lookahead reader registry setting but found it halved our performance to about 50 GB/hr. Disabling the setting brought the aux copy speed back to the 100 GB/hr range.

    I would like to test various multiplexing factors in the future (currently at 3).

    Our magnetic disk is on the low performing side being a Drobo Pro with (8) 2TB disks in a RAID6 array. We may convert our NetApp FAS2020 to be the target sometime in the future and will retest at that time.

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 10-04-2011, 7:06 PM

    When we used Single Instancing in v7, aux copies ran around 60gb/hour.  After some SIS corruptions we got rid of SIS and enjoyed 250gb/hour+ aux copies.

    We moved to 9sp2a and aux copies for DeDupe data ran around 80gb/hour.  After enabling the Look Ahead key it now runs anywhere between 140gb/hour to 200gb/hour which is "acceptable" for the volumes of data we're moving.

    Slightly OT: If anything I think there are trade offs, downsides and risks that need to be understood with using DeDupe itself as v7 Single Instancing was somewhat prone to corruption.  So far we've noticed though that DeDupe certainly saves a HUGE amount of maglib.

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 10-05-2011, 10:23 PM

    Leif,

     

    Your experience is pretty much spot on with what I've been seeing at countless customer sites over the last couple of years (v8 and v9) with deduplication and aux copy to tape.  When dedupe is initially configured, aux copy to tape performance is somewhat slower than without dedupe (perhaps 25% slower), then it gradually plummets to 50% (half speed) or worse, as you noted.  Defragging usually increases performance significantly, but some customers don't like taking the outage to perform the defrag on a regular basis.  Besides defragging, you can also upgrade to a beefier disk target that handles 100% random reads better, or reduce aux copies to tape to once a month rather than once a week.  In the latter case, I have several customers that leverage DASH Copy (aux copy of dedupe data) to a remote site disk target so they only need to cut to tape once a month or so.  Of course that requires a remote site, remote disk target, and another MediaAgent... but it's a great approach if feasible.  Otherwise your best bet is a beefier disk target and/or defrag on a regular basis.  Please share with everyone some of your best aux copy speeds.  100 GB/hr is pretty good, especially for the Drobo with dedupe, let's see if we can get up to 150 GB/hr!!!  It's really interesting to hear that the registry key didn't help you at all.  I still don't have a good understanding on what this registry key does, so I have not told any customers to enable it, and your experience makes me glad I did not recommend it.  I wish CommVault would explain this registry key better.  For example, what are the pros and cons of using this key, when is the key appropriate and when is it not appropriate.  I have no idea.

     

    Bill

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 10-07-2011, 3:28 AM

    I thank you so much!!

    We changed dwMaxAsyncIoRequests to 5 and DataMoverUseLookAheadLinkReader to 1.

    We had speed up to 50GB/hr to LTO4, which is really really slow!
    Now we hit the 200GB/hr! Wow what that’s more than 3 times fast.
    Really impressive with monthly backups (>8TB).

    Our Backupserver:
    HP DL380 G6 / 2*Quad Xeon / 32GB RAM
    Storage: 18TB HP SAS MDL with 300GB FC SAN DDB-LUN
    Tape: HP MSL8096 - 96 Slot / 3 Drive LTO4 SAS
    Global DeDup Store with 128kb Chunk Size
    Simpana 9 SP3a

    Thanks!

    Best regards
    Daniel

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 10-07-2011, 8:57 AM

    Thanks for the feedback Daniel. Of the postings I've read on the forum, I seem to have the only instance where performance decreases with the look-ahead reader enabled. It must have something to do with the magnetic disk I'm using and it's poor scalability. I can only speculate unless I take some measurements comparing what's happening with the look-ahead reader enabled/disabled.

    I wasn't familiar with the dwMaxAsyncIoRequests key before. I see in the documentation that you can adjust it from 2 to 5. I don't have this key enabled in our media agent. Anybody know what value is the default value? Just was curious how I disable it and undo the setting should I decide to test it.

  • Re: Any downsides to using the Look-Ahead Reader?
    Posted: 10-07-2011, 9:48 AM

    Does anyone know if the memory leak issue was ever addressed by Commvault on this? I did try it on my media agents but they began to crash during the backup windows due to the memory leak. Once the key was removed everything returned to normal slow throughput.

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy