Maglib - RAID Controller Stripe Size?

Last post 03-04-2012, 8:45 AM by Paul Hutchings. 50 replies.
Page 1 of 2 (51 items)   1 2 Next >
Sort Posts: Previous Next
  • Maglib - RAID Controller Stripe Size?
    Posted: 02-05-2012, 10:26 AM

    What's the recommended stripe size to use when creating a physical RAID volume on which to put your maglib volumes please?

    I know Commvault suggest 64kb NTFS cluster size, not sure what the recommendation on the RAID is though?

    Thanks,

    Paul

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-05-2012, 7:24 PM

    When we had an engineer on site to setup the system he recommended 4k or 8k stripe size but that could have been specific to our site and may not be a general standard. 

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-05-2012, 7:31 PM

    I believe Paul was referring to the actual stripe size on the RAID array which I believe there are no general recommendation as it depends on your controller and RAID level for optimum performance.

     

    I would check with RAID controller vendor on optimum settings for the RAID array for balanced Read/Write.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-06-2012, 7:39 AM

    Correct yes.  The RAID controller is a PERC H800 so presumably the same as that found in a DL2200 appliance?

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-06-2012, 9:10 AM

    I believe you are correct. Would advise if you could check with Dell on the optimum settings for the RAID level you wish to use with respect with stripe size and RAID policies.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-06-2012, 9:25 AM

    Dell should have a best practice white paper of some sort. I know for us (using HP), it ended up being MUCH faster using the biggest strip size possible.

    What I'd suggest is that you try the smallest size, one in the middle and the biggest size and run some benchmark on all 3 configurations. Then you can try and tweak it a bit if need be.

    Phil

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-06-2012, 9:57 AM

    Anyone have an IOMeter profile that roughly mimicks aux copy from dedupe?

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-06-2012, 10:13 AM

    Commvault has build in tools.

    diskread is one and I believe cvdiskperf is another.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-06-2012, 10:17 AM
  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-06-2012, 10:21 AM

    Paul Hutchings:

     

    I have a case open right now because of slow aux copy performance and the Commvault engineers are using these to test my read speeds.

    Example:

    diskread -bkread -path s:temp -verbose

    Outputs info like Throughput: 291 GB/hr

     

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-06-2012, 10:29 AM

    Thank you very much, now I am confused though, the help refers to CvDiskPerf, your post refers to diskread.exe, and if I look for diskread in BOL it links to diskreadwrite.exe

    3 tools, that all do the same thing?! Smile

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 12:52 PM

    Just got the thing hooked up today.

    600MB/Sec doesn't seem bad for 32kb 100% sequential read from a six drive RAID5 of 7.2k SAS, with background initialisation still going on does it?

    Those are from IOMeter.  Of course an Aux copy won't be 100% sequential.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 1:01 PM

    Hutch try these:

     

    CvDiskPerf -READWRITE -RANDOM -PATH c:\test -OUTFILE c:\temp\perf.txt

    Output:

    DiskPerf Version        : 1.3

    Path Used               : T:\

    Read-Write type         : RANDOM

    Block Size              : 512

    Block Count             : 4096

    File Count              : 1024

    Total Bytes Written     : 2147483648

    Time Taken to Write(S)  : 31.45

    Throughput Write(GB/H)  : 228.93

    Total Bytes Read        : 2147483648

    Time Taken to Read(S)   : 16.88

    Throughput Read(GB/H)   : 426.46

    Time Taken to Create(S) : 18.83

    Throughput Create(GB/H) : 382.35

    Next copy 50gb of V_xxxxx files and run this:

    diskread -bkread -path s:\temp\v_xxxxx -verbose

    Output is more sequential:

    Number of files read: 79

    Number of Bytes read: 53573805348

    time taken: 616 Sec

    Throughput: 291 GB/hr

     

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 1:14 PM

    Paul Hutchings:

    Just got the thing hooked up today.

    600MB/Sec doesn't seem bad for 32kb 100% sequential read from a six drive RAID5 of 7.2k SAS, with background initialisation still going on does it?

    Those are from IOMeter.  Of course an Aux copy won't be 100% sequential.

    It would be nice to get ratios from Commvault but I'd expect an aux copy to be close to 100% random actually. When you think about it, it access deduped blocks spread all over the disks and you probably use more than 1 stream at a time adding to the random stress. Personally, I did my tests using 100% random. I don't think it would be 100% either but I'd expect it to be closer to the truth.

    If anyone from Commvault can actually comment on this, it'd be great. I'm sure there are some stats on what the block sizes, randomness, read vs write ratios are.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 1:20 PM

    PhilippeMorin:

    Paul Hutchings:

    Just got the thing hooked up today.

    600MB/Sec doesn't seem bad for 32kb 100% sequential read from a six drive RAID5 of 7.2k SAS, with background initialisation still going on does it?

    Those are from IOMeter.  Of course an Aux copy won't be 100% sequential.

    It would be nice to get ratios from Commvault but I'd expect an aux copy to be close to 100% random actually. When you think about it, it access deduped blocks spread all over the disks and you probably use more than 1 stream at a time adding to the random stress. Personally, I did my tests using 100% random. I don't think it would be 100% either but I'd expect it to be closer to the truth.

    If anyone from Commvault can actually comment on this, it'd be great. I'm sure there are some stats on what the block sizes, randomness, read vs write ratios are.

    nspljoe, thank you, much appreciate it.

    Phillippe, agreed.  Problem is I can run Commvault tools and if all it's testing is sequential...

    Or I can run IOMeter but then what profile should I be using?

    I figured it will be a mix of random with "bursts" of sequential?  I shall be doing all I can to get my data spread across as many mount paths as possible but it makes a huge difference to any strategy on how to carve up the physical disks if I can't get an idea of real-world performance rehydrating deduped data.

    So far all I can do shoot in the dark and aim to meet the building block specifications so anything any of the Commvault guys can add would be great Smile

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 1:25 PM

    Hutch,

    Its worse then that. As I am finding out the IO is almost all random. It doesn't help that the way Commvault writes to mount paths, data age, write, data age, all in dedup fashion fragments files very badly. I dont think there is a test you can run which will simulate a new SP/Maglib/DDB or one that is 3 months old.

    I would run the Commvault tools for baseline as that is what Commvault uses in its troubleshooting. I would also try IOmeter with a profile of 64kb reads, 90% random, with as many threads as your aux would have.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 1:37 PM

    nspijoe:

    Hutch,

    Its worse then that. As I am finding out the IO is almost all random. It doesn't help that the way Commvault writes to mount paths, data age, write, data age, all in dedup fashion fragments files very badly. I dont think there is a test you can run which will simulate a new SP/Maglib/DDB or one that is 3 months old.

    I would run the Commvault tools for baseline as that is what Commvault uses in its troubleshooting. I would also try IOmeter with a profile of 64kb reads, 90% random, with as many threads as your aux would have.

    Lovely stuff it just gets better Laughing

    Tbh I took a simple view on speccing up this thing:

    Right now everything is on a single 16 spindle SATA Equallogic in RAID6.  

    Aux Copies run at around 150-200gb/hr depending on which storage policy and so on, sometimes more, sometimes less - which isn't great but isn't terrible either.

    Point being that's with iSCSI and MPIO in the way and all the disks in one big disk group/RAID type.

    So the new box more than meets the building block criteria, and I'm assuming that with 24xSAS spindles (still 7.2k) and being able to split them up into separate RAID groups that I'm certainly not going to be any worse off.

    The biggest challenge, I think, will be ensuring that I get the aux copy to populate the thing spread out across all the spindles as good as I can.

    Our biggest backup jobs are Exchange (600gb or so) and our main file server, which is a synthetic full of around 8tb.

    Something Phillippe said on another thread scared the crap out of me, which was that because a Synthetic Full (we do DASH Fulls) is written as a single stream, it's aux copied to tape as a single stream - which suggests that for my single biggest backup I could be dependent on the throughput of just one RAID group at a time when going to tape - this makes me seriously question how to carve up those 24 disks (4x5+1 RAID5, 2x10+2 RAID6 etc.).

    I know I start a lot of threads on this, but the reasons are simple - lack of hard fact from Commvault and the whole "get it wrong and your backups will suck forever" factor Smile

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 1:40 PM

    Stuck on the phone right now so not a whole lot of time to comment but just to be clear, that 1 stream from synthetic full to 1 stream aux copy was for disk to disk. Not sure it applies to tape.

    Back in a bit... interesting stuff being said.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 2:46 PM

    PhilippeMorin:

    Stuck on the phone right now so not a whole lot of time to comment but just to be clear, that 1 stream from synthetic full to 1 stream aux copy was for disk to disk. Not sure it applies to tape.

    Back in a bit... interesting stuff being said.

    Thanks Phillippe, and I guess with a copy to tape the "to tape" part will only be 1 stream per physical tape drive so maybe that bit isn't so much of a problem as I'd first thought.

    I think all my comments and general confusion still stand though about how the heck to get the data sufficiently distributed across all 24 spindles that I'm making good use of them al.

    What seems to happen now is that when an aux copy starts, I get several streams copy to tape in parallell, then the smaller jobs finish copying and I'm left with the 8tb file server aux copy.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 3:02 PM

    White papers (Engineering Docs) on the DL2200 Powered by Commvault Simpana.

    http://support.dell.com/support/edocs/stor-sys/pvdl2200/comvault/en/index.htm

     

     

    Dell PowerEdge RAID Controller Cards H700 and H800 - Technical Guide

     

    http://www.dell.com/downloads/global/products/pvaul/en/perc-technical-guidebook.pdf

     

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 3:02 PM

    Paul Hutchings:

    What seems to happen now is that when an aux copy starts, I get several streams copy to tape in parallell, then the smaller jobs finish copying and I'm left with the 8tb file server aux copy.

    Then you might be seeing somewhat the same behaviour as disk to disk since your 1 stream eventually gets fed by only 1 stream instead of being able to do a 10+ to 1 stream.

    Paul Hutchings:

    I think all my comments and general confusion still stand though about how the heck to get the data sufficiently distributed across all 24 spindles that I'm making good use of them al.

    Unfortunately, beside the spill and fill option, there isn't much control over this. You could always go kind of crazy at first and disable write on some mount paths so that it is spread out properly but that would be a pain to maintain.

    Also, it doesn't mean that because you use all your mount paths at lets say 50% capacity that the load is actually spread evenly. You could still have all the blocks in the first mount path being used in all your backup jobs (going with an extreme example here) and accessed 80% of the time while all the other unique blocks on the other mount paths would barely be accessed because even though they're unique, they aren't shared with many jobs. I think the best you can do is just setup your hardware the best you can (with HP, I found that 1 raid group per mount path was giving the best results), enable spill and fill and then hope for the best! Smile

     

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-07-2012, 3:31 PM

    PhilippeMorin:

    Paul Hutchings:

    What seems to happen now is that when an aux copy starts, I get several streams copy to tape in parallell, then the smaller jobs finish copying and I'm left with the 8tb file server aux copy.

    Then you might be seeing somewhat the same behaviour as disk to disk since your 1 stream eventually gets fed by only 1 stream instead of being able to do a 10+ to 1 stream.

    Can any of the Commvault guys reading confirm this please?

    How can I do Synthetic Full DASH copies but then make best use of multiple streams when going to tape?

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 5:55 AM

    Right a little progress - right now I have an aux copy running.

    It shows a single stream on the "streams" tab.

    However, if I run Process Monitor and monitor disk activity, I can see that auxcopy.exe is pulling data from across all of my maglib volumes.

    Which suggests that it's not as clear cut about which stream/maglib data is being pulled from.

    Paul

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 7:24 AM

    Hi Paul,

     

    A lot of topics have been discussed here regarding performance so apologies if I stray a little from what you require.

     

    1. You want guidelines on performance from CV.

    This is kind of hard to do when CV does not control the hardware aspect of the backup process. From our development perspective, we need to be robust enough that the product works under a whole multitude of hardware configurations as well as trying to do this at a decent speed.

    To achieve this, we reply heavily on the OS abstraction layer. To perform the reads, we call read requests to the OS which then performs the requested job etc. You can imagine a lot of performance variation can occur here when customers have completely different hardware. That's why baseline performance figures are rarely given out as this may set the wrong expectation over something we do not control.

     

    2. Random vs sequential reads/writes

    Deduped CV chunk data writes references down into the chunk metadata files on where the deduped chunk resides. In a way all requests to read a particular chunk may actually traverse a number of directories in order to get there to perform the read. In most cases, raw throughput is not the correct figure to be judging how good/bad a RAID array would perform. I would recommend performing IOPS tests to get a more accurate picture of how this RAID array would perform in real world use.

    E.g. During a DASH copy, the source disk array of 12 drives in RAID6 may only see actual throughput of 2-3MB/s per disk but IOPS has already reached 80-100 per disk. The number of operations a single disk can handle has a direct and usually inversly proportional relationship to the throughput. This is also the reason why most storage vendors quote on IOPS and not throughput MB/s as it is not a true indication of real world usage.

     

    3. Streams!!

    Ok so I understand there is a lot of confusion out there on streams. Phillipe is correct in saying that a single backup using 1 stream will only be able to use 1 stream during aux copy. I know this causes issues with large backup sets but unfortunately the only way I see around it is by using many readers for the backup job in the first place and potentially turn on "Allow multiple readers per mount point" option.

    Eg. Your 8TB backup job using 1 reader will be assigned 1 stream ID as the job is written down. When Aux Copy comes in, because the job has 1 stream, this entire stream will be allocated within one Aux Copy stream. Now if your backup job used 4 readers, it will have 4 stream ID's assigned. Depending on the size of each stream, the Aux Copy job can now break this apart and allocate an appropriate amount of streams to copy this job UP TO 4.

    This same stream number lock applies to Media Refresh. If two tapes were written to with a different stream numbers, during refresh it cannot combine these into 1 tape and stream number. It has to stay consistent. A workaround for this would be to create a additional secondary copy, select the jobs from both tapes to copy and set this copy to combine to 1 stream. This way the destination tape will now have a stream number of 1.

    If you performed the Aux Copy with combine to streams in the first place, you will limit the amount of stream numbers used onto tapes. (eg. combine to 2 streams means there will only exist tapes with stream numbers 1 and 2 greatly alleviatiion the problems during media refresh).

     

    I hope this clears up a few things for everyone. Let me know if you have further questions. It's almost midnight where I am so apologies if I don't respond quickly.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 7:34 AM

    Hi Jordan,

    First off, a huge thanks - that's a very useful perspective on it.

    I can totally see your point about there not being a one-size fits all approach to it.

    If I'm honest I guess where there's a little bit of frustration is that it seems like I have 24 spindles in front of me, and there just doesn't seem a way to do a benchmark of average real-world performance for re-hydrating so I'm kind of left guessing based on benchmarks of tests that may or may not accurately simulate our situation.

    I do get why every situation is different, but you see why it's slightly frustrating hopefully Smile

    Now, on the streams/readers thing.  For our main file server I've always used the "allow multiple readers" option on the backup - I guess this may explain why I'm seeing activity on all the mount paths in my maglib even when the aux copy job appears to show only one stream going to tape?

    Once again, thanks, that reply has filled in a lot of the gaps.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 7:55 AM

    Hi Paul,

    I see the frustration from a customer perspective where if a config does not work it is not CV's fault the performance is bad.

    Let's try to limit this a bit by laying down some general guidelines and real world situations that we have encountered in support.

    1. RAID5 vs RAID6

    Generally we see much better performance in RAID5 then RAID6. I understand RAID6 will give you an extra disk redundancy but you could half offset this by having a hotspare.

    Eg. if you suffered a single drive loss, it would not matter if you were in RAID5 or RAID6 as both will be in degraded/rebuild mode and performance suffers. If you suffered from 1 drive loss followed by a second drive loss shortly after, if your RAID5 has rebuilt then no biggie, and the RAID6 is again merely chugging along and rebuilding. If you suffer the rare occurance of 2 simultaneous drive losses then only RAID6 here will allow no data loss.

    The upside of RAID5 however is the much faster performance as there is much less parity calculation overhead compared to RAID6. Whether you choose 5 or 6 is a business decision you need to make and judge based on what you think is more likely to occur for drive failures.

     

    2.  Judging performance, what is good and what is bad.

    Since you have 24 drive lying there, it's the best time to do some testing :)

    What you need to consider are:

    a) how many RAID arrays will I create 

    b) what RAID level should I use

    For a) the performance would usually be better when you have less RAID sets as there's less parity calculations. It does however mean in case of a multi-drive failure, you are likely to lose a lot more data. For b) as I previously mentioned, depends on business requirements.

    You may want to draw up what the extremities of the business requirements are (eg. how leanient would management allow vs how strict management would require). Based on this, draw up two scenarios of configuration to meet these 2 extremities and see what the IOPS performance is like using IOMeter and other similar tools. From here you usually would have a baseline of how low the performance would be and how high it may be. Usually the business requirement will be somewhere in between so adjust accordingly.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 8:09 AM

    Tbh my concern is read not write - write speed isn't an issue even with the SATA RAID6 Equallogic with the maglib and DDB all on the same spindles Smile

    Aux copy of dedupe data is my single biggest concern here as I want to be feeding the tapes as fast as I can.

    From the IOMeter testing that I've done so far, it seems the two main options are:

    • 4 x 5+1 RAID5 volumes
    • 2 x 10+2 RAID6 volumes

    Sequantial read performance flies with either as you'd expect, but that's not relevant here.

    Random read performance seems to be about equal, which I guess you'd expect as it's the IOPS per spindle that are the bottleneck.

    From a non-Commvault perspective the RAID5 option gives me more scope if I ever need to change things around.

    RAID5 is acceptable for this IMO as the data will always be on tape anyway, so the odds of losing the production server, and losing the backup server before the data has hit tape are slim.

    Is there a suggested way to migrate data to the new MA to optimze the spread of data, other than to simply set "spill and fill"?

    Thanks,

    Paul

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 8:47 AM

    Spill and fill will only work properly if you have enough streams running to utilize all the mount paths. The best way to fill the mount paths evenly would be to use spill and fill as well as using equal or more streams than there are mount paths.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 8:56 AM

    jordanxu:

    Spill and fill will only work properly if you have enough streams running to utilize all the mount paths. The best way to fill the mount paths evenly would be to use spill and fill as well as using equal or more streams than there are mount paths.

    So if I have 16 mount paths use 16 or 32 (sure I read use a multiple of 2 somewhere?) streams on the storage policies?

    Is there a way to specify how much data to write to a mount path before randomly choosing another?

    I'm assuming that with the large client that once it starts to write data to a set of mounth paths it won't stop until those mount paths are full?

    This may not be the best way of explaining it, but if I have 16 mount paths:

    uuuu uuuu uuuu uuuu

    When I run the aux copy to do the initial copy, I don't want to end up with all the big data spread unequally like this:

    uuUu uUUu UUUu uuuU

    If that makes sense Smile

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:02 AM

    I believe you are correct with the multiple of 2.

    So for your intial aux copy from current disk to new disks with 16 mount paths, I would set 32 or more streams in the Storage Policy then kick off the Aux Copy job with 32 streams. This way it should use 2 streams per mount path and will try to be as evenly spread as possible based on the Aux Copy setting of trying for 500GB per stream. 

    Of course if you had 1 stream that is really large, it may not end up being completely evenly spread out unfortunately.

    I don't believe there are any ways of telling aux copy to change mount path as a certain usage point as everything is written down based on streams.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:05 AM

    jordanxu:

    I believe you are correct with the multiple of 2.

    So for your intial aux copy from current disk to new disks with 16 mount paths, I would set 32 or more streams in the Storage Policy then kick off the Aux Copy job with 32 streams. This way it should use 2 streams per mount path and will try to be as evenly spread as possible based on the Aux Copy setting of trying for 500GB per stream. 

    Of course if you had 1 stream that is really large, it may not end up being completely evenly spread out unfortunately.

    I don't believe there are any ways of telling aux copy to change mount path as a certain usage point as everything is written down based on streams.

    Thanks Jordan, so, how is this idea for a "bodge"? Smile

    Create LUNs of 500gb, 16 or so.  500gb is small so if they fill before a stream finishes it will randomly choose another LUN to continue.

    When the aux copy finishes, extend the volumes to the full 2tb.

    Bodge, but should work?

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:10 AM

    Paul Hutchings:

    Thanks Jordan, so, how is this idea for a "bodge"? Smile

    Create LUNs of 500gb, 16 or so.  500gb is small so if they fill before a stream finishes it will randomly choose another LUN to continue.

    When the aux copy finishes, extend the volumes to the full 2tb.

    Bodge, but should work?

    I believe you may be onto something there. Your theory should technically work provided 8TB minus the reserve space for each mount path is enough space for the intial copy. :)

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:23 AM

    jordanxu:

    Paul Hutchings:

    Thanks Jordan, so, how is this idea for a "bodge"? Smile

    Create LUNs of 500gb, 16 or so.  500gb is small so if they fill before a stream finishes it will randomly choose another LUN to continue.

    When the aux copy finishes, extend the volumes to the full 2tb.

    Bodge, but should work?

    I believe you may be onto something there. Your theory should technically work provided 8TB minus the reserve space for each mount path is enough space for the intial copy. :)

    I'm assuming even if I did the sums wrong, just as with tape when the maglibs are full the aux copy would go pending until there was more space, wouldn't it?

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:25 AM

    jordanxu:

    3. Streams!!

    Eg. Your 8TB backup job using 1 reader will be assigned 1 stream ID as the job is written down. When Aux Copy comes in, because the job has 1 stream, this entire stream will be allocated within one Aux Copy stream. Now if your backup job used 4 readers, it will have 4 stream ID's assigned. Depending on the size of each stream, the Aux Copy job can now break this apart and allocate an appropriate amount of streams to copy this job UP TO 4.

    Wow, you guys have been pretty busy since last night! Glad to see Commvault participating actively to this thread Smile

    Jordan, could you clarify the stream usage regarding synthetic backups? You example is for a regular full but I'd love confirmation that the synthetic is indeed only 1 stream and that the only work around would be to create more subclients since you can't tell the synthetic job to use multiple streams.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:28 AM

    Paul Hutchings:

    I'm assuming even if I did the sums wrong, just as with tape when the maglibs are full the aux copy would go pending until there was more space, wouldn't it?

    Yes indeed it will go pending with no resources available error.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:31 AM

    PhilippeMorin:

    Wow, you guys have been pretty busy since last night! Glad to see Commvault participating actively to this thread Smile

    Jordan, could you clarify the stream usage regarding synthetic backups? You example is for a regular full but I'd love confirmation that the synthetic is indeed only 1 stream and that the only work around would be to create more subclients since you can't tell the synthetic job to use multiple streams.

    I will need to find out tomorrow as my knowledge on Synthetic Fulls is limited. Will update once I discuss with some engineers from the other teams. Smile

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:31 AM

    Paul Hutchings:

    Right a little progress - right now I have an aux copy running.

    It shows a single stream on the "streams" tab.

    However, if I run Process Monitor and monitor disk activity, I can see that auxcopy.exe is pulling data from across all of my maglib volumes.

    Which suggests that it's not as clear cut about which stream/maglib data is being pulled from.

    Paul

    Well, even 1 stream will need to read blocks spread all over your volumes because of dedup. I'm pretty sure process monitor will show the average disk activity that occured over a short lap of time so in reality, even if the 1 stream is writing sequentially to the volumes, it will show up in your monitor as simultaneous access. Also, I would think the look ahead key will do exactly that: read ahead the data thus simulating multiple streams in the monitor.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:34 AM

    jordanxu:

    Paul Hutchings:

    I'm assuming even if I did the sums wrong, just as with tape when the maglibs are full the aux copy would go pending until there was more space, wouldn't it?

    Yes indeed it will go pending with no resources available error.

    That sounds perfect.  Thinking about it, I don't know how good/bad Windows would be at growing partitions when they have other adjacent partitions.

    So the sensible option would seem to be to go with 2tb partitions in Windows, and use the "Reserve Space" setting to control how much is actually used.

    I feel like I have more of a plan now Smile

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:35 AM

    Paul Hutchings:

    So if I have 16 mount paths use 16 or 32 (sure I read use a multiple of 2 somewhere?) streams on the storage policies?

    I wasn't aware of this best practice. Where did you read this and do you know the reasoning behind it?

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:37 AM

    PhilippeMorin:

    Paul Hutchings:

    So if I have 16 mount paths use 16 or 32 (sure I read use a multiple of 2 somewhere?) streams on the storage policies?

    I wasn't aware of this best practice. Where did you read this and do you know the reasoning behind it?

    Have a look at this thread, posts by Zahid in particular:

    http://forum.commvault.com/forums/8663/ShowThread.aspx#8663

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 9:54 AM

    Hutch,

    I have thought a bunch about this but havent been able to test like you *may* be able to.

    I was actually thinking for random IO it would be better to have a bunch of smaller raid sets to service the IO. I could be wrong in how modern controllers work but I would think it operate like this:

    request 1, r2, r3 ,r4 ,r5, r6, r7, r8

    Now if all of those are destined for 1 raid 5 set with 10 (9+1) drives say, it will take those requests and work them sequentially. Now it each request is on differen't sectors of each drive or spread across multiple drive the limiting factor in a IO contrainted enviroment and not raw throughput would be seek time to the correct location. The seek time will be limiting factor in a 7.2k drive to service the request, then the next, and next and next.

    Now if you have those same 8 requests in 5 raid 1 sets and if multiplexing and random streams distributed them evenly, those same 8 requests could be serviced in the time 1 request is serviced in the first scenario.

    ^^^^ all of that above is based on my loose understanding of disk/san/IO so it could be all wrong.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 10:15 AM

    jordanxu:

    I believe you are correct with the multiple of 2.

    So for your intial aux copy from current disk to new disks with 16 mount paths, I would set 32 or more streams in the Storage Policy then kick off the Aux Copy job with 32 streams. This way it should use 2 streams per mount path and will try to be as evenly spread as possible based on the Aux Copy setting of trying for 500GB per stream. 

    Of course if you had 1 stream that is really large, it may not end up being completely evenly spread out unfortunately.

    I don't believe there are any ways of telling aux copy to change mount path as a certain usage point as everything is written down based on streams.

    I think this is somewhat wrong (hopefully you aren't and I'm the one mistaken!). Let me explain.

    As we know, the aux copy will use the same number of streams as the original job. It will also use the same stream IDs.

    To my knowledge, Paul uses a lot of synthetic fulls resulting in many big job all using a single stream. A single stream cannot (I think?) be spread accros mount paths. This can be easily seen when looking at the synthetic full jobs on your storage policy primary copy. If you right click one of your huge jobs and select view media, you will notice that the job is using a single mount path. I'm not sure if this is a limitation for backups jobs only or if it would behave the same on secondary copies.

    Now, lets say you set everything up and start your aux copy, it will most likely start copying synthetic fulls (as your real fulls have probably aged by now?) meaning that your baseline (unique blocks) of one of your big synthetics will be written to a single mount path. Of course, the other mount paths will also fill up as you will use multiple streams for your aux copy and you'll have other jobs waiting to be copied but that one huge synthetic will still remain an issue. Now, I'm unsure how the aux copy will react if there isn't enough space on the mount path for that one stream. Will it be able to switch to a new mount path or will go pending?

    The only workaround I could think of is to schedule a real full on all your servers using multiple streams and then set the aux copy to copy only jobs based on the date of your fulls. Once the baseline is completed, you would be able to set the aux copy to copy all other jobs since now your previous synthetics would only need to copy unique blocks, not the whole baseline.

    Hopefully this makes sense!

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 10:25 AM

    Just to confirm a "View Media" does show only one mount path - though of course the data has to be spread across the disks as an 8tb client is backing up to 2tb mount paths.

    I guess the devil will be in the detail here...

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-08-2012, 10:30 AM

    You're right. And only a few GBs were probably written to that mount path. But I still find it somewhat odd that streams always seem to be on a single mount path. Also, have you never got one of those errors which is something like "Job going pending because the mount path is busy. Can't use another mount path on multiplexed jobs". It's definitely not that message but something around those lines which tends to point to a single mount path per stream.

    But, I could be wrong, just something I noticed while looking at my jobs!

    Hopefully Jordan will have some insights on this. Smile

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-09-2012, 6:54 AM

    On a slightly separate note but related to the migration -

    What's the suggested/best way to change the dedupe block size?

    At the time of setting up dedupe everything was set to 64kb other than my VMware policy which was set to 32kb.

    The Building Blocks white paper seems to suggest using 128kb.

    I'm not desparate to change it, but I'm not sure if there is a significant benefit in doing so, or not doing so?

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-14-2012, 1:04 PM

    I feel like this is a bit of a diary/blog but the last bit of kit (the SSD for the DDB) arrived today Smile

    So far my maglib performance is looking pretty good using IOMeter and SQLIO to simulate random read performance.

    The SSD is showing around 13k random 4k write IOPS but I haven't yet filled it or done anything to test worst case scenario performance, which I guess is a nearly full SSD?

    I'm really caught in two minds on whether to mess around running an aux copy, or to just say "what the hell" and start from scratch and run the old and new MA in parallel and let the maglib data age off the old MA whilst it builds up on the new.

    • I have concerns about balancing data if trying to do an aux copy.
    • I'd like to change the dedupe block size to 128kb
    • If I'm doing all of that it throws up questions about whether to look at global dedupe as per the building blocks white paper.

    Ho hum a fun few days....

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-20-2012, 2:02 PM

    Here's where I'm currently at:

    New MA is setup with 16x2tb mount paths.

    DDB is on SSD

    I set the MA to "spill and fill" and set 2 writers maximum per mount path and enabled stream randomisation.

    I created a Global Dedupe policy with 128kb block size - from various advice I don't believe this will leave me any worse off than using a dedicated DDB per Storage Policy.

    Now, this is where I had a change of plan and got creative. Smile

    On my new MA I have a bunch of 1tb "leftovers" on each physical RAID group, so I created a (temporary) 4tb stripe using Windows Disk Manager.

    I then started to restore data to it from our existing MA.

    I then backed that data up again, a bit at a time, using a dedicated subclient on the FS iDA installed on the new MA that uses six random streams.

    I then repeat with some fresh data...

    Net result is that I am able to "pre-seed" my new MA with most of my data so that when I flick the storage policy copies over to the new maglib, the forced Full backup is fairly quick.

    It also seems to give me a little more/better control over getting the data spread over the mount paths on the new MA.

    Early days, but so far, happy bunny Smile

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-20-2012, 2:17 PM

    Good to hear it's going well so far! Did you end up creating only 1 raid group? What strip size did you choose? What raid level?

    When the move is complete, I'd be curious to know your new throughput for synthetic fulls and aux copies if you don't mind! Smile

    Phil

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-20-2012, 3:02 PM

    PhilippeMorin:

    Good to hear it's going well so far! Did you end up creating only 1 raid group? What strip size did you choose? What raid level?

    When the move is complete, I'd be curious to know your new throughput for synthetic fulls and aux copies if you don't mind! Smile

    Phil

    So far 4 x 5+1 RAID5 sets.

    I did deliberate going with RAID6 but decided the disk penalty was too great so it's a compromise but an acceptable one considering everything goes to tape anyway.

    I went with the 64kb stripe size.

    I've not yet done a Synthetic Full, did do a quick aux copy earlier and I think it was up at around 350gb/hr to an LTO4, though it was "only" a small 200gb or so job - that was with it set to combine to 1 stream with a multiplexing of 4.

    I've not played with tweaking any of the aux copy stuff yet though, all I've done is literally point the existing tape copy at the new library, and enable the look ahead reader key on the new MA.

    I'll certainly report back when I have a few larger jobs and when I'm in a position to do a Synthetic Full and a few larger Aux Copies such as those of our full main server backup - just depends if I get the pre-seed done for this weekend or not.

    Paul

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 02-28-2012, 1:03 PM

    If anyone from Commvault can clarify how Synthetic Fulls work in combination with streams I would be very grateful.

    The situation right now is that my initial backup of my main file server to the new MA had to be a full.

    The full had 8 streams, which, combined with all the pre-seeing that I did, seems to have left me with a nice balance of data over all of my mount paths.

    When I did the aux to tape I combined to 1 stream with a multiplexing factor of 4 and I was basically driving a single tape drive at full speed (haven't tried 2 streams to use both tape drives yet).

    Anyway, what will happen with the Aux Copy on Friday when I will now be running a Synthetic Full backup?

    Obviously the source data will still be scattered across all of the mount paths, but I'm unclear on the situation regards streams - if a Synthetic Full is a single stream that suggests I cannot write it across two tape drives at once?

    Thanks very much.

  • Re: Maglib - RAID Controller Stripe Size?
    Posted: 03-04-2012, 8:45 AM

    PhilippeMorin:

    Good to hear it's going well so far! Did you end up creating only 1 raid group? What strip size did you choose? What raid level?

    When the move is complete, I'd be curious to know your new throughput for synthetic fulls and aux copies if you don't mind! Smile

    Phil

    Phil, FYI now I've done a few Synthetic Full backups and a few Aux copies I'm seeing around 900-950gb/hr on my big Synthetic Fulls and around 120MB/Sec to each tape drive on the aux copies.

    I've only had a couple of instances where I've had an Aux Copies from different policies running in parallel but it looks like I can sustain that to two aux copies at once, which was my primary concern with laying the libraries out.  

    This is using a single Global Dedupe Policy with 128kb block size.

    So far, happy camper Smile

Page 1 of 2 (51 items)   1 2 Next >
The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy