Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives

Last post 12-20-2019, 4:41 PM by Creeves2012. 13 replies.
Sort Posts: Previous Next
  • Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-04-2019, 11:47 AM

    We're consistently seeing issues with our copy to tape jobs inside of Commvault running way below the hardware specs for our LTO6 tape write speeds. 

    Currently, we're seeing about 185GB\HR (52MB\s) for our Commvault copy to tape jobs. 

    Our legacy BackupExec enviroment (which we're replacing with Commvault) is using LTO6 tapes tapes as well. But we're seeing around 400-500GB/HR for our copy to tape jobs there.

    We're using a Spectra T950e tape library, with partitions of LTO6 drives for each media agent (each have their own unique 3-4 drives allocated), connected using 8GB fiber through brocade switches

    Same configuration for the fiber zoning, tape firmware, etc between the BE and Commvault enviroment. The Commvault servers are on newer hardware with flash storage all around, while our legacy BackupExec servers are utilizing spinning disks for their disk storage. We would of expected to see the Commvault servers perform much faster for the tape copies here in this case. 

    Anyone else experience similar issues? Documentation and support has been pretty minimal with tape issues when I've entered in requests previously. 

  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-05-2019, 11:11 AM

    Can you please escalae or talk to support? Few questoins. Are both enviornment have same HBA type and driver revision? Are media type same on both? can you check if you are using 1 MB block size (proeprty of copy -> datapath tab)? are you using H/W compression? is data encrypted?

     

    I think if you talk to support. they can guide you better.

  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-08-2019, 2:30 AM

    Hi Creeves2012

    When investigation issue with slowness in Aux Copy to Tape, you will need to consider all the different hopes and the potential bottleneck. 

    Here are some examples:

    • Read from Source Disk Library 
    • Network between Disk Library and the MediaAgent 
    • MediaAgent over Fiber to the Tape Library 
    • Tape Library write speed. 
    Depending on which component could be the bottleneck, it might not be necessaryly the Tape Library, but could be the source read resulting in the slow Aux Copy performance. 
     
    I would recommend to engage Commvault support so they can review the perfomance statistics and identify the bottleneck, so we can target the cause of your issue. 
     
    Regards
     
    Winston 
  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-12-2019, 9:22 AM

    Hi Creeves2012,

     

    I'm struggling with similar issue.

    I've got a setup with TS4500 with 5 LTO7 drives and the auxcopy jobs to tape are very slow.

    Main source is on 24 spindles, but I've been doing a lot of tests placing source on prod disk library from which I get 2-3TB/Hr during backups and even SSDs. Got very confusing results ranging from 90GB/Hr to 1700GB/Hr from the same source and for the same jobs that been AuxCopied!

    After a loooot of tweaking and tuning I now get throughput ranging from very poor 100 to decent 700 GB/Hr on daily aux copy jobs.

    I think this issue occured first after upgrading from SP11 to SP12 if I remember correctly.

    I have a case open with support for a year now but it looks very hard to troubleshoot since the results are not consistent using the same settings.


    Thanks,
    Robert
  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-13-2019, 11:44 AM

    Please escalate this to support so that Dev can look into details/ root cause analysis.

    Thanks.

  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-16-2019, 1:44 PM

    Hi JNH,

     

    Devs are already looking into this, but as far as I know there is not much they see in logs. It's hard to say if the issue is inside or outside Commvault itself at the moment. We will try to engage HW vendor to help look into what's happening after data is out of Commvault and before Tape.

     

    Thanks.


    Thanks,
    Robert
  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-16-2019, 2:19 PM

    Have you set any multiplexing?  On the Aux copy, Media tab are you combining and multiplexing streams?  

  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-16-2019, 2:38 PM

    Thanks for the replies all. I've entered a support ticket in again to investigate to see what we can do. 

    The other issues we see spawn from these slow tape copies, the queue for the Auxillary copies grow so large, they will never effectively finish, which then fills up our disk storage as the Full jobs are retained until they are copied off to tape. 

    Answering previous replies with the information below. 

    For context, we're running all current drivers and HBA configuration, along with the same media with Hardware Compression for the Auxillary jobs. In relation to the block size configuration, we're set with the option of "Use Media Type Setting" for the aux copy block sizes.

    The read source for the libraries are 12GB SAS disk arrays locally attached to each media agent, which is then connected over 8GB fiber lines to the tape library.  Our media agents showing the worst performance all have 7200RPM read intensive SAS drives, we have our newer builds of media agents utilizing read intensive heavy I\O SSD disks, and are not seeing the problem as bad with those, but that is to be expected with SSD vs HDD performance.

    The interesting part with the disk performance in Commvault with the HDD disks, we are seeing as poor as 70GB/hr to 200GB/hr for Aux jobs for our media agents with the HDD disks. Where with the same model of disks in BackupExec for it's tape copies, we average 400-500GB/HR. (keeping in mind, the backup exec servers are on similar hardware, just older all around. Same 8GB fiber to the tape library, we were expecting to see the same speeds with Commvault here...)

    We're on V11 SP 16, we stay current with Hotfixes and are usually n-1 from major service packs. We actually get better performance with Multiplexing OFF as opposed to turned on. 

    I will continue to follow up and post any helpful results or findings. 

     

    Thanks for the replies all. 

  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-16-2019, 3:04 PM

    We used Multiplexing&combining some time ago, but after some tuning it turned out that switching it off gives best results.


    Thanks,
    Robert
  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-17-2019, 7:55 AM

    Hi Creeves2012 

    Have you applied or changed the following tunables:

    • DataMoverLinkAheadReaderSlots 
    This should help in read intensive operation, as it will be reading more blocks from Disk prior to processing the writes.
     
    Regards 
     
    Winston 
  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-17-2019, 4:09 PM

    I am refraining from making any changes until I hear back from support for ideas. So far nothing yet.

    Below are the tape write speeds over the last 7 days for 6 of my Commvault media agents. Ordered lowest speed to highest speed. 1-5 are identical media agent disks for local storage. The 6th media agent is identical hardware except the HDD's are replaced with SSD disks for storage.

    1. 73GB/Hr
    2. 188GB/Hr
    3. 233GB/HR
    4. 259GB/HR
    5. 282GB/HR
    6. 1100GB/HR *** This is our SSD media agent
    Compared to identical hardware on our BackupExec Servers still online receiving speeds
    1. 450GB/hr
    2. 548GB/hr
    3. 447GB/hr
    The above are the remaining Backup Exec servers we're looking to retire and replace with new Commvault servers. 
    Sharing Media agent hardware, as when we have had performance issues we tend to see support pointing at underpowered hardware as the cause.
    Media Agent Hardware
    • 164TB usable 12GB SAS connected 7200RPM Drives
    • Media agent 06 is (SSD is local direct attached read intensitve disks, still 164TB of usable)
    • 2 PCIE DDB disks for each media agent, with rated Commvault IO Performance results below
    IOps Read IOps Write IOps MBps
    102464.3901 51236.99 51227.4 400.2515
    102464.3901 51236.99 51227.4 400.2515
    • Dual 10GB Network Adapters for Backups of clients data shipping from source to media agent disk
    • Dual 8GB fiber connected to brocade fiber switches,  switches then connected to a Spectra T950E tape libray with 12 LTO7 full height tape drives
    • 190GB of Memory per media agent
    • 2 Intel Xeon Silver 4110 CPU @ 2.1GHZ - 2 sockets, 16 cores, 32 logical processors. 
    • R730XD - R740XD models per.
  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-20-2019, 2:18 PM

    To close the loop for this thread. Support has closed my ticket and pointed not surprisingly at hardware as the problem without any guidance on any of the auxillary copy configuration. 

    Running the CVDiskPerf tests on our media agents and BackupExec Servers we're seeing worse performance with our BackupExec servers with read\write speed than our Commvault media agents hardware, but still the BackupExec Tape copies write 3-4x faster than Commvault to tape copies. 

    Will be evaluating other options outside of Commvault to handle our tape processing and backups forward looking as BackupExec 2015 outperforms it. 

  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-20-2019, 2:50 PM

    What is the support ticket ID? I didn't read it earlier?

    Thanks

  • Re: Commvault Extremely Slow Auxiliary Copy to LTO6 Tape Drives
    Posted: 12-20-2019, 4:41 PM

    191217-46 - is the closed ticket.

    Commvault disks have faster read, and write, but are being out performed by slower BackupExec disks for copies to tape, failing to see how there isn't anything else to investigate on Commvault here in it's handling of copying from disk to tape. Again, these are seperate servers. Tests ran with services stopped and no disk activity aside from this test.

    For comparrision - looking at our active jobs on these servers now. There is an auxillary copy to tape job in Commvault using the disks below currently copying at 19GB/HR. The BackupExec disks with performance specs seen below are also running a copy job to it's tape drives, but at 550GB/HR. 

    CVDiskPerf Results Commvault Disks 

    Throughput Create(GB/H)     : 2150.69
    Throughput Write(GB/H) : 2129.77
    Throughput Read(GB/H) : 790.24
    Throughput Delete(GB/H) : 35909.38
    Throughput Create(GB/H)     : 1854.39
    Throughput Write(GB/H) : 2622.62
    Throughput Read(GB/H) : 707.13
    Throughput Delete(GB/H) : 53955.81

     

    CVDiskPerf Results BackupExec Disks

    Throughput Create(GB/H)     : 1198.44
    Throughput Write(GB/H) : 1246.78
    Throughput Read(GB/H) : 505.29
    Throughput Delete(GB/H) : 30698.37
    Throughput Create(GB/H)     : 1198.32
    Throughput Write(GB/H) : 1408.83
    Throughput Read(GB/H) : 572.30
    Throughput Delete(GB/H) : 30706.82
The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy