Throughput/bottleneck

Last post 07-15-2010, 12:03 PM by RKBrower. 7 replies.
Sort Posts: Previous Next
  • Throughput/bottleneck
    Posted: 07-12-2010, 12:25 PM

    I am not able to get more than 1GB of total throughput for aux copies or normal backups.  I have teamed my nics and have multiple client backing up.  Im backuing up to a DataDomain (disk storage group) I was even using a 10GBe connection at one time.  Anyone else have similar issues?

  • Re: Throughput/bottleneck
    Posted: 07-12-2010, 12:33 PM

    Do you have the capabilities to run a backup to a different device other than the Data domain?  This would rule out a potential read\write issue with the data domain.

    You could also try to copy a large file from the client to the media agent via UNC path to test network performance.

    To many variables at this time and we need to figure out where the bottleneck is.


    CommVault Messaging Support
  • Re: Throughput/bottleneck
    Posted: 07-12-2010, 1:11 PM

    Check you NIC utilization during backups and AuxCopy operations.  If it is showing almost no usage compared to the amount of data trying to go through it then this solution may apply.  Note you will usually only see this slow down when more then one job is running.

     

    When troubleshooting throughput issues of backups going to Data Domain devices have the customer check to see if ntfs-acls is enabled on the Data Domain.  Disabling this option has been confirmed to fix throughput issues on two separate cases.

     

    There's a new option in version 4.5.1.2 of the Data Domain OS called ntfs-acls. Having this option enabled (default is enabled) causes a lot of overhead when running multiple backup jobs, and causes extremely poor performance.  The solution was to run the following commands on the DDR:

     

    cifs disable cifs option set ntfs-acls disabled cifs enable

     

    "We can now run multiple backup jobs to our DDRs, and the MediaAgent NIC throughput is over 95%. "

     

     

     

     


    Attachment: FileSecurityWACLs.png
  • Re: Throughput/bottleneck
    Posted: 07-12-2010, 1:13 PM

    Have you tried just copying a large file from your CV box to your DD box? What kind of throughput to you get? Have you looked at the NIC drivers? Are they outdated?

    Just a few suggestions.

  • Re: Throughput/bottleneck
    Posted: 07-12-2010, 1:31 PM

    Hello BRichter,  Many good replies so far.   Also do you have 'Clean Jobs' scheduled to run at any time backups or auxcopies are trying to read or write?  I have seen everything come to a crawl if our software tries to read or write to a Data Domain device while a Clean Job is running.

    Performance issues can take some time to isolate.


    Mark Spencer
    CommVault, Business Critical Support
  • Re: Throughput/bottleneck
    Posted: 07-13-2010, 11:04 AM

    Hello,  There is also a possibility it  may not be related to the DD at all.  You may also want to investigate if TCP Chimney Offloading or ToE is in use in your environment.  This can cause your backups to run much slower because of the impact it has on the network interfaces for clients and mediaagents.  We have at least one kb that touches on this issue.  At the bottom of the kb their are links to some Microsoft articles.  This is just one example of the issues it can cause.  If backup streams are contantly being interrupted at the client or mediaagent level it could force the backup to start at the beginning of the chunk (in the case of indexed based backups like File System).  This will bring down the overall throughput of the job.

    https://ma.commvault.com/KB/Details/11138 

     


    Mark Spencer
    CommVault, Business Critical Support
  • Re: Throughput/bottleneck
    Posted: 07-15-2010, 10:53 AM

    Hello,  Are you having any luck on this issue.  The issue would seem to be something with MediaAgent, Data Domain or the network between since you have the same slow issues with Auxcopy which is reading directly from the Data Domain.  Something also to consider is if you are teaming your NICs on the MediaAgent or Data Domain we have had customer report performance issues and sometimes turned out to be the teamed NIC configuration.   If this is the case could you break the team as a test to see if performance improves? 

     


    Mark Spencer
    CommVault, Business Critical Support
  • Re: Throughput/bottleneck
    Posted: 07-15-2010, 12:03 PM
    Good day Current documentation from DD associated with integration with CommVault
The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy