Backup slowness

Last post 12-18-2018, 9:48 AM by mmagbee. 6 replies.
Sort Posts: Previous Next
  • Backup slowness
    Posted: 10-31-2018, 8:48 AM

    Hi there,

    I am looking for backup throughput enhancement.After taking an infra backup I have observed following values in the job log(mainly in CVPerfMgr.log).

    Pipe ID	Read[DiskRead]		Pipeline write[NetworkWrite]	SDT: Receive Data	Application Data size Processed Data size	New Data size	Writer: DM: Physical Write	DSBackup: Media Write		

    10785 [246.32 GB][990.80 GBPH] [217.35 GB][455.99 GBPH] [218.03 GB] [1645.52 GBPH] 217.35 GB 205.40 GB 11.53 KB [440.35 MB] [1548.10 GBPH] [468.92 MB] [0.68 GBPH]
    10786 [233.14 GB] [1000.37 GBPH] [223.23 GB] [312.57 GBPH] [223.92 GB] [1733.60 GBPH] 223.22 GB 220.87 GB 10.00 GB [10.53 GB] [3158.37 GBPH] [10.54 GB] [15.18 GBPH]
    It is observed that Pipeline write[NetworkWrite] speed is low ,suggestion solicited for enhancing it.
    Another point:
    It is observed that DSBackup: Media Write is very low as compared with Writer: DM: Physical Write	.
    How to enhance it? What do they signify? Any guidance is appreciated.
    Rest regards.
  • Re: Backup slowness
    Posted: 11-08-2018, 4:43 AM

    Hi Suman 

    In regards to Backup slowness, there are various factors that needs to be taken into account. 

    CVPerfMgr, is definitely a log that covers all the basis. 

    From the logs tha tou have provided, the bottleneck can both be the network between the Client and the MediaAgent (in that case it would be good to do a normal copy and paste between the Client and MediaAgent to confirm the throughput).

    If it is the physical write to the Disk Library, you can also try to do a write test from the MediaAgent to the Disk Library to confirm the speed. 

    The above test are scenario's to rule out any environmental issue. 

    Hopefully that helps. 

    Thank you 

    Winston

  • Re: Backup slowness
    Posted: 11-12-2018, 2:10 AM

    We have 10Gbps networking connectivity between the servers. I mean Backup Proxy Server(Client) and Commvault Media agent. Data goes into ScaleIO storage. We have also done iperf3 speed test, all went well(2412GB/Hr). 

    Any other thoughts will be appreciated.

    One more thing.

    For Virtual Machine backup, what are the most effective performance parameters to be tuned in commvault?

     

    Best Regards

  • Re: Backup slowness
    Posted: 11-12-2018, 3:22 AM

    Hi, I won't dig into performance but just a note - if you have 10G network, keep in mind how TCP works. The only way you can load it is to put more sessions on it (if you can read with this speed & write)

    e.g. enable more network agents on storage policy, running multiple subliclients on the same time and etc.with some tweaks and performance optimizations we were able to achive a succsessfull 13TB backup per our over 10G network.

    note: iperf is not the best tool to test how commvault will work. use commvault tools

  • Re: Backup slowness
    Posted: 11-12-2018, 4:06 AM

    Hi, Thanks for your reply.

    We have tweaked Number of data readers with max. allowed Network agents for Infra. backup ,yet to reach 1TB/Hr backup speed.

    please specify the values and the performance parameters you have fine-tuned to get the 13TB/Hr throughput.

    For Tenant/VM backup what are the performance parameters you suggest to fine tune?

    Rest regards

  • Re: Backup slowness
    Posted: 12-17-2018, 12:01 PM

    By my calculation, a single 10gb link would allow for around 3.6TB / hour

    That aside, if you have deduplication turned on, your DDB may not be able to keep up with such a high rate of data coming in, which will slow the backup down. 

    So assuming your destination disk and source disk can keep up with that speed, take a look at the DDB metrics CommVault shows you when you drill into Storage Resources, Deduplication Engines, then select the de-dupe database or partition you're using for this.  There should be numbers like Average Q&I Time.  

  • Re: Backup slowness
    Posted: 12-18-2018, 9:48 AM

    You are correct, that is the avg for a 1gb link. I have adjusted simliar to have Dedupe to run on MA instead of client. That and a teamed 10gb nic on the source disk.

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy