Strong slowdown in backup when using deduplication

Last post 12-04-2018, 1:37 AM by pasko@mkb.ru. 13 replies.
Sort Posts: Previous Next
  • Strong slowdown in backup when using deduplication
    Posted: 11-26-2018, 9:12 AM

    Hello

    There is a slowdown when backing up using the deduplication. DDB is located on Intel SSD NVMe P3700 fast disk, but almost 100 million Unique Blocks are added to DDB every week after backups. Currently, the number of Unique Blocks is 400 million.

    Are there any recommended values for the number of Unique Blocks in DDB, the excess of which causes performance degradation?

    Aleksandr

  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-26-2018, 9:53 AM

    I noticed that if you are simultaneously backing up 2 or more to this DDB, then the slowdown is very strong.

  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-26-2018, 11:33 AM

    DDB doesnt work on database backup or on databbase dump backup. there are couple of performance tunning stuff you can give a try at to improve DDB ratio as well the backup performance

     

    1. use DDB only on File system backup, but exclude any database file

    2. Dont try to use DDB on encrypted data, it wont help and will increase the backup time

    3. Use DDB while Auxing data from onsite to offsite 

    4. use DDB on media agent instead of client machine incase the there are large number of files getting backed up. 

    5. After one traditional full backup use synthetic full backup method, that will improve your DDB performance. 

    6. Use same block size while configring policies. 

  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-26-2018, 11:35 AM

    Just to add

    7. Split the DDB if the size has increased

    8. Dont run DDB backup when normal backup jobs related to it are running. 

  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-27-2018, 12:39 AM

    Hi Aleksandr

    Adding on to Hussain's information, other on-going house-keeping such as Pruning (as part of Data Aging) occurs on the DDB and the Disk Library, and it is during these interval of Pruning (every 60 minutes), coupled with Backup load could severly degrade the performance. 

    You can also consider Operation Windows to schedule time frames where Physical Pruning will occur. Documentation - https://documentation.commvault.com/commvault/v11/article?p=6611.htm

    Thank you 

    Winston 

  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-27-2018, 8:20 AM

    Hi Aleksandr,

    A few questions:

    What are the Media Agent specifications?
    What is the Q&I time shown for this DDB?
    What is the application size for this DDB?
    What is the amount of Pending delete records for this DDB?

    There is a theoretical limit of 750,000,000 unique blocks for a DDB/partition.
    Reaching this limit might suggest performance impact, but this is related to more than just the amount of unique blocks.


    Jos Meijer
    Senior Technical Consultant
  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-28-2018, 3:55 AM

    Hello

    >> 1. use DDB only on File system backup, but exclude any database file

    a separate DDB is used for File backup, a specially dedicated DDB is used for Oracle backup

     

    >> 2. Dont try to use DDB on encrypted data, it wont help and will increase the backup time

    for backup encrypted data do not use the DDB

     

    >> 3. Use DDB while Auxing data from onsite to offsite 

     

    >> 4. use DDB on media agent instead of client machine incase the there are large number of files getting backed up. 

    use deduplication only on Media Agent

     

    >> 5. After one traditional full backup use synthetic full backup method, that will improve your DDB performance. 

    We do not use synthetic backups, as there were problems with them

     

    >> 6. Use same block size while configring policies. 

    use default block size 128K

  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-28-2018, 4:23 AM

    Hi

    >> What are the Media Agent specifications?

    We use 2-partitions DDB on 2 Media Agents: 2 DELL PowerEdge R740xd servers (256 GB RAM, 2 CPU Intel Xeon 5120 @ 2.20 GHz, Windows Server 2016) . Intel SSD NVMe P3700 is used to store the DDB on each server. For Disk Library use storage Dell MD1420 configured as RAID1+0.

    >> What is the Q&I time shown for this DDB?

    Average Q&I Time in last 3 days: 3814 ms (190%)


    >> What is the application size for this DDB?

    Total size of Application data=130 TB

     

    >> What is the amount of Pending delete records for this DDB?

    Number of Pending Delete Records = 0

     

    >> There is a theoretical limit of 750,000,000 unique blocks for a DDB/partition.
    >> Reaching this limit might suggest performance impact, but this is related to more than just the amount of unique blocks.

    Now each partition use Total number  of Unique Blocks ~ 470,000,000

    Now each partition use Size of DDB Path ~ 142 GB

    On the recommendation of the Commvault Support, i conducted a test using a IOMETER on Intel SSD P3700 which the DDB is located. The results are good - 40K Total I/Os per Second, Average I/O response Time (ms)  = 0,12

     

  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-28-2018, 7:37 AM

    i sugest you need to split the load from existing DDB ..create a new DDB and move some machine on a new policy. 

     

    there is no exact aswer for this as far as the hardware goes everything seems correct. 

     

    we have in past seen such scenerios, and balancing the load by splitting has really helped. 

  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-28-2018, 8:02 AM

    Hello

    Thank you for help.Most likely, I will do as you advise - i create a separate DDB into which I will back up only one Oracle database (~22 TB). I am confused by the fact that a couple of weeks ago the same Oracle DB was copied faster to other "old" servers with a other DDB.

    if you have time tell me, please:

    1. If I increase the number of partitions from 2 to 4 in the DDB, will the backup speed up?

    2. If I turn off compression when backup will speed up backup?

    3. May have some parameters to keep the DDB in memory ?

    (we has 256GB RAM per Media Agents)

    4. Сan the speed decrease due to the fact that the DDB is transactional and logging can be disabled?

    Regards, Aleksandr

  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-28-2018, 8:32 AM

    Your hardware is more than enough, one question though. Are there other DDB's hosted on the MA or only the one partition per MA for the DDB in question?

    I would start by checking the media agents on other processes interfering with the DDB, such as anti-virus.

    Then finetuning the DDB related processes.
    Devide backup window from DDB verification and DDB backup.

    You can also temporaraly disable activity for this MA and run a DDB simulator to identify the response /performance of a DDB on that disk without additional activities:

    http://documentation.commvault.com/commvault/v11/article?p=59032.htm

    To answer your questions:

    1. Yes, the load will be devided and therefor the bottleneck in Q&I will be addressed and therefor the speed will increase, asuming your are not already hitting the limits for the client and/or network.

    2. Probably not, I have seen that compression mostly improves the backup performance. Unless the Client is not able to provide resources to compress in a timely fashion.

    3. As you are using transactional DDB's, these are already using in memory processes. Manually altering defaults for the DDB is not advisable, current values are tested on large environments and should work properly.

    4. Yes transactional DDB's have a +/- 10-15% performance impact, but this should not be an issue with nvme hosting them.


    Jos Meijer
    Senior Technical Consultant
  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-28-2018, 8:51 AM

    Hello

    >> Are there other DDB's hosted on the MA or only the one partition per MA for the DDB in question?

    On these Media Agents only this DDB11 is located, each partitions on a separate Media Agent:

    partition1 --> MA with name simpana-ma-11; partition2 --> MA with name simpana-ma-12.

     

    >> I would start by checking the media agents on other processes interfering with the DDB, such as anti-virus.

    Exceptions are configured for the antivirus according to the Commvault documentation.

     

    Now I will make a 4-partitions DDB and restart the backup.

     

    Regards, Aleksandr

  • Re: Strong slowdown in backup when using deduplication
    Posted: 11-28-2018, 8:54 AM

    You probably already know this, but keep in mind that hashes will be distributed over 2 additional partitions and will result in increase of storage usage on the disklibrary, this will level in time.


    Jos Meijer
    Senior Technical Consultant
  • Re: Strong slowdown in backup when using deduplication
    Posted: 12-04-2018, 1:37 AM

    Hello

    Thank you for info. With 4 partitions, the DDB began to work faster (backup speed increased)

    Regards, Aleksandr

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2018 Commvault | All Rights Reserved. | Legal | Privacy Policy