Question about Short Term and Long Term Retention Mixed in the same DDB

Last post 10-07-2019, 9:32 PM by Watson. 3 replies.
Sort Posts: Previous Next
  • Question about Short Term and Long Term Retention Mixed in the same DDB
    Posted: 10-01-2019, 11:04 AM

    All,

     

    I'm having a discussion with some people that you shouldn't mix short term and long term data in the same DDB store but I don't feel I got my point across to the group. I've al;ways been told that with DDBs, if you have a muti-year retention data set then  you wouldn't put that in the in the same DDB as something that is short term retention like 14 or 30 days. This is due to mostly data aging processes and if a seal is needed.

    Is this still the case? Could someone expand on my arguement with details on why you wouldn't do this? I want to make sure I can get my point across to people who don't necessarily understand how data should be managed with DDBs.

     

    Thanks. 

  • Re: Question about Short Term and Long Term Retention Mixed in the same DDB
    Posted: 10-02-2019, 10:03 PM

    Commvault Deduplication has changed a lot since the earlier design considerations.

    DDB 5 technology which was added in Service Pack 14 which has resulted in considerably much smaller DDB sizes (https://documentation.commvault.com/commvault/v11_sp14/article?p=108822.htm).  Essentially the outcome of this has given Commvault devs the confidence to introduce Hozinontal Deduplication Databases in Service Pack 17 (https://documentation.commvault.com/commvault/v11_sp17/article?p=111494.htm) when you create a new storage policy copy, and what you will get is a separate DDB for Virtualization, Databases and File Systems.

    If you are already using DDB 5 tech, then I would not be concerned about mixed retention.  If your DDB version is old, say v10 (or if transactional ddb has been enabled) and the Q&I times are high - then you may have good reason be concerned for the reasons you have stated.

  • Re: Question about Short Term and Long Term Retention Mixed in the same DDB
    Posted: 10-06-2019, 6:31 AM

    Suppose you have 30 daily jobs with monthly retention and 1 job with yearly retention of the same server. After a month, although the 30 jobs will age off the disk, the ref blocks/records on the DDB will not age because the yearly job is referencing those. This can lead to DDB bloat over time hence the recommendation to have separate DDB for jobs retained for longer period

  • Re: Question about Short Term and Long Term Retention Mixed in the same DDB
    Posted: 10-07-2019, 9:32 PM
    Previously when setting up a Deduplication Database (DDB) with long term retention we recommended setting up 

    Primary Copy with 90 day retention
    Secondary Copy with 1 to 5 year retention and to seal every year

    This recommendation was based on managing the DDB size and maintaining performance.

    In terms of DDB size

    - there is no limit to the number of secondary tables
    - a secondary table will not be deleted until all references in it age
    - white space does not get reused
    - a secondary table can grow to a maximum of 4GB

    The secondary tables for a DDB created in version 11 can contain 256 archfiles per secondary table
    The secondary tables for a DDB created in version 11 before Service Pack 14 can contain 16 archfiles per secondary table
    The secondary tables for a DDB created in version 11 after Service Pack 14 contain 1 archfiles per secondary table

    To manage secondary table bloat inside the DDB, on DDBs created prior to Service Pack 14 you can compact the secondary tables inside the DDB as per https://documentation.commvault.com/commvault/v11_sp17/article?p=12613.htm

    In terms of performance we can see degraded performance when a DDB reaches 800 million primary records. With this in mind, as Anthony mentioned, Development have introduced Hozinontal Scaling in Service Pack 17.
    Separate DDB partitions are created for virtual machines, databases and file system agens.
    In SP17, we will also create a new DDB when a DDB reaches the defined threshold. With the auto creation of DDBs, a new DDB will be created when a DDB reaches the defined threshold, which will result in new sub-clients being added to the policy will point to the new DDB. Any existing sub clients will continue to backup to the existing DDB.

    If the DDB is created in SP17 you are not required to create a separate DDB with long term retention.

    There is an additional consideration, if resiliency is enabled on your DDB and 1 partition is corrupt. Any blocks that were written while the partition was offline will be maintained on your disk library until the job meets retention.
The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy