Block settings for DDBs, policies and data paths for AWS S3

Last post 03-29-2019, 11:03 AM by Erc07. 0 replies.
Sort Posts: Previous Next
  • Block settings for DDBs, policies and data paths for AWS S3
    Posted: 03-29-2019, 11:03 AM

    We've been running Commvault 11 for a few years and recently got an Aux Copy to AWS S3 working and tested so we can recover in a full DR scenario.  Now I'm trying to figure out the best settings to make sure we're getting an efficient data transfer and storage.  I've read through many articles and forum posts but nothing that ties everything together and am hoping posting here will help organize what I've read.

    Our environment is two sites with one media agent local to each that handles backups for VMs, file agents and DB agents. Every day the previous night's backups are sync copies to the other site.  We use dedupe locally and Aux copies are DASH.  A single CommServe controls everything and all data is AES encrypted.

    For the copy to AWS S3, all data is sent from media agent A. It sends jobs from the Primary copy for storage policies that use it's deduped disk library, and uses the Secondary copies for jobs that come in from the remote site media agent B.  That works well over our 1Gb synchronous internet connection.

    What I'm struggling to iron out are the detailed settings like block sizes and how to get the most efficient dedupe and job thoughput.

    The different block level dedupe factors we have set are:

    Local global dedupe DBs: 128KB

    AWS S3 global dedupe DB (on media agent A): 512KB

    Storage policies: 128KB

    The block sizes on the copies' data paths are:

    Primary and Secondary copies to disk libaries (on site): Use media type settings (64KB)

    AWS S3 copy: 1024 (automatically set when creating the copy)

    Based on the last bullet point in link 1 below, should I change the storage policies to have a block level of 512KB or is the default 128KB like link 2 shows OK?

    Similarly, should the data path block size for the S3 cloud lirbary be set to 512KB instead of 1024?  I only found reference to the 1024 in link 3 for tuing 10Gb+ connections.

    My plan was to leave storage policy block sizes at 128KB and set the cloud library data path to 512KB. Any advise is welcome. Thanks!

    Note: The S3 cloud copies are DASH Network Optimized and Space optomized Aux Copy is checked.

    Link 1:

    Link 2:

    Link 3:

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy