Resumable Full VM Restores

Last post 06-05-2017, 1:51 AM by MWR_COMM. 3 replies.
Sort Posts: Previous Next
  • Resumable Full VM Restores
    Posted: 06-01-2017, 11:09 PM

    Hi,

    I'd like to request that Full VM restores be made resumeable.

    Currently, if a full VM restore is interupted the job will *restart* but all progress/data written is lost.

    Given nothing else is touching the VM but CV, it should be able to work out where it was up to when the the restore job was suspended/interupted and continue from there.

    This is something you have fixed before by making exchange backups resumable back in CV10? which was a *BIG* quality of life improvemment, can you do the same for Full VM restores.

    Background - Since SP6 came out we've had to reboot out RHEL MA's every couple of weeks as the synth fulls slow down from hundred's of gb/hr to under 10gb/hr (on local SSD DDB/IC) and post reboot are fine.  Restores also have kittens at the same time going extremely slowly despite very low resource usage on the MA's.  The case (170531-32) for this isn't getting anywhere fast hence this as a potential workaround.

  • Re: Resumable Full VM Restores
    Posted: 06-02-2017, 2:33 AM

    Hi Brendan

    The RHEL Media Agent performance issues appear to be the root cause of both the issues you are experiencing. We do have some suggested addtional settings that should result in improved Synth Full performance. In this instance resolving the overall media agent performance issues will result in VM restores completing in a more timely manner

    With regards to the VM restore continuing from the last location if interrupted , this is something that is currently being discussed internally 

    Cheers

  • Re: Resumable Full VM Restores
    Posted: 06-04-2017, 10:56 PM

    Hi MWR,

    Are those RHEL settings in a BoL article somewhere or are they internal?

    Currently using the following additional settings on the MA's:

    UseCacheDB MediaAgent INTEGER 1 True This Object
    UseAuxcopyReadlessPlus MediaAgent INTEGER 1 True This Object
    SignaturePerBatch MediaAgent INTEGER 32 True This Object
    DataMoverUseLookAheadLinkReader MediaAgent INTEGER 1 True This Object
    DataMoverLookAheadLinkReaderSlots MediaAgent INTEGER 32 True This Object

     

    /etc/sysctl.conf:

    # Kernel parameters as per CommVault BoL
    kernel.sem=3450 11481600 3232 3328

    # Controls the maximum shared segment size, in bytes
    #kernel.shmmax = 51539607552

    # Controls the maximum number of shared memory segments, in pages
    #kernel.shmall = 51539607552

    #Controls the CPU time that SoftIRQ gets to drain the recieve buffers
    net.core.netdev_budget=600
    net.core.netdev_max_backlog = 30000

    #Smaller cache for better NFS writes
    vm.dirty_ratio = 10
    vm.dirty_background_ratio = 5

    #Stop VM using OS swap (set to lowest without disabling)
    vm.swappiness = 1

    #Stop "page allocation failure" errors
    vm.min_free_kbytes=4194304

    # Raise generic socket memory useability, and start 'em big
    net.core.rmem_default=12582912
    net.core.wmem_default=12582912
    net.core.rmem_max=16777216
    net.core.wmem_max=16777216

    # Raise tcp memory useability too
    net.ipv4.tcp_rmem=4096 12582912 16777216
    net.ipv4.tcp_wmem=4096 12582912 16777216

    # increase the number of option memory buffers
    net.core.optmem_max=12582912

    # raise the amount of memory for the fragmentation reassembly buffer
    # (if it goes above high_thresh, kernel starts tossing packets until usage
    # goes below low_thresh)
    net.ipv4.ipfrag_high_thresh=8388608
    net.ipv4.ipfrag_low_thresh=524288

    # make sure window scaling is on
    net.ipv4.tcp_window_scaling=1

    #Disable IPv6
    net.ipv6.conf.all.disable_ipv6 = 1

    # max out the number of task request slots in the RPC code. also set here as well /etc/modprobe.d/sunrpc.conf
    sunrpc.tcp_slot_table_entries=128
    sunrpc.udp_slot_table_entries=128

    END

    Also VMXnet3 NIC tunings on /etc/rc.local & fixes for max open files in a couple of different places.

     

    Please pass onto the discussion thread +1 from a customer for the VM full restore continuing from last location.  If a CMR is generated, please let me know.

    It's wierd thought that it takes a couple of weeks before the slow performance eventuates and that at the same time live file recovery breaks.

  • Re: Resumable Full VM Restores
    Posted: 06-05-2017, 1:51 AM

    Hi

    2 of the 3 additional settings suggested are documented in BoL

    http://documentation.commvault.com/additionalsetting/details?name=%22nNumPipelineBuffers%22&id=603

    http://documentation.commvault.com/additionalsetting/details?name=%22SignaturePerBatch%22&id=1015

    These settings were suggested and tested by our senior support team and are specific to slow synthetic full performance on SP6

    There is currently no specific performance tuning guide for Media Agents running on RHEL.

    It is also recommened to add the nPipeLineMode additional setting as a first step and continue to monitor before adding/changing additional settings.  

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy