Slow restore from deduplicated backup

Last post 11-27-2019, 1:26 PM by lukasz.borek. 3 replies.
Sort Posts: Previous Next
  • Slow restore from deduplicated backup
    Posted: 10-25-2019, 7:54 AM

    Hi.

    Some time ago we set up media agent with local stroage (SATA disk and ZFS) and DDB (SSD) as backup destination for mixed clients (MSSQL, virt, filesystem). Our clients notice that restore performance dropped in time. Standard deduplicated restore is very slow (~50GB/hr), but restore from job with deduplication disabled is around 800GB/hr. Same problem can be seen while runing data verification job : deduplicated one is 4 time slower than no-deduplicated.

    From performance perspective :

    CVDiskPerf -PATH /storage/iotest/ -RANDOM -BLOCKCOUNT 163840 -FILECOUNT 10 -THREADCOUNT 10:

    Throughput Create(GB/H) : 1929.49
    Throughput Write(GB/H) : 684.97
    Throughput Read(GB/H) : 1739.77
    Throughput Delete(GB/H) : 299543.70

    From sample restore log :

    7774 979 10/25 11:14:00 2466873 [FSRESTHEAD ] 83-# Stats: AppTy 33, AF 1, Seeks 31873/31873/115195, WrkItms 144650, MxRdr (Bsy-Tot: 1-9, Pnd 0), MsgsRcvd 3604, QLen 2604, LstMsg CLOSE_ARCHIVE
    27774 979 10/25 11:14:00 2466873 [FSRESTHEAD ] 83-# Stats: [AllocBuf Time] Exp Avg [0.01], Total Avg [0.03], Total Time [2563.82], Total Count [102360]
    27774 979 10/25 11:14:00 2466873 [FSRESTHEAD ] 83-# Stats: [AllocBuf Speed] Curr Avg [3.06] MB/Sec, Bytes [1323150192]; Total Avg [2.49] MB/Sec, Bytes [6704170560]
    27774 979 10/25 11:14:00 2466873 [FSRESTHEAD ] 83-# Stats: [MediaRead Time] Exp Avg [0.00], Total Avg [0.00], Total Time [376.58], Total Count [134230]
    27774 979 10/25 11:14:00 2466873 [FSRESTHEAD ] 83-# Stats: [MediaRead Speed] Curr Avg [4.31] MB/Sec, Bytes [848679629]; Total Avg [11.57] MB/Sec, Bytes [4566787582]

    I assume that data data in disk library gets fragmented during time and restore proccess need more time to read and re-hydrate, but what can one do do fix it? DDB Seal each X-months?

    Thanks.

  • Re: Slow restore from deduplicated backup
    Posted: 10-25-2019, 9:42 AM

    For restores we dont need Dedupe database, so as long as your backups are good you dont need to seal store. 

    Restore from dedupe data is random reads, if you are getting good dedupe ratio those read might be high, as restore from most recent full (or synthfull) has to read data from last full and all incrementals,so as you mention framentation might be one of the factor where read comes slow.

    Try following registry key 

    It goes on MediaAgent machine under "MediaAgent" section. its dword (integet if yo put via GUI) key name is "DataMoverLookAheadLinkReaderSlots" set this to 256.

    Which service pack are you on? in recent service pack SP16 we have made some enhancement for look ahead reader. along with above key those changes will help in your restore performance. we have made lookahead read as default 256 inSP16 so you dont need reg key in SP16.

  • Re: Slow restore from deduplicated backup
    Posted: 10-25-2019, 3:33 PM

    Are you using Commvault encryption on the DDB? We've run into an issue where pulling keys adds a considerable amount of time to the restore process on backups that are highly deduplicated.

  • Re: Slow restore from deduplicated backup
    Posted: 11-27-2019, 1:26 PM

    No encryption. Root cause was backednd storage performancde. Together with data frontend grow, SATA drives were not able to read data fast enaught. Data accros years become fragmented and from sequential read it came to random reads. Migrating to NAS with SAS drives fixed the issue. 

    Thanks. 

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy