Aux Copy aka DASH Copy slow

Last post 06-06-2017, 3:47 PM by ashevtekar. 8 replies.
Sort Posts: Previous Next
  • Aux Copy aka DASH Copy slow
    Posted: 06-05-2017, 7:07 PM

    Currently our DASH to Del (via WAN) is taking over 24 hours to complete. We implemented it back in October and until recently I never noticed any issues. 

    I am seeing the following Critical Alerts:

    Critical

    475384

     

    06/05/2017 08:26:29

    MediaManager

    commvault

    13:185

    Auxiliary Copy Fallen Behind alert is generated for the storage policy copy [col-30day/2. DASH to del] which is having 34526.37 GB data to copy.

     
    and

    Critical

    428844

     

    05/31/2017 08:20:40

    MediaManager

    commvault

    32:417

    Deduplication database on MediaAgent [col-cv2] Path [D:\DDB02] is found offline within last [24] hours. Backups/Auxcopy may not be running to this partition. Space for prunable data managed by the DDB will not be freed until the DDB is fully online.

    So I have an Incident opened with Support and they are insisted that it's an I/O Disk issue. We are using a HPE 1.3TB HH/HL LE PCIe Workload Accelerator card attached to an SSD drive. No other programs or data exist on the DDB partition. During implementation we followed our implementators guidelines to the letter of the law. 
    Is i possible that the DB issues are causing the Aux/Dash copy slow performance? They recommened running I/O Meter but then we'd have to migrate the data, buy a replacement drive and test which means an extended outage.
    Anyone else run into this type of issue or know of any other application that can run without migrating the db/ddb? I have attached an excerpt from running a "Critical Alert" Report.
    Thanks.
     
  • Re: Aux Copy aka DASH Copy slow
    Posted: 06-06-2017, 9:54 AM

    Hi,

     

    Is this the destination ddb of the secondary copy to for which you are seeing the auxcopy fallen behind alert? How big is the ddb? Can you please let us know the support ticket number so we have more details to understand the issue.

     

    Thanks,

    Amey 

  • Re: Aux Copy aka DASH Copy slow
    Posted: 06-06-2017, 11:04 AM

    Hi Amey,

    The incident number is 170531-245. The aux copy (ddb) is the one copying from our primary site (Columbus) to our warm/secondary site (Delaware). 

    I have attached a screen shot of the active job. It's showing 18TB of data to process. The number seems to drop each time and I'm assuming because each job iteration it is catching up and less amount of changes.

    Kind Regards,

    Bill


    Attachment: dbb.jpg
  • Re: Aux Copy aka DASH Copy slow
    Posted: 06-06-2017, 11:42 AM
    Hi Bill,


    Thanks and yes as auxcopy continues to copy data total data processed keeps increasing and it eventually catches up to total data to process.

    For the ddb of copy "2. DASH to del" can you please send us screenshots of following page
    Storage Resources -> Deduplication Engines -> Storage Policy Name/Copy NAme - Click on each partition - Goto Charts -> Display Options select Average Q & I time and Chart Options last 30 days. If you are using GDSP you need to get the chart for GDSP partitions.

    Thanks,
    Amey
  • Re: Aux Copy aka DASH Copy slow
    Posted: 06-06-2017, 12:13 PM

    Amey,

    I have attached several screen shots per your request and I'm sure more information then you may need but may be benefical.

    Thank you,

    Bill


  • Re: Aux Copy aka DASH Copy slow
    Posted: 06-06-2017, 12:59 PM

    Hi Bill,

     

    Your ddb seem ok to us. Since your ddb was offline few days, did your auxcopy not run? If yes, that is why it got fallen behind. So your auxcopy throughput is low compared to before? If yes, please escalate with DB and logs. We will check.

     

    Thanks,

    Amey

  • Re: Aux Copy aka DASH Copy slow
    Posted: 06-06-2017, 1:24 PM

    Amey,

    Where did you notice th ddb was offline? I didn't catch that so I would like to know where to look.

    Also looping back around the assigned SE mentioned the following:

    "The performance statistics we reviewed are directly related to the performance of the DDB disk query and insert times.

    .007 was seen as an average time, for optimal performance this should run around .000

    Is the hardware configuration the current config where the DDB sits?

    del-cv1\SIDBEngine.log ################### 2000 7b8 05/31 08:11:02 ##### 2-0-3-0 LogCtrs 2093 [Query And Insert Handler], Exp Avg [0.007611], Samples [1000], Total Time [227.901453], Total Invoked [31765], Total Avg [0.007175] 2000 7b8 05/31 08:21:02 ##### 2-0-3-0 LogCtrs 2093 [Query And Insert Handler], Exp Avg [0.007470], Samples [1000], Total Time [388.095790], Total Invoked [59120], Total Avg [0.006565] 2000 7b8 05/31 08:41:02 ##### 2-0-3-0 LogCtrs 2093 [Query And Insert Handler], Exp Avg [0.003989], Samples [1000], Total Time [957.283794], Total Invoked [174753], Total Avg [0.005478] 2000 7b8 05/31 08:51:02 ##### 2-0-3-0 LogCtrs 2093 [Query And Insert Handler], Exp Avg [0.006505], Samples [1000], Total Time [1346.487689], Total Invoked [252809], Total Avg [0.005326]

    Log analysis:

    QI times very high: [0.007175] "

    But based on the screen shots the do the Q&I times look average or above what is recommended for a Large environment. Do you see a need to run IOMETER to eliminate the disk as the culprit?

    Thanks,

    Bill

  • Re: Aux Copy aka DASH Copy slow
    Posted: 06-06-2017, 1:29 PM

    Hi Bill,

     

    It was in the alert doc you sent in first reply. Let me get back on other question.

    Thanks, Amey

     

    Critical

    428843

     

    05/31/2017 08:20:40

    MediaManager

    commvault

    32:417

    Deduplication database on MediaAgent [col-cv1] Path [D:\DDB01] is found offline within last [24] hours. Backups/Auxcopy may not be running to this partition. Space for prunable data managed by the DDB will not be freed until the DDB is fully online.

  • Re: Aux Copy aka DASH Copy slow
    Posted: 06-06-2017, 3:47 PM

    Hi Bill,

     

    For Q&I, I advise you to work with SE that you are working with and for auxcopy slowness just escalate as an TR and we will check.

     

    Thanks,

    Amey

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy