AUX copy throwing chunk errors after moving to sp12

Last post 10-16-2019, 2:10 PM by Arvind Bingi. 11 replies.
Sort Posts: Previous Next
  • AUX copy throwing chunk errors after moving to sp12
    Posted: 10-08-2019, 9:37 AM

    hi 

     

    we have a two way firewall configured in our environment and is up and working from past 10yrs, recently we upgraded to service sp12 of version 11 and since than we are facing lot chunk issues on AUX jobs and those errors are intermittent. it comes on policy today and than on another policy tomorrow. we have raised a ticket with CV support and the reply we got is to move to one way firewall configuraiton

    somehow i dont feel this is right as support is suggesting me to change the design without providing the actual reason behind the error 

     

    i want to know if anyone else over here has faced any issue after movin to sp12 of v11?

     

    Regards

     

    Hussain R

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-08-2019, 9:21 PM

    You are correct.   These look like disk related errors and not related to network configuration (firewall)

     

    do work with commvault support to look into disk issues.  Look for those errors in your OS events/logs as well

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-09-2019, 1:02 PM

    Thanks for the post.  Our Support Team will do a deeper dive into your open case.

    I just spoke with the engineer's manager who will review and assist you further.

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-11-2019, 8:36 AM

    Thanks

    we have increased the buffer time as suggested by support, looking forward to the AUX copies progress right now. will keep everyone posted

     

    Hussain

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-12-2019, 1:38 PM

    Below is CVJobReplicattorODS.log :

     

    11616 2420  10/10 05:01:23 X CPipelayer::InitializeSdtHead Cannot initialize the SDT connection to the tail. Probably authentication has failed. Err Code [90], Err Msg [Recv. timeout]
    11616 2420  10/10 05:01:23 X SdtBase::relRef: Going to delete SdtBase as ref count is down to 0. RCId [42676579]
    11616 2420  10/10 05:01:23 X SdtBase::() - SdtBase is being destroyed...
    11616 2420  10/10 05:01:23 X CCVAPipelayer::StartPipeline() - Failed to initiate pipeline
    11616 2420  10/10 05:01:24 X CVArchive::StartPipeline() - Startup of DataPipe failed
    11616 2420  10/10 05:01:24 X [Reader_337] Failed to setup copy pipeline for copy [888] stream [23]
    11616 2420  10/10 05:01:24 X ~CVArchive() - Destroying CVArchive. This=000000000549FBB0
    11616 2420  10/10 05:01:24 ####### SdtTailSrvPool::Rel: Resetting SrvPool as ref. count is 0.
    11616 2420  10/10 05:01:24 X [Reader_337] Cannot setup the pipeline.

    Frequently seeing below errors : 

    Error Code: [13:138]    --> This error code followed by communication error as below -

    Description: Error occurred while processing chunk [176492743] in media [V_20511752], at the time of error in library [DL_SourceMA] and mount path [[SorceMA] D:\Mount\Lun-7], for storage policy [SP-PD_ABC_PQR] copy [2. Offsite] MediaAgent [DestMA IP]: Backup job [X]. Unable to setup the copy pipeline. .
    Source: Commserve, Process: AuxCopyMg

    Unable to communicate with the remote machine [DestMA IP] to start the Data Pipe. Please check the network connectivity between the local machine and the remote machine and verify this product's Communications Service is running on the remote machine.Check streams information for more details
    Source: SourceMA, Process: AuxCopy

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-12-2019, 2:00 PM
    • Aplynx is not online. Last active: 10-16-2019, 2:44 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,723

    The errors here aren't the same, but the firewall explanation is applicable. 

    https://ma.commvault.com/Article/Details/53517

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-12-2019, 2:28 PM

    Thanks Liam for your reply.

    The exiting setup we have is two-way. That is since start of the environment. But we started facing this kind of SDT connection failures since last couple of months. Probably after installing SP12 hotfix pack released in March 2019. But not sure.... might be conincidence.

     

    Is the established SDT connections going down/failed becuase of no data recieved from the source ?

    What is the default SDT time out value ?

    FYI that we do also use encryption of primary data before it is getting AUX copied. Is the Encryption taking time and ultimately timing out the SDT connection ?

     

     

     

     

     

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-12-2019, 2:49 PM
    • Aplynx is not online. Last active: 10-16-2019, 2:44 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,723

    It's certainly possible as there is a timeout being reported. Setting one of the media agents incoming connections of the other as blocked will set the connection as one way which uses a persistent tunnel that will have a higher timeout threshold. 

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-12-2019, 3:23 PM

    Ok.

    Instead of changing from 2-way to 1-way firewall configuration, can we just add nTCP_KEEPALIVE_TIMEOUT aditional setting on source and destination media agent groups even if we are on V11 SP12 ? Will it take effects ?

    And if we use this setting, hope no extra load will be placed in terms of system resource on CommServe and media agents.

     

     

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-12-2019, 4:55 PM
    • Aplynx is not online. Last active: 10-16-2019, 2:44 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,723

    You can certainly try that as well. 

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-16-2019, 1:32 PM

    nTCP_KEEPALIVE_TIMEOUT did not help. Still experieicing the same issue.

  • Re: AUX copy throwing chunk errors after moving to sp12
    Posted: 10-16-2019, 2:10 PM

    TCP time out value is set to 1200 on media agent group level.

     

    Below is the AuxCopy.log on source server: 

    Anyone has any idea why we get such errors ?

    80  43ec  10/16 16:43:03 7712037 Reader [22] <Copy/Stream> Source <491/38> Target <500/38>: Reporting FAIL to AuxcpyMgr, Err [0/0]. Chnk [176802553], bytes copied [0]
    9380  43ec  10/16 16:43:03 7712037 Failed to setup the Auxcopy pipeline for NAS index.
    9380  43ec  10/16 16:43:03 7712037 Failed to Init the Auxcopy Data Reader
    9380  43ec  10/16 16:43:03 7712037 Going to bring down the auxcopy pipeline
    9380  43ec  10/16 16:43:03 7712037 stat- ID [Next chunk recv times], Job Id [7712037], Samples [1], Time [0.000002] Sec(s), Average [0.000002] Sec/Sample
    9380  4920  10/16 16:43:05 7712037 CPipelayer::InitializeSdtHead Cannot initialize the SDT connection to the tail. Probably authentication has failed. Err Code [90], Err Msg [Recv. timeout]
    9380  4920  10/16 16:43:05 7712037 SdtBase::relRef: Going to delete SdtBase as ref count is down to 0. RCId [42893780]
    9380  4920  10/16 16:43:05 7712037 SdtBase::() - SdtBase is being destroyed...
    9380  4920  10/16 16:43:05 7712037 CCVAPipelayer::StartPipeline() - Failed to initiate pipeline
    9380  4920  10/16 16:43:05 7712037 CVArchive::StartPipeline() - Startup of DataPipe failed
    9380  4920  10/16 16:43:05 7712037 Reader [23] <Copy/Stream> Source <491/41> Target <500/41>: Failed to setup copy pipeline for copy [500] stream [41]
    9380  4920  10/16 16:43:05 7712037 ~CVArchive() - Destroying CVArchive. This=0000000002D5CA00
    9380  4920  10/16 16:43:05 7712037 Reader [23] <Copy/Stream> Source <491/41> Target <500/41>: Reporting FAIL to AuxcpyMgr, Err [0/0]. Chnk [176802551], bytes copied [0]
    9380  4920  10/16 16:43:05 7712037 Failed to setup the Auxcopy pipeline for NAS index.
    9380  4920  10/16 16:43:05 7712037 Failed to Init the Auxcopy Data Reader
    9380  4920  10/16 16:43:05 7712037 Going to bring down the auxcopy pipeline
    9380  4920  10/16 16:43:05 7712037 stat- ID [Next chunk recv times], Job Id [7712037], Samples [1], Time [0.000003] Sec(s), Average [0.000003] Sec/Sample
    9380  29b0  10/16 16:43:06 7712037 Received FREE STREAM Request for readerId [18]
    9380  29b0  10/16 16:43:06 7712037 Reader [23] <Copy/Stream> Source <491/41> Target <500/41>: Reporting FREE_STREAM to AuxcpyMgr, Err [0/0]. Chnk [176802551], bytes copied [0]
    9380  29b0  10/16 16:43:06 7712037 Reader [22] <Copy/Stream> Source <491/38> Target <500/38>: Reporting FREE_STREAM to AuxcpyMgr, Err [0/0]. Chnk [176802553], bytes copied [0]
    9380  1230  10/16 16:43:07 7712037 CPipelayer::InitializeSdtHead Cannot initialize the SDT connection to the tail. Probably authentication has failed. Err Code [90], Err Msg [Recv. timeout]
    9380  1230  10/16 16:43:07 7712037 SdtBase::relRef: Going to delete SdtBase as ref count is down to 0. RCId [42893782]
    9380  1230  10/16 16:43:07 7712037 SdtBase::() - SdtBase is being destroyed...
    9380  1230  10/16 16:43:07 7712037 CCVAPipelayer::StartPipeline() - Failed to initiate pipeline

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy