Sporadic Firewall Connection Issues

Last post 12-04-2017, 4:11 AM by franzr77. 7 replies.
Sort Posts: Previous Next
  • Sporadic Firewall Connection Issues
    Posted: 11-15-2017, 2:34 AM

    Hi!

    I have aCommCell in my Datacenter where customers connect their clients and mediaagents to via one-way firewall. so all clients cann open a connection to commserv via a port-forwarding gateway.

    every some days i see in the logs that one client looses the connection and reopens a tunnel. this is not a big problem when no backup or restore is running. when it happens during a backup the current task gets an error. when it happens during a restore the restore might fail.

    this is an example of the error:

    1488 1058 11/14 00:35:47 TN:00020 ######## ERROR: cvfwd_iot_wait(): Got READ error on DYNAMIC tunnel from "pinf06" to "10092_psrv01" via (172.25.0.6, 212.186.139.62) on fd=6128: The specified network name is no longer available.
    1488 1058 11/14 00:35:47 TN:00020 ######## ERROR: cvfwd_reset_tunnel(): Ungraceful termination of DYNAMIC tunnel from "pinf06" to "10092_psrv01" via (172.25.0.6, 212.186.139.62)
    1488 1058 11/14 00:35:47 TN:00020 ######## ERROR: cvfwd_reset_tunnel(): Resetting DYNAMIC tunnel from "pinf06" to "10092_psrv01" via (172.25.0.6, 212.186.139.62)

    Client-readyness is always fine and it affects any client so its not related to a customer or client. it happens on all of my commcells and there are various network-paths so its not related to one network path.

    what i seen so far is that its worst when there is high network load between the client and the commserv.

    for example when i do a persitent recovery from the mediaagent at a client-site to the comserv. then this happens more often.

    the customers are conneted with slow internet-connections < 10MBit, sometimes even <2MBit 

     

    has anyone of you seen the same errors in his firewall-log. are there any parameters to tune? i allready changed the keep-alive interval and encryption but it doesnot help.

    i had a support ticket open for this problem but the support couldnot really help.

    It seems that commvault bas a problem under slow or loaded network-conditions.

     

    CommVault Version: v11 sp9 - all hotfixes installed

    Clients, Mediaagents and CommServ are Windows Systems (mostly 2016)

     

    Greetings,

    Franz

     

     

     

  • Re: Sporadic Firewall Connection Issues
    Posted: 11-20-2017, 4:02 PM

    Hi Franz,

     

    From the logs you had pasted, the issue is with the underlying network interface that keeps vanishing/disconnecting. From the application layer I doubt we will be able to do anything here to hold the connection. Do you see this happening with better network connections? I would even recommend talking with your network provided for the network stability.

    If you feel that this is not an issue with the network, please escalate a TR so that we could take a look.

     

    With regards,

    Prakash



    -Prakash
  • Re: Sporadic Firewall Connection Issues
    Posted: 11-23-2017, 12:44 PM
    • Ali is not online. Last active: 12-06-2017, 11:24 AM Ali
    • Top 10 Contributor
    • Joined on 08-05-2010

    Typically even if its a 'slow' network say, relatively it shouldn't impact Commvault, if there is an ungraceful shutdown as seen above that means Commvault is detecting it then closing the backup stream, not the other way around (this is all presumably of course).

    Would suggest running Wireshark and engaging the network admins for sure, and see if there are any WAN accelerators which may have 'tipping-points' configured or even some firewalls have this I believe to ensure the data size being moved during these windows isn't triggering some limit set on the hardware side.

  • Re: Sporadic Firewall Connection Issues
    Posted: 11-23-2017, 12:47 PM
    • Aplynx is not online. Last active: 12-08-2017, 10:24 AM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Expert
    • Points 1,171

    Try setting outgoing routes to raw and see if that stays open. 

  • Re: Sporadic Firewall Connection Issues
    Posted: 12-04-2017, 3:19 AM

    Hi.

    I see basically the same behaviour.

    My configuration is as below:

    CommServe in Cloud

    Media Agent & Cllient on Premise

    Below on Media Agent.

    1368 04d0 12/04 10:06:47 ######## ######## Detected a change in IP configuration. Sending KEEP_ALIVEs through all tunnels.
    1368 04d0 12/04 10:06:47 ######## ######## Detected a change in IP configuration. Re-reading config files.
    1368 04d0 12/04 10:06:47 TN:00005 ######## ERROR: cvfwd_reset_tunnel(): Resetting PERSISTENT tunnel from "bibackup1" to "proxy fqdn" via (ANY, "Proxy IP")
    1368 04d0 12/04 10:06:47 TN:00006 ######## ERROR: cvfwd_reset_tunnel(): Resetting PERSISTENT tunnel from "bibackup1" to "proxy" via (ANY, "Proy IP")

     

    So no real data traverses the link as it is only job control data.

    Small backup jobs complete.

     

    Any help would be appreciated.

     

    Thanks.

  • Re: Sporadic Firewall Connection Issues
    Posted: 12-04-2017, 3:25 AM

    I discussed the problem with support and they said that it is a network problem.

    What helped a littelbit was to chanage the tunnels from authenticated to encrypted.

    in my opinion the commvailt firewall tunnel architecture is not resilent enough.

  • Re: Sporadic Firewall Connection Issues
    Posted: 12-04-2017, 4:07 AM

    Hi.

     

    The connection is via a Proxy so I changed the Default Outgoing Route on both the CS and Client Group on the Options tab to Encrypted.

     

    I will see if this makes a difference.

     

    Cheers.

  • Re: Sporadic Firewall Connection Issues
    Posted: 12-04-2017, 4:11 AM

    Allso check the fallback=1 route in the FwConfig.txt on the client and change it also to encrypted

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2017 Commvault | All Rights Reserved. | Legal | Privacy Policy