Sporadic Firewall Connection Issues

Last post 03-12-2018, 2:55 PM by Vercetti. 14 replies.
Sort Posts: Previous Next
  • Sporadic Firewall Connection Issues
    Posted: 11-15-2017, 2:34 AM

    Hi!

    I have aCommCell in my Datacenter where customers connect their clients and mediaagents to via one-way firewall. so all clients cann open a connection to commserv via a port-forwarding gateway.

    every some days i see in the logs that one client looses the connection and reopens a tunnel. this is not a big problem when no backup or restore is running. when it happens during a backup the current task gets an error. when it happens during a restore the restore might fail.

    this is an example of the error:

    1488 1058 11/14 00:35:47 TN:00020 ######## ERROR: cvfwd_iot_wait(): Got READ error on DYNAMIC tunnel from "pinf06" to "10092_psrv01" via (172.25.0.6, 212.186.139.62) on fd=6128: The specified network name is no longer available.
    1488 1058 11/14 00:35:47 TN:00020 ######## ERROR: cvfwd_reset_tunnel(): Ungraceful termination of DYNAMIC tunnel from "pinf06" to "10092_psrv01" via (172.25.0.6, 212.186.139.62)
    1488 1058 11/14 00:35:47 TN:00020 ######## ERROR: cvfwd_reset_tunnel(): Resetting DYNAMIC tunnel from "pinf06" to "10092_psrv01" via (172.25.0.6, 212.186.139.62)

    Client-readyness is always fine and it affects any client so its not related to a customer or client. it happens on all of my commcells and there are various network-paths so its not related to one network path.

    what i seen so far is that its worst when there is high network load between the client and the commserv.

    for example when i do a persitent recovery from the mediaagent at a client-site to the comserv. then this happens more often.

    the customers are conneted with slow internet-connections < 10MBit, sometimes even <2MBit 

     

    has anyone of you seen the same errors in his firewall-log. are there any parameters to tune? i allready changed the keep-alive interval and encryption but it doesnot help.

    i had a support ticket open for this problem but the support couldnot really help.

    It seems that commvault bas a problem under slow or loaded network-conditions.

     

    CommVault Version: v11 sp9 - all hotfixes installed

    Clients, Mediaagents and CommServ are Windows Systems (mostly 2016)

     

    Greetings,

    Franz

     

     

     

  • Re: Sporadic Firewall Connection Issues
    Posted: 11-20-2017, 4:02 PM

    Hi Franz,

     

    From the logs you had pasted, the issue is with the underlying network interface that keeps vanishing/disconnecting. From the application layer I doubt we will be able to do anything here to hold the connection. Do you see this happening with better network connections? I would even recommend talking with your network provided for the network stability.

    If you feel that this is not an issue with the network, please escalate a TR so that we could take a look.

     

    With regards,

    Prakash



    -Prakash
  • Re: Sporadic Firewall Connection Issues
    Posted: 11-23-2017, 12:44 PM
    • Ali is not online. Last active: 03-05-2018, 11:33 PM Ali
    • Top 10 Contributor
    • Joined on 08-05-2010

    Typically even if its a 'slow' network say, relatively it shouldn't impact Commvault, if there is an ungraceful shutdown as seen above that means Commvault is detecting it then closing the backup stream, not the other way around (this is all presumably of course).

    Would suggest running Wireshark and engaging the network admins for sure, and see if there are any WAN accelerators which may have 'tipping-points' configured or even some firewalls have this I believe to ensure the data size being moved during these windows isn't triggering some limit set on the hardware side.

  • Re: Sporadic Firewall Connection Issues
    Posted: 11-23-2017, 12:47 PM
    • Aplynx is not online. Last active: 06-20-2018, 11:01 AM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Expert
    • Points 1,312

    Try setting outgoing routes to raw and see if that stays open. 

  • Re: Sporadic Firewall Connection Issues
    Posted: 12-04-2017, 3:19 AM

    Hi.

    I see basically the same behaviour.

    My configuration is as below:

    CommServe in Cloud

    Media Agent & Cllient on Premise

    Below on Media Agent.

    1368 04d0 12/04 10:06:47 ######## ######## Detected a change in IP configuration. Sending KEEP_ALIVEs through all tunnels.
    1368 04d0 12/04 10:06:47 ######## ######## Detected a change in IP configuration. Re-reading config files.
    1368 04d0 12/04 10:06:47 TN:00005 ######## ERROR: cvfwd_reset_tunnel(): Resetting PERSISTENT tunnel from "bibackup1" to "proxy fqdn" via (ANY, "Proxy IP")
    1368 04d0 12/04 10:06:47 TN:00006 ######## ERROR: cvfwd_reset_tunnel(): Resetting PERSISTENT tunnel from "bibackup1" to "proxy" via (ANY, "Proy IP")

     

    So no real data traverses the link as it is only job control data.

    Small backup jobs complete.

     

    Any help would be appreciated.

     

    Thanks.

  • Re: Sporadic Firewall Connection Issues
    Posted: 12-04-2017, 3:25 AM

    I discussed the problem with support and they said that it is a network problem.

    What helped a littelbit was to chanage the tunnels from authenticated to encrypted.

    in my opinion the commvailt firewall tunnel architecture is not resilent enough.

  • Re: Sporadic Firewall Connection Issues
    Posted: 12-04-2017, 4:07 AM

    Hi.

     

    The connection is via a Proxy so I changed the Default Outgoing Route on both the CS and Client Group on the Options tab to Encrypted.

     

    I will see if this makes a difference.

     

    Cheers.

  • Re: Sporadic Firewall Connection Issues
    Posted: 12-04-2017, 4:11 AM

    Allso check the fallback=1 route in the FwConfig.txt on the client and change it also to encrypted

  • Re: Sporadic Firewall Connection Issues
    Posted: 03-01-2018, 6:16 PM

    Hi franzr77,

    Did you find a solution for this? I'm having the same exact issue with the same configuration but I can't get commvault support to take a better look since they only say it's a network thing, hope you have foud a solution and I'd be thankful if you shared your comments on this.

  • Re: Sporadic Firewall Connection Issues
    Posted: 03-02-2018, 5:40 AM

    Not really.

    What definetely helped was to set the tunnel to encrypted.

    common problems we see in our configurations are that firewalls in between do some sort of "intelligent" ips things an may block the commvault traffic. 

    it is crucial to deactivate any firewall "intelligence" or algs.

    then it is important to set the tunnel to encrypted to avoid "optimisations" of network provideres.

    BUT in my opinion, as i said before, the commvault firewall is not resilent enough to work over a wan network.

    A really big problem for us is that we cannot do a restore job to our datacenter because commvault uses a persistent-recovery to mount the data on the mediaagent in the datacenter. during al long-running restore of eg. an Exchange Mailbox, the probability is higt that the firewall connection resets and then the persistent-recovery stops with an error.

    so thats our main problem now in our scenario.

    to conclude: everywhere where we have a 100% reliable and performant WAN connection (eg MPLS) ,with ALL firewalling disabled, the commvault firewall works without a problem. if the WAN connection is not perfect, we get lots of problems when we do restores over WAN.

     

    Greetings,Franz

  • Re: Sporadic Firewall Connection Issues
    Posted: 03-02-2018, 8:37 PM

    Vercetti,

    Please escalate the issue. We would like to check your configurations and logs to assist further. Based on that, we will let you know if there is an underlying network issue to be addressed or any configuration change required to adapt to your network setting.

    Thanks,

    Sonia

     

  • Re: Sporadic Firewall Connection Issues
    Posted: 03-07-2018, 4:08 PM

    I tried all the options without luck.

    The only thing that really made a difference in my scenario was to disable IPv6 on the network adapters which I don't use.

     

    Been much more solid.

  • Re: Sporadic Firewall Connection Issues
    Posted: 03-12-2018, 12:18 PM

    Hi Sonia,

     

    How do I do that? via the commvault chat? this is the case nunber 180227-457

  • Re: Sporadic Firewall Connection Issues
    Posted: 03-12-2018, 12:51 PM
    • Aplynx is not online. Last active: 06-20-2018, 11:01 AM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Expert
    • Points 1,312

    I'll let the ticket owner know that the escalation request is being made. 

  • Re: Sporadic Firewall Connection Issues
    Posted: 03-12-2018, 2:55 PM

    Thank you Liam, support reached out to me already.

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2018 Commvault | All Rights Reserved. | Legal | Privacy Policy