cvfwd & Microsoft Failover Cluster Virtual Adapter

Last post 05-28-2020, 11:59 AM by Liam. 10 replies.
Sort Posts: Previous Next
  • cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-20-2020, 12:56 PM
    Hi All

    Today tried to change up some of our Commvault Firewall Rules to One-Way and got horrible amount of errors.
    In cvfwd.log I found that somehow our MediaAgent Cluster Nodes somehow try to use an IP adress that is nowhere configured in the System

    25356 5e10 05/20 18:30:33 TN:00959 ######## ERROR: cvfwd_iot_wait(): Socket connect failed. Got CONNECT error on ON_DEMAND control tunnel from "hmaeh02" to "hmazf02" via (169.254.3.73, 172.31.29.35) on fd=4704: The network location cannot be reached. For information about network troubleshooting, see Windows Help.
    25356 5e10 05/20 18:30:34 TN:00964 ######## ERROR: cvfwd_iot_wait(): Socket connect failed. Got CONNECT error on ON_DEMAND control tunnel from "hmaeh02" to "hmazf04" via (169.254.3.73, 172.31.29.37) on fd=4708: The network location cannot be reached. For information about network troubleshooting, see Windows Help.

    Found that this belongs to "Microsoft Failover Cluster Virtual Adapter" but nowhere find an option to disable it or to prevent Commvault from using it.
    Found some older articles regarding binding order changes in Windows, but that Dialog doesn't exist on our Win 2016 Systems. Another MS Article claimed that Adapter no longer exists on Win 2016, but still is there.

    Does anybody know how to prevent Commvault from ever trying to use this Adapter in first place? DIP is no option since it would require massive amount of config.

    Thanks in Advance
    Stefan
  • Re: cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-20-2020, 1:10 PM

    Hi Stephan,

     

      Are you talking about this IP 169.254.3.73 on hmaeh02?

     

      The hostname to be used by hmaeh02 is present under Machines/hmaeh02/sHOSTNAME in the registry.

     

    Thanks

     

      

  • Re: cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-20-2020, 2:05 PM

    hmaeh01-04 are the 4 Nodes of the MediaAgent Cluster, these are the only names used for communication between Servers and they have only one IP in 172.x.x.x Range, the 169.x.x.x is some randomly created thing for the Cluster Interface and can only be used for internal use (not routed).

    That doesn't hinder Commvault from randomly using it for whatever reason. Attempts to create an outgoing socket via that will always fail. Have to prevent that and hope someone here knows how.

  • Re: cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-20-2020, 10:20 PM

    I'd recommend that you set up Data Interface Pairs on those clients to use the routable 172.x.x.x range to communicate to and from the CommServe/MediaAgent Infrastructure.

    https://documentation.commvault.com/commvault/v11/article?p=7141.htm

  • Re: cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-21-2020, 2:58 AM

    Hi Anthony

    This is one of the MediaAgents and that is why it is so impossible to do DIPs. If I need IP2IP Rules I'd need aprox 4500 of them for this Server alone. If the IP2Group Thing works I might get it down to 20-30. 

    Is there no way to blacklist specific interfaces? also don't understand why CV would use the Cluster Adapter when this is just a fancy loopback.

    Cheers, Stefan

  • Re: cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-21-2020, 3:01 PM

    If you want to use specific interface (like 172.x.x.x) on a machine you can set sBindToInterface with required IP. In this case, it won't use the cluster adapter.

    http://documentation.commvault.com/commvault/v11_sp18/article?p=8559.htm

  • Re: cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-22-2020, 4:31 AM

    Excelent hint, that should work for the time beeing.

    Unfortunately that will break function once we switch to dedicated backend storage network. Wish Commvault would simply stick with the routing defined on OS level and not try to be clever and re-invent the wheel over again by making it square. 

    The 169.x.x.x Addresses are also by definition Link Local, nothing should ever try using that for external (routed) connections. So not sure if that is actually a bug in CV Network Stack in addition to ignoring System Settings. 

    Do you know if that Additional Setting allows a list of IPs/Names or whether it is one-value only?

  • Re: cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-22-2020, 10:53 AM

    Stefan,

    Commvault does not control which interface is used (except for DIPS) we really on the OS to tell us. 

    In Windows Server 2016 (as well as Windows 10), there are no components that still use the network binding order. Hence, Microsoft decided to remove it. By default, Windows uses the Interface Metric property of a network adapter to determine which route has the highest priority. The lower the Interface Metric property value, the higher the priority.

    Also, the value of the Interface Metric property for a network adapter is automatically assigned, by default, and is based on the link speed. This is described in this Microsoft KB article: An explanation of the Automatic Metric feature for IPv4 routes.

    I suspect at somepoint the 169 address was presented as a default by the OS and we may be caching it in cvfwd. Have you tried restarting the service for cvfwd?

    also, you may want to consider possily disabling IPV6 if that is not used on the MA for anything. I have seen that help in the past.


    Gary Seibak
    Technical Account Manager - Commvault
  • Re: cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-22-2020, 12:22 PM
    Commvault couldn't use that IP if it would adher to what Windows defines. You may bind your Service to it, but by the routing defintion you would never use it to talk to any other System in our Env, either you go f.E. 172.27.29.x via the primary NIC, or you go to Default Gateway via primary NIC.
    PS C:\windows\system32> route print
    ===========================================================================
    Interface List
    4...48 df 37 39 7d 60 ......Microsoft Network Adapter Multiplexor Driver
    10...02 48 00 3d 5c f9 ......Microsoft Failover Cluster Virtual Adapter
    1...........................Software Loopback Interface 1
    12...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #3
    14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
    ===========================================================================

    IPv4 Route Table
    ===========================================================================
    Active Routes:
    Network Destination Netmask Gateway Interface Metric
    0.0.0.0 0.0.0.0 172.27.29.254 172.27.29.16 266
    127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
    127.0.0.1 255.255.255.255 On-link 127.0.0.1 331
    127.255.255.255 255.255.255.255 On-link 127.0.0.1 331
    169.254.0.0 255.255.0.0 On-link 169.254.1.201 271
    169.254.1.201 255.255.255.255 On-link 169.254.1.201 271
    169.254.255.255 255.255.255.255 On-link 169.254.1.201 271
    172.27.29.0 255.255.255.0 On-link 172.27.29.16 266
    172.27.29.16 255.255.255.255 On-link 172.27.29.16 266
    172.27.29.34 255.255.255.255 On-link 172.27.29.16 266
    172.27.29.35 255.255.255.255 On-link 172.27.29.16 266
    172.27.29.255 255.255.255.255 On-link 172.27.29.16 266
    224.0.0.0 240.0.0.0 On-link 127.0.0.1 331
    224.0.0.0 240.0.0.0 On-link 169.254.1.201 271
    224.0.0.0 240.0.0.0 On-link 172.27.29.16 266
    255.255.255.255 255.255.255.255 On-link 127.0.0.1 331
    255.255.255.255 255.255.255.255 On-link 169.254.1.201 271
    255.255.255.255 255.255.255.255 On-link 172.27.29.16 266
    ===========================================================================
    Persistent Routes:
    Network Address Netmask Gateway Address Metric
    0.0.0.0 0.0.0.0 172.27.29.254 Default
    0.0.0.0 0.0.0.0 172.27.29.254 Default
    ===========================================================================
    That's what makes it odd. And since you do have a ways to ignore OS-Settings namely DIP I ask myself if there is leaking or missbehaving code. 
    PS: couldn't find and detailed docu on how DIP works on a technical level so far, do you know of any Whitepapers, Tech-Specs or Procedural diagrams?
    cvfwd gets restarted regularly, since its Windows at least 1-2 times per month with the Server. Did check the Logs before and after restart to cover that, doesn't seem to be it. Still, suggestion to restart services or reboot server is always good suggestion on windows :-)
    will skip the IPv6 part, unrealistic suggestion when this will be the only way going forwards. Network Guys want to switch first zones to v6-only end of year, will see how much is really ready for it.

  • Re: cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-28-2020, 11:50 AM
    • Aplynx is not online. Last active: 06-04-2020, 4:59 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,838

    DIPs are used when you have a backup IP that you want to use between the MA and Client, you would override the name resolution by telling CommVault to use a particular IP. 

    http://documentation.commvault.com/commvault/v11/article?p=7140.htm

    Principal is still the same though. Basically override name resolution with IP. 

  • Re: cvfwd & Microsoft Failover Cluster Virtual Adapter
    Posted: 05-28-2020, 11:59 AM
    • Aplynx is not online. Last active: 06-04-2020, 4:59 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,838

    I haven't seen this as an issue for a couple of CV versions now. Probably worth opening a case to review. 

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy