Thankyou for sharing the details on how it was previously setup.
With a single VIC which has two physical ports, the following method could have been adopted for better resilience. (Please refer to the cabling picture I shared earlier for a mapping of the "Ethernet" port numbers below):
VIC Physical port #1:
Ethernet #0: Data port to SW1
Ethernet #1: HB port to SW1
VIC Physical port #2:
Ethernet #2: Data port to SW2
Ethernet #3: HB port to SW2
similarly on other two nodes. Please note, the same VIC, which is a Converged Network Adaptor (CNA), can also contain what Cisco calls virtual HBA (vHBA) for Fiber-Channel traffic, if required. This assumes the customer is using fabric-interconnects(F.I) to manage the servers.