Unix Client Failing Check Readiness

Last post 02-27-2020, 1:21 PM by pereirath. 6 replies.
Sort Posts: Previous Next
  • Unix Client Failing Check Readiness
    Posted: 02-25-2020, 5:37 PM

    Anyone seeing this before? Client was working previously and stopped. Other clients wiht the same agent working fine.

    Error: Failed to start phase [Backup] on [clienthostname] due to network error [Remote machine [clienthostname]. The socket connect failed. Error returned [-1=CON6239/clienthostname*clienthostname*8400*8402: All proxies reported clienthostname as unreachable]. Check Network Connectivity.]. Will attempt to restart. Please check if this product's services are running on the remote host.

     

    Validated the following

    • Services running on the client
    • 8400 port is listening on the client
    • Services were restarted
    • Client has valid backups and just stop communicating

    ##############

    CVD.log

    7591 a    02/25 14:54:59 ########## TPool [N/A]. Ser# [1] Tot [3], Pend [0], Comp [3], Max Par [1], Time (Serial) [0.008814]s, Time (Parallel) [0.008814]s, Wait [0.000446]s

    7591 5    02/25 14:55:42 ########## connect() - ERROR: Connect to 127.0.0.1:54474 failed: Connection timed out

    7591 5    02/25 14:55:42 ########## EvSocket::doConnect() - Could not connect to clienthostname(clienthostname):EvMgrS: Connect to 127.0.0.1:54474 failed: Connection timed out

    7591 5    02/25 14:55:42 ########## checkEventSocket() - Unable to setup event connection to EvMgrS to send data.

    7591 5    02/25 14:55:42 ########## logMessageToInbox() - Saving unset message to Inbox: file "/var/log/commvault/Log_Files/EV_Inbox/_EV_u9elLa".

    7591 a    02/25 14:59:59 ########## TPool [N/A]. Ser# [1] Tot [4], Pend [1], Comp [3], Max Par [2], Time (Serial) [0.010473]s, Time (Parallel) [15.012826]s, Wait [0.000621]s

    7591 213e 02/25 15:01:09 ########## connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.

    7591 213e 02/25 15:03:09 ########## connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.

    7591 213e 02/25 15:03:09 ########## pool/CvNetworkPool.cpp(950): -Debug-: Critical -- pool/CvNetworkPool.cpp(632): -Error-: getSessionObject() - Failed to start a Session. Error Message: 0xE8090010:{CvNetworkPool::startASession(510)} + {CCvNetwork::Attach_RemoteService(1512)} + {CCvNetwork::Attach_RemoteService(1481)} + {CCvNetwork::LookupRegisteredPorts(4531)} + {CCvNetwork::LookupRegisteredPorts_Remote(4326)} + {CSessionConnectionSocket::LookupPortRemotely(1117)} + {CCvNetwork::Attach_RemotePort(1116)} + {CCvNetwork::GetConnection(2954)} + {CSessionConnectionSocket::GetConnection(189)/EventMessage.0x09000010.9:16}

    7591 213e 02/25 15:03:09 ########## [ClientSessionWrapper::connectToCS] Connnecting to [AppManager network access] service on CommServer.

    7591 a    02/25 15:04:59 ########## TPool [N/A]. Ser# [1] Tot [4], Pend [1], Comp [3], Max Par [2], Time (Serial) [0.010866]s, Time (Parallel) [300.058787]s, Wait [0.000507]s

    7591 213e 02/25 15:05:09 ########## connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.

    7591 213e 02/25 15:07:09 ########## connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.

    7591 213e 02/25 15:07:09 ########## [ClientSessionWrapper::connectToCS] Error connecting to svc [AppManager network access] on CS. [0xE8090010:{CCvNetwork::Attach_RemoteService(1512)} + {CCvNetwork::Attach_RemoteService(1481)} + {CCvNetwork::LookupRegisteredPorts(4531)} + {CCvNetwork::LookupRegisteredPorts_Remote(4326)} + {CSessionConnectionSocket::LookupPortRemotely(1117)} + {CCvNetwork::Attach_RemotePort(1116)} + {CCvNetwork::GetConnection(2954)} + {CSessionConnectionSocket::GetConnection(189)/EventMessage.0x09000010.9:16}]

    7591 213e 02/25 15:07:09 ########## ClientInterfaceNet::lookup() - [ClientSessionWrapper::connectToCS] Error connecting to svc [AppManager network access] on CS. [0xE8090010:{CCvNetwork::Attach_RemoteService(1512)} + {CCvNetwork::Attach_RemoteService(1481)} + {CCvNetwork::LookupRegisteredPorts(4531)} + {CCvNetwork::LookupRegisteredPorts_Remote(4326)} + {CSessionConnectionSocket::LookupPortRemotely(1117)} + {CCvNetwork::Attach_RemotePort(1116)} + {CCvNetwork::GetConnection(2954)} + {CSessionConnectionSocket::GetConnection(189)/EventMessage.0x09000010.9:16}]

    7591 213e 02/25 15:07:09 ---------- IndexCacheMaintenance() - Error: Unable to obtain ID for tyhe client name [clienthostname]

    7591 a    02/25 15:09:59 ########## TPool [N/A]. Ser# [1] Tot [4], Pend [0], Comp [4], Max Par [1], Time (Serial) [480.053478]s, Time (Parallel) [164.991998]s, Wait [0.000465]s

    7591 216c 02/25 15:13:15 ---------- CCVDSpaceCheck::CleanupIdleEventConnections() - Invoking CleanUp Idle Event Connections.

    7591 216c 02/25 15:13:15 ########## periodicallyCleanUpIdleEventConnections() - CleanUpIdleEventConnections  started

     

    ClOraAgent.log

    1521 1   02/25 15:00:31 9484725 ::Initialize1() - ClientHostName=clienthostname CtrlClientName=clienthostname*clienthostname*8400*8402, ServerHostName=commserve.domain.com

    1521 1   02/25 15:00:31 9484725 ::Initialize1() - getting args (for GUI case) db_name=[(null)] logsBackup=[FALSE] mode=0 op = [CrossCheck] ...

    1521 1   02/25 15:00:31 9508605 ::Initialize1() - m_totalBytesDiff=[524288000].

    1521 1   02/25 15:00:31 9508605 ::Initialize1() - secondsToWait=30 mode=0 m_backup=8000040C.

    1521 1   02/25 15:00:31 9508605 OraBaseContext::Initialize() - request for job: 9508605

    1521 1   02/25 15:00:31 9508605 OraBaseContext::setJobModes() - m_oraObjNameList = [B F A F]

    1521 1   02/25 15:00:31 9508605 OraBaseContext::InitAppMgrForCrossCheckORCloneClean() - clientId:3235

    1521 1   02/25 15:02:31 9508605 connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.

    1521 1   02/25 15:04:31 9508605 connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.

    1521 1   02/25 15:04:31 9508605 pool/CvNetworkPool.cpp(950): -Debug-: Critical -- pool/CvNetworkPool.cpp(632): -Error-: getSessionObject() - Failed to start a Session. Error Message: 0xE8090010:{CvNetworkPool::startASession(510)} + {CCvNetwork::Attach_RemoteService(1512)} + {CCvNetwork::Attach_RemoteService(1481)} + {CCvNetwork::LookupRegisteredPorts(4531)} + {CCvNetwork::LookupRegisteredPorts_Remote(4326)} + {CSessionConnectionSocket::LookupPortRemotely(1117)} + {CCvNetwork::Attach_RemotePort(1116)} + {CCvNetwork::GetConnection(2954)} + {CSessionConnectionSocket::GetConnection(189)/EventMessage.0x09000010.9:16}

    1521 1   02/25 15:04:31 9508605 [ClientSessionWrapper::connectToCS] Connnecting to [AppManager network access] service on CommServer.

    1521 1   02/25 15:06:31 9508605 connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.

    1521 1   02/25 15:08:31 9508605 connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.

    1521 1   02/25 15:08:31 9508605 [ClientSessionWrapper::connectToCS] Error connecting to svc [AppManager network access] on CS. [0xE8090010:{CCvNetwork::Attach_RemoteService(1512)} + {CCvNetwork::Attach_RemoteService(1481)} + {CCvNetwork::LookupRegisteredPorts(4531)} + {CCvNetwork::LookupRegisteredPorts_Remote(4326)} + {CSessionConnectionSocket::LookupPortRemotely(1117)} + {CCvNetwork::Attach_RemotePort(1116)} + {CCvNetwork::GetConnection(2954)} + {CSessionConnectionSocket::GetConnection(189)/EventMessage.0x09000010.9:16}]

    1521 1   02/25 15:08:31 9508605 ClientInterfaceNet::lookup() - [ClientSessionWrapper::connectToCS] Error connecting to svc [AppManager network access] on CS. [0xE8090010:{CCvNetwork::Attach_RemoteService(1512)} + {CCvNetwork::Attach_RemoteService(1481)} + {CCvNetwork::LookupRegisteredPorts(4531)} + {CCvNetwork::LookupRegisteredPorts_Remote(4326)} + {CSessionConnectionSocket::LookupPortRemotely(1117)} + {CCvNetwork::Attach_RemotePort(1116)} + {CCvNetwork::GetConnection(2954)} + {CSessionConnectionSocket::GetConnection(189)/EventMessage.0x09000010.9:16}]

    1521 1   02/25 15:10:31 9508605 connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.

    1521 1   02/25 15:12:31 9508605 connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.

    1521 1   02/25 15:12:31 9508605 pool/CvNetworkPool.cpp(950): -Debug-: Critical -- pool/CvNetworkPool.cpp(632): -Error-: getSessionObject() - Failed to start a Session. Error Message: 0xE8090010:{CvNetworkPool::startASession(510)} + {CCvNetwork::Attach_RemoteService(1512)} + {CCvNetwork::Attach_RemoteService(1481)} + {CCvNetwork::LookupRegisteredPorts(4531)} + {CCvNetwork::LookupRegisteredPorts_Remote(4326)} + {CSessionConnectionSocket::LookupPortRemotely(1117)} + {CCvNetwork::Attach_RemotePort(1116)} + {CCvNetwork::GetConnection(2954)} + {CSessionConnectionSocket::GetConnection(189)/EventMessage.0x09000010.9:16}

    1521 1   02/25 15:12:31 9508605 [ClientSessionWrapper::connectToCS] Connnecting to [AppManager network access] service on CommServer.

    1521 1   02/25 15:14:31 9508605 connect() - ERROR: Connect to 127.0.0.1:54474 timed-out.


     
  • Re: Unix Client Failing Check Readiness
    Posted: 02-25-2020, 8:13 PM
    • efg is not online. Last active: 03-24-2020, 6:10 PM efg
    • Top 10 Contributor
    • Joined on 02-02-2010
    • CommVault Tinton Falls NJ
    • Master
    • Points 1,720

    A couple things...   Looks like the connection is timing out.   Is this a Linux server?   Was the server rebooted and then the connections started failing?   Could it be that the firewall services were manually stopped but not disabled, so that on reboot they started again and now the connections are blocked.   Try running the command "iptables -L" as root to see if the firewall is on.  You may need to stop the services.   One other thing I noticed is that the connection to local loopback (127.0.0.1) is timing out.   Can you verify that the "lo" device is enabled on the server?  I have seen cases where the loopback device got disabled somehow on the server which caused similar problems.

    Let us know if this helps...

    BTW...  on most redhat systems the command "systemctl stop firewalld" will stop the firewall and "systemctl disable firewalld" will disable on boot.


    Ernst F. Graeler
    Senior Engineer III
    Development
  • Re: Unix Client Failing Check Readiness
    Posted: 02-25-2020, 8:34 PM

    Thanks EFG for your suggestion. This isnt the first time I have this issue, and I dont want to just reinstall the agent.

    Do you have the commands for Solaris client?

  • Re: Unix Client Failing Check Readiness
    Posted: 02-25-2020, 9:54 PM

    Can you provide the FwConfig.txt for the Solaris Server?

    Can you perform reverse DNS lookups from both client and server?

    Can you successfully cvping the client hostname from the CommServe and vice versa?

  • Re: Unix Client Failing Check Readiness
    Posted: 02-26-2020, 12:04 PM

    Io looks good and firewall is not enable  on the client. reponse from solaris admin.

     

    The logs from CVfwd.log reveal something interesting that Im not entirely sure I fully understand. Seems like a few ports that commvault uses were being refused connection or in use (8400/8403) and that was blocking the connection perhaps? Which then force the client to try a different host and pointed to my CommServe DR and it connected? Maybe? 

    • From a CV Admin, it is very complicated for me to troubleshoot this since I dont have much solaris access, how can we setup perhaps reset connections from commvault side proactively to avoid clients communication issues? The restart services obvioulsy wont work from commvault since the communication is interrupted.
    • I searched for the errors below and nothing came up on CV. 
    • Seems like Commvault client got confused and couldnt firmly re-estabilshed the connection on 8400/8403.
    • The other thing im not sure is if the DR server actially played negativily here in "connecting" with the client.

    Thoughts? 

    BEFORE CS Service Restart

    CVFwd.log

    7600 0001 02/23 22:17:44 TN:00438 CN:32980 Completed tunneled connection {e589ade8-96f2-5d72-9f42-faca973772ab} to local:59376

    7600 0001 02/23 22:29:17 TN:00439 ######## ERROR: cvfwd_reset_tunnel(): Ungraceful termination of PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE"" via (IP_ADDRESS)

    7600 0001 02/23 22:30:27 TN:00439 CN:27383 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32804 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32910 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32921 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32925 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32941 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32940 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32950 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32953 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32956 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32964 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32965 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00439 CN:32968 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00438 ######## ERROR: cvfwd_reset_tunnel(): Ungraceful termination of PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)

    7600 0001 02/23 22:30:27 TN:00438 CN:27461 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00438 CN:32903 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00438 CN:32912 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00438 CN:32966 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00438 CN:32967 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00438 CN:32969 Parking client connection due to a tunnel failure

    7600 0001 02/23 22:30:27 TN:00438 CN:32981 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32982 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32983 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32984 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32985 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32986 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32987 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32988 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32989 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32990 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32991 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32992 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:30:27 TN:00438 CN:32993 Client connection dropped due to tunnel disconnect [PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)]

    7600 0001 02/23 22:31:27 TN:00647 ######## Initialized new PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE_Instace2" via (IP_ADDRESS)

    7600 0001 02/23 22:31:27 TN:00646 ######## Initialized new PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE" via (IP_ADDRESS)

    7600 0001 02/23 22:31:57 ######## CN:27461 Replacing parked client connection with CN:33024

    7600 0001 02/23 22:31:57 TN:00646 CN:33024 Taking over parked client connection CN:27461

     

    AFTER CS Service RESTART 

    17842 0001 02/26 08:11:12 ######## ######## tunnel_ports=8403

    17842 0001 02/26 08:11:12 ######## ######## [outgoing]

    17842 0001 02/26 08:11:12 ######## ######## solarisclient commserve remote_guid=8B5D7011-69FC-4819-AA47-BEDD036EEE2A fallback=1 type=ondemand proto=httpsa cvfwd=commserve.domain.com:8403 encryption=med extraports=8400,8601-8620

    17842 0001 02/26 08:11:12 ######## ######## solarisclient commserve remote_guid=8B5D7011-69FC-4819-AA47-BEDD036EEE2A proxy=Commserve_Instance2

    17842 0001 02/26 08:11:12 ######## ######## solarisclient commserve remote_guid=8B5D7011-69FC-4819-AA47-BEDD036EEE2A proxy=DRCOMMSERVE

    17842 0001 02/26 08:11:12 ######## ######## solarisclient Commserve_Instance2 remote_guid=E684500C-6EB5-496F-B3F6-0A723937F73A type=persistent proto=httpsa cvfwd=commserve.domain.com:8408 encryption=med

    17842 0001 02/26 08:11:12 ######## ######## solarisclient DRCOMMSERVE remote_guid=500B9A9E-1E67-A3EB-9801-4F270347D297 type=persistent proto=httpsa cvfwd=DRCOMMSERVE.domain.com:8408 encryption=med

    17842 0001 02/26 08:11:12 ######## ######## Creating IPv4 socket for listening to client connections.

    17842 0001 02/26 08:11:12 ######## ######## Client fd=4 bound to 0.0.0.0:63594.

    17842 0001 02/26 08:11:12 ######## ######## Creating IPv6 socket for listening to client connections.

    17842 0001 02/26 08:11:12 ######## ######## Client fd=6 bound to :::53286.

    17842 0001 02/26 08:11:12 ######## ######## ERROR: cvfwd_process_tunnel_ports(): IPv4 port 8403 is in use. Will retry in 60 seconds.

    17842 0001 02/26 08:11:12 ######## ######## Start listening to incoming messages.

    17842 0001 02/26 08:11:13 ######## ######## Initializing SSL subsystem

    17842 0001 02/26 08:11:13 TN:00001 ######## Initialized new PERSISTENT tunnel from "solarisclient" to "Commserve_Instance2" via (IPADDRESS)

    17842 0001 02/26 08:11:13 TN:00002 ######## Initialized new PERSISTENT tunnel from "solarisclient" to "DRCOMMSERVE" via (IPADDRESS)

     

     

    KILLED COMMVAULT PROCESS AND KICKED OFF AGAIN

    user@solarisclient:~$ netstat -an | grep LISTEN

           *.8403               *.*                  0      0  256000      0 LISTEN

          *.8400               *.*                  0      0  262640      0 LISTEN

    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    user@solarisclient:~$ netstat  -an | grep 840
    *.8403 *.* 0 0 256000 0 LISTEN
    IP.64176 COMMSERVE.8408 64660 0 65160 0 ESTABLISHED
    IP.37349 DR COMMSERVE.8408 63952 0 65160 0 ESTABLISHED
    *.8400 *.* 0 0 262640 0 LISTEN

     

    Everything is back working now. 

  • Re: Unix Client Failing Check Readiness
    Posted: 02-26-2020, 1:58 PM

    Port 8408 is used for the Highly Available CommServe on both your CommServe and DR CommServe. These are a small cut down version of of Commvault running just the core Commvault services with the failover package on the second instance (hence why the port is in 8405-8409 range). Clients will communicate with both CommServe's over 8408 continuously unless the CommServe communication was interrupted (eg patching, service restarts, networking, or otherwise). To help further, can you supply the FwConfig.txt for the CommServe, DR Commserve, and the client? What Commvault version, service pack and hotfix level are running on these machines.

  • Re: Unix Client Failing Check Readiness
    Posted: 02-27-2020, 1:21 PM

    So I was able to pin point the exact time where commvault loses connection on the client side. So often times we have a security vulnerability scan performed in our environment by a solution called Nessus from Tenable. The day this client stopped working, was the same date this software was kicked off, so I internally asked these guys to re-run the scan so I could evaluate Commvault logging behaviour and this is the output cvfwd.txt 

     

    This is a common behaviour among all the solaris servers. 

     

    TN:00002 CN:00119 Completed tunneled connection {8bfb79d2-8152-f785-a4-f0f6b56a2715} to local:8400

    TN:00001 CN:00123 Completed tunneled connection {93ae76a7-392e-0735-0e5b9fa47d673} to local:8400

     ######## ######## ERROR: cvfwd_validate_command(): Invalid opening message signature 0x47

    TN:00003 ######## ERROR: cvfwd_dispatch_one_tunnel(): Received invalid command from fd=12. Drop the socket.

     ######## ########   47 45 54 20 2f 20 48 54  54 50 2f 31 2e 30 0d 0a  |GET / HTTP/1.0..|

     ######## ########   0d 0a                                             |..|

     ######## ######## ERROR: cvfwd_validate_command(): Invalid opening message signature 0x48

    TN:00004 ######## ERROR: cvfwd_dispatch_one_tunnel(): Received invalid command from fd=12. Drop the socket.

     ######## ########   48 45 4c 50 0d 0a                                 |HELP..|

     TN:00002 CN:00145 Completed tunneled connection {3932d40e-cf13-c6ac-b88a-4c515f78} to local:8400

     TN:00002 CN:00146 Completed tunneled connection {0e453e32-980f-f23b-7d33-91e0da212} to local:8400

     TN:00001 CN:00149 Completed tunneled connection {b8bc5c2d-3343-dfce-0caf-b42b07eb} to local:8400

    TN:00002 CN:00155 Completed tunneled connection {5dc950a4-81f0-8699-2e45-144df36f1} to local:8400

    TN:00001 CN:00159 Completed tunneled connection {f22ff81b-dc05-d1ed-9450-db75989ae} to local:8400

     TN:00002 CN:00165 Completed tunneled connection {ae9e3af1-4abe-fbe4-e05f-15b934ec27} to local:8400

     ######## ######## ERROR: cvfwd_validate_command(): Invalid opening message signature 0x47

    TN:00005 ######## ERROR: cvfwd_dispatch_one_tunnel(): Received invalid command from fd=42. Drop the socket.

    ######## ########   47 45 54 20 2f 20 48 54  54 50 2f 31 2e 30 0d 0a  |GET / HTTP/1.0..|

    ######## ########   0d 0a                                             |..|

     ######## ######## ERROR: cvfwd_validate_command(): Invalid opening message signature 0x48

    TN:00006 ######## ERROR: cvfwd_dispatch_one_tunnel(): Received invalid command from fd=42. Drop the socket.

     ######## ########   48 45 4c 50 0d 0a                                 |HELP..|

    TN:00001 CN:00242 Completed tunneled connection {426e4f-8c-e75d-ccc3-c1707a9dbecd} to local:8400

    TN:00001 CN:00243 Completed tunneled connection {1dada8-8dd7-2a6f-0952-146c0c062ebe} to local:8400

     TN:00002 CN:00251 Completed tunneled connection {433fcb-6ae5-549e-04bb-6ddc81288a6a} to local:8400

     TN:00002 CN:00261 Completed tunneled connection {787dd1-bfca-6b27-b9bf-11904e00df08} to local:8400

    TN:00001 CN:00262 Completed tunneled connection {379078-2022-6e13-a2ac-5142108c668d} to local:8400

     

    So Nessus on the other side, tests many different ports, tries to connect, read and test. Once it finishes it releases the port, but commvault seems to not understand that and simply stop working. 

     

    - main(): execution starting...

    -Port 8400 tests starting...

    -Port 8400 state S_STATE_CONNECTING starting...

    - port_new(): Opened socket to port 8400

    -Port 8403 tests starting...

    -Port 8403 state S_STATE_CONNECTING starting...

    - port_new(): Opened socket to port 8403

    - port_process(): Port 8400 test starting...

    -Port 8400 state S_STATE_CONNECTING completed in 0.022051 seconds.

    -Port 8400 state S_STATE_READING starting...

    - port_process(): Port 8400 test completed in 0.001057 seconds.

    - port_process(): Port 8403 test starting...

    -Port 8403 state S_STATE_CONNECTING completed in 0.021054 seconds.

    -Port 8403 state S_STATE_READING starting...

    - port_process(): Port 8403 test completed in 0.000538 seconds.

    .  port_done(): Closed socket to port 22

    .  port_send_get(): Port 8400 test starting...

    . Port 8400 state S_STATE_READING completed in 1.398179 seconds.

    . Port 8400 state S_STATE_READING_W_GET starting...

    .  port_send_get(): Port 8400 test completed in 0.000680 seconds.

    .  port_send_get(): Port 8403 test starting...

    . Port 8403 state S_STATE_READING completed in 1.398361 seconds.

    . Port 8403 state S_STATE_READING_W_GET starting...

    .  port_send_get(): Port 8403 test completed in 0.000801 seconds.

    . Port 515 state S_STATE_READING_W_GET completed in 0.007756 seconds.

    . Port 515 state set to S_STATE_DONE.

    .  port_done(): Closed socket to port 515

    . Port 515 tests completed in 1.426273 seconds.

    . Port 8400 state S_STATE_READING_W_GET completed in 0.008466 seconds.

    . Port 8400 state set to S_STATE_DONE.

    .  port_done(): Closed socket to port 8400

    . Port 8400 tests completed in 1.430724 seconds.

    . Port 8403 state S_STATE_READING_W_GET completed in 0.008536 seconds.

    . Port 8403 state set to S_STATE_DONE.

    .  port_done(): Closed socket to port 8403

    . Port 8403 tests completed in 1.429519 seconds.

    .  main(): select() loop completed in 1.420362 seconds.

    .  main(): execution completed in 1.438703 seconds.

    . find_service.nasl completed in 1.440292 seconds.

     

    To summarise the client is working and this is definitely helps us to better troubleshoot this in the future, however I would be interested to hear what causes Commvault to at some point stop persisting the port and just crashes. Thoughts?

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy