Backup failing at scan phase

Last post 08-12-2019, 4:06 AM by Anthony.Hodges. 13 replies.
Sort Posts: Previous Next
  • Backup failing at scan phase
    Posted: 07-16-2019, 4:18 AM

    Hi Team, 

    One of my HPUX server backup is failing on below error.

    I installed the agent V10 SP9 on it from Command line and it went well.

    I can cvping as

    CS to client - yes

    Client to CS - Yes

    MA to Client - yes

    Client to MA - Yes

    Client OS : HP-UX B.11.31

    Platform: ia64

    Commserver version - V11SP12

    Failed to start phase [Scan] on [hpxi001a] due to network error [Remote system [hpxi001a*hpxi001a*8400*8402]. Communications Service returned error when executing remote command [ifind.exe] arguments [-j 4254299 -a 2:3742 -t 2 -d inpnqvsam08*inpnqvsam08*8400*8402 -r 1559369352 -ab 0 -i 1 -cs inpnqcmvs01 -s "default"  -jt 4254299:4:15  -idxma inpnqvsam08*inpnqvsam08*8400*8402  -mountPath  -seb  -ffe 1  -lf 0 -li 4203835 -ls 4193281  -attrEx 0 ].]. Will attempt to restart. Please check if this product's services are running on the remote host.

     

    Any help/ Guidance is highly appreciated

  • Re: Backup failing at scan phase
    Posted: 07-19-2019, 2:42 AM

    Are the services running on the hpxi001a box?

    sudo /opt/commvault/simpana/status

    sudo /opt/commvault/simpana/list

    Any reason you are running v10sp9?  SP15 was the last SP for v10.

  • Re: Backup failing at scan phase
    Posted: 07-19-2019, 3:38 AM

    THere can be multiple cause for the issue.

     

    1. Check if you are able to telnet  8400-8404 ports from master to client and media agent to client. May be be it is able to connect with other ports but for the posrts it is trying to scan is used by other application or the port is somehow down

    2.PLease check/share CVMA,CVD,Filescan logs for more details.

  • Re: Backup failing at scan phase
    Posted: 07-19-2019, 3:54 AM
    hpxi001a:/root >sudo /opt/commvault/simpana/status
    sudo: /etc/sudoers is mode 0644, should be 0440
    hpxi001a:/root >Jun 22 16:44:14 hpxi001a sudo: root : /etc/sudoers is mode 0644, should be 0440 ; TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/opt/commvault/simpana/status
    ^C
    hpxi001a:/root >sudo /opt/commvault/simpana/list
    sudo: /etc/sudoers is mode 0644, should be 0440
    hpxi001a:/root >Jun 22 16:47:27 hpxi001a sudo: root : /etc/sudoers is mode 0644, should be 0440 ; TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/opt/commvault/simpana/list

     
    Only reason to use V10SP9 is its an old vm running on HP-UX - HP-UX B.11.31 - ia64
  • Re: Backup failing at scan phase
    Posted: 07-19-2019, 5:03 AM
    Sorry those commands where wrong. :(

    Double-check the paths, but it should be something like this…

    /opt/commvault/Base/simpana status

    /opt/commvault/Base/simpana list

    The list command should show the Commvault services that are running.

    If there are no services running , perhaps just run

    /opt/commvault/simpana start
  • Re: Backup failing at scan phase
    Posted: 07-22-2019, 3:16 AM

    Actually we have HitachiHDS agetninstalled on the machine and not simpana.

    Below is the output.

     

    hpxi001a:/root >HitachiHDS status
    --------------------------------- Instance001 ---------------------------------
    [ General ]
    Version = 11(BUILD80) SP12 (1303739)
    CommServe = inpnqcmvs01.corp.amdocs.com
    Home Directory = /opt/hds/Base
    Log Directory = /var/log/hds/Log_Files
    Core Directory = /opt/hds
    Temp Directory = /opt/hds/Base/Temp
    Platform Type = 4
    Cvd Port Number = 8400
    [ Package ]
    1002/CVGxBase 11(BUILD80) = File System Core
    [ Physical Machine/Cluster Groups ]
    Display Name = hpxi001a
    - Client Hostname = hpxi001a
    - Job Results Directory = /opt/hds/iDataAgent/jobResults
    hpxi001a:/root >HitachiHDS list

    *** Instance001, inpnqcmvs01 ***
    +-----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Service | PID | Description |
    +-----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | cvlaunchd | 3441 | Data Protection Suite Launch Daemon |
    +-----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | cvd | 3442 | Data Protection Suite Communications Service |
    +-----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    hpxi001a:/root >
  • Re: Backup failing at scan phase
    Posted: 07-22-2019, 3:50 AM

    Also while doing check readyness I am getting below result.

    Client Not Ready. 1. System Temp Folder has low capacity.


    though /opt/has enough space. 

     
     
    hpxi001a:/opt/hds/Base >HitachiHDS status
    --------------------------------- Instance001 ---------------------------------
    [ General ]
    Version = 11(BUILD80) SP12 (1303739)
    CommServe = inpnqcmvs01.corp.amdocs.com
    Home Directory = /opt/hds/Base
    Log Directory = /var/log/hds/Log_Files
    Core Directory = /opt/hds
    Temp Directory = /opt/hds/Base/Temp
    Platform Type = 4
    Cvd Port Number = 8400
    [ Package ]
    1002/CVGxBase 11(BUILD80) = File System Core
    [ Physical Machine/Cluster Groups ]
    Display Name = hpxi001a
    - Client Hostname = hpxi001a
    - Job Results Directory = /opt/hds/iDataAgent/jobResults
    hpxi001a:/opt/hds/Base >
     
    hpxi001a:/opt/hds/Base >df -k /opt
    /opt (/dev/vg00/lvol6 ) : 31413736 total allocated Kb
    5691024 free allocated Kb
    25722712 used allocated Kb
    82 % allocation used
    hpxi001a:/opt/hds/Base >
  • Re: Backup failing at scan phase
    Posted: 07-22-2019, 6:07 PM

    Hi Atul K

    The issue at hand from the errors reported from the JobManager looks to indicate an communication issue between CommServe and Client (as it is stuck in the filescan phase).

    To rule out any potential network issue, could you put in place a Commvault Network Route Configuration (V11 - formerly known as CommVault Firewalls) between Client and CommServe - http://documentation.commvault.com/commvault/v11/article?p=59407.htm

    The purpose is to create an application consistent tunnel, and if their are actual communication issues (with in the environment) the additional logging from Commvault (in cvfwd.log) will also help 

    Can you please do the above and share the outcome

    Regards

    Winston

     

  • Re: Backup failing at scan phase
    Posted: 07-23-2019, 10:37 AM
    • efg is not online. Last active: 08-22-2019, 5:13 PM efg
    • Top 10 Contributor
    • Joined on 02-02-2010
    • CommVault Tinton Falls NJ
    • Expert
    • Points 1,672

    Hi Atul,

    Based on the output from the "HitachiHDS status" command the client actually has V11 SP12 installed, but only the "File System Core" package.

    hpxi001a:/root >HitachiHDS status
    --------------------------------- Instance001 ---------------------------------
    [ General ]
     Version = 11(BUILD80) SP12 (1303739)
     CommServe = inpnqcmvs01.corp.amdocs.com
     Home Directory = /opt/hds/Base
     Log Directory = /var/log/hds/Log_Files
     Core Directory = /opt/hds
     Temp Directory = /opt/hds/Base/Temp
     Platform Type = 4
     Cvd Port Number = 8400
    [ Package ]
     1002/CVGxBase 11(BUILD80) = File System Core
    [ Physical Machine/Cluster Groups ]
     Display Name = hpxi001a
      - Client Hostname = hpxi001a
      - Job Results Directory = /opt/hds/iDataAgent/jobResults
    hpxi001a:

    Please check to make sure to install the FULL File system package as "core" has limited functionality and is more meant for "basic" laptop based installations.  Can you try and "push" the standard FS package from the commcell?  

    I also didn't see if you tried running the "Check Readiness" from the commcell...   If that succeeds then you should be able to push the install directly from the commcell.

     

    Let us know how you make out.


    Ernst F. Graeler
    Senior Engineer III
    Development
  • Re: Backup failing at scan phase
    Posted: 08-09-2019, 5:45 AM

    Hi Friend

    I managed to install V11 SP8+ on this server. Check readyness is OK as well.

    But while i start the full backup of the server it fails on below error.

            Error Code: [19:1304]
    Description: Failed to connect to Communications Service to register the process.
    Source: hpxi001a, Process: FileScan     

    --------------------------------------------******--------------------------------------------

    hpxi001a:/system_kits/CommVault/CV_V11SP8_Unix/11.0.0/Unix >HitachiHDS status
    --------------------------------- Instance001 ---------------------------------
    [ General ]
    Version = 11(BUILD80) SP8
    CommServe = inpnqcmvs01
    Home Directory = /opt/hds/Base
    Log Directory = /var/log/hds/Log_Files
    Core Directory = /opt/hds
    Temp Directory = /opt/hds/Base/Temp
    Platform Type = 4
    Cvd Port Number = 8400
    [ Package ]
    1002/CVGxBase 11(BUILD80) = File System Core
    1101/CVGxIDA = File System
    [ Physical Machine/Cluster Groups ]
    Display Name = hpxi001a
    - Client Hostname = hpxi001a
    - Job Results Directory = /opt/hds/iDataAgent/jobResults
    hpxi001a:/system_kits/CommVault/CV_V11SP8_Unix/11.0.0/Unix >
  • Re: Backup failing at scan phase
    Posted: 08-09-2019, 1:48 PM
    • efg is not online. Last active: 08-22-2019, 5:13 PM efg
    • Top 10 Contributor
    • Joined on 02-02-2010
    • CommVault Tinton Falls NJ
    • Expert
    • Points 1,672

    Check to see if there may be more than one version of commvault running on the server.  I have seen cases where after several failed install attempts where there were "ghost" cvd processes still running (resident in memory) along with the "real" cvd process (for the valid install).  These "ghost" processes would interfere with starting/running jobs, while a check-readiness would succeed.

    Try stopping the commvault services (use "HitachiHDS stop")  then use the "ps" command to make sure there are no other commvault processes still running.   For example:

    # ps -aef |grep "/opt/hds"

    or

    # ps -aef |grep "cvd"

    If there arey any additional processes, kill them using "kill -9"

    then restart the services and see if the backup runs.

    The only other problem I can think of would be if the ifind process is crashing (for file scan) as that could produce a similar error message.  Check if there is a FileScan.log under "/var/log/hds/Log_Files" to see if there are any errors logged there.

    You may also want to push out the latest service pack to the client.  From what I recall from an earlier post your CS is at SP12.

     

    If you are still encountering problems I would recommend getting a support ticket opened.


    Ernst F. Graeler
    Senior Engineer III
    Development
  • Re: Backup failing at scan phase
    Posted: 08-11-2019, 3:13 AM

    Do you what ports are open between the HPUX server and the CommServe & MediaAgents?  Is it possible that you have chosen a Storage Policy where the Primary Copy is in a network that is not accessible to the HPUX by accident?

    What is in the /Base/FWConfig.txt file on the HPUX?

    Are you using hosts files instead of DNS?  Can you do reverse DNS lookups from both ends?

  • Re: Backup failing at scan phase
    Posted: 08-12-2019, 2:49 AM

    I am getting attached error while i start the job.

    Attachment: Commvault error.txt
  • Re: Backup failing at scan phase
    Posted: 08-12-2019, 4:06 AM

    From the looks of it to me it could not get an SSL connection because the time difference betwen the CommServe and the Client appears to be large.

     

    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    Machine : inpnqcmvs01
    File    : JobManager_2019_08_12_12_00_26.log
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

    11148 3fc8  08/12 11:57:17 4335999 Delay Msg  successfully deleteSavedJMJobUpdateMsgByJob
    11148 3fc8  08/12 11:57:17 ####### Servant    Kill operation completed in [0] seconds. Number of jobs affected=[1]
    11148 37b8  08/12 11:57:17 4335999 Service    JMDataLinkRelated ret 2 open=0 close=0 update=0 links=0 instance=0 data=0 logs=0 bkpLevel=2
    11148 37b8  08/12 11:57:18 4335999 Scheduler   -> KILLED BY USER <- Completed JobType[Backup] CompletionStatus[Killed] Backup job [4335999] has been killed by NTNET\AKASAR. Reason:[]. Client [hpxi001a], Agent Type [HP-UX File System], Subclient [ArchiveLogs], Backup Level [Incremental], Duration [00:06:23].

     

    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    Machine : hpxi001a
    File    : FileScan.log
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

    5982 1   07/16 13:02:12 4335999 CvClntCert::loadLib() - Successfully loaded /opt/hds/Base/libCvClntCertImpl.so
    5982 1   07/16 13:02:13 4335999 Cvcl::init() - CVCL: Running in FIPS Mode
    5982 1   07/16 13:02:13 4335999 FindUNIX::FindUNIX(676) - Will use Case Sensitive matching for wildcards, filters and exceptions
    5982 1   07/16 13:02:13 4335999 File System Scan Started
    5982 1   07/16 13:02:13 4335999 Started Scan... argc =  =39
    5982 1   07/16 13:02:13 4335999 Performance COUNTER ANALYSIS will be turned ON
    5982 1   07/16 13:02:13 4335999 Performance COUNTER ANALYSIS will be printed every 1800 seconds
    5982 1   07/16 13:02:13 4335999 Performance COUNTERS WOULD NOT BE RESET FOR EVERY FILE
    5982 1   07/16 13:02:13 4335999 Performing an Incremental Search.
    5982 1   07/16 13:02:13 4335999 Find::inits() - Debug level is 1
    5982 1   07/16 13:02:13 4335999 Find::inits() - CheckAppleDouble is not set
    5982 1   07/16 13:02:13 4335999 Find::inits() - SYMLINKBKP nBackupSymLinkData is not set
    5982 1   07/16 13:02:13 4335999 CommServer Name before Job object init = inpnqcmvs01
    5982 1   07/16 13:02:13 4335999 JM Client  CVBkpJobClient::init(): Initializing job object with token [4335999:4:1].
    5982 1   07/16 13:02:13 4335999 Initializing Job Control object...
    5982 1   07/16 13:02:13 4335999 Init() - Initializing job control [token=4335999:4:1,cn=hpxi001a], serverName [inpnqcmvs01], ControlFlag [1], Job Id [4335999]
    5982 1   07/16 13:02:13 4335999 CVJobCtrlClient::startSession() - Failed to start a session with CVD. SessionError[150995042]::Error String[SSL authentication with [hpxi001a] has failed: [0xEDDD0068:{CCvNetwork::AuthenticateOutGoingConnection(6879)/Couldn't perform SSL authentication. Connection:[** RemoteHost:hpxi001a RemoteClient:hpxi001a RemoteProcess:todo RemoteProcessId:5548 RemoteThreadId:8 RemoteService:JobControlCVD RemotePort:Type=Socket. IPv4 port=52705, IPv6 port=0]} + {cccAuthSocket(1041)} + {_cccAuthSocket(829)/SSL authentication has failed} + {cccSend(1154)} + {cccGetSSLErrors(298)/Int.104.0x68-error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (s3_clnt.c:1253)}]], timedOut[FALSE]
    5982 1   07/16 13:02:13 4335999 EvEvent::setMsgEventArguments() - MsgId[0x18050013]: Args Table corrupt.
    5982 1   07/16 13:02:13 4335999 Init() - Error registering JOB with the Communications Service for JobID [4335999]
    5982 1   07/16 13:02:13 4335999 SetupConnectionWithCVD() - SetupConnectionWithCVD failed. mangledHostName=hpxi001a*hpxi001a*8400*8402 localClientName=hpxi001a TimeOut=False, error=0xEDDD0068:{CCvNetwork::Attach_RemoteService(1473)} + {CCvNetwork::Attach_RemoteService(1416)} + {CCvNetwork::Attach_RemotePort(1078)} + {CCvNetwork::AuthenticateOutGoingConnection(6980)} + {CCvNetwork::AuthenticateOutGoingConnection(6879)/Couldn't perform SSL authentication. Connection:[** RemoteHost:hpxi001a RemoteClient:hpxi001a RemoteProcess:todo RemoteProcessId:5548 RemoteThreadId:8 RemoteService:JobControlCVD RemotePort:Type=Socket. IPv4 port=52705, IPv6 port=0]} + {cccAuthSocket(1041)} + {_cccAuthSocket(829)/SSL authentication has failed} + {cccSend(1154)} + {cccGetSSLErrors(298)/Int.104.0x68-error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (s3_clnt.c:1253)}
    5982 1   07/16 13:02:13 4335999 SendToCVD() - Couldn't setup the connection
    5982 1   07/16 13:02:13 4335999 Init() - registerJob failed. Returning failure
    5982 1   07/16 13:02:13 4335999 Could not initialize job control.


The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy