Disk library access failures?

Last post 01-31-2020, 7:12 AM by Wwong. 3 replies.
Sort Posts: Previous Next
  • Disk library access failures?
    Posted: 01-22-2020, 7:27 AM
    Hi everybody!

    I'm having problems with accessing a shared disk library. I'm stumped.
    We're using an FS8600 NAS gateway, connected to an SC3000 array with iSCSI. The disk library is accessed through SMB-shares (UNC paths) from 4 media agents.

    The sidbprune.log files on all media agents are riddled with access errors. Some examples:

    3404 9a4 01/22 13:06:25 ### CIndexFile::Open:1294 Failed to delete the change file [\\dellfs2\DDL38\DDL38\CV_MAGNETIC\V_3676172\CHUNK_44852833\SFILE_CONTAINER.idx.change]. Error [0x80070005:{CQiFile::Delete(1267)} + {CQiPath::GrantDeletePermissions(686)/W32.5.(Access is denied.)-Failed to get the security descriptor of path [\\dellfs-dhg2-2213\DDL_DHG_MP38\DDL_DHG_MP38\CV_MAGNETIC\V_3676172\CHUNK_44852833\SFILE_CONTAINER.idx.change]}]
    3404 9a4 01/22 13:06:25 ### 75-3 OpenIdx:3074 Cannot open Sfile Idx [\\dellfs2\DDL38\DDL38\CV_MAGNETIC\V_3676172\CHUNK_44852833\SFILE_CONTAINER.idx], SysErr [5], Err [0x80070005:{CQiFile::Delete(1267)} + {CQiPath::GrantDeletePermissions(686)/W32.5.(Access is denied.)-Failed to get the security descriptor of path [\\dellfs-dhg2-2213\DDL_DHG_MP38\DDL_DHG_MP38\CV_MAGNETIC\V_3676172\CHUNK_44852833\SFILE_CONTAINER.idx.change]}]

    or

    3404 1ee0 01/22 13:12:59 ### 75-3 FinalizeSFile:3520 Error in removing [\\dellfs2\DDL27\DDL27\CV_MAGNETIC\V_3677830\CHUNK_44877902\SFILE_CONTAINER_027], Error [0x80070005:{CQiFile::Delete(1267)} + {CQiPath::GrantDeletePermissions(686)/W32.5.(Access is denied.)-Failed to get the security descriptor of path [\\dellfs-dhg2-2213\DDL27\DDL27\CV_MAGNETIC\V_3677830\CHUNK_44877902\SFILE_CONTAINER_027]}]

    If i try to mount the library path from the media agents directly, i get errors trying to create a new file in \DDL27.
    The error says:

    An unexpected error is keeping you from creating the file. If you continue to receive this error, you can use the error code to search for help with this problem.
    Error 0x8000FFFF: Catastrophic failure

    There's no such problem on the root folder of the SMB-share; only within in the library folder.
    I can actually create a new share for the same NAS volume, which then doesn't have the catastrophic failure rror.
    But if i then redirect Commvault disk library config to point to the other SMB-share, the error appears in the new SMB share and disappears in the old SMB share.

    So what might be causing such a problem? Is this perhaps expected behaviour?
  • Re: Disk library access failures?
    Posted: 01-27-2020, 7:32 AM

    Hi Martanwb

    The reason why you might be getting the error "Catastrophic failure", could be because of Ransomware Protection (Commvault feature) that is enabled on the MediaAgent which is preventing external intervention. It is an expected behavior if the feature is enabled. (You can check whether the feature is enabled on the MediaAgent, via the Properties -> Advanced)

    Going back to your initial errors, it looks like the MediaAgent is having accessing issue when attempt to read a file, and this could be due to the amount of active connections to the CIF share. Might be advisable to check with your Storage Team of the Storage Vendor what are the limitation of CIF share

    Regards

    Winston

  • Re: Disk library access failures?
    Posted: 01-30-2020, 1:48 AM

    Hi Winston.

    That turns out to be exactly right. I tried turning ransomware protection on/off, and it has an immediate effect. I.e. "catastrophic failure" versus working correctly.

    Sounds like a good way to protect those libraries. Although it doesn't seem to help from other media agents which don't have that library configured, but can still reach. (same VLAN).

    I hope to be able to read somewhere about what the ransomware protection does exactly. I'm curious now.

    Too bad it doesn't help at all against those other errors. Might be library corruption, but i'm not sure how to arrange shutting down backups for possibly weeks to get it all verified.

    Thanks for your help.

    Regards,

    Martijn.

  • Re: Disk library access failures?
    Posted: 01-31-2020, 7:12 AM

    Hi Martin

    Ransomware in Commvault leverages a Filter Driver that initializes a honeypot mechanise is a which will detect, deflect, or, in some manner, counteract attempts at unauthorized use of the Mount Path. 

    In saying this the feature will only come into affect if the Commvault Services are running on the MediaAgent that is connected to the MediaAgent. 

    It would also be advisable to restrict or limit permission access at the Storage Level, so other Servers that have access to the Mount Path/Storage won't cause abnormal behavior to the Data at rest

    Regards

    Winston 

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy