I have opened several Incidents with Support but we cannot seem to find a resolution. I keep getting the Media Mount and Usage error alerts on all 3 media agents and pretty much all paths over the past several weeks/months.

The latest SE noticed the following message:

[TYPE] Information [TIME] 2019/09/09 01:06:43  [SOURCE] Galaxy [COMPUTER] xxx.xxx.local[DESCRIPTION] Failed in acquiring the lock on file [E:\Program Files\Commvault\ContentStore\Log Files\PerformanceMetrics.log] Module [PerformanceMetrics] Error[0x80070306:{CQiLogger::LockLockfile(1326)} + {CQiLogger::LockLockfile(1320)/W32.774.(One or more errors occurred while processing the request.)-Timed out acquiring mutex on Global\QiLogger_PerformanceMetrics}]

Has anyone else encountered several Media Mount and Usage alerts and or know what the "mutex on Global" even means?

Thanks. 

Is the CommServe or are the MediaAgents clustered?

The errors are showing up on our Media agents. We have 2 on one campus and a single media agent at a different campus. They are not clustered and the OS is Win2k12 R2 and the hardware is HPE Proliant 380DL Gen9. Each campus location has its own dedicated Storage (Netapp e2700 series). So single Commcell environment. 

Given that it is happening on the three mediaagents, it is possible that the problem is coming from the CommServe.  What version+servicepack of Commvault are you running, and what is the Build Number of the SQL Server on the CommServe?

I am running v11 sp15 HPK21. The SQL build number on the CommServe is Server 2014 v12.0.2269.0 (X64) Jun 10 2015.

So there is no Anti-Virus on the media agents but I happened to notice that SEP is installed but disabed on the CommServe. Should it be completely uninstalled? 

Ok thanks. That is an old revision of SQL server 2014. Can you see errors in the Program Files\Microsoft SQL Server\MSSQL.n\MSSQL\LOG\ERRORLOG file?

I checked the SQL ERRORLOG files on both commserve servers (active & standby) and I didn't see any errors around the date/time the media mount and usage alert came out. 

Support is seeing this when they parse the most recent logs:

redatacted\cvd

1760  2150  09/09 01:27:50 1611889 6700604-# [DM_BASE    ] **ERROR** MountActiveVolume Failed to mount volume 0  MediaGroupId = 205 MA hostname- redacted MediaManager returned error [752], errorString [A mount path already exists at the given location. Cannot create a file when that file already exists. ]

1760  2150  09/09 01:27:50 1611889 6700604-# [DM_BASE    ] Going to check whether the mount should be retried for the above mediamanager error

1760  2150  09/09 01:27:50 1611889 6700604-# [DM_BASE    ] **ERROR** MountActiveVolume Failed to mount volume 0  MediaGroupId = 205 MA hostname- redacted MediaManager returned error [752], errorString [A mount path already exists at the given location. Cannot create a file when that file already exists. ]

The error code 183 returned by the OS indicates that the file already exists:

C:\>net helpmsg 183

Cannot create a file when that file already exists.

This might happen if the device did not respond, timed out and CV eventually retried the same file.

The health check on the dedicated storage device showed no issues. 

Hi wsimps01

The original error looks to be an issue when updating logs, and could be related to slowness on the Disk where MediaAgent Binary is installed. 

The subsequent mount errors looks like volume exist in the current mount path and this type of issue could be exhibited due to:

• Slow storage at the time of writing, Commvault acknowledges the volume is already created or
• A DR restore on the Commserve is done, when it attempts to re-write the same volume, as the volume already exist on Disk it will report the error and skip writing.