Issue Description:

RCA of Getting Event ID 5120 on a Cluster Shared Volume.

Initial Description:

As per the article: https://blogs.msdn.microsoft.com/clustering/2014/02/26/event-id-5120-in-system-event-log/

Auto Pause due to STATUS_IO_TIMEOUT

When conducting backups of a Windows Server 2012 or later Failover Cluster using Cluster Shared Volumes (CSV), you may encounter the following event in the System event log:

Log Name:  System
Source:  Microsoft-Windows-FailoverClustering
Event ID:  5120
Task Category: Cluster Shared Volume
Level:  Error
Description:  Cluster Shared Volume 'VolumeName' ('ResourceName') is no longer available on this node because of 'STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.

Having an Event ID 5120 logged may or may not be the sign of a problem with the cluster, based on the error code logged.  Having an Event 5120 with an error code of STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR or the error code c0130021 may be expected and can be safely ignored in most situations.

An Event ID 5120 with an error code of STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR is logged on the node which owns the cluster Physical Disk resource when there was a VSS Software Snapshot which clustering knew of, but the software snapshot was deleted.  When a snapshot is deleted which Failover Clustering had knowledge of, clustering must resynchronize its state of the view of the snapshots.

One scenario where an Event ID 5120 with an error code of STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR may be logged is when using System Center Data Protection Manager (DPM), and DPM may delete a software snapshot once a backup has completed.  When DPM requests deletion of a software snapshot, volsnap will mark the software snapshot for deletion.  However volsnap conducts deletion in an asynchronous fashion which occurs at a later point in time.  Even though the snapshot has been marked for deletion, Clustering will detect that the software snapshot still exists and needs to handle it appropriately.  Eventually volsnap will perform the actual deletion operation of the software snapshot.  When clustering then notices that a software snapshot it knew of was deleted, it must resynchronize its view of the snapshots.

Think of it as clustering getting surprised by an un-notified software snapshot deletion, and the cluster service telling the various internal components of the cluster service that they need to resynchronize their views of the snapshots.

There are also a few other expected scenarios where volsnap will delete snapshots, and as a result clustering will need to resynchronize its snapshot view.  Such as if a copy on write fails due to lack of space or an IO error. In these conditions volsnap will log an event in the system event log associated with those failures.  So review the system event logs for other events accompanying the event 5120, this could be logged on any node in the cluster.

_____________________________________________________________________________________________________________________________________________

System Information: QVH6
OS Name        Microsoft Windows Server 2012 R2 Datacenter
Version        6.3.9600 Build 9600
Other OS Description         Not Available
OS Manufacturer        Microsoft Corporation
System Name        QVH6
System Manufacturer        HP
System Model        ProLiant BL465c G7
System Type        x64-based PC
System SKU        518857-B21
Processor        AMD Opteron(tm) Processor 6174, 2200 Mhz, 12 Core(s), 12 Logical Processor(s)
Processor        AMD Opteron(tm) Processor 6174, 2200 Mhz, 12 Core(s), 12 Logical Processor(s)
BIOS Version/Date        HP A19, 02/02/2014

Application Events:

• Tried to find the events at the time of issue but was not able to find anything specific.

System Events:

• Checked the event logs on the QVH6 and found that the Physical Server is going low on Physical RAM due to which the Live migrated VMs were not able to come online.

• Checked the event logs on the QVH6 and found that the Physical Server is going low on Physical RAM due to which the Live migrated VMs were not able to come online.

Provider: disk
Event ID: 154
Level: Error
Text: “The IO operation at logical block address <LBA> for Disk <Disk Number> (PDO name: <PDO Name>) failed due to a hardware error.”
Logged when: A fatal hardware error is reported by the device.
Sense Key: Byte 0x2D in the binary data
Add’l Sense Code: Byte 0x2E in the binary data
Add’l Sense Code Qualifier: Byte 0x2F in the binary data

OS Name        Microsoft Windows Server 2012 R2 Datacenter
Version        6.3.9600 Build 9600
Other OS Description         Not Available
OS Manufacturer        Microsoft Corporation
System Name        QVH9
System Manufacturer        HP
System Model        ProLiant BL465c Gen8
System Type        x64-based PC
System SKU        708931-B21
Processor        AMD Opteron(tm) Processor 6344, 2600 Mhz, 6 Core(s), 12 Logical Processor(s)
Processor        AMD Opteron(tm) Processor 6344, 2600 Mhz, 6 Core(s), 12 Logical Processor(s)
BIOS Version/Date        HP A26, 02/11/2014

err(vista).exe' 0xc0000203
# for hex 0xc0000203 / decimal -1073741309
  STATUS_USER_SESSION_DELETED                                    ntstatus.h
# The remote user session has been deleted.
# 1 matches found for "0xc0000203"
PS C:\Users\adix5025.INDIA\Downloads\ERR>
0xc0000203 = The remote user session has been deleted.

OS Name Microsoft Windows Server 2012 R2 Datacenter
Version 6.3.9600 Build 9600
Other OS Description Not Available
OS Manufacturer Microsoft Corporation
System Name QVH15
System Manufacturer HP
System Model ProLiant BL465c Gen8
System Type x64-based PC
System SKU 708931-B21
Processor AMD Opteron(tm) Processor 6344, 2600 Mhz, 6 Core(s), 12 Logical Processor(s)
Processor AMD Opteron(tm) Processor 6344, 2600 Mhz, 6 Core(s), 12 Logical Processor(s)
BIOS Version/Date HP A26, 02/11/2014

OS Name Microsoft Windows Server 2012 R2 Datacenter
Version 6.3.9600 Build 9600
Other OS Description Not Available
OS Manufacturer Microsoft Corporation
System Name QVH1
System Manufacturer HP
System Model ProLiant BL465c G7
System Type x64-based PC
System SKU 518857-B21
Processor AMD Opteron(tm) Processor 6174, 2200 Mhz, 12 Core(s), 12 Logical Processor(s)
Processor AMD Opteron(tm) Processor 6174, 2200 Mhz, 12 Core(s), 12 Logical Processor(s)
BIOS Version/Date HP A19, 02/02/2014