Issue Description:

Cluster Nodes evicted with event id 1135 on Cluster Name: aaborhv-clstr-2 Running a copy of Microsoft Windows Server 2012 R2 Standard Version 6.3.9600 Build 9600 on 20th July

Initial Description:

As we know that in this case the resources failover from one  node to another this generally happens when the node on which the resource was running is no more capable of running that resource. This may be due to lack of essential components like unable to access storage or Loss of network connectivity. Sometimes the Node on which the resource was running gets evicted from the failover clustering membership (event id 1135) which makes the resources to failover to another node.

Why is Event ID 1135 Logged ?

This event will be logged on all nodes in the Cluster except for the node that was removed. The reason for this event is because one of the nodes in the Cluster marked that node as down. It then notifies all of the other nodes of the event. When the nodes are notified, they discontinue and tear down their heartbeat connections to the downed node.

What caused the node to be marked down?

All nodes in a Windows 2008 or 2008 R2 Failover Cluster talk to each other over the networks that are set to Allow cluster network communication on this network. The nodes will send out heartbeat packets across these networks to all of the other nodes. These packets are supposed to be received by the other nodes and then a response is sent back. Each node in the Cluster has its own heartbeats that it is going to monitor to ensure the network is up and the other nodes are up. The example below should help clarify this:

If any one of these packets are not returned, then the specific heartbeat is considered failed. For example, W2K8-R2-NODE2 sends a request and receives a response from W2K8-R2-NODE1 to a heartbeat packet so it determines the network and the node is up.  If W2K8-R2-NODE1 sends a request to W2K8-R2-NODE2 and W2K8-R2-NODE1 does not get the response, it is considered a lost heartbeat and W2K8-R2-NODE1 keeps track of it.  This missed response can have W2K8-R2-NODE1 show the network as down until another heartbeat request is received.

By default, Cluster nodes have a limit of 5 failures in 5 seconds before the connection is marked down. So if W2K8-R2-NODE1 does not receive the response 5 times in the time period, it considers that particular route to W2K8-R2-NODE2 to be down.  If other routes are still considered to be up, W2K8-R2-NODE2 will remain as an active member.

If all routes are marked down for W2K8-R2-NODE2, it is removed from active Failover Cluster membership and the Event 1135 that you see in the first section is logged. On W2K8-R2-NODE2, the Cluster Service is terminated and then restarted so it can try to rejoin the Cluster.

Reference :

                Having a problem with nodes being removed from active Failover Cluster membership?

                http://blogs.technet.com/b/askcore/archive/2012/02/08/having-a-problem-with-nodes-being-removed-from-active-failover-cluster-membership.aspx

Scenario 1

Issue happened on 25th .

__________________________________________________________________________________________

System Information: ABCOUHV01

OS Name        Microsoft Windows Server 2012 R2 Standard

Version        6.3.9600 Build 9600

Other OS Description         Not Available

OS Manufacturer        Microsoft Corporation

System Name        ABCOUHV01

System Manufacturer        HP

System Model        ProLiant BL460c Gen8

System Type        x64-based PC

System SKU        641016-B21

Processor        Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz, 2500 Mhz, 6 Core(s), 12 Logical Processor(s)

Processor        Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz, 2500 Mhz, 6 Core(s), 12 Logical Processor(s)

BIOS Version/Date        HP I31, 6/1/2015

System Events:

• Check the Machine and found that we are getting event id 1135 which says that the Cluster node: ABCOUHV03 was removed from the Cluster.

• Based on the error of the VM failure VM failed because it was not able to find the configuration file. As it was missing : The system cannot find the file specified

Application Events:

• Checked the application events but was not able to find anything specific related to the issue.

List of outdated drivers:

Cluster Logs:

00005548.000051a8::2016/07/25-03:48:53.654 INFO  [RES] Physical Disk: Supplied device path Q: is a disk path, status 2

00005548.000051a8::2016/07/25-03:48:53.654 INFO  [RES] Physical Disk: Failed to open device Q:, status 3

00005548.000051a8::2016/07/25-03:48:53.654 WARN  [RHS] Error 3 from resource type control for restype Physical Disk.

00003328.00003618::2016/07/25-03:48:53.785 INFO  [GEM] Node 2: Deleting [3:54298 , 3:54310] (both included) as it has been ack’d by every node

00003328.00004648::2016/07/25-03:53:27.359 ERR   [REMP] HandleHwprvPostCommitSnapshot: unknown snapshot set 7488383a-f77f-49c6-abd5-3f30519b790c

00003328.00001338::2016/07/25-03:53:27.359 INFO  [WRTA] OnEndLocalSnapshotSet: snapshot set 97a00019-cb80-40ad-b304-5d5f70da06e1, initiator SnapshotInitiatorAgentHwProvider

00003328.00004648::2016/07/25-03:53:27.381 INFO  [DCM] ClusterSnapshotSetHandler(exit): 97a00019-cb80-40ad-b304-5d5f70da06e1, isLocal false, event HwPostCommitSnapshot, HrError(0x00000000)

00003328.00001338::2016/07/25-03:53:27.359 INFO  [WRTA] OnEndLocalSnapshotSet: snapshot set 97a00019-cb80-40ad-b304-5d5f70da06e1, initiator SnapshotInitiatorAgentHwProvider

00003328.00004648::2016/07/25-03:53:27.381 INFO  [DCM] ClusterSnapshotSetHandler(exit): 97a00019-cb80-40ad-b304-5d5f70da06e1, isLocal false, event HwPostCommitSnapshot, HrError(0x00000000)

____________________________________________________________________________________________

System
Information: ABCOUHV07

OS Name        Microsoft
Windows Server 2012 R2 Standard

Version        6.3.9600 Build
9600

Other OS Description        
Not Available

OS Manufacturer       
Microsoft Corporation

System Name        ABCOUHV07

System Manufacturer        HP

System Model        ProLiant
BL460c Gen8

System Type        x64-based
PC

System SKU        641016-B21

Processor        Intel(R)
Xeon(R) CPU E5-2640 0 @ 2.50GHz, 2500 Mhz, 6 Core(s), 12 Logical Processor(s)

Processor        Intel(R)
Xeon(R) CPU E5-2640 0 @ 2.50GHz, 2500 Mhz, 6 Core(s), 12 Logical Processor(s)

BIOS Version/Date        HP
I31, 6/1/2015

System
Events:

Application
Events:

• Checked the
  application events but was not able to find anything specific related to
  the issue.

List
of outdated drivers: