Hostname: c871abc.int.abcde.com
Build Number: VMware ESXi 6.5.0 build-10719125
Licence Mode: 00000-00000-00000-00000-00000 : Evaluation Mode
|
DVPort ID |
In Use |
Client |
|
99 |
1 |
vmnic4 |
|
100 |
1 |
vmnic5 |
|
101 |
1 |
vmnic6 |
|
102 |
1 |
vmnic7 |
|
104 |
1 |
vmnic0 |
|
105 |
1 |
vmnic1 |
|
11 |
1 |
vmk1 |
|
3 |
1 |
vmk0 |
|
19 |
1 |
vmk2 |
|
27 |
1 |
vmk3 |
|
35 |
1 |
vmk4 |
|
43 |
1 |
vmk5 |
|
endpoint |
portKey |
portID |
portgroupKey |
DvsPortset |
teamUplink |
teaming |
name |
vlan |
|
vmk0 |
3 |
50331664 |
dvportgroup-3159 |
DvsPortset-0 |
lag1* |
lag1 |
DPortGroup_MGMT_VCB111CKG-300-ABCD_C |
No VLAN |
|
vmk1 |
11 |
50331665 |
dvportgroup-3160 |
DvsPortset-0 |
lag1* |
lag1 |
DPortGroup_vMotion_VCB111CKG-300-ABCD_C |
VLAN 2423 |
|
vmk2 |
19 |
50331666 |
dvportgroup-3161 |
DvsPortset-0 |
fallback |
Uplink 5 |
DPortGroup_iSCSIA_VCB111CKG-300-ABCD_C |
No VLAN |
|
vmk3 |
27 |
50331667 |
dvportgroup-3162 |
DvsPortset-0 |
vmnic7 |
Uplink 7 |
DPortGroup_iSCSIB_VCB111CKG-300-ABCD_C |
No VLAN |
|
vmk4 |
35 |
50331668 |
dvportgroup-3163 |
DvsPortset-0 |
vmnic4 |
Uplink 4 |
DPortGroup_iSCSIC_VCB111CKG-300-ABCD_C |
No VLAN |
|
vmk5 |
43 |
50331669 |
dvportgroup-3164 |
DvsPortset-0 |
vmnic6 |
Uplink 6 |
DPortGroup_iSCSID_VCB111CKG-300-ABCD_C |
No VLAN |
|
Adapter |
Vmknic |
IPv4 |
IPv4 Subnet Mask |
MTU |
Vlan ID |
TOE |
TSO |
Current Speed |
Compliant Status |
|
vmhba64 |
vmk2 |
192.111.1.115 |
255.255.255.0 |
9000 |
0 |
FALSE |
TRUE |
0 |
compliant |
|
vmhba64 |
vmk3 |
192.111.2.115 |
255.255.255.0 |
9000 |
0 |
FALSE |
TRUE |
10000 |
compliant |
|
vmhba64 |
vmk4 |
192.111.3.115 |
255.255.255.0 |
9000 |
0 |
FALSE |
TRUE |
10000 |
compliant |
|
vmhba64 |
vmk5 |
192.111.4.115 |
255.255.255.0 |
9000 |
0 |
FALSE |
TRUE |
10000 |
compliant |
VOBD Logs:
2019-05-29T02:48:12.121Z:
[netCorrelator] 51138068us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate
up
2019-05-29T02:48:12.130Z: [netCorrelator]
51146980us: [vob.net.vmnic.linkstate.down] vmnic vmnic6 linkstate down
2019-05-29T02:48:12.139Z: [netCorrelator]
51155663us: [vob.net.vmnic.linkstate.down] vmnic vmnic7 linkstate down
2019-05-29T02:48:13.002Z: [netCorrelator] 52018578us:
[esx.clear.net.vmnic.linkstate.up] Physical NIC vmnic0 linkstate is up
2019-05-29T02:48:13.002Z: An event (esx.clear.net.vmnic.linkstate.up) could not
be sent immediately to hostd; queueing for retry.
2019-05-29T02:48:13.002Z:
[netCorrelator] 52018810us: [esx.clear.net.vmnic.linkstate.up] Physical NIC
vmnic5 linkstate is up
2019-06-06T02:36:39.330Z:
[netCorrelator] 213812985425us: [vob.net.dvport.uplink.transition.down] Uplink:
vmnic5 is down. Affected dvPort: 3/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25 68
8e. 2 uplinks up. Failed criteria: 128
2019-06-06T02:36:39.330Z:
[netCorrelator] 213812985432us: [vob.net.dvport.uplink.transition.down] Uplink:
vmnic5 is down. Affected dvPort: 11/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25
68 8e. 2 uplinks up. Failed criteria: 128
2019-06-06T02:36:39.330Z:
[netCorrelator] 213812985436us: [vob.net.dvport.uplink.transition.down] Uplink:
vmnic5 is down. Affected dvPort: 19/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25
68 8e. 0 uplinks up. Failed criteria: 128
2019-06-06T02:36:39.330Z:
[netCorrelator] 213812985438us: [vob.net.dvport.uplink.transition.down] Uplink:
vmnic5 is down. Affected dvPort: 27/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25
68 8e. 1 uplinks up. Failed criteria: 128
2019-06-06T02:36:39.330Z:
[netCorrelator] 213812985441us: [vob.net.dvport.uplink.transition.down] Uplink:
vmnic5 is down. Affected dvPort: 35/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25
68 8e. 1 uplinks up. Failed criteria: 128
2019-06-06T02:36:39.330Z:
[netCorrelator] 213812985444us: [vob.net.dvport.uplink.transition.down] Uplink:
vmnic5 is down. Affected dvPort: 43/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25
68 8e. 1 uplinks up. Failed criteria: 128
2019-06-06T02:36:39.330Z:
[netCorrelator] 213812985570us: [vob.net.vmnic.linkstate.down] vmnic vmnic5
linkstate down
2019-06-06T02:36:41.001Z:
[netCorrelator] 213822808291us: [esx.problem.net.dvport.connectivity.lost] Lost
network connectivity on DVPorts: “19/50 1a 8d fc dd 2a fd 7b-97 40 5f df
96 25 68 8e”. Physical NIC vmnic5 is down.
2019-06-06T02:36:41.001Z:
[netCorrelator] 213822808353us: [esx.problem.net.dvport.redundancy.lost] Lost
uplink redundancy on DVPorts: “27/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96
25 68 8e”, “35/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25 68 8e”,
“43/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25 68 8e”. Physical NIC
vmnic5 is down.
2019-06-06T02:36:41.001Z:
[netCorrelator] 213822808389us: [esx.problem.net.dvport.redundancy.degraded]
Uplink redundancy degraded on DVPorts: “3/50 1a 8d fc dd 2a fd 7b-97 40 5f
df 96 25 68 8e”, “11/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25 68
8e”. Physical NIC vmnic5 is down.
2019-06-06T02:36:48.831Z:
[vmfsCorrelator] 213822486403us: [vob.vmfs.heartbeat.timedout]
5cf53b90-f375c98a-cd26-e4434b3e0220
rfc4122.1ff6e29a-b524-4d24-a5ee-3498f5c7e752
2019-06-06T02:36:48.831Z: [vmfsCorrelator] 213830638745us:
[esx.problem.vmfs.heartbeat.timedout] 5cf53b90-f375c98a-cd26-e4434b3e0220
rfc4122.1ff6e29a-b524-4d24-a5ee-3498f5c7e752
2019-06-06T02:37:03.580Z: [iscsiCorrelator]
213837234523us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for
vmhba64:C0:T0
2019-06-06T02:37:03.580Z: [iscsiCorrelator]
213837234601us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for
vmhba64:C4:T0
2019-06-06T02:37:03.683Z: [vmfsCorrelator] 213837337140us:
[vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume
5cf53b90-f375c98a-cd26-e4434b3e0220
(rfc4122.1ff6e29a-b524-4d24-a5ee-3498f5c7e752): [Timeout] [HB state abcdef02
offset 4165632 gen 3 stampUS 213837336633 uuid
5cf53996-fa599fd2-3777-e4434b1fe2a0 jrnl <FB 28800> drv 14.81]
2019-06-06T02:37:03.683Z: [vmfsCorrelator] 213845489977us:
[esx.problem.vmfs.heartbeat.recovered] 5cf53b90-f375c98a-cd26-e4434b3e0220
rfc4122.1ff6e29a-b524-4d24-a5ee-3498f5c7e752
2019-06-06T02:37:13.582Z: [iscsiCorrelator]
213847236392us: [vob.iscsi.session.recovery.timeout] iScsi session recovery
timeout for vmhba64:C0:T0:
2019-06-06T02:37:13.582Z: [iscsiCorrelator]
213847236397us: [vob.iscsi.session.recovery.timeout] iScsi session recovery
timeout for vmhba64:C4:T0:
2019-06-06T02:37:13.583Z:
[scsiCorrelator] 213847237679us: [vob.scsi.scsipath.pathstate.dead] scsiPath
vmhba64:C0:T0:L254 changed state from on
2019-06-06T02:37:13.585Z: [scsiCorrelator]
213855392879us: [esx.problem.storage.redundancy.degraded] Path redundancy to
storage device naa.624a93703a72032fc5d247c8000113ee degraded. Path
vmhba64:C0:T0:L254 is down. Affected datastores: Unknown.
2019-06-06T02:37:13.586Z:
[scsiCorrelator] 213847237683us: [vob.scsi.scsipath.pathstate.dead] scsiPath
vmhba64:C4:T0:L254 changed state from on
2019-06-06T02:37:13.586Z: [scsiCorrelator]
213855393741us: [esx.problem.storage.redundancy.degraded] Path redundancy to
storage device naa.624a93703a72032fc5d247c8000113ee degraded. Path
vmhba64:C4:T0:L254 is down. Affected datastores: Unknown.
Vmkernel Logs:
2019-06-06T02:36:39.330Z
cpu70:66341)i40en: i40en_UpdateUplinkLinkStatus:3897: Link is down for device
vmnic5 (0x43085e97b040)
2019-06-06T02:36:39.330Z
cpu70:66341)netschedHClk: NetSchedHClkNotify:2892: vmnic5: link down
notification
2019-06-06T02:36:48.831Z cpu13:65726)VVol: VVolTMHandler:5843: VVolTMHandler
called for device (rfc4122.1ff6e29a-b524-4d24-a5ee-3498f5c7e752) RESET TM spec
(0x0, -1)
2019-06-06T02:36:52.655Z cpu78:69571)HBX: 2958:
‘rfc4122.1ff6e29a-b524-4d24-a5ee-3498f5c7e752’: HB at offset 4165632 – Waiting
for timed out HB:
2019-06-06T02:36:52.655Z cpu78:69571)
[HB state abcdef02 offset 4165632 gen 3 stampUS 213813486372 uuid
5cf53996-fa599fd2-3777-e4434b1fe2a0 jrnl <FB 28800> drv 14.81 lockImpl 4
ip 10.185.232.53]
2019-06-06T02:36:57.834Z
cpu42:66081)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba64:CH:4 T:0 L:254
: Task mgmt “Abort Task” with itt=0x45b4 (refITT=0x45b3) timed out.
2019-06-06T02:37:03.580Z
cpu59:66863)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba64:CH:0 T:0 CN:0:
iSCSI connection is being marked “OFFLINE” (Event:4)
2019-06-06T02:37:03.580Z
cpu59:66863)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID:
00023d000021 TARGET: iqn.2010-06.com.purestorage:flasharray.27a1d97c46f37adc
TPGT: 1 TSIH: 0]
2019-06-06T02:37:03.580Z
cpu59:66863)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.111.1.115:55998
R: 192.111.1.211:3260]
2019-06-06T02:37:03.580Z
cpu59:66863)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba64:CH:4 T:0 CN:0:
iSCSI connection is being marked “OFFLINE” (Event:4)
2019-06-06T02:37:03.580Z
cpu59:66863)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID:
00023d000031 TARGET: iqn.2010-06.com.purestorage:flasharray.27a1d97c46f37adc
TPGT: 1 TSIH: 0]
2019-06-06T02:37:03.580Z
cpu59:66863)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.111.1.115:42094
R: 192.111.1.213:3260]
2019-06-06T02:37:03.580Z
cpu13:65726)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba64:CH:4 T:0 L:254
: Task mgmt “Abort Task” with itt=0x45b6 (refITT=0x45b3) timed out.
2019-06-06T02:37:03.580Z
cpu13:65726)NMP: nmp_ThrottleLogForDevice:3647: Cmd 0x89 (0x439e0d823e40,
65623) to dev “naa.624a93703a72032fc5d247c8000113ee” on path
“vmhba64:C4:T0:L254” Failed: H:0x8 D:0x0 P:0x0 Invalid sense data:
0x0 0x0 0x0. Act:EVAL
2019-06-06T02:37:03.580Z cpu13:65726)WARNING:
NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device
“naa.624a93703a72032fc5d247c8000113ee” state in doubt; requested fast
path state update…
2019-06-06T02:37:03.580Z
cpu13:65726)ScsiDeviceIO: 2918: Cmd(0x439e0d823e40) 0x89, CmdSN 0x1d85d from
world 65623 to dev “naa.624a93703a72032fc5d247c8000113ee” failed
H:0x8 D:0x0 P:0x0
2019-06-06T02:37:03.580Z
cpu42:66081)VVol: 2205: [vvolDev: rfc4122.1ff6e29a-b524-4d24-a5ee-3498f5c7e752]
WRITE command (ioFlags 0x10, cmd 39) failed: Timeout (0xbad0021)
2019-06-06T02:37:03.682Z cpu13:65726)HBX: 283:
‘rfc4122.1ff6e29a-b524-4d24-a5ee-3498f5c7e752’: HB at offset 4165632 –
Reclaimed heartbeat [Timeout]:
2019-06-06T02:37:03.682Z cpu13:65726)
[HB state abcdef02 offset 4165632 gen 3 stampUS 213837336633 uuid
5cf53996-fa599fd2-3777-e4434b1fe2a0 jrnl <FB 28800> drv 14.81 lockImpl 4
ip 10.185.232.53]
2019-06-06T12:19:53.851Z
cpu2:127963)Tcpip_Vmk: 129: get connection pkt trace failed with error code
195887136
2019-06-06T12:19:53.851Z cpu2:127963)Tcpip_Vmk:
129: get connection pkt trace failed with error code 195887136
2019-06-06T12:19:53.851Z cpu2:127963)Tcpip_Vmk:
96: get connection stats failed with error code 195887136
2019-06-06T12:19:53.851Z cpu2:127963)Tcpip_Vmk:
129: get connection pkt trace failed with error code 195887136
2019-06-06T12:19:53.851Z cpu2:127963)Tcpip_Vmk:
129: get connection pkt trace failed with error code 195887136
Hostd:
2019-06-06T02:36:41.001Z
info hostd[12F85B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 525 : Lost
network connectivity on DVPorts: “19/50 1a 8d fc dd 2a fd 7b-97 40 5f df
96 25 68 8e”. Physical NIC vmnic5 is down.
2019-06-06T02:36:41.002Z info
hostd[12F85B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 526 : Lost
uplink redundancy on DVPorts: “27/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96
25 68 8e”, “35/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25 68 8e”,
“43/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25 68 8e”. Physical NIC
vmnic5 is down.
2019-06-06T02:36:41.002Z info
hostd[127C1B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 527 : Uplink
redundancy degraded on DVPorts: “3/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96
25 68 8e”, “11/50 1a 8d fc dd 2a fd 7b-97 40 5f df 96 25 68 8e”.
Physical NIC vmnic5 is down.
Conclusion:
- The
Driver version of few of the network Drivers are different from the
others:
- As per
the List below:
|
vmnic |
PCI bus address |
link |
speed |
duplex |
MTU |
driver |
driver |
version |
firmware version |
MAC address |
VID |
DID |
SVID |
SDID |
name |
|
vmnic0 |
0000:18:00.0 |
Up |
10000 |
Full |
9000 |
i40en |
1.07.17 |
|
6.80 0x80003d74 18.8.9 |
e4:43:4b:1f:e2:a0 |
8086 |
1572 |
1028 |
1f9c |
Intel(R) Ethernet Controller X710 for 10GbE SFP+ |
|
vmnic1 |
0000:18:00.1 |
Up |
10000 |
Full |
9000 |
i40en |
1.07.17 |
|
6.80 0x80003d74 18.8.9 |
e4:43:4b:1f:e2:a1 |
8086 |
1572 |
1028 |
0 |
Intel(R) Ethernet Controller X710 for 10GbE SFP+ |
|
vmnic2 |
0000:18:00.2 |
Down |
0 |
Half |
1500 |
i40en |
1.07.17 |
|
6.80 0x80003d74 18.8.9 |
e4:43:4b:1f:e2:a2 |
8086 |
1572 |
1028 |
0 |
Intel(R) Ethernet Controller X710 for 10GbE SFP+ |
|
vmnic3 |
0000:18:00.3 |
Down |
0 |
Half |
1500 |
i40en |
1.07.17 |
|
6.80 0x80003d74 18.8.9 |
e4:43:4b:1f:e2:a3 |
8086 |
1572 |
1028 |
0 |
Intel(R) Ethernet Controller X710 for 10GbE SFP+ |
|
vmnic4 |
0000:3b:00.0 |
Up |
10000 |
Full |
9000 |
i40en |
1.07.17 |
|
6.00 0x80003a2a 18.3.6 |
f8:f2:1e:5a:8f:60 |
8086 |
1572 |
8086 |
6 |
Intel(R) Ethernet Controller X710 for 10GbE SFP+ |
|
vmnic5 |
0000:3b:00.1 |
Down |
0 |
Half |
9000 |
i40en |
1.07.17 |
|
6.00 0x80003a2a 18.3.6 |
f8:f2:1e:5a:8f:61 |
8086 |
1572 |
8086 |
0 |
Intel(R) Ethernet Controller X710 for 10GbE SFP+ |
|
vmnic6 |
0000:5e:00.0 |
Up |
10000 |
Full |
9000 |
i40en |
1.07.17 |
|
6.00 0x80003a2a 18.3.6 |
f8:f2:1e:5a:8e:d0 |
8086 |
1572 |
8086 |
6 |
Intel(R) Ethernet Controller X710 for 10GbE SFP+ |
|
vmnic7 |
0000:5e:00.1 |
Up |
10000 |
Full |
9000 |
i40en |
1.07.17 |
|
6.00 0x80003a2a 18.3.6 |
f8:f2:1e:5a:8e:d1 |
8086 |
1572 |
8086 |
0 |
Intel(R) Ethernet Controller X710 for 10GbE SFP+ |
- We are
using Network Driver Version: 1.07.17, with this version of drivers we
should be running at 18.8.9 Firmware version, However few of the Network Adaptors are
running on an older version which is 18.3.6.
- As per the Article: VMware Compatibility Guide below are the Firmware and Driver version
associated with the Adaptors that we are using:
|
Device Driver(s) |
Firmware Version |
Additional Firmware Version |
Type |
|
|
ESXi 6.5 U2 |
i40en version 1.7.11 |
18.08.00 |
N/A |
Partner Async, native |
|
ESXi 6.5 U2 |
i40en version 1.5.8 |
18.05.00 |
N/A |
Partner Async, native |
|
ESXi 6.5 U2 |
i40en version 1.5.6 |
18.03.00 |
N/A |
Partner Async, native |
|
ESXi 6.5 U2 |
i40en version 1.4.3 |
18.03.00 |
N/A |
Partner Async, native |
