RCA 45: vMotion Operation Taking a Long Time to Complete

 

VM Name: abc20109891

Datastore Name: ABCD-CL64-440-094-GSDE-VMAX    naa.60000970000497200440533030384430

 

 

Vmware Logs:

 

  • From the Vmware Logs we can see that the before vMotion VM was residing on abc3es0052s

 

2021-06-20T19:32:08.904Z| vmx| I125: Hostname=abc3es0052s.xyz.adroot.com
2021-10-04T12:07:14.326Z| vmx| I125: VigorTransportProcessClientPayload: opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01-c0-f3aa seq=12926617: Receiving Migrate.PrepareSource request.
2021-10-04T12:07:14.326Z| vmx| I125: MigrateVMXdrToSpec: type: 1 srcIp=<130.50.35.1> dstIp=<130.50.35.3> mid=396a5bf65149e8fc uuid=36323437-3735-4753-4838-333159395242 priority=no checksumMemory=no maxDowntime=0 encrypted=0 resumeDuringPageIn=no latencyAware=yes diskOpFile= srcLogIp=<<unknown>> dstLogIp=<<unknown>> ftPrimaryIp=<<unknown>> ftSecondaryIp=<<unknown>>
2021-10-04T12:07:14.326Z| vmx| I125: Received migrate ‘to’ request for mid id 4137220321177102588, src ip <130.50.35.1>, dst ip <130.50.35.3>(invalidate source config).

 

  • vMotion Operation Initiated for the VM to ESXi Host ABC3ES0054s.

 

2021-10-04T12:07:14.327Z| vmx| I125: MigrateSetInfo: state=1 srcIp=<130.50.35.1> dstIp=<130.50.35.3> mid=4137220321177102588 uuid=36323437-3735-4753-4838-333159395242 priority=low
2021-10-04T12:07:14.327Z| vmx| I125: MigrateSetState: Transitioning from state 0 to 1.
2021-10-04T12:07:14.331Z| vmx| I125: MigrateReadHostLog: Hostlog’s migID 4137223290241240859 doesn’t match active migID 4137220321177102588.  Resetting hostlog state.
2021-10-04T12:07:14.331Z| vmx| I125: MigrateReadHostLog: Hostlog_Dump: Hostlog abc20109891-7e6a4b25.hlog
2021-10-04T12:07:14.331Z| vmx| I125:    OpType: vmotion
2021-10-04T12:07:14.352Z| vmx| I125: VigorTransport_ServerSendResponse opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01-c0-f3aa seq=12926617: Completed Migrate request.
2021-10-04T12:07:14.833Z| vmx| I125: Received migrate ‘start’ request for mig id 4137220321177102588, dest world id 4295804.
2021-10-04T12:07:14.833Z| vmx| I125: MigrateSetState: Transitioning from state 1 to 2.
2021-10-04T12:07:14.836Z| vmx| I125: MigratePlatformInitMigration:  DiskOp file set to /vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891-diskOp.tmp
2021-10-04T12:07:14.845Z| vmx| I125: DISKLIB-LIB_MISC   : DiskLib_SetOpenFlags: Failed to set open flags: Operation not supported 11.
2021-10-04T12:07:14.845Z| vmx| I125: DISKLIB-LIB_MISC   : DiskLib_SetOpenFlags: Failed to set open flags: Operation not supported 11.
2021-10-04T12:07:14.845Z| vmx| I125: DISKLIB-LIB_MISC   : DiskLib_SetOpenFlags: Failed to set open flags: Operation not supported 11.
2021-10-04T12:07:14.845Z| vmx| I125: MigratePrepareEventLog: Prepare multiwriter disk handoff complete.
2021-10-04T12:07:14.880Z| vmx| I125: MigrateSetState: Transitioning from state 2 to 3.

 

 

  • Checked the vMotion Transition from State 3 to Stage 4 took almost 22 Minutes.

 

2021-10-04T12:29:30.400Z| vcpu-0| I125: MigrateSetState: Transitioning from state 3 to 4.

2021-10-04T12:29:30.400Z| vcpu-0| I125: Migrate: VM starting stun, waiting 100 seconds for go/no-go message.
2021-10-04T12:29:30.415Z| vcpu-0| I125: Closing disk ‘scsi0:2’
2021-10-04T12:29:30.416Z| vcpu-0| I125: Closing disk ‘scsi0:1’
2021-10-04T12:29:30.418Z| vcpu-0| I125: Closing disk ‘scsi0:0’
2021-10-04T12:29:30.419Z| vcpu-0| I125: Migrate_Open: Migrating to <130.50.35.3> with migration id 4137220321177102588
2021-10-04T12:29:30.420Z| vcpu-0| I125: Progress -1% (msg.checkpoint.saveStatus)
2021-10-04T12:29:30.437Z| vcpu-0| I125: Progress 0% (none)
2021-10-04T12:29:30.587Z| vcpu-0| I125: Progress 101% (none)

 

  • The delay of this 22 Minutes can be seen in the logs:

 

2021-10-04T12:29:30.590Z| vcpu-0| I125: Migrate: VM successfully stunned.
2021-10-04T12:29:30.590Z| vcpu-0| I125: MigrateSetState: Transitioning from state 4 to 5.
2021-10-04T12:29:30.590Z| vcpu-0| I125: Closing all the disks of the VM.
2021-10-04T12:29:30.749Z| worker-2119793| I125: Migrate: Remote Log: Destination waited for 1335.87 seconds.

 

 

2021-10-04T12:29:30.750Z| worker-2119793| I125: Migrate: Remote Log: Beginning checkpoint restore.
2021-10-04T12:29:30.750Z| worker-2119793| I125: Migrate: Remote Log: Switching to checkpoint state.
2021-10-04T12:29:31.511Z| vmx| I125: MigrateSetStateFinished: type=1 new state=6
2021-10-04T12:29:31.511Z| vmx| I125: MigrateSetState: Transitioning from state 5 to 6.
2021-10-04T12:29:31.511Z| vmx| I125: Migrate: Powering off
2021-10-04T12:29:31.521Z| vmx| I125: MigrateSetState: Transitioning from state 6 to 0.
2021-10-04T12:29:31.521Z| vmx| I125: SVMotion_PowerOff: Not running Storage vMotion. Nothing to do
2021-10-04T12:29:32.090Z| vmx| I125: Vix: [mainDispatch.c:4258]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.
2021-10-04T12:29:32.090Z| vmx| A100: ConfigDB: Setting cleanShutdown = “TRUE”
2021-10-04T12:29:32.090Z| vmx| I125: Vix: [mainDispatch.c:4258]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.
2021-10-04T12:29:32.090Z| vmx| I125: Transitioned vmx/execState/val to poweredOff
2021-10-04T12:29:32.090Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=5 on unregistered device ‘Migrate’ (cmd=(null))
2021-10-04T12:29:32.095Z| vmx| I125: VigorTransport_ServerCloseClient: Closing transport 8EB146AA20 (err = 0)
2021-10-04T12:29:32.097Z| vmx| W115: VMX has left the building: 0.

 

Vmware Logs:

 

 

2021-10-04T12:07:14.606Z| vmx| I125: Hostname=ABC3ES0054s.xyz.adroot.com
2021-10-04T12:07:14.607Z| vmx| I125: System uptime 8999314083232 us
2021-10-04T12:07:14.704Z| vmx| I125: MigrateVMXdrToSpec: type 1 unsharedSwap 0 memMinToTransfer 0 cpuMinToTransfer 0 numDisks 0 numStreamIps 2 numFtStreamIps 0
2021-10-04T12:07:14.704Z| vmx| I125: Received migrate ‘from’ request for mid id 4137220321177102588, src ip <130.50.35.1>.
2021-10-04T12:07:14.704Z| vmx| I125: MigrateSetInfo: state=8 srcIp=<130.50.35.1> dstIp=<130.50.35.3> mid=4137220321177102588 uuid=36323437-3735-4753-4838-333159395343 priority=low
2021-10-04T12:07:14.705Z| vmx| I125: MigrateSetState: Transitioning from state 0 to 8.
2021-10-04T12:07:14.705Z| vmx| I125: MigrateReadHostLog: Won’t persist hostlog to disk.
2021-10-04T12:07:14.727Z| vmx| W115: PowerOn
2021-10-04T12:07:14.807Z| vmx| I125: MigrateSetState: Transitioning from state 8 to 9.

  • In the Logs we can see the delay happening at Stage 10 of the vMotion Operation:

 

2021-10-04T12:07:14.989Z| vmx| I125: MigrateSetState: Transitioning from state 9 to 10.

2021-10-04T12:07:19.657Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘Tools’ (cmd=queryFields)
2021-10-04T12:07:19.657Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestInfo’ (cmd=queryFields)
2021-10-04T12:07:19.657Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:07:19.657Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestAppMonitor’ (cmd=queryFields)
2021-10-04T12:07:19.657Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘CrashDetector’ (cmd=queryFields)
2021-10-04T12:09:05.456Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘Tools’ (cmd=queryFields)
2021-10-04T12:20:01.004Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestInfo’ (cmd=queryFields)
2021-10-04T12:20:01.004Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:20:01.004Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestAppMonitor’ (cmd=queryFields)
2021-10-04T12:20:01.004Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘CrashDetector’ (cmd=queryFields)
2021-10-04T12:21:24.146Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘Tools’ (cmd=queryFields)

  • The Same delay can be seen here in the Destination Vmware logs at stage 10:

 

2021-10-04T12:29:30.748Z| vmx| I125: MigrateWaitForData: Waited for 1335.87 seconds.
2021-10-04T12:29:30.748Z| vmx| I125: MigrateRPC_DrainPendingWork: Draining pending remote user messages before restore…
2021-10-04T12:29:30.749Z| vmx| I125: MigrateRPC_DrainPendingWork: All pending work completed.
2021-10-04T12:29:30.749Z| vmx| I125: MigrateSetState: Transitioning from state 10 to 11.
2021-10-04T12:29:30.749Z| vmx| I125: SVMotionFixParentPaths: No snapshot paths need to be validated
2021-10-04T12:29:30.749Z| vmx| I125: Migrate_Open: Restoring from <130.50.35.1> with migration id 4137220321177102588
2021-10-04T12:29:30.757Z| worker-4296255| I125: DISK: failed to create nomad vob context.
2021-10-04T12:29:30.783Z| vmx| I125: Migrate_Open: Restoring from <130.50.35.1> with migration id 4137220321177102588
2021-10-04T12:29:30.787Z| vmx| I125: Progress -1% (msg.checkpoint.restoreStatus)
2021-10-04T12:29:30.797Z| vmx| I125: Progress 20% (none)
2021-10-04T12:29:30.802Z| vmx| I125: Progress 60% (none)
2021-10-04T12:29:30.806Z| vmx| I125: Progress 83% (none)
2021-10-04T12:29:30.811Z| vmx| I125: Progress 101% (none)

2021-10-04T12:29:31.505Z| vmx| I125: MigrateSetStateFinished: type=2 new state=12

 

  • Transition from the State 11 to State 12 is within seconds.

 

2021-10-04T12:29:31.505Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.
2021-10-04T12:29:31.505Z| vmx| I125: Migrate: Restoring original message callbacks.
2021-10-04T12:29:31.505Z| vmx| I125: Migrate_SetSuccess: Reopening config file.
2021-10-04T12:29:31.531Z| vcpu-0| I125: Migrate: cleaning up migration state.
2021-10-04T12:29:31.531Z| vcpu-0| I125: Migrate: Final status reported through Vigor.
2021-10-04T12:29:31.531Z| vcpu-0| I125: MigrateSetState: Transitioning from state 12 to 0.

 

 

Source Hostname: abc3es0052s.xyz.adroot.com

Esxi Version: ESXi 6.7 P05

 

Hostd Logs:

 

  • We can see the start of the vMotion Operation for the VM:

 

2021-10-04T12:07:14.321Z info hostd[2102690] [Originator@6876 sub=Vcsvc.VMotion opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01-c0-f3aa user=vpxuser] PrepareSourceEx [4137220321177102588], VM = ’62’
2021-10-04T12:07:14.321Z info hostd[2102690] [Originator@6876 sub=Vcsvc.VMotionSrc.4137220321177102588 opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01-c0-f3aa user=vpxuser] VMotionEntry: migrateType = 1
2021-10-04T12:07:14.322Z info hostd[2102690] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01-c0-f3aa user=vpxuser] State Transition (VM_STATE_ON -> VM_STATE_EMIGRATING)

2021-10-04T12:07:14.381Z info hostd[2102156] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx] VMotionStatusCb [4137220321177102588] : Prepare task completed successfully
2021-10-04T12:07:14.829Z info hostd[2104248] [Originator@6876 sub=Vcsvc.VMotion opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01-8e-f3cf user=vpxuser] InitiateSource [4137220321177102588], WID = 4295804
2021-10-04T12:07:14.831Z info hostd[2104248] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01-8e-f3cf user=vpxuser] VMotionInitiateSrc (4137220321177102588): wid=4295804

 

  • As soon as the vMotion initiated as Source we can see the events where the Device for the Virtual Machine is showing as Busy:

 

2021-10-04T12:29:35.600Z info hostd[2101947] [Originator@6876 sub=vm] DictionaryLoad: Cannot open file “/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx”: Device or resource busy.
2021-10-04T12:29:35.602Z info hostd[2101947] [Originator@6876 sub=vm] VigorOffline_GenSecPolicy: retry reading /vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx
2021-10-04T12:29:35.613Z info hostd[2542484] [Originator@6876 sub=vm] Cnx_Connect: Error message: There is no VMware process running for config file /vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx
2021-10-04T12:29:35.613Z info hostd[2542484] [Originator@6876 sub=vm] VigorTransportClientManageConnection: VM /vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx does not appear to be running
2021-10-04T12:29:39.613Z info hostd[2101947] [Originator@6876 sub=vm] DictionaryLoad: Cannot open file “/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx”: Device or resource busy.
2021-10-04T12:29:39.614Z info hostd[2101947] [Originator@6876 sub=vm] VigorOffline_GenSecPolicy: retry reading /vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx
2021-10-04T12:29:39.614Z info hostd[2102692] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx] Device not attached – cannot read online fields
2021-10-04T12:29:43.619Z info hostd[2101947] [Originator@6876 sub=vm] DictionaryLoad: Cannot open file “/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx”: Device or resource busy.
2021-10-04T12:29:43.620Z warning hostd[2102692] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx] Query VMX about hlstate failed Fault cause: vim.fault.InvalidPowerState
2021-10-04T12:29:43.622Z info hostd[2102692] [Originator@6876 sub=Vcsvc.VMotionSrc.4137220321177102588] ResolveCb: Succeeded
2021-10-04T12:29:43.622Z info hostd[2102692] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx] State Transition (VM_STATE_EMIGRATING -> VM_STATE_OFF)
2021-10-04T12:29:43.622Z error hostd[2101947] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx] Could not perform config check (storage accessible): Fault cause: vim.fault.GenericVmConfigFault
2021-10-04T12:29:43.623Z info hostd[2102692] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx] State Transition (VM_STATE_OFF -> VM_STATE_UNREGISTERING)
2021-10-04T12:29:43.629Z warning hostd[2102692] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20109891/abc20109891.vmx] Failed to find activation record, event user unknown.
2021-10-04T12:29:43.630Z info hostd[2102692] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 2664 : Removed abc20109891 on abc3es0052s.xyz.adroot.com from ha-datacenter

 

  • Total VM Downtime came out to be 0.02 Seconds.

 

2021-10-04T12:29:43.630Z info hostd[2102692] [Originator@6876 sub=Vcsvc.VMotionSrc.4137220321177102588] CompleteOp: Vmotion task succeeded with result: (vim.host.VMotionManager.SrcVMotionResult) {

–>    vmDowntime = 20220,
–>    vmPrecopyStunTime = 13851033,
–>    vmPrecopyBandwidth = 104299923
2021-10-04T12:29:43.767Z info hostd[2543975] [Originator@6876 sub=Vcsvc.VMotion opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01-5e-1b69 user=vpxuser] CompleteSource [4137220321177102588]

 

VMKernel Logs:

 

2021-10-04T12:07:14.836Z cpu42:2117674)Migrate: vm 2117675: 3885: Setting VMOTION info: Source ts = 4137220321177102588, src ip = <130.50.35.1> dest ip = <130.50.35.3> Dest wid = 4295804 using SHARED swap, encrypted
2021-10-04T12:07:14.841Z cpu42:2117674)Hbr: 3561: Migration start received (worldID=2117675) (migrateType=1) (event=0) (isSource=1) (sharedConfig=1)
2021-10-04T12:07:14.841Z cpu115:4249814)MigrateNet: 1751: 4137220321177102588 S: Successfully bound connection to vmknic vmk1 – ‘130.50.35.1’
2021-10-04T12:07:14.842Z cpu115:4249814)MigrateNet: 1751: 4137220321177102588 S: Successfully bound connection to vmknic vmk1 – ‘130.50.35.1’
2021-10-04T12:07:14.842Z cpu8:2101456)MigrateNet: vm 2101456: 3263: Accepted connection from <::ffff:130.50.35.3>
2021-10-04T12:07:14.842Z cpu8:2101456)MigrateNet: vm 2101456: 3351: dataSocket 0x43130ea88e90 receive buffer size is 563272
2021-10-04T12:07:14.842Z cpu8:2101456)Migrate: 358: Remote machine is ESX 6.5 or newer.
2021-10-04T12:07:14.842Z cpu115:4249814)VMotionUtil: 5199: 4137220321177102588 S: Stream connection 1 added.
2021-10-04T12:07:14.843Z cpu115:4249814)MigrateNet: 1751: 4137220321177102588 S: Successfully bound connection to vmknic vmk2 – ‘130.50.35.85’
2021-10-04T12:07:14.843Z cpu115:4249814)VMotionUtil: 5199: 4137220321177102588 S: Stream connection 2 added.

 

  • In the Vmkernel Logs we can see that the Time delay happened because of the Pre-Copy Operation which was running on the Pages to copy from Source to Destination.

 

2021-10-04T12:28:21.528Z cpu14:2117675)VMotion: 5367: 4137220321177102588 S: Another pre-copy iteration needed with 1698915 pages left to send (prev2 47185920, prev 47185920, pages dirtied by pass through device 0, network bandwidth ~86.697 MB/s, 18649%$
2021-10-04T12:29:21.242Z cpu17:2117675)VMotion: 5367: 4137220321177102588 S: Another pre-copy iteration needed with 211492 pages left to send (prev2 47185920, prev 1698915, pages dirtied by pass through device 0, network bandwidth ~115.554 MB/s, 744% t2$
2021-10-04T12:29:28.800Z cpu36:2117675)VMotion: 5367: 4137220321177102588 S: Another pre-copy iteration needed with 31608 pages left to send (prev2 1698915, prev 211492, pages dirtied by pass through device 0, network bandwidth ~115.450 MB/s, 560% t2d)
2021-10-04T12:29:30.267Z cpu19:2117675)VMotion: 5277: 4137220321177102588 S: Stopping pre-copy: only 1094 pages left to send, which can be sent within the switchover time goal of 0.500 seconds (network bandwidth ~89.719 MB/s, 3228% t2d)
2021-10-04T12:29:30.408Z cpu43:2119797)VSCSI: 6669: handle 8605(vscsi0:0):Destroying Device for world 2117675 (pendCom 0)
2021-10-04T12:29:31.381Z cpu115:4249814)VMotionSend: 5095: 4137220321177102588 S: Sent all modified pages to destination (network bandwidth ~112.399 MB/s)

 

 

vCenter Logs:

 

  • No Unusual Events can be seen in the vCenter Logs.

 

2021-10-04T12:07:14.208Z info vpxd[08641] [Originator@6876 sub=vpxLro opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03] [VpxLRO] — BEGIN task-296057 — abc20109891 — Drm.ExecuteVMotionLRO —
2021-10-04T12:07:14.208Z info vpxd[08641] [Originator@6876 sub=drsExec opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03] Executing DRM recommended vMotion, migrating [vim.VirtualMachine:vm-4981,abc20109891] from vim.HostSystem:host-4959 to vim.HostSystem:host-5073

 

2021-10-04T12:07:14.214Z info vpxd[08641] [Originator@6876 sub=vpxLro opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01] [VpxLRO] — BEGIN lro-9055758 —  — VmprovWorkflow —
2021-10-04T12:07:14.214Z info vpxd[08641] [Originator@6876 sub=pbm opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01] migrate callback is skipped – Same-VC migration with home and all disks and profiles the same
2021-10-04T12:07:14.214Z info vpxd[08641] [Originator@6876 sub=VmProv opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01] Starting Local-VC Migrate of poweredOn vm abc20109891 on vim.HostSystem:host-4959(130.50.34.1) with ds ds:///vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/ to vim.HostSystem:host-5073(130.50.34.3) with ds ds:///vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/
2021-10-04T12:07:14.286Z info vpxd[08641] [Originator@6876 sub=pbm opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01] pre migrate callback is skipped – for cached from pre migrate check reason

2021-10-04T12:29:43.754Z info vpxd[08641] [Originator@6876 sub=pbm opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01] post migrate callback is skipped – for cached from pre migrate reason
2021-10-04T12:29:43.977Z warning vpxd[08641] [Originator@6876 sub=VpxProfiler opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01] VpxLro::LroMain [TotalTime] took 1349763 ms
2021-10-04T12:29:43.977Z info vpxd[08641] [Originator@6876 sub=vpxLro opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03-01] [VpxLRO] — FINISH lro-9055758
2021-10-04T12:29:43.982Z warning vpxd[08641] [Originator@6876 sub=VpxProfiler opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03] VpxLro::LroMain [TotalTime] took 1349773 ms
2021-10-04T12:29:43.982Z info vpxd[08641] [Originator@6876 sub=vpxLro opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-03] [VpxLRO] — FINISH task-296057

 

 

 

VM Name: abc20110123

Datastore Name: 5ef4b713-49c335ca-33ca-48df373666b0

 

 

Vmware Logs:

 

 

2021-06-20T17:29:29.621Z| vmx| I125: Hostname=abc3es0052s.xyz.adroot.com
2021-06-20T17:29:29.621Z| vmx| I125: System uptime 2785153454 us

2021-10-04T12:07:14.468Z| vmx| I125: VigorTransportProcessClientPayload: opID=ku9y3x0q-2006536-auto-17095-h5:70046217-33-05-02-01-fc-f3bf seq=12773804: Receiving Migrate.PrepareSource request.

2021-10-04T12:07:14.468Z| vmx| I125: MigrateSetInfo: state=1 srcIp=<130.50.35.1> dstIp=<130.50.35.4> mid=4137220321177088244 uuid=36323437-3735-4753-4838-333159395247 priority=low
2021-10-04T12:07:14.468Z| vmx| I125: MigrateSetState: Transitioning from state 0 to 1.
2021-10-04T12:07:14.868Z| vmx| I125: Received migrate ‘start’ request for mig id 4137220321177088244, dest world id 4287296.
2021-10-04T12:07:14.868Z| vmx| I125: MigrateSetState: Transitioning from state 1 to 2.

 

2021-10-04T12:07:14.871Z| vmx| I125: MigratePlatformInitMigration:  DiskOp file set to /vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20110123/abc20110123-diskOp.tmp
2021-10-04T12:07:14.878Z| vmx| I125: MigratePrepareEventLog: Prepare multiwriter disk handoff complete.

  • Same events can be seen here we the delay comes at Stage 3.

 

2021-10-04T12:07:14.939Z| vmx| I125: MigrateSetState: Transitioning from state 2 to 3.
2021-10-04T12:26:31.450Z| vcpu-0| I125: MigrateSetState: Transitioning from state 3 to 4.

 

2021-10-04T12:26:31.450Z| vcpu-0| I125: Migrate: Preparing to suspend.
2021-10-04T12:26:31.450Z| vcpu-0| I125: Migrate: VM starting stun, waiting 100 seconds for go/no-go message.
2021-10-04T12:26:31.465Z| vcpu-0| I125: Closing disk ‘scsi0:2’
2021-10-04T12:26:31.466Z| vcpu-0| I125: Closing disk ‘scsi0:1’
2021-10-04T12:26:31.468Z| vcpu-0| I125: Closing disk ‘scsi0:0’
2021-10-04T12:26:31.469Z| vcpu-0| I125: Migrate_Open: Migrating to <130.50.35.4> with migration id 4137220321177088244
2021-10-04T12:26:31.469Z| vcpu-0| I125: Progress -1% (msg.checkpoint.saveStatus)
2021-10-04T12:26:31.496Z| vcpu-0| I125: Progress 0% (none)
2021-10-04T12:26:31.721Z| vcpu-0| I125: Progress 101% (none)
2021-10-04T12:26:31.721Z| vcpu-0| I125: Migrate: VM successfully stunned.
2021-10-04T12:26:31.721Z| vcpu-0| I125: MigrateSetState: Transitioning from state 4 to 5.

 

2021-10-04T12:26:31.721Z| vcpu-0| I125: Closing all the disks of the VM.
2021-10-04T12:26:32.265Z| vmx| I125: MigrateSetState: Transitioning from state 5 to 6.
2021-10-04T12:26:32.265Z| vmx| I125: Migrate: Powering off
2021-10-04T12:26:32.275Z| vmx| I125: MigrateSetState: Transitioning from state 6 to 0.
2021-10-04T12:26:32.275Z| vmx| I125: SVMotion_PowerOff: Not running Storage vMotion. Nothing to do
2021-10-04T12:26:32.682Z| vmx| W115: VMX has left the building: 0.

 

 

Destination Vmware Logs:

 

  • On the vmware logs we can see the same set of events at stage 10 we can see the delay being noticed.

 

2021-10-04T12:07:14.674Z| vmx| I125: Hostname=ABC3ES0055s.xyz.adroot.com
2021-10-04T12:07:14.674Z| vmx| I125: System uptime 8583710750932 us
2021-10-04T12:07:14.680Z| vmx| I125: Transitioned vmx/execState/val to poweredOff
2021-10-04T12:07:14.765Z| vmx| I125: MigrateVMXdrToSpec: type: 1 srcIp=<130.50.35.1> dstIp=<130.50.35.4> mid=396a5bf65149b0f4 uuid=36323437-3735-4753-4838-333159395343 priority=no checksumMemory=no maxDowntime=0 encrypted=0 resumeDuringPageIn=no latencyAware=yes diskOpFile= srcLogIp=<<unknown>> dstLogIp=<<unknown>> ftPrimaryIp=<<unknown>> ftSecondaryIp=<<unknown>>
2021-10-04T12:07:14.765Z| vmx| I125: MigrateVMXdrToSpec: type 1 unsharedSwap 0 memMinToTransfer 0 cpuMinToTransfer 0 numDisks 0 numStreamIps 2 numFtStreamIps 0
2021-10-04T12:07:14.765Z| vmx| I125: Received migrate ‘from’ request for mid id 4137220321177088244, src ip <130.50.35.1>.
2021-10-04T12:07:14.765Z| vmx| I125: MigrateSetInfo: state=8 srcIp=<130.50.35.1> dstIp=<130.50.35.4> mid=4137220321177088244 uuid=36323437-3735-4753-4838-333159395343 priority=low
2021-10-04T12:07:14.765Z| vmx| I125: MigrateSetState: Transitioning from state 0 to 8.
2021-10-04T12:07:14.789Z| vmx| W115: PowerOn
2021-10-04T12:07:14.839Z| vmx| I125: MigratePlatformInitMigration:  DiskOp file set to /vmfs/volumes/5ef4b713-49c335ca-33ca-48df373666b0/abc20110123/abc20110123-diskOp.tmp
2021-10-04T12:07:14.848Z| vmx| I125: MigrateSetState: Transitioning from state 8 to 9.
2021-10-04T12:07:15.065Z| vmx| I125: MigrateSetState: Transitioning from state 9 to 10.

 

 

2021-10-04T12:24:33.260Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘Tools’ (cmd=queryFields)
2021-10-04T12:24:33.260Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestInfo’ (cmd=queryFields)
2021-10-04T12:24:33.260Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:24:33.260Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestAppMonitor’ (cmd=queryFields)
2021-10-04T12:24:33.260Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘CrashDetector’ (cmd=queryFields)

2021-10-04T12:07:18.868Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘Tools’ (cmd=queryFields)
2021-10-04T12:07:18.868Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestInfo’ (cmd=queryFields)
2021-10-04T12:07:18.868Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:08:44.966Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestAppMonitor’ (cmd=queryFields)
2021-10-04T12:09:20.808Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:09:28.097Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘CrashDetector’ (cmd=queryFields)
2021-10-04T12:09:59.555Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:11:28.846Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestAppMonitor’ (cmd=queryFields)
2021-10-04T12:11:50.931Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘CrashDetector’ (cmd=queryFields)
2021-10-04T12:13:51.989Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:16:46.429Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:18:07.086Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestInfo’ (cmd=queryFields)
2021-10-04T12:26:25.193Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestAppMonitor’ (cmd=queryFields)
2021-10-04T12:26:25.193Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘CrashDetector’ (cmd=queryFields)

 

  • The Delay is around 20 Minutes.

 

2021-10-04T12:26:31.878Z| vmx| I125: MigrateWaitForData: Waited for 1156.98 seconds.
2021-10-04T12:26:31.878Z| vmx| I125: MigrateRPC_DrainPendingWork: Draining pending remote user messages before restore…

2021-10-04T12:26:31.878Z| vmx| I125: MigrateSetState: Transitioning from state 10 to 11.
2021-10-04T12:26:31.912Z| vmx| I125: Progress -1% (msg.checkpoint.restoreStatus)
2021-10-04T12:26:31.921Z| vmx| I125: Progress 2% (none)
2021-10-04T12:26:31.921Z| vmx| I125: Progress 5% (none)
2021-10-04T12:26:31.921Z| vmx| I125: Progress 7% (none)
2021-10-04T12:26:31.928Z| vmx| I125: Progress 60% (none)
2021-10-04T12:26:31.931Z| vmx| I125: Progress 83% (none)
2021-10-04T12:26:31.937Z| vmx| I125: Progress 101% (none)
2021-10-04T12:26:32.259Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.
2021-10-04T12:26:32.286Z| vcpu-0| I125: MigrateSetState: Transitioning from state 12 to 0.

 

 

Source Hostname: abc3es0052s.xyz.adroot.com

 

 

Destination Hostname: ABC3ES0055s.xyz.adroot.com

 

 

 

VM Name: abc20110089

Datastore: 5f7df7ca-87929102-d3a4-48df373666b0

 

 

Vmware Logs:

 

  • From the Vmware Logs we can see the same set of events:

 

2021-06-20T18:10:49.207Z| vmx| I125: Hostname=abc3es0052s.xyz.adroot.com
2021-06-20T18:10:49.208Z| vmx| I125: System uptime 5264739920 us
2021-10-04T12:07:14.418Z| vmx| I125: Received migrate ‘to’ request for mid id 4137220321177100492, src ip <130.50.35.1>, dst ip <130.50.35.4>(invalidate source config).
2021-10-04T12:07:14.418Z| vmx| I125: MigrateSetInfo: state=1 srcIp=<130.50.35.1> dstIp=<130.50.35.4> mid=4137220321177100492 uuid=36323437-3735-4753-4838-333159395247 priority=low
2021-10-04T12:07:14.418Z| vmx| I125: MigrateSetState: Transitioning from state 0 to 1.
2021-10-04T12:07:14.866Z| vmx| I125: Received migrate ‘start’ request for mig id 4137220321177100492, dest world id 4287276.
2021-10-04T12:07:14.866Z| vmx| I125: MigrateSetState: Transitioning from state 1 to 2.
2021-10-04T12:07:14.939Z| vmx| I125: MigratePrepareEventLog: Prepare event complete.

  • The Delay happened at the same stage: Stage 3.

 

2021-10-04T12:07:14.939Z| vmx| I125: MigrateSetState: Transitioning from state 2 to 3.
2021-10-04T12:30:34.781Z| vcpu-0| I125: MigrateSetState: Transitioning from state 3 to 4.
2021-10-04T12:30:34.781Z| vcpu-0| I125: Migrate: Preparing to suspend.
2021-10-04T12:30:34.781Z| vcpu-0| I125: Migrate: VM starting stun, waiting 100 seconds for go/no-go message.
2021-10-04T12:30:34.794Z| vcpu-0| I125: Closing disk ‘scsi0:2’
2021-10-04T12:30:34.795Z| vcpu-0| I125: Closing disk ‘scsi0:1’
2021-10-04T12:30:34.797Z| vcpu-0| I125: Closing disk ‘scsi0:0’
2021-10-04T12:30:34.798Z| vcpu-0| I125: Migrate_Open: Migrating to <130.50.35.4> with migration id 4137220321177100492
2021-10-04T12:30:34.883Z| vcpu-0| I125: MigrateSetState: Transitioning from state 4 to 5.
2021-10-04T12:30:34.883Z| vcpu-0| I125: Closing all the disks of the VM.
2021-10-04T12:30:35.355Z| vmx| I125: MigrateSetState: Transitioning from state 5 to 6.
2021-10-04T12:30:35.355Z| vmx| I125: Migrate: Powering off
2021-10-04T12:30:35.365Z| vmx| I125: MigrateSetState: Transitioning from state 6 to 0.
2021-10-04T12:30:35.956Z| vmx| W115: VMX has left the building: 0.

 

  • At the Destination Host we can see the same set of events where the Delay is of 23 Minutes.

 

2021-10-04T12:07:14.660Z| vmx| I125: Hostname=ABC3ES0055s.xyz.adroot.com
2021-10-04T12:07:14.661Z| vmx| I125: System uptime 8583710737076 us
2021-10-04T12:07:14.752Z| vmx| I125: Received migrate ‘from’ request for mid id 4137220321177100492, src ip <130.50.35.1>.
2021-10-04T12:07:14.752Z| vmx| I125: MigrateSetInfo: state=8 srcIp=<130.50.35.1> dstIp=<130.50.35.4> mid=4137220321177100492 uuid=36323437-3735-4753-4838-333159395343 priority=low
2021-10-04T12:07:14.752Z| vmx| I125: MigrateSetState: Transitioning from state 0 to 8.
2021-10-04T12:07:14.838Z| vmx| I125: MigrateSetState: Transitioning from state 8 to 9.
2021-10-04T12:07:15.058Z| vmx| I125: MigrateSetState: Transitioning from state 9 to 10.
2021-10-04T12:07:18.867Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘Tools’ (cmd=queryFields)
2021-10-04T12:07:18.867Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestInfo’ (cmd=queryFields)
2021-10-04T12:07:18.867Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:07:18.867Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestAppMonitor’ (cmd=queryFields)
2021-10-04T12:07:18.867Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘CrashDetector’ (cmd=queryFields)
2021-10-04T12:13:51.987Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:13:51.987Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestAppMonitor’ (cmd=queryFields)
2021-10-04T12:13:51.987Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘CrashDetector’ (cmd=queryFields)
2021-10-04T12:14:06.834Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘Tools’ (cmd=queryFields)
2021-10-04T12:14:06.834Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestInfo’ (cmd=queryFields)
2021-10-04T12:14:06.834Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestPeriodic’ (cmd=queryFields)
2021-10-04T12:14:06.834Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestAppMonitor’ (cmd=queryFields)
2021-10-04T12:28:45.301Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘GuestAppMonitor’ (cmd=queryFields)
2021-10-04T12:28:45.301Z| vmx| I125: Vigor_ClientRequestCb: failed to do op=3 on unregistered device ‘CrashDetector’ (cmd=queryFields)
2021-10-04T12:30:34.979Z| vmx| I125: MigrateWaitForData: Waited for 1400.08 seconds.
2021-10-04T12:30:34.979Z| vmx| I125: MigrateSetState: Transitioning from state 10 to 11.
2021-10-04T12:30:35.027Z| vmx| I125: Progress 34% (none)
2021-10-04T12:30:35.034Z| vmx| I125: Progress 83% (none)
2021-10-04T12:30:35.040Z| vmx| I125: Progress 101% (none)
2021-10-04T12:30:35.350Z| vmx| I125: MigrateSetStateFinished: type=2 new state=12
2021-10-04T12:30:35.350Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.
2021-10-04T12:30:35.377Z| vcpu-0| I125: MigrateSetState: Transitioning from state 12 to 0.

 

 

Conclusion:

 

  • Based on the logs we can conclude that the Delay is happening at State 3 on the Source and State 10 on the Destination.

 

Migrate State 3 = MIGRATE_TO_VMX_PRECOPY

Migrate State 10 = MIGRATE_FROM_VMX_PRECOPY

 

  • This is the time when the Pre-Copy Operation is running. Generally this issue could happen because of  long data transfers between hosts or the VM is very busy as the Pre-Copy operation has to copy all the RAM Changes before it can perform the switchover which can be seen as below:

 

2021-10-04T12:28:21.528Z cpu14:2117675)VMotion: 5367: 4137220321177102588 S: Another pre-copy iteration needed with 1698915 pages left to send (prev2 47185920, prev 47185920, pages dirtied by pass through device 0, network bandwidth ~86.697 MB/s, 18649%$
2021-10-04T12:29:21.242Z cpu17:2117675)VMotion: 5367: 4137220321177102588 S: Another pre-copy iteration needed with 211492 pages left to send (prev2 47185920, prev 1698915, pages dirtied by pass through device 0, network bandwidth ~115.554 MB/s, 744% t2$
2021-10-04T12:29:28.800Z cpu36:2117675)VMotion: 5367: 4137220321177102588 S: Another pre-copy iteration needed with 31608 pages left to send (prev2 1698915, prev 211492, pages dirtied by pass through device 0, network bandwidth ~115.450 MB/s, 560% t2d)
2021-10-04T12:29:30.267Z cpu19:2117675)VMotion: 5277: 4137220321177102588 S: Stopping pre-copy: only 1094 pages left to send, which can be sent within the switchover time goal of 0.500 seconds (network bandwidth ~89.719 MB/s, 3228% t2d)
2021-10-04T12:29:30.408Z cpu43:2119797)VSCSI: 6669: handle 8605(vscsi0:0):Destroying Device for world 2117675 (pendCom 0)
2021-10-04T12:29:31.381Z cpu115:4249814)VMotionSend: 5095: 4137220321177102588 S: Sent all modified pages to destination (network bandwidth ~112.399 MB/s)

 

  • Time taken by during transition from state 11 to state 12

 

abc20109891 : 1 Sec

2021-10-04T12:29:30.749Z| vmx| I125: MigrateSetState: Transitioning from state 10 to 11.

2021-10-04T12:29:31.505Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.

 

abc20110123 : 1 Sec

2021-10-04T12:26:31.878Z| vmx| I125: MigrateSetState: Transitioning from state 10 to 11.

2021-10-04T12:26:32.259Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.

 

abc20110089 : 1 Sec

2021-10-04T12:30:34.979Z| vmx| I125: MigrateSetState: Transitioning from state 10 to 11

2021-10-04T12:30:35.350Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.

 

 

Action Plan:

 

  • In order to Test this you can try to perform the vMotion Operation at the time when there is not enough load on the VM and test the vMotion Operation.
  • Also in order to isolate the Load on the ESXi Host, please keep few VMs over the Host and then try to perform a vMotion operation.

 

 

Note: Please note that this may or may not be related to the issue, but Its always recommended to keep the Driver/Firmware Version to the Recommended state.

 

Device

Description

Installed Firmware

Installed Driver

VCG Driver

vmhba1

Emulex LPe12000 8Gb PCIe Fibre Channel Adapter

2.10X6

lpfc 12.6.182.2-1OEM.670.0.0.8169922

12.8.317.0

vmhba2

Emulex LPe12000 8Gb PCIe Fibre Channel Adapter

2.10X6

lpfc 12.6.182.2-1OEM.670.0.0.8169922

12.8.317.0

vmhba3

Emulex LPe12000 8Gb PCIe Fibre Channel Adapter

2.10X6

lpfc 12.6.182.2-1OEM.670.0.0.8169922

12.8.317.0

vmhba4

Emulex LPe12000 8Gb PCIe Fibre Channel Adapter

2.10X6

lpfc 12.6.182.2-1OEM.670.0.0.8169922

12.8.317.0

 

Ashutosh Dixit

I am currently working as a Senior Technical Support Engineer with VMware Premier Services for Telco. Before this, I worked as a Technical Lead with Microsoft Enterprise Platform Support for Production and Premier Support. I am an expert in High-Availability, Deployments, and VMware Core technology along with Tanzu and Horizon.

Leave a Reply