More clues: in vcenter's vpxd log, it looks like those two 1800 seconds were spent on below but anyone know what they mean? What service was it tried to log in but failed?
* Right after the first 1800 seconds:
2019-03-04T22:16:02.693Z error vpxd[19613] [Originator@6876 sub=sms opID=jry8o5o4-426463-auto-9528-h5:70045337-da-01] [ConnectLocked] Failed to login to service: N7Vmacore16TimeoutExceptionE(Operation timed out)
2019-03-04T22:16:02.697Z error vpxd[19613] [Originator@6876 sub=sms opID=jry8o5o4-426463-auto-9528-h5:70045337-da-01] Received exception from SMS: N7Vmacore9ExceptionE(Operation timed out)
2019-03-04T22:16:02.700Z warning vpxd[19613] [Originator@6876 sub=VmProv opID=jry8o5o4-426463-auto-9528-h5:70045337-da-01] InvokeListenerPreCallback [StorageListeners] took 1800029 ms
* right after the second 1800 seconds:
2019-03-04T22:46:12.626Z error vpxd[19613] [Originator@6876 sub=sms opID=jry8o5o4-426463-auto-9528-h5:70045337-da-01] [ConnectLocked] Failed to login to service: N7Vmacore16TimeoutExceptionE(Operation timed out)
2019-03-04T22:46:12.629Z error vpxd[19613] [Originator@6876 sub=sms opID=jry8o5o4-426463-auto-9528-h5:70045337-da-01] Received exception from SMS: N7Vmacore9ExceptionE(Operation timed out)
2019-03-04T22:46:12.633Z warning vpxd[19613] [Originator@6876 sub=VmProv opID=jry8o5o4-426463-auto-9528-h5:70045337-da-01] InvokeListenerPostCallback [StorageListeners] took 1800033 ms
* this looks like a summary: 2 x 1800 secs ~= 3601 secs
2019-03-04T22:46:12.759Z warning vpxd[19613] [Originator@6876 sub=VpxProfiler opID=jry8o5o4-426463-auto-9528-h5:70045337-da-01] VpxLro::LroMain [TotalTime] took 3610111 ms
Thanks,