Hi,
I have configured vSAN on a new VMKernel Adapter on the vSwitch0, I can access the vSANDataStore but cannot create folder in it, I simply receive error "Cannot complete file operation".
I tried this this : https://kb.vmware.com/s/article/2105091 no change occurred.The below logs are obtained from one of the ESXi hosts /var/log/osfsd.log file as instructed in the link.
The logs show the following with only 1 type of error repeated, "VsanMountInt: Failed to resolve .vsan.stats: Not found", from the asterisk below the is the repeated error written here once.
[root@ESXi1:/var/log] cat osfsd.log
2019-02-15T19:17:45Z mark: storage-path-claim-completed
2019-02-15T20:25:07Z mark: storage-path-claim-completed
2019-02-15T20:49:39Z mark: storage-path-claim-completed
2019-02-16T07:33:00Z mark: storage-path-claim-completed
2019-02-16T07:49:56Z mark: storage-path-claim-completed
2019-02-16T10:20:16.089Z 2104183:main:203: Starting
2019-02-16T10:20:16.089Z 2104183:Event_Init:274: Initializing event system
2019-02-16T10:20:16.091Z 2104183:Vmklink_Init:131: Initializing Vmklink Request Slab
2019-02-16T10:20:16.091Z 2104183:Vmklink_Init:151: Initializing Vmklink connection
2019-02-16T10:20:16.091Z VmklinkMPI-LIB : Vmklink MPI initialized : osfs-vmklink with queue size = 16
2019-02-16T10:20:16.091Z 2104183:Vmklink_Init:179: Vmklink connection setup completed
2019-02-16T10:20:17.092Z 2104183:AnnounceReady:103: Emitted pid '2104183' (8)
2019-02-16T10:20:20.169Z VmklinkMPI-LIB : Received request ID 100 of size 165
2019-02-16T10:20:20.169Z VmklinkMPI-LIB : { : ID: 0x100, bufLen: 165 }
2019-02-16T10:20:20.169Z VmklinkMPI-LIB : Msg received. Calling recv Handler
2019-02-16T10:20:20.169Z VmklinkMPI-LIB : { : ID: 0x100, bufLen: 165 }
2019-02-16T10:20:20.169Z 2104183:DebugDumpVmklinkRequest:1063: {ID: 0x100; type: LOOKUP; pid: [ vsan]; cid: 52dcf84bacfe6581-5b9491c54bbdc952; status: 0; bufLen: 0; mf: 1684761974; o: 7368053 }
2019-02-16T10:20:20.169Z 2104183:Provider_Lookup_Load:502: Provider [ vsan] not found, try to load it.
2019-02-16T10:20:20.275Z 2104183:Provider_LoadPlugin:168: Loaded entry point (/usr/lib/vmware/osfs/lib/vsan.so): 0xdf49938050
2019-02-16T10:20:20.275Z 2104183:ProviderInit:3073: VsanProviderInit()
2019-02-16T10:20:20.275Z 2104183:VsanGetMinUnmountRetryDelay:970: VSAN min delay between unmount retries set to: 1
2019-02-16T10:20:20.275Z 2104183:VsanGetMaxUnmountTries:935: VSAN max unmount tries set to: 4
2019-02-16T10:20:20.275Z 2104183:VsanGetMaxMountTries:368: Max mount retries: 5
2019-02-16T10:20:20.275Z 2104183:VsanGetMinMountRetryDelay:390: Min mount retry delay: 1
2019-02-16T10:20:20.275Z Creating threadpool osfsdVsan with 4 threads
2019-02-16T10:20:20.276Z OBJLIB-LIB: ObjLibPlugins_Load: Loading plugin directory '/usr/lib64/vmware/plugin/objLib'
2019-02-16T10:20:20.276Z OBJLIB-LIB: ObjLibLoadPlugin: Loading plugin 'objLib/vsanObjBE.so'
2019-02-16T10:20:20.303Z PluginLdr_Load: Loaded plugin 'objLib/vsanObjBE.so' from '/usr/lib64/vmware/plugin/objLib/vsanObjBE.so'
2019-02-16T10:20:20.304Z ObjLibPluginInit: Initialized plugin
2019-02-16T10:20:20.304Z VsanBackend_GetAPI: Received request for API com.vmware.plugin.objlib.backend.api@1.0
2019-02-16T10:20:20.304Z OBJLIB-LIB: ObjLib_RegisterDynamicBE: Back-end 'vsan' found and registered as type 3
2019-02-16T10:20:20.304Z 2104183:VVOLLIB : VVolLib_IpcInit:208: IPC timeouts set to: connect = 30 sec, send = 30 sec, receive = 200 sec
2019-02-16T10:20:20.305Z 2104183:VVOLLIB : VVolLib_GetSoapContext:379: Using 30 secs for soap connect timeout.
2019-02-16T10:20:20.305Z 2104183:VVOLLIB : VVolLib_GetSoapContext:380: Using 200 secs for soap receive timeout.
2019-02-16T10:20:20.305Z OBJLIB-LIB: Objlib initialized.
2019-02-16T10:20:20.306Z 2104183:VsanGetAutounmountTimerValue:248: VSAN autounmount timer value: 60
2019-02-16T10:20:20.306Z 2104183:VsanGetForceAutounmountDelayValue:273: VSAN autounmount force delay: 900
2019-02-16T10:20:20.306Z 2104183:VsanGetMaxMountedVolumeThreshold:299: Max mounted VSAN volumes threshold: 512
2019-02-16T10:20:20.306Z 2104183:VsanDir_Init:1083: Initializing CMMDS library
2019-02-16T10:20:20.309Z 2104183:VsanDirCMMDSStatusCb:1034: CMMDS enabled: setting up subscriptions
2019-02-16T10:20:20.310Z 2104183:VsanDir_Init:1109: CMMDS library init done
2019-02-16T10:20:20.310Z 2104183:Provider_Register:338: Registering provider ID [ vsan] as driver #0 with 8 normal and 0 priority workers
2019-02-16T10:20:20.310Z Creating threadpool osfsd-nvsan with 8 threads
2019-02-16T10:20:20.310Z 2104183:AutoUnmountGetPeriodicity:185: [ vsan] auto-unmount timer set to: 60
2019-02-16T10:20:20.311Z 2104183:AutoUnloadGetPeriodicity:169: [ vsan] auto-unload timer set to: 60
2019-02-16T10:20:20.311Z 2104183:ProviderInit:3188: Vsan ProviderInit succeeded
2019-02-16T10:20:20.311Z 2104183:Provider_LoadPlugin:177: Initialization complete (/usr/lib/vmware/osfs/lib/vsan.so)
2019-02-16T10:20:20.311Z 2104183:Provider_Lookup:469: Found matching driver for ID [ vsan]
2019-02-16T10:20:20.311Z Enqueue a new work item in threadpool osfsd-nvsan
2019-02-16T10:20:20.311Z Add the new item into the pending list
2019-02-16T10:20:20.312Z Adding a thread to threadpool osfsd-nvsan
2019-02-16T10:20:20.312Z Starting main loop for the worker thread osfsd-nvsan
2019-02-16T10:20:20.312Z Added a new thread to threadpool (osfsd-nvsan), numThreadsInPool (1)
2019-02-16T10:20:20.312Z 2104249:Lookup:660: Got lookup request for "vmkdump"
2019-02-16T10:20:20.313Z 2104249:Provider_Lookup:469: Found matching driver for ID [ vsan]
2019-02-16T10:20:20.313Z 2104249:VsanMountInt:2406: VsanMountInt: request for [vmkdump]
2019-02-16T10:20:20.313Z 2104249:VsanDir_FriendlyNameToUUID:1696: Name [vmkdump] not found in directory
2019-02-16T10:20:20.313Z 2104249:VsanMountInt:2443: VsanMountInt: Failed to resolve vmkdump: Not found
2019-02-16T10:20:20.313Z 2104249:LookupCompletion:698: lookup completion: buf =
2019-02-16T10:20:20.313Z 2104249:PostVmklinkSendEvent:352: Create ordered event to send response.
2019-02-16T10:20:20.313Z 2104249:DebugDumpVmklinkResponse:1097: {ID: 0x100; type: LOOKUP; pid: [ vsan]; cid: 52dcf84bacfe6581-5b9491c54bbdc952; status: Not found; bufLen: 0 }
2019-02-16T10:20:20.313Z Completed main loop for the worker thread osfsd-nvsan
2019-02-16T10:20:20.314Z VmklinkMPI-LIB : VmklinkMPILib_Send called for 0x100,
2019-02-16T10:20:20.314Z VmklinkMPI-LIB : Matching request found for 0x100,
2019-02-16T10:20:20.314Z VmklinkMPI-LIB : Send got req = DF07A885D0 (ID: 100)
2019-02-16T10:20:20.314Z VmklinkMPI-LIB : Vmklink send sent total: 186 bytes of 186 needed
2019-02-16T10:20:20.315Z VmklinkMPI-LIB : Sent request ID 100 size 165
2019-02-16T10:20:20.315Z VmklinkMPI-LIB : { : ID: 0x100, bufLen: 165 }
2019-02-16T10:20:20.315Z 2104183:OSFSVmklinkResourceAvailableCBHandler:519: resource available callback received
2019-02-16T10:20:20.315Z 2104183:VsanDirClusterMembershipChangeCb:650: Subscription for sub-cluster membership entry is now active
2019-02-16T10:20:20.315Z 2104183:VsanDirOsfsNameChangeCb:812: Subscription for OSFS_NAME entry changes is now active
2019-02-16T10:20:20.316Z 2104183:VsanDirDomNameChangeCb:748: Subscription for DOM_NAME entry changes is now active
2019-02-16T10:20:21.173Z VmklinkMPI-LIB : Received request ID 101 of size 165
2019-02-16T10:20:21.173Z VmklinkMPI-LIB : { : ID: 0x101, bufLen: 165 }
2019-02-16T10:20:21.176Z VmklinkMPI-LIB : Msg received. Calling recv Handler
2019-02-16T10:20:21.184Z VmklinkMPI-LIB : { : ID: 0x101, bufLen: 165 }
2019-02-16T10:20:21.184Z 2104183:DebugDumpVmklinkRequest:1063: {ID: 0x101; type: LOOKUP; pid: [ vsan]; cid: 52dcf84bacfe6581-5b9491c54bbdc952; status: 0; bufLen: 0; mf: 1634956846; o: 32497584236080750 }
2019-02-16T10:20:21.184Z 2104183:Provider_Lookup:469: Found matching driver for ID [ vsan]
2019-02-16T10:20:21.185Z Enqueue a new work item in threadpool osfsd-nvsan
2019-02-16T10:20:21.185Z Add the new item into the pending list
2019-02-16T10:20:21.185Z Adding a thread to threadpool osfsd-nvsan
2019-02-16T10:20:21.185Z Starting main loop for the worker thread osfsd-nvsan
2019-02-16T10:20:21.185Z Added a new thread to threadpool (osfsd-nvsan), numThreadsInPool (1)
2019-02-16T10:20:21.185Z 2104256:Lookup:660: Got lookup request for ".vsan.stats"
2019-02-16T10:20:21.185Z 2104256:Provider_Lookup:469: Found matching driver for ID [ vsan]
2019-02-16T10:20:21.185Z 2104256:VsanMountInt:2406: VsanMountInt: request for [.vsan.stats]
2019-02-16T10:20:21.186Z 2104256:VsanDir_FriendlyNameToUUID:1696: Name [.vsan.stats] not found in directory
2019-02-16T10:20:21.186Z 2104256:VsanMountInt:2443: VsanMountInt: Failed to resolve .vsan.stats: Not found
2019-02-16T10:20:21.186Z 2104256:LookupCompletion:698: lookup completion: buf =
2019-02-16T10:20:21.186Z 2104256:PostVmklinkSendEvent:352: Create ordered event to send response.
2019-02-16T10:20:21.186Z 2104256:DebugDumpVmklinkResponse:1097: {ID: 0x101; type: LOOKUP; pid: [ vsan]; cid: 52dcf84bacfe6581-5b9491c54bbdc952; status: Not found; bufLen: 0 }
2019-02-16T10:20:21.186Z Completed main loop for the worker thread osfsd-nvsan
2019-02-16T10:20:21.186Z VmklinkMPI-LIB : VmklinkMPILib_Send called for 0x101,
2019-02-16T10:20:21.186Z VmklinkMPI-LIB : Matching request found for 0x101,
2019-02-16T10:20:21.187Z VmklinkMPI-LIB : Send got req = DF07A885D0 (ID: 101)
2019-02-16T10:20:21.187Z VmklinkMPI-LIB : Vmklink send sent total: 186 bytes of 186 needed
2019-02-16T10:20:21.187Z VmklinkMPI-LIB : Sent request ID 101 size 165
2019-02-16T10:20:21.187Z VmklinkMPI-LIB : { : ID: 0x101, bufLen: 165 }
2019-02-16T10:20:21.187Z 2104183:OSFSVmklinkResourceAvailableCBHandler:519: resource available callback received
2019-02-16T10:20:22.188Z Enqueue a new work item in threadpool osfsdVsan
2019-02-16T10:20:22.188Z Add the new item into the pending list
2019-02-16T10:20:22.188Z Adding a thread to threadpool osfsdVsan
2019-02-16T10:20:22.188Z Added a new thread to threadpool (osfsdVsan), numThreadsInPool (1)
2019-02-16T10:20:22.188Z Starting main loop for the worker thread osfsdVsan
2019-02-16T10:20:22.188Z 2104276:VSANEA_ForEachObjectApplyFunction:185: Querying CMMDS
2019-02-16T10:20:26.192Z VmklinkMPI-LIB : Received request ID 102 of size 165
2019-02-16T10:20:26.192Z VmklinkMPI-LIB : { : ID: 0x102, bufLen: 165 }
2019-02-16T10:20:26.192Z VmklinkMPI-LIB : Msg received. Calling recv Handler
2019-02-16T10:20:26.192Z VmklinkMPI-LIB : { : ID: 0x102, bufLen: 165 }
2019-02-16T10:20:26.192Z 2104183:DebugDumpVmklinkRequest:1063: {ID: 0x102; type: LOOKUP; pid: [ vsan]; cid: 52dcf84bacfe6581-5b9491c54bbdc952; status: 0; bufLen: 0; mf: 1634956846; o: 32497584236080750 }
********************************************************************************
2019-02-16T10:22:55.476Z 2104183:Provider_Lookup:469: Found matching driver for ID [ vsan]
2019-02-16T10:22:55.476Z Enqueue a new work item in threadpool osfsd-nvsan
2019-02-16T10:22:55.476Z Add the new item into the pending list
2019-02-16T10:22:55.476Z Adding a thread to threadpool osfsd-nvsan
2019-02-16T10:22:55.476Z Starting main loop for the worker thread osfsd-nvsan
2019-02-16T10:22:55.476Z Added a new thread to threadpool (osfsd-nvsan), numThreadsInPool (1)
2019-02-16T10:22:55.476Z 2105216:Lookup:660: Got lookup request for ".vsan.stats"
2019-02-16T10:22:55.476Z 2105216:Provider_Lookup:469: Found matching driver for ID [ vsan]
2019-02-16T10:22:55.476Z 2105216:VsanMountInt:2406: VsanMountInt: request for [.vsan.stats]
2019-02-16T10:22:55.476Z 2105216:VsanDir_FriendlyNameToUUID:1696: Name [.vsan.stats] not found in directory
2019-02-16T10:22:55.476Z 2105216:VsanMountInt:2443: VsanMountInt: Failed to resolve .vsan.stats: Not found
2019-02-16T10:22:55.476Z 2105216:LookupCompletion:698: lookup completion: buf =
2019-02-16T10:22:55.476Z 2105216:PostVmklinkSendEvent:352: Create ordered event to send response.
2019-02-16T10:22:55.476Z 2105216:DebugDumpVmklinkResponse:1097: {ID: 0x121; type: LOOKUP; pid: [ vsan]; cid: 52dcf84bacfe6581-5b9491c54bbdc952; status: Not found; bufLen: 0 }
2019-02-16T10:22:55.477Z 2104183:Event_Pump:386: PumpEvents: Interrupted system call, continuing
2019-02-16T10:22:55.477Z VmklinkMPI-LIB : VmklinkMPILib_Send called for 0x121,
2019-02-16T10:22:55.477Z VmklinkMPI-LIB : Matching request found for 0x121,
2019-02-16T10:22:55.477Z VmklinkMPI-LIB : Send got req = DF07A885D0 (ID: 121)
2019-02-16T10:22:55.477Z VmklinkMPI-LIB : Vmklink send sent total: 186 bytes of 186 needed
2019-02-16T10:22:55.477Z VmklinkMPI-LIB : Sent request ID 121 size 165
2019-02-16T10:22:55.477Z VmklinkMPI-LIB : { : ID: 0x121, bufLen: 165 }
2019-02-16T10:22:55.477Z 2104183:OSFSVmklinkResourceAvailableCBHandler:519: resource available callback received
[root@ESXi1:/var/log]