I've been able to get both AMD and NVIDIA cards to work. but this is with ESXI 6.7U1. Seems like folks have seen differences between 6.5 and 6.7.
I've been using more late model cards (GTX 1080ti, RTX 2080, 2080ti, Vega 64). Don't know if older ones have different issues or not.
AMD(Radeon) card seems to work fairly well most of the time. It has an annoying feature that comes up in 30Hz mode and I have to play with settings to get 60hz to stay.
NVIDIA: This one seems to work once fine, and then won't work again (ie restart or power down/up) until you reboot ESXI host. The workaround to this is to disable the graphics card on shutdown and then reenable on startup. Sounds like I pain but I scripted it on my system and it does it automatically so I don't have to remember. It has been working reliably.
-m