I had a similar issue in my new 6.5 deployment. After removing a host and migrating VMK adapters, it still showed that there were active ports. My workaround was to edit the port group and block all ports. That released the ghost "active" ports and then I was able to delete the port group and VDswitch.
This worked for me!!!