Hello Karthi,
Thank you for your post.
I noticed that you have enabled Custom SSO configuration. However, I see that you have not filled anything in the Advanced options.
Could you please try entering all the information and then try adding the vcenter server? Please use FQDN instead of IP address in all the fields and try adding the vcenter server.
If you need any assistance please let me know and we shall have a web-ex session.