Symptoms

  • IO Filters marked offline in a vSAN cluster. One of the reasons can be  the missing Self Signed SSL certificate in /etc/vmware/ssl/castore.pem
  • Before making any changes in the system , please validate if the customer is using the third party certificates
  • Configuring Custom Certificates on ESXi hosts to authenticate vSAN hosts https://kb.vmware.com/s/article/56441

Cause

  • The vSAN GUI (   vCenter > Configure > Storage Providers ) may show all the hosts IOfilter storage providers as “offline” 
  • The Re-scan or re-synchronize of the VASA providers does not make any change to  the state of IOfilter.  
  • The upgrade of ESXi does not resolve the issue.

You may find the below instances showing that SSL certs are not being verified for the hosts in the IOfiltervpd logs (ESXi :  /var/log/iofiltervpd.log )

2019-03-07T04:12:50Z iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can’t be verified
2019-03-07T04:13:00Z iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can’t be verified
2019-03-07T04:13:00Z iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can’t be verified
2019-03-07T04:13:10Z iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can’t be verified
2019-03-07T04:13:20Z iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can’t be verified

  • The newly added host may show the IOfilters proviiders as “online”
  • You may find that the certificates located at  /etc/vmware/ssl/castore.pem , have missing the “Self Signed Certificate” for the host with IOfilters  offline
  • Following is Self signed certificate which must be part of the /etc/vmware/ssl/castore.pem file 

Resolution

You may follow the below steps :

  • Put the host in Maintenance mode with Ensure Accessibility one at a time
  • Take backup of the current of cert file  /etc/vmware/ssl/castore.pem 
  • Copy the file /etc/vmware/ssl/castore.pem to the Hosts from the working host.
  • Run command to replace the older file with newer one : cp /tmp/castore.pem /etc/vmware/ssl/castore.pem

  • Reboot the hosts one at a time.

The providers should show as “online” in vCenter > Configure > Storage Providers

Advertisement