Today I was working on replacing SSL certificates on my vRealize IaaS server so as to complete the SSL certificate replacement on all vRealize components that I have deployed in my lab.
I am not going to cover here the steps needed to generate signed SSL certificates as I have covered them in my earlier blog posts. If you are looking for Step by Step demonstration of the same please have a look on this Article Grant Orchard.
After replacing the IaaS SSL certificates, IaaS Server needs to re-register the new certificates to the vRA Appliance. The commands used to achieve this is as follows:
# cd C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\Cafe
# vcac-config RegisterEndpoint --EndpointAddress https://vra1.alex.local/vcac --Endpoint ui -v
# vcac-config RegisterEndpoint --EndpointAddress https://vra1.alex.local/vcac/Repository --Endpoint repo -v
# vcac-config RegisterEndpoint --EndpointAddress https://vra1.alex.local/vcac/WAPI --Endpoint wapi -v
# vcac-config RegisterEndpoint --EndpointAddress https://vra1.alex.local/vcac/WAPI/api/status --Endpoint status -v
Note: vra1.alex.local is the name of my vRA appliance. Replace this with your appliance FQDN.
When I ran the above commands all of them was failing with below error messages.
Note: I have not included the full error message and only included the part which were making some sense (in Italics) as its too long to paste entire error message here.
System.Data.Services.Client.DataServiceQueryException: An error occurred while processing this request. ---> System.Data.Services.Client.DataServiceClientException:
An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.ComponentModel.Win32Exception: The system cannot find the file specified
[SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)]
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) +6675286
System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover) +219
System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout) +6703968
System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance) +6704427
As you can see from above error message that something was wrong with my SQL Server. I was scratching my head what could have gone wrong. I decided to test my SQL server connectivity first.
I launched SQL management studio and tried connecting to the vRA-IaaS DB instance using ‘sa’ user. It failed and I was wondering why. Then I decided to give a shot to the windows authentication method (as sometimes my ‘sa’ user have thrown tantrum at me)
To my surprise windows authentication was also not working.
Then I checked whether or not SQL services are running (of course this I should have checked at first place) and found SQL services were dead.
I understood what was wrong. Before replacing the SSL certs when I tried login to IaaS server using the service account ‘svcvcac’ as visible in above screenshots, Windows was telling me that password has expired and needs to be changed.
I forgot that I have configured the same service account to be used in my SQL server (I should have used a separate dedicated one for sql) and due to password change SQL services died as SQL is using this service account to run as.
I changed the password to original password (at the time of IaaS installation) and restarted my SQL services which started without any issues.
Now it was time to re-run the certificate re-register commands. This time all the commands completed successfully.
Lessons learnt from this mistake can be summarized as:
1: Service accounts should have “Password Never Expire” or longer duration for password expiry then any other normal domain user accounts
2: Use a dedicated service accounts for SQL servers.
Again I would say mistakes are essential for us to teach us really good lessons.
I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂