Troubleshooting xConnect certificate issues

Symptoms:

Exceptions in the log files:

[Experience Analytics]: Failed to synchronize segments. Message: The certificate was not found. Store: My, Location: CurrentUser, FindType: FindByThumbprint, FindValue: 83DCC21BBF54D76F71D7B67EA2319273BCDA8E10, InvalidAllowed: True.. Details:    at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.GetAll(NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.GetAll(NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsManager.GetSegmentsToSynchronize()
   at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsManager.SynchronizeAllSegments()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsProcessor.<process>d__4.MoveNext()

Exception System.InvalidOperationException: The certificate was not found. Store: My, Location: CurrentUser, FindType: FindByThumbprint, FindValue: 83DCC21BBF54D76F71D7B67EA2319273BCDA8E10, InvalidAllowed: True.
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetReportSegments()
   at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetRequestType()
   at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.GetModelFromBindingContext(HttpActionContext actionContext, ModelBindingContext bindingContext)
   at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.BindModel(HttpActionContext actionContext, ModelBindingContext bindingContext)
   at System.Web.Http.ModelBinding.ModelBinderParameterBinding.ExecuteBindingAsync(ModelMetadataProvider metadataProvider, HttpActionContext actionContext, CancellationToken cancellationToken)
   at System.Web.Http.Controllers.HttpActionBinding.<executebindingasynccore>d__12.MoveNext()
   ...

Exception System.InvalidOperationException: Ensure definition type did not complete successfully. StatusCode: 401, ReasonPhrase: 'Invalid certificate', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Pragma: no-cache
  Cache-Control: no-cache
  Date: Thu, 02 Jan 2020 15:03:36 GMT
  Server: Microsoft-IIS/10.0
  X-AspNet-Version: 4.0.30319
  X-Powered-By: ASP.NET
  Content-Length: 0
  Expires: -1
}
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
   at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetReportSegments()
   at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetRequestType()
   at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.GetModelFromBindingContext(HttpActionContext actionContext, ModelBindingContext bindingContext)
   at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.BindModel(HttpActionContext actionContext, ModelBindingContext bindingContext)
   at System.Web.Http.ModelBinding.ModelBinderParameterBinding.ExecuteBindingAsync(ModelMetadataProvider metadataProvider, HttpActionContext actionContext, CancellationToken cancellationToken)
   at System.Web.Http.Controllers.HttpActionBinding.<executebindingasynccore>d__12.MoveNext()
   ...

Solution:

1. Ensure that certificate thumbprint in the connection strings of Sitecore XP roles matches the thumbprint of the certificate in use:
   

   <add name="xconnect.collection.certificate" 
   connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=83DCC21BBF54D76F71D7B67EA2319273BCDA8E19" />

   <add name="sitecore.reporting.client.certificate" 
   connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=83DCC21BBF54D76F71D7B67EA2319273BCDA8E19" />

   <add name="xdb.marketingautomation.operations.client.certificate" 
   connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=83DCC21BBF54D76F71D7B67EA2319273BCDA8E19" />

   <add name="xdb.marketingautomation.reporting.client.certificate" 
   connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=83DCC21BBF54D76F71D7B67EA2319273BCDA8E19" />

   <add name="xdb.referencedata.client.certificate" 
   connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=83DCC21BBF54D76F71D7B67EA2319273BCDA8E19" />

2. Ensure that validateCertificateThumbprint value of xConnect and Marketing Automation roles matches the thumbprint of the certificate in use. The setting can be found in the following file: \App_Config\AppSettings.config
3. Ensure that xconnect.collection.certificate connection string of the AutomationEngine job has correct certificate thumbprint:
   \App_Data\jobs\continuous\AutomationEngine\App_Config\ConnectionStrings.config
4. [For Sitecore XP 9.1+] Ensure that xconnect.collection.certificate, xconnect.configuration.certificate, and xconnect.search.certificate connection strings of the ProcessingEngine job have the correct thumbprints: \App_Data\jobs\continuous\ProcessingEngine\App_Config\ConnectionStrings.config
5. Ensure that the AllowInvalidClientCertificates app setting is true on Sitecore XP roles if your certificate is self-signed (the setting can be found in the Web.config file by default):
   

   <add key="AllowInvalidClientCertificates" value="True" />

6. Ensure that AllowInvalidClientCertificates app setting is true on xConnect roles if your certificate is self signed. The setting can be found in the following files by default:
   \App_Config\AppSettings.config
   \App_Data\jobs\continuous\JOB_NAME_GOES_HERE\App_Config\AppSettings.config

Depending on the implementation of the specific solution, do the following:

• For on-premise:
  
  1. Check the certificate purpose using PowerShell:
     

     $cert = Get-ChildItem -Path Cert:\LocalMachine\My | Where-Object {$_.Thumbprint -match "XXXX...XXXX"} 
     foreach($key in $cert.Extensions){if('EnhancedKeyUsages' -in $key.psobject.Properties.Name){ $key.EnhancedKeyUsages.FriendlyName }}
     

     where XXXX...XXXX is the thumbprint from connection string.
     
     Note that the example uses "LocalMachine" StoreLocation and "My" StoreName. If you have different values in the connection strings, update the script accordingly:

     <add name="xconnect.collection.certificate" connectionString="StoreName=My;StoreLocation=LocalMachine;FindType=FindByThumbprint;FindValue=XXXX...XXXX" />

     The script must output the Client Authentication in the list:
     
     
  2. Ensure that the process that runs the Sitecore XP and xConnect application has access to client certificate private keys:
     
     

     2.1. Open the Windows Run window and enter mmc command.

     2.2. In the opened window click File, in the drop-down list select Add/Remove Snap In, Among the Available snap-ins select Certificates, click Add. In the pop-up window, select the Computer Account radio button and click Finish.

     2.3. In the left-side Console Root area, expand Certificates (Local Computer), expand Personal folder, click the Certificates folder. In the central window area, find your client certificate in the table (to ensure that the certificate is correct, double click it, select Details and select Thumbprint in the list. It must be the same as in the connection strings).
     

     2.4. Right click on the certificate, select All Tasks in the drop-down list, select Manage Private Keys:
     

     2.5. Ensure that the users that run the applications are present in the list:
     
     
     If the user is missing, click Add, and in the opened window, specify server root in the From this location field and type the user name in the search window. In the search window, type "IIS AppPool\UserName", where UserName is the name of the user that is used for running the application:
     
     
     Task Manager can be used for finding a user that runs the application. For example, the user name is "sc902.xconnect" here.
     

  3. Ensure that trusted root does not contain non-self-signed certificates using the following PowerShell script:
     

     Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}

     If there are any, move them to the Intermediate Certification Authorities:
     

     Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Move-Item -Destination Cert:\LocalMachine\CA

  4. Ensure that xConnect instances accept Client Certificates:
     

     4.1. Select xConnect site in IIS.

     4.2. Click the SSL Settings icon in the IIS section and ensure that the configuration looks as follows:
     

  For Azure Web Apps:
  

  1. Ensure that Incoming Client certificate is enabled on xConnect services:
     
     
  2. Ensure that the certificate is present in the private certificates list of all Sitecore and xConnect Web Apps:
     
  3. Ensure that the WEBSITE_LOAD_CERTIFICATES app setting is defined for all Sitecore and xConnect Web Apps and contains the same thumbprint as the certificate has: