Troubleshooting list indexing issues

  • Description

    The List Manager application creates and manages lists that other applications and modules use, for example EXM. List Manager uses sitecore_analytics_index as a data source. During processing, the sitecore_analytics_index populates from the analytics database. List locking ensure that all contacts in the list are indexed and ready. Lists lock after every add ,remove, or clear operation. A special agent checks if all operations have finished and unlocks the lists.

    In Sitecore XP 8.2 Update-2 and later versions, you can disable list locking by enabling the \App_Config\Include\ListManagement\Sitecore.ListManagement.DisableListLocking.config.disabled file.

    This article describes the most common issues that can make list indexing stuck, describes how the list can be re-indexed, and contains the instructions about the information that Sitecore Support requires for troubleshoot issues with the List Manager.

  • General recommendations

    Scenario: List indexing does not finish.

    What can be checked and fixed:

    • Ensure that at least one server performs the processing role. Configuration files with Sitecore.Analytics.Processing.* names are enabled on the server.
    • If there are several environments (testing/development/production etc.), each environment must have its own set of Mongo databases.
    • Check that the server did not restart during the list upload process.
    • Make sure that the CD servers are configured properly and the processing role is disabled for them. The easiest way to check it is to make sure that the Sitecore.Analytics.Processing.Aggregation.Services.config file is disabled. If the file is enabled, please follow this article to configure the servers properly.
    • Make sure that the sitecore_analytics_index uses the Sitecore.ContentSearch.Maintenance.Strategies.TimedIndexRefreshStrategy strategy.
    • Make sure that your dedicated EXM dispatch server does not perform the processing role. The Sitecore.Analytics.Processing.Aggregation.Services.config file must be disabled.
    • If you use Lucene as your search provider for the sitecore_analytics_index, there must be only one Content Management server. This server must perform the processing role. You cannot have any dedicated processing servers.
    • If there is a dedicated processing server or more than one Content Management instance, you must use SOLR  as your search provider.
    • Make sure that the Analytics.ClusterName setting value is set on the Content Delivery servers as described in this article.
    • If you have more than one Content Delivery server, make sure that the Shared Session state is stored in an out-of-process storage. for example MongoDB, SQL or Redis. Refer to this documentation section for details.

    After checking the listed suggestions, try to re-index the list as described in the "How To Re-Index The Affected List" section below. If re-indexing does not help or the issue happens frequently, please refer the "Troubleshooting Data" section.

  • How to re-index the affected list

    To re-index the list:

    1. Install the patch that corresponds to your Sitecore XP version:
    2. Open to the /sitecore/admin/listmanager.aspx page.
    3. Select the affected list from the drop-down menu and click the Check button.
    4. Depending on the results:
      • If the number of contacts in the index is smaller than the number in the analytics database, click the Re-index button and wait for the process to finish.
      • If the number of contacts in the analytics database is smaller than the number in the index, it may mean that some contacts were removed manually from the analytics database or the data was not re-indexed after changing. If the data wasn't removed from the database manually, click the Remove obsolete contacts button. If the data was removed manually, you will need to rebuild the reporting database.
      • If the number of contacts in the analytics database and index is the same, but differs from the number on the list item, locate the list item under the /sitecore/system/List Manager/All Lists one and change the Recipients field to the correct value. In most cases, the list must be unlocked after 15 seconds. Otherwise, you can unlock it manually from List Manager dashboard.

    Note: If the number of contacts in the analytics database differs from the number of contacts on the list item, the issue is most likely caused by incorrect code that was used to add contacts to the list. If a contact is added to a list using the analytics API (by modifying contact tags), it is important to increment the Recipient field value of the list item in the master database. This action may be difficult to perform on Content Delivery servers where the master database is not available. For scaled environments, you should use the EXM API, which performs all needed operations.

  • Collecting Troubleshooting Data

    If you performed the previous actions and the issue still exists and is reproducible, contact Sitecore Support and provide the following information:

    • The way that you add contacts to the list (List Manager UI/custom code/EXM API etc).
    • Is the issue reproducible for all lists?
    • Does the number of indexed contacts always remain at 0 of n, or do some contacts get processed?
    • The output of the /sitecore/admin/listmanager.aspx page after you execute of the Check operation for the affected list (see the "How To Re-Index The Affected List" section above).
    • What servers are present in the environment and their roles.
    • Basic information from each server.
    • The code that you use to add contact to a list and an archive of the \bin folder if the contacts are added to the list using custom code.
  • Enabling extended logging [Optional]

    Sometimes, when troubleshooting list indexing issues, you must enable extended logging to find the root cause of the problem. This section is optional and you can perform the steps below if a support engineer requests this from you:

    1. In the \App_Config\Sitecore.config file (for XP 8.1 and later) or the \Web.config file (for XP 8.0), change the priority value to DEBUG for the <root> node:
      <root>
        <priority value="DEBUG" />
        <appender-ref ref="LogFileAppender" />
      </root>
    2. In the \App_Config\Include\Sitecore.ContentSearch.config file, change the level value to DEBUG for the Crawling logger:
      <logger name="Sitecore.Diagnostics.Crawling" additivity="false">
        <level value="DEBUG"/>
        <appender-ref ref="CrawlingLogFileAppender"/>
      </logger>
    3. Enable the \App_Config\Include\Sitecore.ContentSearch.VerboseLogging.config file.

Applies to:

CMS 8.0 - 8.2

November 13, 2017
November 13, 2017

Keywords: 

  • Search and Indexing