Excessive load on ASP.NET Session State store

Expand all | Collapse all

  • Description

    Collapse

    Implementation details

    Sitecore relies on the default implementation of the ASP.NET SessionStateModule for session state management.

    To avoid data collisions in the session store and unexpected session state behavior, the SessionStateModule and SessionStateStoreProviderBase classes include functionality that exclusively locks the session store item for a particular session during the execution of an ASP.NET page.

    If the SessionStateModule instance encounters locked session data during a call to either the GetItemExclusive or GetItem methods, it re-requests the session data at half-second intervals until either the lock is released or the amount of time specified in the ExecutionTimeout property has elapsed.

    This behavior is fully described in the Locking Session-Store Data MSDN article. You can obtain detailed information about the specific ASP.NET SessionStateModule behavior by providing the reference number 117091416336585 to Microsoft Support.

    Excessive load nature

    If multiple requests arrive with the same ASP.NET session identifier, each of them tries to obtain the exclusive lock for the session item, even though only one can succeed at a time. This causes an excessive load on the session state store produced by multiple threads that try to simultaneously lock the same unit.

  • Symptoms

    Collapse

    You can observe the following symptoms when the session state store is under heavy load:

    • A spontaneous spike in the number of concurrent requests sent to the session state store. For example, a spike of GetItemExclusive commands for MS SQL Session state provider.
    • Session state store response time increases, or reaches DTU quota in your Azure SQL Database service.
    • Various messages in the Sitecore log files show database connectivity issues originating from the SessionStateModule or SessionState classes. For example: 
      Message: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
      Source: System.Data
         at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
         ... 
         at System.Web.SessionState.SqlSessionStateStore.SqlStateConnection..ctor(SqlPartitionInfo sqlPartitionInfo, TimeSpan retryInterval)
      Message: Unable to connect to SQL Server session database.
      Source: System.Web
         at System.Web.SessionState.SqlSessionStateStore.ThrowSqlConnectionException(SqlConnection conn, Exception e)
         at System.Web.SessionState.SqlSessionStateStore.SqlStateConnection..ctor(SqlPartitionInfo sqlPartitionInfo, TimeSpan retryInterval)
         ...
         at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
         at System.Web.SessionState.SessionStateModule.PollLockedSessionCallback(Object state)
      Exception type: TimeoutException 
          Exception message: Timeout performing EVAL, inst: 14, mgr: Inactive, err: never, queue: 71, qu: 0, qs: 71, qc: 0, wr: 0, wq: 0, in: 65536, ar: 0, IOCP: (Busy=1,Free=999,Min=8,Max=1000), WORKER: (Busy=19,Free=32748,Min=8,Max=32767), clientName: RD0003FF85401C
         at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
         at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
         at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags)
         at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.<>c__DisplayClass7.<Eval>b__6()
         at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.RetryForScriptNotFound(Func`1 redisOperation)
         at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.RetryLogic(Func`1 redisOperation)
         at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.Eval(String script, String[] keyArgs, Object[] valueArgs)
         at Sitecore.SessionProvider.Redis.RedisConnectionWrapper.TryTakeWriteLockAndGetData(String sessionId, DateTime lockTime, Object& lockId, ISessionStateItemCollection& data, Int32& sessionTimeout)
         at Sitecore.SessionProvider.Redis.RedisSessionStateProvider.GetItemFromSessionStore(Boolean isWriteLockRequired, HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
         at Sitecore.SessionProvider.Redis.RedisSessionStateProvider.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
         at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
      ERROR Application error.
      Exception: System.TimeoutException
      Message: Timeout waiting for a MongoConnection.
      Source: MongoDB.Driver
         at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection(AcquireConnectionOptions options)
         ...
         at Sitecore.SessionProvider.MongoDB.MongoSessionStateProvider.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
         at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
         at System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData)
         at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
         at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

      ERROR Application error.
      Exception: MongoDB.Driver.MongoConnectionException
      Message: Too many threads are already waiting for a connection.
      Source: MongoDB.Driver
         at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection(AcquireConnectionOptions options)
         at MongoDB.Driver.MongoServerInstance.AcquireConnection()
         at MongoDB.Driver.MongoServer.AcquireConnection(ReadPreference readPreference)
         at MongoDB.Driver.MongoCollection.Update(IMongoQuery query, IMongoUpdate update, MongoUpdateOptions options)
         at Sitecore.SessionProvider.MongoDB.MongoSessionStateStore.InsertItem(String application, String id, Int32 flags, SessionStateStoreData sessionState)
         at System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs)
         at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
         at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
      ERROR Application error.
      Exception: System.Web.HttpException
      Message: The request queue limit of the session is exceeded.
      Source: System.Web
   at System.Web.SessionState.SessionStateModule.QueueRef()
         at System.Web.SessionState.SessionStateModule.PollLockedSession()
         at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
         at System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData)
         at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
         at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

  • Solution

    Collapse

    To resolve this issue:

    • Ensure that your firewall is configured to protect the application from DoS attacks. IIS has a Dynamic IP Address Restrictions module that provides a set of features to protect from DoS attacks.

    • Avoid storing custom objects in the session. The session mechanism can become a bottleneck if there is a large session size. Use custom caches or shared storages instead.

    • Avoid using session state or explicitly specify to use it in read-only mode wherever possible by utilizing ReadOnly MVC controllers or use the EnableSessionState="Readonly" attribute on a page directive for ASP.NET Web Forms pages.

    • Split the Sitecore.Sessions database (in SQLServer, MongoDB, or Redis) into two dedicated databases:

      • Sitecore.Private.Sessions
      • Sitecore.Shared.Sessions

    • In the Web.config and Sitecore.Analytics.Tracking.config files, increase the pollingInterval attribute value on both Private and Shared session state provider definitions from 2 to 60 seconds.

    • In the Web.config file, on the httpRuntime element, reduce the executionTimeout attribute value to 110 seconds (default ASP.NET value). This property controls the allowed execution time for the request, and the maximum exclusive lock duration for a request. A lower lock duration ensures that other requests can obtain the lock faster if the lock was not released previously.

    • Increase the default polling interval for Sitecore Shared session state if Analytics tracking is enabled:

      1. In the Sitecore.Analytics.Tracking.config file, find the timeoutBetweenLockAttempts parameter.
      2. Increase its value from 10 to 200 milliseconds.

      Note: This increases the waiting time between attempts to lock contact in the shared session state.

    • Configure the throttle concurrent requests per session and set a lower value than the default one (50):

      1. Install .NET Framework 4.7 or higher on the server with Sitecore Content Delivery instance.

        Note: To determine which .NET Framework versions are installed, see this article.
      2. In the Web.config file, find the appSettings element, and check the following setting: 
        <add key="aspnet:RequestQueueLimitPerSession" value="25"/>
      3. Keep the .NET Framework version on the <httpruntime /> and <compilation /> elements that Sitecore XP ships with: 
        <configuration>
        ...
          <system.web>
          ...
            <compilation targetFramework="4.5.2"/>
            <httpRuntime targetFramework="4.5.2"/>
          ...
          </system.web>
        ...
        </configuration>

    • Switch the Private Session State to In-process mode and configure a Load Balancer to use sticky sessions:

      1. In the Web.config file, find the sessionState element.
      2. Change the mode attribute value to InProc.
    • Do not send multiple requests from ASPX or CSHTML pages back to Sitecore XP using JavaScript (for example, Ajax calls).

Applies to:

CMS 6+

November 09, 2017
November 20, 2019

Reference number:

153901, 179898, 196356, 196647, 200264

Keywords: 

  • Performance,
  • Scaling