SQL Server Agent Jobs Queued on Maximum User Working Threads (Fix)

🚨Part of the SQL Server Errors series, the exact messages and what actually causes them.In: Maintenance & Automation › SQL Agent & Jobs

SQL Server Agent [398]message template · read from SQLAGENT.RLL
[398] The user job (%s) has been queued because the maximum number of user working threads (%ld) are already running. This user job will be executed as soon as one of the working threads finishes execution.
⚡Run sp_help_job @execution_status = 2 to see what is waiting for a thread right now, then read the Agent log. [398] is the Agent-wide limit on user jobs, [251] is one subsystem’s limit. max worker threads in sp_configure is a different limit and changing it will not help.

SQL Server Agent does not always tell you a job failed. Sometimes it tells you the job is queued, and then nothing runs. Nothing is broken and nothing has errored. Agent has run out of the threads it uses to start user jobs, so the next one goes on a queue.


Check What Is Queued Right Now

Ask Agent first. sp_help_job reads the live job state from Agent, and MS Docs lists execution status 2 as Waiting for thread, so this answers “is anything stuck now”, not “did something wait last night”:

EXEC msdb.dbo.sp_help_job @execution_status = 2;   -- 2 = Waiting For Thread
sp_help_job on the lab instanceexample output, not something to copy · 32 columns, first 4 shown
job_id originating_server name enabled ------ ------------------ ---- -------

No rows is the answer you want. On the same instance, @execution_status = 4 (Idle) returns all 26 jobs, so an empty result means nothing is waiting, not a filter that matched nothing. A row here is a job Agent has accepted and not started: note its name, then go to the Agent log to see which limit is holding it.


Find the Message in the Agent Log

This is in the SQL Server Agent error log, not the SQL Server error log, which is why a search of the usual place turns up nothing. In SSMS it is under SQL Server Agent, Error Logs. From a query window:

CREATE TABLE #agentlog (LogDate DATETIME, ErrorLevel INT, [Text] NVARCHAR(MAX));

INSERT  #agentlog
EXEC msdb.dbo.sp_readerrorlog 0, 2;   -- 2 = the SQL Server Agent log

SELECT  CONVERT(VARCHAR(19), LogDate, 120) AS LogDate, [Text]
FROM    #agentlog
WHERE   [Text] LIKE '%working threads%'
   OR   [Text] LIKE '%is being queued for the%'
ORDER BY LogDate DESC;

DROP TABLE #agentlog;

The number in parentheses after user working threads is the limit you have reached, so the message carries its own answer to “what is my ceiling”. Which line you find tells you which limit it is:

  • [398] The user job … has been queued: the Agent-wide limit on user jobs, the message at the top of this page.
  • [501] The system job … has been queued: the same message for system working threads, which is a separate pool.
  • [251] Step n of job x is being queued for the y subsystem: one subsystem is full while Agent itself is not. Read your subsystem limits.

The wording of all 3 is read from the Agent message file, SQLAGENT.RLL, on SQL Server 2025. sp_readerrorlog with 2 as the second parameter is what points the query at the Agent log.


Three Limits With Almost the Same Name

Agent is telling you it is out of threads. It is not telling you which ceiling you hit. Be sure which of these you are looking at before you change anything, because the one everybody reaches for first, max worker threads, has nothing to do with Agent.

Limit Where it is set What it caps
Agent user working threads MaxWorkerThreads under the SQLServerAgent registry key. Not present on a clean install; Microsoft Support puts the default at 100 x CPUs. How many user job steps Agent will have running before it queues the next one. This is the limit message [398] is about.
Subsystem concurrent steps msdb.dbo.syssubsystems.max_worker_threads Concurrent steps for one subsystem: Distribution, LogReader, TSQL, PowerShell and the rest, each with their own number. Logged as [251].
Engine worker threads sp_configure 'max worker threads' Worker threads inside the database engine, for sessions and tasks. Nothing to do with Agent starting jobs.

The third row is the trap. Microsoft documents the max worker threads server configuration option as defaulting to 0, meaning SQL Server sizes the pool itself at startup from the CPU count and architecture. It governs the engine. If that pool were the problem you would be looking at sessions unable to get a worker and a THREADPOOL wait climbing, not at an Agent message about jobs.


Read Your Own Subsystem Limits

The second row is the one you can query, and it takes 5 seconds. per_cpu is the limit divided by the CPU count, which is how the defaults are worked out:

SELECT  s.subsystem,
        s.max_worker_threads,
        i.cpu_count,
        s.max_worker_threads / i.cpu_count AS per_cpu
FROM    msdb.dbo.syssubsystems AS s
CROSS JOIN sys.dm_os_sys_info AS i
ORDER BY s.subsystem_id;

Microsoft documents max_worker_threads in dbo.syssubsystems as the maximum number of concurrent steps for a given subsystem. Only members of sysadmin can read the table.

syssubsystems on the lab instanceexample output, not something to copy · SQL Server 2025, 8 CPUs
subsystem max_worker_threads cpu_count per_cpu --------------- ------------------ --------- ------- TSQL 160 8 20 CmdExec 80 8 10 Snapshot 800 8 100 LogReader 200 8 25 Distribution 800 8 100 Merge 800 8 100 QueueReader 800 8 100 ANALYSISQUERY 800 8 100 ANALYSISCOMMAND 800 8 100 SSIS 800 8 100 PowerShell 2 8 0

The grid says 2 useful things. Distribution sits at 800 here, so on this instance the subsystem is not what would hold back 100 distribution agents. And PowerShell is 2, which is the limit people meet by accident: schedule 3 PowerShell steps together and the third one waits, with a [251] line in the Agent log and no error anywhere.

The defaults are multipliers, not fixed numbers: TSQL 20 x CPUs, CmdExec 10 x, LogReader 25 x, the other replication subsystems, Analysis Services and SSIS 100 x, and PowerShell a flat 2. They come from msdb.dbo.sp_verify_subsystems, which is undocumented, read here from its definition on SQL Server 2025 CU8.

That procedure keeps whatever values the table already holds when it refreshes it, so CPUs added after install do not raise the limits. If per_cpu is lower than those multipliers, the server has grown since the table was written, so read yours rather than trusting mine.


Raise a Subsystem Limit

If the log shows [251] for a subsystem, raise that subsystem’s number and restart SQL Server Agent, which reads the table when it starts. MS Docs documents the column but no procedure for changing it; the steps come from Microsoft Support’s archived article on Agent MaxWorkerThreads (2011):

UPDATE  msdb.dbo.syssubsystems
SET     max_worker_threads = 400          -- your new limit
WHERE   subsystem = N'Distribution';

A raised value survives the refresh described above. Restarting Agent stops every job that is running, replication agents included, so on a distributor this is a planned change, not a mid-incident one.


What Microsoft Suggested, and Why It Was Not the Answer

On a distributor running 100 or more Distribution and Log Reader Agent jobs, the queue does not drain, because the agents already holding the threads are not finishing.

I hit this during an OS upgrade on a 2-node Windows failover cluster acting as the distributor. After upgrading the first node and flipping the workload onto it, the distribution agent jobs would not all run at the same time, and the behaviour stayed inconsistent through several rounds of troubleshooting. The thing that fixed it was not in SQL Server at all.

During the support case, the suggestion was to set a registry value under the Agent key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL14.MSSQLSERVER\SQLServerAgent

MaxWorkerThreads = 250   (REG_DWORD)

MSSQL14.MSSQLSERVER is the default instance of SQL Server 2017. Microsoft documents the instance ID format as MSSQL{nn}.INSTANCENAME, where nn is 14 for 2017, 15 for 2019, 16 for 2022 and 17 for 2025, so substitute your own version and instance name rather than copying that line.

Be clear about what that value is before you go looking for it. It is not a documented setting, it is not exposed anywhere in SSMS, and it will not already be there: a clean SQL Server 2025 install puts 60 values under that key and MaxWorkerThreads is not one of them. The only Microsoft description of it is the archived Support article above, which names it as the Agent-wide limit behind [398]. Applying the suggestion means creating a new DWORD rather than editing an existing one, then restarting the SQL Server Agent service. Treat it as what it is, an override handed over inside a support case for a specific problem.

It did not fix my problem. I am including it because it is a reasonable thing to try, and because if you have already tried it and are still queueing, you are in the same place I was.


The Fix: a Windows Value, Not a SQL Server One

The value that ended it lives in the Windows Session Manager configuration:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems

Value name: Windows        Type: REG_EXPAND_SZ

That single string holds the configuration for the Windows subsystem. Read off a current 64-bit install it looks like this, wrapped here to fit:

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows
SharedSection=1024,20480,768 Windows=On SubSystemType=Windows
ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3
ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16

The number that matters is the third one in SharedSection. Microsoft’s guidance on desktop heap allocation failures describes it as the size, in KB, of the desktop heap for each desktop associated with a non-interactive window station, and gives 768 as the 64-bit default. Services run in session 0, which is exactly that non-interactive world, and the same article warns that exhausting a desktop heap surfaces as out of memory errors and unexpected behaviour rather than as anything that names the real cause.

Raising the third number and rebooting cleared the queueing on my distributor, and it stayed clear. My note from the time records a value of around 1000 against a default of 768.

What Is Documented, and What Is Mine

Keeping those apart matters here more than usual, so:

  • Documented: what the third SharedSection value is, its default, and that desktop heap exhaustion produces misleading failures.
  • Documented: replication agents are standalone programs that run as jobs under SQL Server Agent, and each database published transactionally gets its own Log Reader Agent. That is why a distributor accumulates agent jobs in the first place, and it means every running agent is a separate Windows process, not a thread inside sqlservr.
  • Mine, not documented: the link between that heap and Agent’s user working threads message. I have 1 cluster, 1 fix and a result that held. I cannot show you a document that joins them, so treat it as a thing to try after the SQL Server side has come up empty, not as a known cause.

Check for the Symptom Before You Edit Anything

Desktop heap exhaustion has a documented fingerprint in the System event log: Event ID 243, source Win32k, level Warning, “A desktop heap allocation failed”. Look for it before touching the registry:

Get-WinEvent -FilterHashtable @{ LogName = 'System'; Id = 243 } -MaxEvents 20 -ErrorAction SilentlyContinue |
    Where-Object ProviderName -match 'Win32k' |
    Select-Object TimeCreated, ProviderName, Message

Finding it is a strong signal. Not finding it is weaker than it looks, because the same article notes the event is logged only once per session, so it can be sitting hours behind the symptom or already rolled out of the log.

Read before you change this
  • This is a registry edit on a production node and it takes effect after a restart, so it is maintenance window work, not something to try mid incident if you can avoid it.
  • Export the key first. The Windows value is one long string and a typo in it is a boot problem, not a SQL Server problem.
  • Change the third SharedSection number only. Leave every other parameter in the string exactly as it is.
  • Do not raise the second number past 20480. That is the one cap Microsoft states outright, and the same guidance says not to increase the desktop heap unless you need to.

Back Up the Value Before the Next OS Upgrade

The value was reset by the in-place OS upgrade. That is what put the distributor into this state to begin with, and it is the part worth carrying away, because the same upgrade is still ahead of you on the other node.

Before an OS upgrade on a distributor that runs a lot of agent jobs, export the key so you can compare afterwards:

reg export "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems" C:\temp\SubSystems-before-upgrade.reg

Then read it back after the upgrade and before you hand the node any replication work. A value that used to be tuned and has quietly gone back to its default is invisible until the workload lands on it, which on a cluster is the moment you fail over.


Frequently Asked Questions

Is this the same as max worker threads in sp_configure?
No, and changing that setting will not help. max worker threads sizes the database engine’s own thread pool for sessions and tasks, and its default of 0 lets SQL Server work the number out at startup. Message [398] comes from SQL Server Agent, which schedules jobs and is a separate process with its own limits.
The jobs are queued, so will they run eventually?
That is what the message promises, and on a server full of short T-SQL steps it is true. It is not much comfort on a distributor, because Distribution and Log Reader Agents are usually configured to run continuously. A thread taken by an agent that never finishes is a thread that never comes back, so the queue behind it does not move.
Why is nothing in the SQL Server error log?
Because it is not a SQL Server message. It is written by SQL Server Agent to the Agent log, which is a separate file. Read it in SSMS under SQL Server Agent, Error Logs, or with EXEC msdb.dbo.sp_readerrorlog 0, 2 where the second parameter selects the Agent log.
How many agent jobs does it take to hit this?
There is no single number, because the limit is on steps running at the same time, not on jobs defined. A distributor with 200 agent jobs where most are scheduled and idle may never see it, and one with far fewer continuous agents can. The message tells you the actual ceiling in its own text, so start there rather than guessing from the job count.
Do I have to reboot after changing SharedSection?
Plan for one. That string is read as part of bringing the Windows subsystem up, and a reboot is what I did on the node in question. On a failover cluster that is a planned failover, not a surprise, so treat this as maintenance window work rather than something to try during the incident if you can help it.
Does a new subsystem limit need an Agent restart?
Yes. Microsoft Support’s guidance for both the subsystem table and the MaxWorkerThreads registry value is to stop and start SQL Server Agent afterwards, and restarting Agent stops the jobs that are running at the time. Do it in a window.

Where To Go Next

A queued job is a symptom you can see. These are the checks that tell you what is actually running underneath it, and which ceiling you are near.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *