Insufficient System Memory and Memory Grant Timeouts (Errors 701 and 8645)

🚨Part of the SQL Server Errors series, the exact messages and what actually causes them.

Error 701Msg 701  ·  Level 19  ·  logged
There is insufficient system memory in resource pool ‘default’ to run this query.
Error 8645Msg 8645  ·  Level 17  ·  logged
A timeout occurred while waiting for memory resources to execute the query in resource pool ‘default’. Rerun the query.
Neither of these means “buy more RAM”, and the difference between them is who is at fault. 701 is the server unable to find memory to start. 8645 is the server having the memory but giving it to someone else for too long. One is a capacity problem, the other is usually one badly estimated query.

These two arrive together often enough that they get treated as one symptom with one answer, and the one answer is usually wrong. Adding memory to a server hitting 8645 can change nothing at all, because the memory was already there.


701 and 8645 Are Different Failures

MsgWhat happenedWhere to look first
701
severity 19
SQL Server could not allocate the memory it needed at all. Not a queue, not a wait. The request could not be satisfied.Instance-wide. max server memory, what else is on the box, and whether something outside SQL Server is taking the machine’s memory.
8645
severity 17
The query queued for a memory grant, waited past the timeout, and gave up. The memory exists. It was committed to other queries.A single query, usually. Which grants are outstanding, how large they are, and whether the estimate that produced them was sane.

The severities carry the same message. 701 is severity 19, which is a resource error the server logs as serious. 8645 is severity 17, an insufficient-resources condition that resolves on its own. The wording of 8645 says as much: “Rerun the query.” SQL Server is telling you the condition was temporary. 701 makes no such offer.

Worth separating from both, because it sits in the same mental bucket and is neither: error 8623, “the query processor ran out of internal resources and could not produce a query plan”, severity 16. That is not memory pressure. It is a statement too complex to compile, nearly always a generated query or an enormous IN list, and no amount of RAM changes it.


Do Not Diagnose This From Cumulative Wait Stats

Every article about 8645 tells you to look at RESOURCE_SEMAPHORE in sys.dm_os_wait_stats. That is the right wait type and the wrong source, and it is worth showing why with a real measurement rather than asserting it.

Here is what that DMV said on an instance that had been up for one day. The last column is the one that matters: cumulative wait time as a percentage of how long the instance had been running.

wait_typewaitsmax single wait% of uptime
SOS_WORK_DISPATCHER89,878,1011.0 hr2424.3%
SLEEP_TASK165,7151.1 hr429.6%
LOGMGR_QUEUE391,6861.0 hr200.1%
RESOURCE_SEMAPHORE2168.0 hr163.9%

Read that RESOURCE_SEMAPHORE row cold and it is alarming: 216 queries, one of them waiting eight hours for a memory grant. It is also meaningless. The instance had been up 24 hours. A single wait cannot last eight hours inside a window where nothing waited eight hours, and a cumulative total cannot be 164% of the time available to accumulate it.

Two things produce that, and both are ordinary:

  • Waits accumulate per task, in parallel. Many things wait at once, so totals routinely exceed wall-clock time. SOS_WORK_DISPATCHER at 2424% is the clearest tell: that is background workers idling, and it is present on every healthy instance.
  • The wait clock does not stop when the machine does. Anything that suspends the host, a VM pause, a laptop sleeping, a snapshot, leaves a wait running across the gap and records a single enormous value. That eight-hour maximum is a machine that was asleep, not a query that was patient.
Why this is in the post This was not a prepared example. I ran sys.dm_os_wait_stats while writing, saw 216 RESOURCE_SEMAPHORE waits totalling more than 18 hours, and very nearly used it as a worked example of memory pressure. Checking it against uptime is what stopped that. The number was accurate and it meant nothing, which is the exact failure mode this section exists to warn about.

So the sanity check comes first, before any wait-stats conclusion:

-- Any wait whose total exceeds uptime is telling you about accumulation or a
-- suspended host, not about pressure. Check this BEFORE reading the numbers.
DECLARE @up_ms bigint = (SELECT DATEDIFF_BIG(ms, sqlserver_start_time, GETDATE())
                         FROM sys.dm_os_sys_info);

SELECT TOP 10 wait_type,
       waiting_tasks_count AS waits,
       wait_time_ms,
       max_wait_time_ms,
       CAST(wait_time_ms * 100.0 / NULLIF(@up_ms,0) AS decimal(10,1)) AS pct_of_uptime
FROM sys.dm_os_wait_stats
WHERE waiting_tasks_count > 0
ORDER BY wait_time_ms DESC;

The usable ways to measure memory grant pressure are all narrower in time than “since the instance started”:

  • Take a delta. Snapshot sys.dm_os_wait_stats, wait a known interval, snapshot again, subtract. A window you chose is interpretable in a way that “since startup” is not.
  • Look at what is happening right now, with the two DMVs below. Nothing cumulative, nothing to misread.
  • Use Query Store if the database has it on. It keeps per-query memory grant history with timestamps, which is what you actually want when the 8645 happened at 3am.

What To Run While It Is Happening

-- 1. Who is holding a grant, and who is queued behind them.
--    grant_time IS NULL means this query is waiting. That is your 8645 in progress.
SELECT r.session_id,
       g.requested_memory_kb / 1024  AS requested_mb,
       g.granted_memory_kb  / 1024   AS granted_mb,
       g.used_memory_kb     / 1024   AS used_mb,
       g.ideal_memory_kb    / 1024   AS ideal_mb,
       g.queue_id, g.wait_order, g.wait_time_ms,
       CASE WHEN g.grant_time IS NULL THEN 'WAITING' ELSE 'granted' END AS state,
       t.text
FROM sys.dm_exec_query_memory_grants g
LEFT JOIN sys.dm_exec_requests r ON r.session_id = g.session_id
OUTER APPLY sys.dm_exec_sql_text(g.sql_handle) t
ORDER BY g.requested_memory_kb DESC;

-- 2. The semaphore itself: how much grant memory exists, and how much is spoken for.
SELECT pool_id, resource_semaphore_id,
       target_memory_kb     / 1024 AS target_mb,
       max_target_memory_kb / 1024 AS max_target_mb,
       available_memory_kb  / 1024 AS available_mb,
       granted_memory_kb    / 1024 AS granted_mb,
       grantee_count, waiter_count
FROM sys.dm_exec_query_resource_semaphores;

The column that usually solves it is the gap between requested_mb and used_mb. A query that asked for 4 GB and used 40 MB did not need the memory, it needed a better estimate, and while it held that grant everything behind it was queuing. That single query is the cause of the 8645s, and it will not look like a problem in any duration-based report, because it ran fine.

Measured on a healthy instance sys.dm_exec_query_memory_grants returned 0 rows waiting and 0 rows granted, and sys.dm_exec_query_resource_semaphores showed grantee_count and waiter_count both at 0. Empty is the normal reading. Run these once now so that a non-empty result during an incident means something to you.

The Settings Worth Checking

SELECT name, value_in_use
FROM sys.configurations
WHERE name IN ('max server memory (MB)', 'min server memory (MB)',
               'min memory per query (KB)', 'index create memory (KB)');

SELECT total_physical_memory_kb / 1024 AS server_physical_mb,
       available_physical_memory_kb / 1024 AS server_available_mb
FROM sys.dm_os_sys_memory;

SELECT physical_memory_in_use_kb / 1024 AS sql_using_mb,
       large_page_allocations_kb / 1024 AS large_pages_mb,
       memory_utilization_percentage
FROM sys.dm_os_process_memory;

Two settings cause more of these than hardware does:

  • max server memory set low and forgotten. Often set during a build, or copied from a smaller server, and never revisited. On the instance measured for this post it was 2048 MB on a machine with 7968 MB physical. Perfectly reasonable for a lab, and exactly the shape that produces grant queuing on a server people expect to be sized for the work.
  • min memory per query raised. The default is 1024 KB. Every query gets at least this, so raising it multiplies across concurrency and shrinks what is left for the queries that genuinely need a grant.

For 701 specifically, also look outside SQL Server. It is an allocation failure, so anything else on the box competing for memory belongs in the investigation: another instance, an antivirus scan, a backup agent, or a VM whose memory is being reclaimed by the host.


What Not To Do

  • Do not add RAM as the first move on an 8645. The memory existed. Unless you also raise max server memory, the instance will not use it, and if one query is requesting a grossly oversized grant it will simply take a bigger share of a bigger pool.
  • Do not raise min memory per query to “give queries more memory”. It does the opposite under concurrency, by reserving a floor for every query including the ones that need nothing.
  • Do not clear the cache to make it go away. DBCC FREEPROCCACHE and friends destroy the evidence and the plans, and the next execution of the same query will request the same oversized grant.
  • Do not read cumulative wait stats without checking them against uptime. As above, and it is the most common way this gets diagnosed wrongly.
  • Do not treat 8623 as a memory problem. Different error, different cause, and the fix is in the application that generated the statement.

Common Questions

Will more RAM fix error 8645?
Usually not on its own. 8645 means the memory was already committed to other queries, so the question is which query took it and whether it deserved it. Adding RAM without raising max server memory changes nothing at all, and adding both still leaves an oversized grant taking an oversized share.
Which is worse, 701 or 8645?
701 is severity 19 and 8645 is severity 17, and that ordering reflects it. 701 means an allocation could not be satisfied, which is an instance-wide condition. 8645 is a timeout on a queue, and SQL Server’s own message text tells you to rerun the query. Repeated 8645s still matter, they are just a different investigation.
My RESOURCE_SEMAPHORE waits look enormous. Is that the cause?
Check the total against instance uptime before concluding anything. Measured while writing this post: 216 waits totalling over 18 hours on an instance up for 24 hours, with a single wait recorded at 8 hours. Both numbers were artifacts of parallel accumulation and a host that had been suspended. Take a delta over a window you choose, or read the live DMVs instead.
What is a normal reading for sys.dm_exec_query_memory_grants?
Empty. On an instance with nothing queued it returns no rows, and waiter_count on the semaphore is 0. That is worth confirming on your own servers while they are healthy, so that rows appearing during an incident are immediately meaningful rather than something you have to interpret from scratch.
How do I find the query that caused it after the fact?
Query Store, if the database has it enabled, because it keeps memory grant figures per query with timestamps. Without it the live DMVs are gone the moment the grant is released, and the error log gives you the time of the 8645 but not the query that was holding the memory.
Is error 8623 the same problem?
No. 8623 is severity 16 and means the query processor could not produce a plan because the statement was too complex, typically a generated query or a very large IN list. It is a compilation limit, not memory pressure, and it is fixed in the application rather than on the server.

Related Scripts

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *