Microsoft.Data.SqlClient.SqlException: Execution Timeout Expired during EndExecuteNonQueryAsync

[gidasaramsft]

We are using Microsoft.SqlClient 5.2.0 - We have couple of pipelines that run DML (update/delete/insert) queries on a SQL database.
Recently we are intermittently hitting the below error when running the pipelines:

The queries get a Sql server connection and use ExecuteNonQueryAsync to run the SqlCommand.

Query #923 update set_list set song_id = 13 where dml_flag = 1 and gig_id < 13 failed unexpectedly with exception: System.AggregateException: One or more errors occurred. ---> Microsoft.Data.SqlClient.SqlException: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out
--- End of inner exception stack trace ---
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Polaris.Test.Utilities.SqlServerUtils.<GetSqlServerFrontendConnectionAsync>d__11.MoveNext() in C:__w\1\s\test\TestUtilities\SqlServerUtils.cs:line 256
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Polaris.Utilities.SyncAwaiter.<>c__DisplayClass1_01.<<SyncAwaitThrow>b__0>d.MoveNext() in C:\__w\1\s\src\Utilities\SyncAwaiter.cs:line 0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Polaris.Utilities.SyncAwaiter.SyncAwaitThrow[T](Func1 taskToAwaitOn) in C:__w\1\s\src\Utilities\SyncAwaiter.cs:line 44
at Microsoft.Polaris.Test.Utilities.SqlServerUtils.GetSqlServerFrontendConnection(String connectionDatabase, HashSet1 enableFeTraceFlags, HashSet1 disableFeTraceFlags) in C:__w\1\s\test\TestUtilities\SqlServerUtils.cs:line 276
at Microsoft.Polaris.Test.IntegrationTest.Utils.ResultRowSet.d__11.MoveNext() in C:__w\1\s\test\IntegrationTest\Utils\ResultRowSet.cs:line 0
--- End of inner exception stack trace ---
---> (Inner Exception #0) Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Polaris.Test.Utilities.SqlServerUtils.d__11.MoveNext() in C:__w\1\s\test\TestUtilities\SqlServerUtils.cs:line 256
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Polaris.Utilities.SyncAwaiter.<>c__DisplayClass1_01.<<SyncAwaitThrow>b__0>d.MoveNext() in C:\__w\1\s\src\Utilities\SyncAwaiter.cs:line 0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Polaris.Utilities.SyncAwaiter.SyncAwaitThrow[T](Func1 taskToAwaitOn) in C:__w\1\s\src\Utilities\SyncAwaiter.cs:line 44
at Microsoft.Polaris.Test.Utilities.SqlServerUtils.GetSqlServerFrontendConnection(String connectionDatabase, HashSet1 enableFeTraceFlags, HashSet1 disableFeTraceFlags) in C:__w\1\s\test\TestUtilities\SqlServerUtils.cs:line 276
at Microsoft.Polaris.Test.IntegrationTest.Utils.ResultRowSet.d__11.MoveNext() in C:__w\1\s\test\IntegrationTest\Utils\ResultRowSet.cs:line 0
ClientConnectionId:c7a38229-4986-468a-b723-bd0e794291af
Error Number:-2,State:0,Class:11<---
System.AggregateException: One or more errors occurred. ---> Microsoft.Data.SqlClient.SqlException: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out
--- End of inner exception stack trace ---
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Polaris.Test.Utilities.SqlServerUtils.d__11.MoveNext() in C:__w\1\s\test\TestUtilities\SqlServerUtils.cs:line 256
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Polaris.Utilities.SyncAwaiter.<>c__DisplayClass1_01.<<SyncAwaitThrow>b__0>d.MoveNext() in C:\__w\1\s\src\Utilities\SyncAwaiter.cs:line 0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Polaris.Utilities.SyncAwaiter.SyncAwaitThrow[T](Func1 taskToAwaitOn) in C:__w\1\s\src\Utilities\SyncAwaiter.cs:line 44
at Microsoft.Polaris.Test.Utilities.SqlServerUtils.GetSqlServerFrontendConnection(String connectionDatabase, HashSet1 enableFeTraceFlags, HashSet1 disableFeTraceFlags) in C:__w\1\s\test\TestUtilities\SqlServerUtils.cs:line 276
at Microsoft.Polaris.Test.IntegrationTest.Utils.ResultRowSet.d__11.MoveNext() in C:__w\1\s\test\IntegrationTest\Utils\ResultRowSet.cs:line 0
--- End of inner exception stack trace ---
---> (Inner Exception #0) Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Polaris.Test.Utilities.SqlServerUtils.d__11.MoveNext() in C:__w\1\s\test\TestUtilities\SqlServerUtils.cs:line 256
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Polaris.Utilities.SyncAwaiter.<>c__DisplayClass1_01.<<SyncAwaitThrow>b__0>d.MoveNext() in C:\__w\1\s\src\Utilities\SyncAwaiter.cs:line 0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Polaris.Utilities.SyncAwaiter.SyncAwaitThrow[T](Func1 taskToAwaitOn) in C:__w\1\s\src\Utilities\SyncAwaiter.cs:line 44
at Microsoft.Polaris.Test.Utilities.SqlServerUtils.GetSqlServerFrontendConnection(String connectionDatabase, HashSet1 enableFeTraceFlags, HashSet1 disableFeTraceFlags) in C:__w\1\s\test\TestUtilities\SqlServerUtils.cs:line 276
at Microsoft.Polaris.Test.IntegrationTest.Utils.ResultRowSet.d__11.MoveNext() in C:__w\1\s\test\IntegrationTest\Utils\ResultRowSet.cs:line 0
ClientConnectionId:c7a38229-4986-468a-b723-bd0e794291af
Error Number:-2,State:0,Class:11<---

Further technical details

Microsoft.Data.SqlClient version: 5.2.0

[David-Engel]

That is the expected error when the command takes longer than the timeout. Have you tried increasing your CommandTimeout?