watcher: System.Net.Http.HttpRequestException: Error while copying content to a stream.

[karok2m]

Describe the bug

OnError callback of Watcher<T>() reports the following error after ~40 minutes when watching for a Kubernetes resource for which no instance exists in the cluster and no new one is created during the runtime. After the error watcher is closed (OnClosed callback is invoked).

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   at k8s.LineSeparatedHttpContent.PeekableStreamReader.PeekLineAsync()
   at k8s.LineSeparatedHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult\](Task waitTask, TState state, Func`2 returnFunc)
   at k8s.Kubernetes.CreateResultAsync[T\](HttpRequestMessage httpRequest, HttpResponseMessage httpResponse, Nullable`1 watch, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.ICustomObjectsOperations_ListClusterCustomObjectWithHttpMessagesAsync[T\](String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.ListClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.WatcherExt.<>c__DisplayClass1_0`2.<<MakeStreamReaderCreator>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at k8s.Watcher`1.<>c.<CreateWatchEventEnumerator>b__21_1[TR\](Task`1 t)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+MoveNext()
   at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)
   at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)

Kubernetes C# SDK Client Version
13.0.1

Server Kubernetes Version
1.26.10

Dotnet Runtime Version
net7.0

To Reproduce
After reading description of #828 I think that Kubernetes API server must close watch connection.

Expected behavior
Even if connection is closed by Kubernetes API server, the watcher should be able to recover and no attempt should be made to read past end of stream.

KubeConfig
N/A

Where do you run your app with Kubernetes SDK (please complete the following information):

• OS: Debian 11
• Environment: container running in Kubernetes cluster
• Azure, AKS

Additional context
N/A

[tg123]

could you please put a retry outside your watcher?

[karok2m]

@tg123, I have a watcher recreation logic that runs after OnClose() is invoked. That is my current workaround for this exception. But Attempted to read past the end of the stream. error still seems to be something that needs to be addressed.

[tg123]

this seems not a client side issue, probably server side closed the connection which is out of client side's control

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale