grpc authentication using htpasswd

[abcdefstar]

Hi

Its not an issue, may be im missing something. I'm setting up the cache in EKS with an ALB and using grpcs for the remote cache. I wanted to use htpasswd based authentication for authenticating. The authentication works well when used https instead of grpcs but with grpcs i get an error as unimplemented code. Is there any specific configuration for grpcs needs to be done?
without auth, it works fine and i could upload to the s3 bucket the cache files.

Im using build --remote_cache=grpcs://user:pwd@host:9092 --remote_header=key=value.

Also I would like to know is it possible to pass directly as grpcs://user:pwd@host:9092/value instead of using a remote_header?

[mostynb]

Hi, could you copy+paste the exact error message you get on the bazel side? And any relevant logs from bazel-remote when this happens.

[abcdefstar]

Hi, so when i pass any command with bazel i get the same response when its passed with userid and pwd, eg bazel clean
bazel clean --verbose_failures
Starting local Bazel server and connecting to it...
INFO: Invocation ID: ac53db1e-9b48-4eaf-9a2e-4a07b3a9c669
ERROR: java.io.IOException: io.grpc.StatusRuntimeException: UNIMPLEMENTED: HTTP status code 404
invalid content-type: text/plain; charset=utf-8
trailers: Metadata(:status=404,server=awselb/2.0,date=Sat, 03 Sep 2022 16:57:59 GMT,content-type=text/plain; charset=utf-8,content-length=0)
at com.google.devtools.build.lib.remote.RemoteServerCapabilities.get(RemoteServerCapabilities.java:90)
at com.google.devtools.build.lib.remote.RemoteModule.verifyServerCapabilities(RemoteModule.java:197)
at com.google.devtools.build.lib.remote.RemoteModule.beforeCommand(RemoteModule.java:489)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:386)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:231)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:614)
at io.grpc.Context$1.run(Context.java:579)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.grpc.StatusRuntimeException: UNIMPLEMENTED: HTTP status code 404
invalid content-type: text/plain; charset=utf-8
trailers: Metadata(:status=404,server=awselb/2.0,date=Sat, 03 Sep 2022 16:57:59 GMT,content-type=text/plain; charset=utf-8,content-length=0)
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
at build.bazel.remote.execution.v2.CapabilitiesGrpc$CapabilitiesBlockingStub.getCapabilities(CapabilitiesGrpc.java:227)
at com.google.devtools.build.lib.remote.RemoteServerCapabilities.lambda$get$0(RemoteServerCapabilities.java:85)
at com.google.devtools.build.lib.remote.ReferenceCountedChannel.lambda$withChannelBlocking$2(ReferenceCountedChannel.java:84)
at com.google.devtools.build.lib.remote.ReferenceCountedChannel.lambda$withChannel$4(ReferenceCountedChannel.java:107)
at io.reactivex.rxjava3.internal.operators.single.SingleUsing.subscribeActual(SingleUsing.java:59)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback.onSuccess(SingleFlatMap.java:85)
at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onSuccess(SingleFlatMap.java:112)
at io.reactivex.rxjava3.internal.operators.single.SingleMap$MapSingleObserver.onSuccess(SingleMap.java:65)
at io.reactivex.rxjava3.internal.operators.single.SingleDoOnDispose$DoOnDisposeObserver.onSuccess(SingleDoOnDispose.java:84)
at io.reactivex.rxjava3.internal.operators.single.SingleDoOnError$DoOnError.onSuccess(SingleDoOnError.java:52)
at io.reactivex.rxjava3.internal.operators.observable.ObservableSingleSingle$SingleElementObserver.onComplete(ObservableSingleSingle.java:110)
at io.reactivex.rxjava3.internal.observers.DeferredScalarDisposable.complete(DeferredScalarDisposable.java:85)
at io.reactivex.rxjava3.subjects.AsyncSubject.subscribeActual(AsyncSubject.java:233)
at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13176)
at io.reactivex.rxjava3.internal.operators.observable.ObservableSingleSingle.subscribeActual(ObservableSingleSingle.java:36)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.internal.operators.single.SingleDoOnError.subscribeActual(SingleDoOnError.java:35)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.internal.operators.single.SingleDoOnDispose.subscribeActual(SingleDoOnDispose.java:38)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.internal.operators.single.SingleMap.subscribeActual(SingleMap.java:35)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback.onSuccess(SingleFlatMap.java:85)
at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:68)
at com.google.devtools.build.lib.remote.grpc.TokenBucket$1.onNext(TokenBucket.java:79)
at io.reactivex.rxjava3.internal.util.NotificationLite.accept(NotificationLite.java:247)
at io.reactivex.rxjava3.subjects.BehaviorSubject$BehaviorDisposable.test(BehaviorSubject.java:507)
at io.reactivex.rxjava3.subjects.BehaviorSubject$BehaviorDisposable.emitFirst(BehaviorSubject.java:468)
at io.reactivex.rxjava3.subjects.BehaviorSubject.subscribeActual(BehaviorSubject.java:224)
at io.reactivex.rxjava3.core.Observable.subscribe(Observable.java:13176)
at com.google.devtools.build.lib.remote.grpc.TokenBucket.lambda$acquireToken$0(TokenBucket.java:64)
at io.reactivex.rxjava3.internal.operators.single.SingleCreate.subscribeActual(SingleCreate.java:40)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap.subscribeActual(SingleFlatMap.java:37)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.internal.operators.single.SingleDefer.subscribeActual(SingleDefer.java:43)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap.subscribeActual(SingleFlatMap.java:37)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.core.Single.blockingGet(Single.java:3644)
at com.google.devtools.build.lib.remote.ReferenceCountedChannel.withChannelBlocking(ReferenceCountedChannel.java:84)
at com.google.devtools.build.lib.remote.RemoteServerCapabilities.lambda$get$1(RemoteServerCapabilities.java:84)
at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:244)
at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:125)
at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:114)
at com.google.devtools.build.lib.remote.RemoteServerCapabilities.get(RemoteServerCapabilities.java:82)
... 10 more

ERROR: Failed to query remote execution capabilities: UNIMPLEMENTED: HTTP status code 404
invalid content-type: text/plain; charset=utf-8
trailers: Metadata(:status=404,server=awselb/2.0,date=Sat, 03 Sep 2022 16:57:59 GMT,content-type=text/plain; charset=utf-8,content-length=0)

So its like whenever any additional parameter passed along with the url in case of grpc, it throws this error.

[mostynb]

The .bazelci/basic-auth-tests.sh script that's run in CI does bazel build //:bazel-remote --remote_cache=grpc://$USER:$PASS@localhost:9092, which is similar to your example except it's unencrypted grpc.

I'm not sure what the key/value are in your question.

trailers: Metadata(:status=404,server=awselb/2.0,date=Sat, 03 Sep 2022 16:57:59 GMT,content-type=text/plain; charset=utf-8,content-length=0)

I wonder if the line above means that the load balancer is returning an error, without reaching bazel-remote? Do you get any logs on the bazel-remote side when trying this?

[abcdefstar]

The only logs i could see is 2022/09/06 11:28:55 GET 400 172.24.211.176 /
2022/09/06 11:29:21 GET 400 172.24.211.176 /
2022/09/06 11:29:29 GET 400 172.24.212.124 /

Yes it just says that my LB doesnt have such url and rejecting the request. Do i need to have nginx ingress as well along with my ALB?

[mostynb]

Those bazel-remote logs are from bazel-remote's http server/port, for which / isn't a valid request URL path. If you're trying to use bazel with a grpc or grpcs URL for the remote cache and you see these bazel-remote logs, then it sounds like your load balancer might be forwarding to the wrong bazel-remote port. Or maybe you're specifying the wrong port in the bazel flag?

[abcdefstar]

It was related to my htpasswd issue and once changed it, i dont see any 400 error and i could see bazel logs in my console. And also it doesnt throw me an error when i pass grpcs://user:pwd@host:port --remote-header... But i see this error in my bazel build as
Suppressed: java.io.IOException: Error while uploading artifact with digest '55a767d4532a150fd837f8b390ff2e33b724ccbddf4888236512e5ab3b277922/23345'
at com.google.devtools.build.lib.remote.ByteStreamUploader.lambda$uploadBlobAsync$1(ByteStreamUploader.java:277)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:192)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:179)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:124)
... 56 more
Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED: no authentication metadata found
at io.grpc.Status.asRuntimeException(Status.java:526)
... 10 more

So my auth credentials arent working. Do i need to create a secret\something? I have it as part of my s3 folder.

[mostynb]

What are your bazel and bazel-remote command lines?