Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

springboot 版本升级,启动报错 #1210

Closed
lhzsdnu opened this issue Nov 25, 2023 · 10 comments
Closed

springboot 版本升级,启动报错 #1210

lhzsdnu opened this issue Nov 25, 2023 · 10 comments
Labels
bug Something isn't working

Comments

@lhzsdnu
Copy link

lhzsdnu commented Nov 25, 2023

Describe the bug
springboot 2.7.17 启动正常,springboot 2.7.18 启动报错,错误信息:ERR-1004(API_TIMEOUT): instance register request timeout.

To Reproduce
Steps to reproduce the behavior.

Expected behavior
A clear and concise description of what you expected to happen.

Environment

  • Version: [e.g. v1.0.0]
  • OS: [e.g. CentOS8]

Additional context
Add any other context about the problem here.

@lhzsdnu lhzsdnu added the bug Something isn't working label Nov 25, 2023
@karl-chanel
Copy link

我从3.1.5升到3.1.6也是同样的错误,退回3.1.5就全部OK。

@herodotus-ecosystem
Copy link
Contributor

我也遇到错误,不过和上面两位的不同。

错误日志

com.tencent.polaris.api.exception.RetriableException: ERR-1007(NETWORK_ERROR): failed to load config file. namespace = herodotus, group = sct-quickstart-caller, file = application-default.yml, cause: UNKNOWN
	at com.tencent.polaris.plugins.configuration.connector.polaris.PolarisConfigFileConnector.getConfigFile(PolarisConfigFileConnector.java:85) ~[polaris-configuration-connector-1.14.3.jar:na]
	at com.tencent.polaris.configuration.client.internal.RemoteConfigFileRepo.lambda$doPull$0(RemoteConfigFileRepo.java:126) ~[polaris-configuration-client-1.14.3.jar:na]
	at com.tencent.polaris.api.plugin.filter.ConfigFileFilterChain.execute(ConfigFileFilterChain.java:49) ~[polaris-plugin-api-1.14.3.jar:na]
	at com.tencent.polaris.configuration.client.internal.RemoteConfigFileRepo.doPull(RemoteConfigFileRepo.java:126) ~[polaris-configuration-client-1.14.3.jar:na]
	at com.tencent.polaris.configuration.client.internal.AbstractConfigFileRepo.pull(AbstractConfigFileRepo.java:46) ~[polaris-configuration-client-1.14.3.jar:na]
	at com.tencent.polaris.configuration.client.internal.RemoteConfigFileRepo.<init>(RemoteConfigFileRepo.java:77) ~[polaris-configuration-client-1.14.3.jar:na]
	at com.tencent.polaris.configuration.client.internal.ConfigFileManager.createConfigKVFile(ConfigFileManager.java:147) ~[polaris-configuration-client-1.14.3.jar:na]
	at com.tencent.polaris.configuration.client.internal.ConfigFileManager.getConfigKVFile(ConfigFileManager.java:102) ~[polaris-configuration-client-1.14.3.jar:na]
	at com.tencent.polaris.configuration.client.flow.DefaultConfigFileFlow.getConfigYamlFile(DefaultConfigFileFlow.java:65) ~[polaris-configuration-client-1.14.3.jar:na]
	at com.tencent.polaris.configuration.client.DefaultConfigFileService.getConfigYamlFile(DefaultConfigFileService.java:76) ~[polaris-configuration-client-1.14.3.jar:na]
	at com.tencent.polaris.configuration.client.DefaultConfigFileService.getConfigYamlFile(DefaultConfigFileService.java:70) ~[polaris-configuration-client-1.14.3.jar:na]
	at com.tencent.cloud.polaris.config.adapter.PolarisConfigFileLocator.loadPolarisPropertySource(PolarisConfigFileLocator.java:223) ~[spring-cloud-starter-tencent-polaris-config-1.12.4-2022.0.4.jar:1.12.4-2022.0.4]
	at com.tencent.cloud.polaris.config.adapter.PolarisConfigFileLocator.initInternalConfigFiles(PolarisConfigFileLocator.java:122) ~[spring-cloud-starter-tencent-polaris-config-1.12.4-2022.0.4.jar:1.12.4-2022.0.4]
	at com.tencent.cloud.polaris.config.adapter.PolarisConfigFileLocator.locate(PolarisConfigFileLocator.java:88) ~[spring-cloud-starter-tencent-polaris-config-1.12.4-2022.0.4.jar:1.12.4-2022.0.4]
	at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:50) ~[spring-cloud-context-4.0.4.jar:4.0.4]
	at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:46) ~[spring-cloud-context-4.0.4.jar:4.0.4]
	at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.doInitialize(PropertySourceBootstrapConfiguration.java:121) ~[spring-cloud-context-4.0.4.jar:4.0.4]
	at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:111) ~[spring-cloud-context-4.0.4.jar:4.0.4]
	at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:611) ~[spring-boot-3.1.6.jar:3.1.6]
	at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:391) ~[spring-boot-3.1.6.jar:3.1.6]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-3.1.6.jar:3.1.6]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-3.1.6.jar:3.1.6]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) ~[spring-boot-3.1.6.jar:3.1.6]
	at com.example.SpringCloudTencentTestApplication.main(SpringCloudTencentTestApplication.java:12) ~[classes/:na]
Caused by: io.grpc.StatusRuntimeException: UNKNOWN
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.50.0.jar:1.50.0]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.50.0.jar:1.50.0]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.50.0.jar:1.50.0]
	at com.tencent.polaris.specification.api.v1.config.manage.PolarisConfigGRPCGrpc$PolarisConfigGRPCBlockingStub.getConfigFile(PolarisConfigGRPCGrpc.java:461) ~[polaris-specification-1.4.0.jar:na]
	at com.tencent.polaris.plugins.configuration.connector.polaris.PolarisConfigFileConnector.getConfigFile(PolarisConfigFileConnector.java:75) ~[polaris-configuration-connector-1.14.3.jar:na]
	... 23 common frames omitted
Caused by: java.lang.UnsupportedOperationException: null
	at io.grpc.netty.AbstractHttp2Headers.isEmpty(AbstractHttp2Headers.java:40) ~[grpc-netty-1.50.0.jar:1.50.0]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:419) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:352) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:188) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) ~[netty-codec-http2-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]

错误分析

最初还是以为是自己的代码没有更新及时,或者哪里配置不对,还参照官网文档,一步一步重新搭建了一个 spring boot 工程验证。这里面有一个需要注意的点。

最初我是各个包独立依赖,如下面所示:

     <dependency>
            <groupId>com.tencent.cloud</groupId>
            <artifactId>spring-cloud-starter-tencent-polaris-discovery</artifactId>
        </dependency>
        <dependency>
            <groupId>com.tencent.cloud</groupId>
            <artifactId>spring-cloud-starter-tencent-polaris-config</artifactId>
        </dependency>
        <dependency>
            <groupId>com.tencent.cloud</groupId>
            <artifactId>spring-cloud-starter-tencent-polaris-circuitbreaker</artifactId>
        </dependency>
        <dependency>
            <groupId>com.tencent.cloud</groupId>
            <artifactId>spring-cloud-starter-tencent-polaris-ratelimit</artifactId>
        </dependency>
        <dependency>
            <groupId>com.tencent.cloud</groupId>
            <artifactId>spring-cloud-starter-tencent-metadata-transfer</artifactId>
        </dependency>

使用上面的依赖,只要 spring boot 从 3.1.5 改到 3.1.6 就会抛错。

但是官网文档中,建议依赖 spring-cloud-starter-tencent-all。所以在验证的时候,又尝试改用依赖 spring-cloud-starter-tencent-all

经过测试发现使用 spring-cloud-starter-tencent-all 时,即使 spring boot 从 3.1.5 改到 3.1.6 也不会出错。因为 spring-cloud-starter-tencent-all 是 uber.jar,推测是部分 jar 被打入了 spring-cloud-starter-tencent-all ,即使更新 spring boot 版本,其中的依赖的部分 jar 并没有变化导致。

临时解决

按照上面的思路进行了部分跟踪,目前发现,升级 spring boot 版本会导致依赖的 netty 版本,从 4.1.100.Final 升级到 4.1.101.Final。这就导致了出错。

奈何能力有限,具体原因还不清楚,更不知道怎么改。目前能想到的临时解决办法:

  1. 将 spring cloud tencent 依赖修改为统一使用 依赖 spring-cloud-starter-tencent-all
  2. 手动降级 netty 版本,降到 4.1.100.Final 版本。

具体怎么解决就得麻烦项目大佬了。

@SkyeBeFreeman
Copy link
Collaborator

目前看是netty版本的冲突,可以现在polaris-java项目尝试升级,不过不确定这个netty版本能否兼容旧版本。

@herodotus-ecosystem
Copy link
Contributor

目前看是netty版本的冲突,可以现在polaris-java项目尝试升级,不过不确定这个netty版本能否兼容旧版本。

是的。核心是 grpc 依赖了 netty。升级了 Spring Boot 版本,就将netty 的版本同步给升。之前初步跟了一下代码,是netty的一个类在新版有了一些变化,拿不到 content-type header 出现抛错。后面我也再详细研究研究。

@SkyeBeFreeman
Copy link
Collaborator

@herodotus-cloud 看了一下,是netty-codec-http2的4.1.101.Final版本中的
io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.FrameReadListener#onHeadersRead(io.netty.channel.ChannelHandlerContext, int, io.netty.handler.codec.http2.Http2Headers, int, short, boolean, int, boolean)
方法多了一个下面这个代码的处理,而headers.isEmpty抛出了UnsupportedOperationException的异常。

     } else if (validateHeaders && !headers.isEmpty()) {
                // Need to check trailers don't contain pseudo headers. According to RFC 9113
                // Trailers MUST NOT include pseudo-header fields (Section 8.3).
                for (Iterator<Entry<CharSequence, CharSequence>> iterator =
                    headers.iterator(); iterator.hasNext();) {
                    CharSequence name = iterator.next().getKey();
                    if (Http2Headers.PseudoHeaderName.hasPseudoHeaderFormat(name)) {
                        throw streamError(stream.id(), PROTOCOL_ERROR,
                                "Found invalid Pseudo-Header in trailers: %s", name);
                    }
                }
            }

详情可以查看polaris-java的日志,
image

@herodotus-ecosystem
Copy link
Contributor

@herodotus-cloud 看了一下,是netty-codec-http2的4.1.101.Final版本中的 io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.FrameReadListener#onHeadersRead(io.netty.channel.ChannelHandlerContext, int, io.netty.handler.codec.http2.Http2Headers, int, short, boolean, int, boolean) 方法多了一个下面这个代码的处理,而headers.isEmpty抛出了UnsupportedOperationException的异常。

     } else if (validateHeaders && !headers.isEmpty()) {
                // Need to check trailers don't contain pseudo headers. According to RFC 9113
                // Trailers MUST NOT include pseudo-header fields (Section 8.3).
                for (Iterator<Entry<CharSequence, CharSequence>> iterator =
                    headers.iterator(); iterator.hasNext();) {
                    CharSequence name = iterator.next().getKey();
                    if (Http2Headers.PseudoHeaderName.hasPseudoHeaderFormat(name)) {
                        throw streamError(stream.id(), PROTOCOL_ERROR,
                                "Found invalid Pseudo-Header in trailers: %s", name);
                    }
                }
            }

详情可以查看polaris-java的日志, image

对的,就是这里。4.1.100.Final 是没有 else if 这段代码的。

@herodotus-ecosystem
Copy link
Contributor

herodotus-ecosystem commented Nov 27, 2023

@herodotus-cloud 看了一下,是netty-codec-http2的4.1.101.Final版本中的 io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.FrameReadListener#onHeadersRead(io.netty.channel.ChannelHandlerContext, int, io.netty.handler.codec.http2.Http2Headers, int, short, boolean, int, boolean) 方法多了一个下面这个代码的处理,而headers.isEmpty抛出了UnsupportedOperationException的异常。

     } else if (validateHeaders && !headers.isEmpty()) {
                // Need to check trailers don't contain pseudo headers. According to RFC 9113
                // Trailers MUST NOT include pseudo-header fields (Section 8.3).
                for (Iterator<Entry<CharSequence, CharSequence>> iterator =
                    headers.iterator(); iterator.hasNext();) {
                    CharSequence name = iterator.next().getKey();
                    if (Http2Headers.PseudoHeaderName.hasPseudoHeaderFormat(name)) {
                        throw streamError(stream.id(), PROTOCOL_ERROR,
                                "Found invalid Pseudo-Header in trailers: %s", name);
                    }
                }
            }

详情可以查看polaris-java的日志, image

应该是 Netty 为了解决这个问题 grpc/grpc-java#13647,所以在 Netty 4.1.101 Final 中增加了 else if 这个判断。

而 grpc 中,目前是不支持 isEmpty 这个操作的。

image

image

image

@SkyeBeFreeman
Copy link
Collaborator

看了一下grpc-java的社区,这个问题已经修复了,后面就等发布相应的更新。grpc/grpc-java#10663

@herodotus-ecosystem
Copy link
Contributor

看了一下grpc-java的社区,这个问题已经修复了,后面就等发布相应的更新。grpc/grpc-java#10663

哦,好的。感谢!辛苦啦。

@SkyeBeFreeman
Copy link
Collaborator

grpc-java 1.59.1已发布,使用新版本spring boot时,可以指定grpc-java的版本到最新的,https://github.com/grpc/grpc-java/releases/tag/v1.59.1

<dependency>
			<groupId>io.grpc</groupId>
			<artifactId>grpc-all</artifactId>
			<version>1.59.1</version>
			<exclusions>
				<exclusion>
					<groupId>io.grpc</groupId>
					<artifactId>grpc-xds</artifactId>
				</exclusion>
			</exclusions>
		</dependency>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants