feat: support grpc stream for data plane #352

hwjiangkai · 2022-12-17T08:12:34Z

What problem does this PR solve?

transport layer performance optimization

Issue Number: close #xxx

Problem Summary

What is changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

codecov · 2022-12-17T13:11:39Z

Codecov Report

Merging #352 (73b43c0) into main (09f3748) will decrease coverage by 3.70%.
The diff coverage is 51.84%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #352      +/-   ##
==========================================
- Coverage   62.72%   59.02%   -3.71%     
==========================================
  Files         133      172      +39     
  Lines       12128    14220    +2092     
==========================================
+ Hits         7607     8393     +786     
- Misses       3977     5225    +1248     
- Partials      544      602      +58

Impacted Files	Coverage Δ
internal/controller/eventbus/server/instance.go	`72.04% <0.00%> (ø)`
internal/controller/trigger/storage/secret.go	`48.48% <0.00%> (ø)`
...nternal/controller/trigger/storage/subscription.go	`54.71% <0.00%> (ø)`
...ernal/controller/trigger/storage/trigger_worker.go	`63.41% <0.00%> (ø)`
...rnal/primitive/interceptor/errinterceptor/error.go	`0.00% <0.00%> (ø)`
.../primitive/interceptor/memberinterceptor/member.go	`0.00% <0.00%> (ø)`
internal/primitive/vanus/id.go	`19.68% <0.00%> (-0.65%)`	⬇️
internal/raft/log/recovery.go	`52.27% <ø> (ø)`
internal/raft/log/wal.go	`100.00% <ø> (ø)`
internal/store/block/entry.go	`70.96% <ø> (ø)`
... and 164 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b23e5a...73b43c0. Read the comment docs.

client/internal/vanus/store/block_store.go

internal/store/segment/api.go

proto/proto/segment.proto

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

internal/store/segment/api.go

ifplusor

don't serialize on a stream.

…m-v2 Signed-off-by: jyjiangkai <jyjiangkai@163.com>

…m-v2

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

internal/store/block/raft/appender.go

internal/store/block/block.go

internal/store/segment/api.go

internal/store/segment/server.go

internal/timer/timingwheel/bucket.go

pkg/errors/errors.go

client/pkg/eventbus/eventbus.go

client/internal/vanus/store/block_store.go

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

proto/proto/segment.proto

pkg/errors/app.go

client/internal/vanus/store/block_store.go

ifplusor · 2022-12-23T07:39:57Z

client/internal/vanus/store/block_store.go

+		if stderr.Is(err, io.EOF) {
+			s.appendStream.CloseSend()
+			s.appendStream = nil
+			c, _ := s.appendCallbacks.LoadAndDelete(opaqueID)


delete for all error

client/internal/vanus/store/block_store.go

internal/controller/trigger/controller.go

internal/store/block/raft/appender.go

internal/store/segment/api.go

internal/store/segment/server.go

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

…m-v2 Signed-off-by: jyjiangkai <jyjiangkai@163.com>

client/internal/vanus/store/block_store.go

internal/store/segment/api.go

client/pkg/eventlog/log_segment.go

…m-v2 Signed-off-by: jyjiangkai <jyjiangkai@163.com>

client/internal/vanus/store/block_store.go

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

internal/store/segment/server.go

internal/store/segment/api.go

ifplusor · 2022-12-30T09:30:49Z

internal/store/segment/api.go

+			errCode = errpb.ErrorCode_UNKNOWN
+			errMsg = "unknown"


remains detail of ErrorType.

client/internal/vanus/store/block_store.go

internal/store/segment/api.go

ifplusor · 2023-01-10T00:44:46Z

internal/store/segment/api.go

+						errCode = errpb.ErrorCode_APPEND_TO_BLOCK_FAILED
+						errMsg = "append to block failed"
+					}
+					log.Error(ctx, "append to block failed", map[string]interface{}{


use Warning, not Error

ifplusor · 2023-01-10T00:48:48Z

internal/store/segment/api.go

+			request, err := stream.Recv()
+			if err != nil {
+				if stderr.Is(err, io.EOF) {
+					log.Warning(ctx, "append stream closed", map[string]interface{}{


EOF is ok, don't use Warning and Error

client/pkg/eventlog/eventlog.go

client/internal/vanus/store/block_store.go

client/pkg/eventlog/eventlog_impl.go

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai added 2 commits December 15, 2022 18:54

feat: transport layer protocol optimization

fd98fab

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

feat: transport layer performance optimization

6e9856b

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai requested review from ifplusor and wenfengwang as code owners December 17, 2022 08:12

hwjiangkai marked this pull request as draft December 17, 2022 08:12

hwjiangkai force-pushed the clientstream-v2 branch from f285759 to 64ce0a2 Compare December 17, 2022 13:05

feat: transport layer performance optimization

72be0cc

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai force-pushed the clientstream-v2 branch from 64ce0a2 to 72be0cc Compare December 17, 2022 13:08

hwjiangkai marked this pull request as ready for review December 17, 2022 13:12

wenfengwang suggested changes Dec 19, 2022

View reviewed changes

feat: fix review comments

023a19f

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai requested a review from xdlbdy as a code owner December 19, 2022 16:26

ifplusor reviewed Dec 20, 2022

View reviewed changes

internal/store/segment/api.go Outdated Show resolved Hide resolved

ifplusor requested changes Dec 20, 2022

View reviewed changes

hwjiangkai marked this pull request as draft December 20, 2022 03:32

Merge branch 'main' of github.com:linkall-labs/vanus into clientstrea…

804f744

…m-v2 Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai force-pushed the clientstream-v2 branch from ed2616f to 804f744 Compare December 20, 2022 09:39

hwjiangkai added 3 commits December 20, 2022 17:42

Merge branch 'main' of github.com:linkall-labs/vanus into clientstrea…

c59a3ee

…m-v2

feat: async append for store

c384253

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

feat: async append for store

394164f

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai marked this pull request as ready for review December 21, 2022 06:10

ifplusor changed the title ~~feat: transport layer performance optimization~~ feat: support grpc stream for data plane Dec 22, 2022

ifplusor requested changes Dec 22, 2022

View reviewed changes

feat: fix review comments

2278c46

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai force-pushed the clientstream-v2 branch from 734120b to 2278c46 Compare December 22, 2022 08:40

ifplusor requested changes Dec 23, 2022

View reviewed changes

hwjiangkai force-pushed the clientstream-v2 branch from 16661cb to b704d31 Compare December 23, 2022 07:17

ifplusor reviewed Dec 23, 2022

View reviewed changes

client/internal/vanus/store/block_store.go Outdated Show resolved Hide resolved

hwjiangkai force-pushed the clientstream-v2 branch from b704d31 to 6843243 Compare December 23, 2022 08:51

ifplusor requested changes Dec 23, 2022

View reviewed changes

hwjiangkai force-pushed the clientstream-v2 branch from 6843243 to 101e46b Compare December 23, 2022 18:11

revert tryagain error

83363a1

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai force-pushed the clientstream-v2 branch from 101e46b to 83363a1 Compare December 24, 2022 19:09

Merge branch 'main' of github.com:linkall-labs/vanus into clientstrea…

39a3140

…m-v2 Signed-off-by: jyjiangkai <jyjiangkai@163.com>

ifplusor reviewed Dec 28, 2022

View reviewed changes

client/internal/vanus/store/block_store.go Outdated Show resolved Hide resolved

ifplusor reviewed Dec 28, 2022

View reviewed changes

client/internal/vanus/store/block_store.go Outdated Show resolved Hide resolved

ifplusor reviewed Dec 28, 2022

View reviewed changes

client/internal/vanus/store/block_store.go Outdated Show resolved Hide resolved

ifplusor reviewed Dec 28, 2022

View reviewed changes

client/internal/vanus/store/block_store.go Outdated Show resolved Hide resolved

ifplusor requested changes Dec 28, 2022

View reviewed changes

hwjiangkai force-pushed the clientstream-v2 branch 2 times, most recently from c527fad to e3f573e Compare December 28, 2022 12:43

Merge branch 'main' of github.com:linkall-labs/vanus into clientstrea…

98dc0f9

…m-v2 Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai force-pushed the clientstream-v2 branch from e3f573e to 98dc0f9 Compare December 30, 2022 03:42

ifplusor requested changes Dec 30, 2022

View reviewed changes

fix review comments

86a6f0e

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai force-pushed the clientstream-v2 branch from 6ce31f8 to 86a6f0e Compare December 30, 2022 08:06

hwjiangkai force-pushed the clientstream-v2 branch 2 times, most recently from 1460868 to 1f7ad23 Compare January 6, 2023 09:46

Merge branch 'main' into clientstream-v2

b4423f7

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai force-pushed the clientstream-v2 branch from 1f7ad23 to b4423f7 Compare January 6, 2023 09:58

ifplusor requested changes Jan 9, 2023

View reviewed changes

hwjiangkai force-pushed the clientstream-v2 branch from a46f7e6 to 905abf6 Compare January 9, 2023 12:40

ifplusor requested changes Jan 10, 2023

View reviewed changes

hwjiangkai force-pushed the clientstream-v2 branch from 905abf6 to 1913655 Compare January 10, 2023 03:24

fix review comments

73b43c0

Signed-off-by: jyjiangkai <jyjiangkai@163.com>

hwjiangkai force-pushed the clientstream-v2 branch from 1913655 to 73b43c0 Compare January 10, 2023 11:38

wenfengwang assigned ifplusor Jan 30, 2023

ifplusor marked this pull request as draft February 14, 2023 02:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support grpc stream for data plane #352

feat: support grpc stream for data plane #352

hwjiangkai commented Dec 17, 2022

codecov bot commented Dec 17, 2022 •

edited

ifplusor left a comment

ifplusor Dec 23, 2022

ifplusor Dec 30, 2022

ifplusor Jan 10, 2023

ifplusor Jan 10, 2023

feat: support grpc stream for data plane #352

Are you sure you want to change the base?

feat: support grpc stream for data plane #352

Conversation

hwjiangkai commented Dec 17, 2022

What problem does this PR solve?

Problem Summary

What is changed and how does it work?

Check List

Tests

codecov bot commented Dec 17, 2022 • edited

Codecov Report

ifplusor left a comment

Choose a reason for hiding this comment

ifplusor Dec 23, 2022

Choose a reason for hiding this comment

ifplusor Dec 30, 2022

Choose a reason for hiding this comment

ifplusor Jan 10, 2023

Choose a reason for hiding this comment

ifplusor Jan 10, 2023

Choose a reason for hiding this comment

codecov bot commented Dec 17, 2022 •

edited