Sidecar Flag: Random delay before upload blocks on sidecar #7239

Vanshikav123 · 2024-03-26T16:31:44Z

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

#7184
Implemented the logic to sleep for a random time between 0 and the specified jitter before uploading blocks.Defined a flag to specify the maximum random delay before uploading blocks.

Verification

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

Vanshikav123 · 2024-03-26T17:07:54Z

Hello ! @yeya24 PTAL

yeya24 · 2024-03-27T05:43:40Z

cmd/thanos/sidecar.go

@@ -506,7 +510,30 @@ type sidecarConfig struct {
 	storeRateLimits store.SeriesSelectLimits
 }

+type durationValue time.Duration


We should be able to set duration via flag directly without implementing interface.

https://github.com/thanos-io/thanos/blob/main/cmd/thanos/receive.go#L968

yeya24 · 2024-03-27T05:49:18Z

cmd/thanos/sidecar.go

@@ -363,6 +364,9 @@ func runSidecar(
 				uploadCompactedFunc, conf.shipper.allowOutOfOrderUpload, metadata.HashFunc(conf.shipper.hashFunc), conf.shipper.metaFileName)

 			return runutil.Repeat(30*time.Second, ctx.Done(), func() error {
+				// Generate random delay using upload jitter.
+				jitter := time.Duration(rand.Int63n(int64(conf.shipper.uploadJitter)))
+				time.Sleep(jitter)


Instead of sleeping, can we do something similar to this https://github.com/cortexproject/cortex/blob/master/pkg/util/time.go#L73?

I image we will add a new function called runutil.RepeatWithJitter which takes the same parameters as runutil.Repeat but also takes variancePerc.
We can just hardcode variancePerc to a sane value like 0.05 so allows [-1.5s, 1.5s] jitter.

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

…thanos into patch-sidecar_flag

fpetkovski

Thanks for this contribution. I am curious what is the benefit to adding jitter to the upload.

fpetkovski · 2024-04-04T07:00:26Z

cmd/thanos/sidecar.go

@@ -105,6 +106,19 @@ func registerSidecar(app *extkingpin.App) {
 	})
 }

+// DurationWithJitter returns random duration from "input - input*variancePerc" to "input + input*variancePerc" interval.
+func DurationWithJitter(input time.Duration, variancePerc float64) time.Duration {
+


Suggested change

fpetkovski · 2024-04-04T07:01:45Z

cmd/thanos/sidecar.go

@@ -518,4 +538,7 @@ func (sc *sidecarConfig) registerFlag(cmd extkingpin.FlagClause) {
 	sc.storeRateLimits.RegisterFlags(cmd)
 	cmd.Flag("min-time", "Start of time range limit to serve. Thanos sidecar will serve only metrics, which happened later than this value. Option can be a constant time in RFC3339 format or time duration relative to current time, such as -1d or 2h45m. Valid duration units are ms, s, m, h, d, w, y.").
 		Default("0000-01-01T00:00:00Z").SetValue(&sc.limitMinTime)
+	conf := &sidecarConfig{}
+	cmd.Flag("upload-jitter", "Maximum random delay before uploading blocks.").Default("0s").DurationVar(&uploadJitter)


Could we bind the flag directly to conf.shipper.uploadJitter?

yess i will do it

fpetkovski · 2024-04-04T07:02:12Z

pkg/runutil/runutil.go

+		case <-ctx.Done():
+			return nil
+		default:
+


Suggested change

Vanshikav123 · 2024-04-04T07:13:23Z

Thanks for this contribution. I am curious what is the benefit to adding jitter to the upload.

adding jitter prevents network blockage as it introduces random delays to the uploads

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

Vanshikav123 · 2024-04-04T12:23:23Z

Hello! @fpetkovski @yeya24 resolved the reviews PTAL!

yeya24 · 2024-04-04T14:53:48Z

cmd/thanos/sidecar.go

+				jitter := DurationWithJitter(conf.shipper.uploadJitter, 0.05)
+				if jitter > 0 {
+					time.Sleep(jitter)
+				}


Why we need to add another jitter here? Isn't RepeatWithJitter enough

should i also remove DurationWithJitter function or keep it for future use ?

Let's remove that

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

yeya24 · 2024-04-05T18:17:48Z

cmd/thanos/sidecar.go

@@ -362,7 +375,7 @@ func runSidecar(
 			s := shipper.New(logger, reg, conf.tsdb.path, bkt, m.Labels, metadata.SidecarSource,
 				uploadCompactedFunc, conf.shipper.allowOutOfOrderUpload, metadata.HashFunc(conf.shipper.hashFunc), conf.shipper.metaFileName)

-			return runutil.Repeat(30*time.Second, ctx.Done(), func() error {
+			return runutil.RepeatWithJitter(30*time.Second, ctx, 0.05, func() error {


0.05 variance seems too small as it is only 30s * 0.05 = 1.5s?
I am thinking about 0.2 here maybe

Resolved all !

yeya24 · 2024-04-05T18:18:11Z

cmd/thanos/sidecar.go

@@ -518,4 +531,6 @@ func (sc *sidecarConfig) registerFlag(cmd extkingpin.FlagClause) {
 	sc.storeRateLimits.RegisterFlags(cmd)
 	cmd.Flag("min-time", "Start of time range limit to serve. Thanos sidecar will serve only metrics, which happened later than this value. Option can be a constant time in RFC3339 format or time duration relative to current time, such as -1d or 2h45m. Valid duration units are ms, s, m, h, d, w, y.").
 		Default("0000-01-01T00:00:00Z").SetValue(&sc.limitMinTime)
+	conf := &sidecarConfig{}
+	cmd.Flag("upload-jitter", "Maximum random delay before uploading blocks.").Default("0s").DurationVar(&conf.shipper.uploadJitter)


Let's remove this if it is not used anymore

yeya24 · 2024-04-05T18:18:25Z

cmd/thanos/config.go

@@ -164,6 +164,7 @@ type shipperConfig struct {
 	allowOutOfOrderUpload bool
 	hashFunc              string
 	metaFileName          string
+	uploadJitter          time.Duration


Remove this if not needed

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

yeya24 · 2024-04-05T21:47:26Z

docs/components/sidecar.md

@@ -228,6 +228,7 @@ Flags:
                                 configuration. See format details:
                                 https://thanos.io/tip/thanos/tracing.md/#configuration
      --tsdb.path="./data"       Data directory of TSDB.
+      --upload-jitter=0s         Maximum random delay before uploading blocks.


Need to fix docs

Also it looks like this flag is never consumed

Also added nowhere

yess will remove it

MichaHoffmann · 2024-04-06T09:26:58Z

cmd/thanos/sidecar.go

@@ -362,7 +362,7 @@ func runSidecar(
 			s := shipper.New(logger, reg, conf.tsdb.path, bkt, m.Labels, metadata.SidecarSource,
 				uploadCompactedFunc, conf.shipper.allowOutOfOrderUpload, metadata.HashFunc(conf.shipper.hashFunc), conf.shipper.metaFileName)

-			return runutil.Repeat(30*time.Second, ctx.Done(), func() error {
+			return runutil.RepeatWithJitter(30*time.Second, ctx, 0.2, func() error {


Does this really help though if we are jittering only 6s; at least for us block uploads can take far longer

We can increase the jitter. What value do you think that makes sense?

I don't know, the whole method feels kinda weird. It was about in house blob storage and blocks being uploaded at same time right?

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

hanjm · 2024-04-08T10:36:04Z

pkg/runutil/runutil.go

+// RepeatWithJitter executes f with a random jitter added to the interval between each execution.
+// It continues until ctx is done or f returns an error.
+// The jitter factor should be between 0 and 1, where 0 means no jitter and 1 means the interval can vary from 0 to 2 times the original interval.
+func RepeatWithJitter(interval time.Duration, ctx context.Context, jitterFactor float64, f func() error) error {


The ctx should be always first parameter. https://pkg.go.dev/context#pkg-overview

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

random delay

fd63531

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

pull-request-size bot added the size/M label Mar 26, 2024

yeya24 reviewed Mar 27, 2024

View reviewed changes

Vanshikav123 and others added 4 commits April 2, 2024 22:39

Merge branch 'thanos-io:main' into patch-sidecar_flag

6d18845

Merge branch 'thanos-io:main' into patch-sidecar_flag

c9c9076

added new jitter function

e09f3f8

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

Merge branch 'patch-sidecar_flag' of https://github.com/Vanshikav123/…

15eceec

…thanos into patch-sidecar_flag

fpetkovski reviewed Apr 4, 2024

View reviewed changes

resolved review changes

cced736

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

yeya24 reviewed Apr 4, 2024

View reviewed changes

Vanshikav123 and others added 2 commits April 5, 2024 22:30

Merge branch 'thanos-io:main' into patch-sidecar_flag

cfca6cb

remove unnecessary jitter

24eafbe

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

yeya24 reviewed Apr 5, 2024

View reviewed changes

final changes

3566f56

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

yeya24 reviewed Apr 5, 2024

View reviewed changes

MichaHoffmann reviewed Apr 6, 2024

View reviewed changes

doc changes

ab01ce9

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

hanjm reviewed Apr 8, 2024

View reviewed changes

Vanshikav123 and others added 2 commits April 8, 2024 16:18

context position

da63e08

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>

Merge branch 'thanos-io:main' into patch-sidecar_flag

386dfee

Vanshikav123 requested review from fpetkovski and yeya24 May 8, 2024 18:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sidecar Flag: Random delay before upload blocks on sidecar #7239

Sidecar Flag: Random delay before upload blocks on sidecar #7239

Vanshikav123 commented Mar 26, 2024 •

edited

Vanshikav123 commented Mar 26, 2024

yeya24 Mar 27, 2024

yeya24 Mar 27, 2024

fpetkovski left a comment

fpetkovski Apr 4, 2024

fpetkovski Apr 4, 2024

Vanshikav123 Apr 4, 2024

fpetkovski Apr 4, 2024

Vanshikav123 commented Apr 4, 2024

Vanshikav123 commented Apr 4, 2024

yeya24 Apr 4, 2024

Vanshikav123 Apr 5, 2024

yeya24 Apr 5, 2024

yeya24 Apr 5, 2024

Vanshikav123 Apr 5, 2024

yeya24 Apr 5, 2024

yeya24 Apr 5, 2024

yeya24 Apr 5, 2024

MichaHoffmann Apr 6, 2024

MichaHoffmann Apr 6, 2024

Vanshikav123 Apr 8, 2024

MichaHoffmann Apr 6, 2024

yeya24 Apr 6, 2024

MichaHoffmann Apr 6, 2024

hanjm Apr 8, 2024

Sidecar Flag: Random delay before upload blocks on sidecar #7239

Are you sure you want to change the base?

Sidecar Flag: Random delay before upload blocks on sidecar #7239

Conversation

Vanshikav123 commented Mar 26, 2024 • edited

Changes

Verification

Vanshikav123 commented Mar 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fpetkovski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vanshikav123 commented Apr 4, 2024

Vanshikav123 commented Apr 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vanshikav123 commented Mar 26, 2024 •

edited