Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Externalize compilation cache by compilers #747

Merged
merged 18 commits into from Aug 18, 2022
Merged

Externalize compilation cache by compilers #747

merged 18 commits into from Aug 18, 2022

Conversation

mathetake
Copy link
Member

@mathetake mathetake commented Aug 17, 2022

This adds the experimental support of the file system compilation cache.
Notably, experimental.WithCompilationCacheDirName allows users to configure
where the compiler writes the cache into.

Versioning/validation of binary compatibility has been done via the release tag
(which will be created from the end of this month). More specifically, the cache
file starts with header with the hardcoded wazero version.

Note the cache implementation doesn't do any security mitigation like signing/verification,
and that would be a future TODO once we reach some stability.

The end-to-end test will be added in a subsequent PR.

Fixes #618

Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
@@ -50,11 +52,39 @@ func BenchmarkInitialization(b *testing.B) {
}
}

func runInitializationBench(b *testing.B, r wazero.Runtime) {
func BenchmarkCompilation(b *testing.B) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Result:

BenchmarkCompilation
BenchmarkCompilation/with_extern_cache
BenchmarkCompilation/with_extern_cache-10         	    2008	    526860 ns/op
BenchmarkCompilation/without_extern_cache
BenchmarkCompilation/without_extern_cache-10      	     165	   7080493 ns/op

Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
@mathetake mathetake marked this pull request as ready for review August 18, 2022 07:02
Copy link
Collaborator

@codefromthecrypt codefromthecrypt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unleashing comments to allow you to drift things while I read the rest

experimental/compilation_cache.go Outdated Show resolved Hide resolved
experimental/compilation_cache.go Outdated Show resolved Hide resolved
internal/compilationcache/compilationcache.go Show resolved Hide resolved
internal/compilationcache/compilationcache.go Show resolved Hide resolved
internal/engine/compiler/engine_cache.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@codefromthecrypt codefromthecrypt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more comments

internal/compilationcache/compilationcache.go Outdated Show resolved Hide resolved
internal/compilationcache/compilationcache.go Outdated Show resolved Hide resolved
internal/compilationcache/compilationcache.go Outdated Show resolved Hide resolved
internal/compilationcache/compilationcache.go Outdated Show resolved Hide resolved
internal/compilationcache/compilationcache.go Outdated Show resolved Hide resolved
mathetake and others added 3 commits August 18, 2022 17:57
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Co-authored-by: Crypt Keeper <64215+codefromthecrypt@users.noreply.github.com>
Copy link
Collaborator

@codefromthecrypt codefromthecrypt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think exposing this as an experimental hook simply with a dir is a great way to start. There's some more polishing to do on internal code, but this can allow users to start giving feedback meanwhile.

internal/compilationcache/file_cache.go Outdated Show resolved Hide resolved
internal/compilationcache/file_cache.go Outdated Show resolved Hide resolved
internal/compilationcache/file_cache.go Outdated Show resolved Hide resolved
mathetake and others added 7 commits August 18, 2022 18:29
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Co-authored-by: Crypt Keeper <64215+codefromthecrypt@users.noreply.github.com>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Takeshi Yoneda <takeshi@tetrate.io>
@mathetake mathetake merged commit 3b32c20 into main Aug 18, 2022
@mathetake mathetake deleted the externcache branch August 18, 2022 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Serialize a compiled module
2 participants