Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch to base64 crate #1294

Closed
wants to merge 4 commits into from
Closed

Switch to base64 crate #1294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
66766c3
Switch to `base64` crate
david-perez Mar 28, 2022
bd04a0f
Update changelog
david-perez Apr 1, 2022
a5e5373
Merge remote-tracking branch 'awslabs/main' into davidpz-base64
david-perez Apr 11, 2022
f1c00cb
Add changelog
david-perez Apr 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.next.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@
# meta = { "breaking" = false, "tada" = false, "bug" = false }
# author = "rcoh"

[[smithy-rs]]
message = "Switch to `base64` crate"
references = ["smithy-rs#1294"]
meta = { "breaking" = true, "tada" = false, "bug" = false }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do I add this message?

The base64 implementation from aws-smithy-types has been removed. Please switch to another implementation (e.g. the base64 crate).

I wanted to add it to aws/SDK_CHANGELOG.md, but that file is autogenerated from update-changelog from tools/sdk-lints. How are the migration guides from that file added if the file is autogenerated? Do they get added manually just before preparing a release?

Copy link
Collaborator

@rcoh rcoh Apr 1, 2022 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an example above—[[aws-sdk-rust]] instead of smithy-rs.

Anything you want in the message goes in the message field—TOML allows multiline strings. However: I think we should keep the same function signatures and just delegate to the base64 crate behind the scenes. Having a "blessed" base64 implementation inside of aws-smithy-types is probably generally a good thing to have.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an example above—[[aws-sdk-rust]] instead of smithy-rs.

But how do I add a longer migration guide e.g. https://github.com/awslabs/smithy-rs/blob/775d227c5fbc46c2e3c2878b270b8e8226ab7ef1/aws/SDK_CHANGELOG.md?plain=1#L5-L33 ? Do I plop all of that Markdown into message?

However: I think we should keep the same function signatures and just delegate to the base64 crate behind the scenes. Having a "blessed" base64 implementation inside of aws-smithy-types is probably generally a good thing to have.

I thought a bit about this before making the change and concluded that there's no benefit. base64 is standard and I can't foresee us or Smithy wanting to customize anything that is not already customizable via the base64 crate. I also don't think that SDK users should have public access to a base64-encoding routine from a runtime crate; they should have no need for it after all, since they should just be using aws_smithy_types::Blob. I think having a "blessed" base64 implementation is unnecessarily increasing public API surface.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe blob should add a from_base64 method...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, just use triple quote strings for long messages

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've added it here f1c00cb

author = "david-perez"

[[smithy-rs]]
message = "Fix link to Developer Guide in crate's README.md"
references = ["smithy-rs#1262"]
Expand Down Expand Up @@ -58,3 +64,9 @@ message = "Update urlencoding crate to v2.1.0"
references = ["smithy-rs#1301"]
meta = { "breaking" = false, "tada" = false, "bug" = false }
author = "benesch"

[[smithy-rs]]
message = "The base64 implementation from `aws-smithy-types` has been removed. Please switch to another implementation (e.g. the [base64](https://crates.io/crates/base64) crate)."
references = ["smithy-rs#1294"]
meta = { "breaking" = true, "tada" = false, "bug" = false }
author = "david-perez"
3 changes: 2 additions & 1 deletion ...azon/smithy/rust/codegen/server/smithy/generators/protocol/ServerProtocolTestGenerator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -663,7 +663,7 @@ class ServerProtocolTestGenerator(
FailingTest(RestJson, "RestJsonWithBodyExpectsApplicationJsonAccept", TestType.MalformedRequest),
FailingTest(RestJson, "RestJsonWithPayloadExpectsImpliedAccept", TestType.MalformedRequest),
FailingTest(RestJson, "RestJsonWithPayloadExpectsModeledAccept", TestType.MalformedRequest),
FailingTest(RestJson, "RestJsonBodyMalformedBlobInvalidBase64_case1", TestType.MalformedRequest),
// TODO(https://github.com/awslabs/smithy/pull/1168)
FailingTest(RestJson, "RestJsonBodyMalformedBlobInvalidBase64_case2", TestType.MalformedRequest),
FailingTest(RestJson, "RestJsonBodyByteMalformedValueRejected_case2", TestType.MalformedRequest),
FailingTest(RestJson, "RestJsonBodyByteMalformedValueRejected_case6", TestType.MalformedRequest),
Expand Down Expand Up @@ -700,6 +700,7 @@ class ServerProtocolTestGenerator(
FailingTest(RestJson, "RestJsonBodyShortUnderflowOverflow_case2", TestType.MalformedRequest),
FailingTest(RestJson, "RestJsonBodyShortUnderflowOverflow_case3", TestType.MalformedRequest),
FailingTest(RestJson, "RestJsonHeaderMalformedStringInvalidBase64MediaType_case1", TestType.MalformedRequest),
// TODO(https://github.com/awslabs/smithy/pull/1168)
FailingTest(RestJson, "RestJsonBodyTimestampDateTimeRejectsUTCOffsets_case0", TestType.MalformedRequest),
FailingTest(RestJson, "RestJsonBodyTimestampDefaultRejectsMalformedEpochSeconds_case5", TestType.MalformedRequest),
FailingTest(RestJson, "RestJsonBodyTimestampDefaultRejectsMalformedEpochSeconds_case7", TestType.MalformedRequest),
Expand Down
1 change: 1 addition & 0 deletions codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/rustlang/CargoDependency.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ data class CargoDependency(
}

companion object {
val Base64: CargoDependency = CargoDependency("base64", CratesIo("0.13"))
val Bytes: CargoDependency = CargoDependency("bytes", CratesIo("1"))
val BytesUtils: CargoDependency = CargoDependency("bytes-utils", CratesIo("0.1.1"))
val FastRand: CargoDependency = CargoDependency("fastrand", CratesIo("1"))
Expand Down
15 changes: 2 additions & 13 deletions codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/smithy/RuntimeTypes.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,19 +193,8 @@ data class RuntimeType(val name: String?, val dependency: RustDependency?, val n
fun QueryFormat(runtimeConfig: RuntimeConfig, func: String) =
RuntimeType(func, CargoDependency.SmithyHttp(runtimeConfig), "${runtimeConfig.crateSrcPrefix}_http::query")

fun Base64Encode(runtimeConfig: RuntimeConfig): RuntimeType =
RuntimeType(
"encode",
CargoDependency.SmithyTypes(runtimeConfig),
"${runtimeConfig.crateSrcPrefix}_types::base64"
)

fun Base64Decode(runtimeConfig: RuntimeConfig): RuntimeType =
RuntimeType(
"decode",
CargoDependency.SmithyTypes(runtimeConfig),
"${runtimeConfig.crateSrcPrefix}_types::base64"
)
val Base64Encode = RuntimeType("encode", CargoDependency.Base64, "base64")
val Base64Decode = RuntimeType("decode", CargoDependency.Base64, "base64")

fun TimestampFormat(runtimeConfig: RuntimeConfig, format: TimestampFormatTrait.Format): RuntimeType {
val timestampFormat = when (format) {
Expand Down
2 changes: 1 addition & 1 deletion ...oftware/amazon/smithy/rust/codegen/smithy/customizations/HttpChecksumRequiredGenerator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ class HttpChecksumRequiredGenerator(
""",
"md5" to CargoDependency.Md5.asType(),
"http" to CargoDependency.Http.asType(),
"base64_encode" to RuntimeType.Base64Encode(codegenContext.runtimeConfig),
"base64_encode" to RuntimeType.Base64Encode,
"BuildError" to codegenContext.runtimeConfig.operationBuildError()
)
}
Expand Down
4 changes: 2 additions & 2 deletions ...kotlin/software/amazon/smithy/rust/codegen/smithy/generators/http/HttpBindingGenerator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,7 @@ class HttpBindingGenerator(
.and_then(|bytes|String::from_utf8(bytes).map_err(|_|#{header}::ParseError::new_with_message("base64 encoded data was not valid utf-8")))
).collect();
""",
"base_64_decode" to RuntimeType.Base64Decode(runtimeConfig),
"base_64_decode" to RuntimeType.Base64Decode,
"header" to headerUtil
)
rust("let $parsedValue = $parsedValue?;")
Expand Down Expand Up @@ -555,7 +555,7 @@ class HttpBindingGenerator(
return when {
target.isStringShape -> {
if (target.hasTrait<MediaTypeTrait>()) {
val func = writer.format(RuntimeType.Base64Encode(runtimeConfig))
val func = writer.format(RuntimeType.Base64Encode)
"$func(&$targetName)"
} else {
quoteValue("AsRef::<str>::as_ref($targetName)")
Expand Down
2 changes: 1 addition & 1 deletion ...tware/amazon/smithy/rust/codegen/smithy/protocols/parse/XmlBindingTraitParserGenerator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -644,7 +644,7 @@ class XmlBindingTraitParserGenerator(
)
}
is BlobShape -> {
withBlock("#T(", ")", RuntimeType.Base64Decode(runtimeConfig)) {
withBlock("#T(", ")", RuntimeType.Base64Decode) {
provider()
}
rustTemplate(
Expand Down
5 changes: 1 addition & 4 deletions ...software/amazon/smithy/rust/codegen/smithy/protocols/serialize/JsonSerializerGenerator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -346,10 +346,7 @@ class JsonSerializerGenerator(
smithyTypes.member("Number")
)
}
is BlobShape -> rust(
"$writer.string_unchecked(&#T(${value.asRef()}));",
RuntimeType.Base64Encode(runtimeConfig)
)
is BlobShape -> rust("$writer.string_unchecked(&#T(${value.asRef()}));", RuntimeType.Base64Encode)
is TimestampShape -> {
val timestampFormat =
httpBindingResolver.timestampFormat(context.shape, HttpLocation.DOCUMENT, EPOCH_SECONDS)
Expand Down
5 changes: 1 addition & 4 deletions ...oftware/amazon/smithy/rust/codegen/smithy/protocols/serialize/QuerySerializerGenerator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,10 +222,7 @@ abstract class QuerySerializerGenerator(codegenContext: CodegenContext) : Struct
smithyTypes.member("Number")
)
}
is BlobShape -> rust(
"$writer.string(&#T(${value.name}));",
RuntimeType.Base64Encode(runtimeConfig)
)
is BlobShape -> rust("$writer.string(&#T(${value.name}));", RuntimeType.Base64Encode)
is TimestampShape -> {
val timestampFormat = determineTimestampFormat(context.shape)
val timestampFormatType = RuntimeType.TimestampFormat(runtimeConfig, timestampFormat)
Expand Down
2 changes: 1 addition & 1 deletion ...azon/smithy/rust/codegen/smithy/protocols/serialize/XmlBindingTraitSerializerGenerator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,7 @@ class XmlBindingTraitSerializerGenerator(
CargoDependency.SmithyTypes(runtimeConfig).asType().member("primitive::Encoder")
)
}
is BlobShape -> rust("#T($input.as_ref()).as_ref()", RuntimeType.Base64Encode(runtimeConfig))
is BlobShape -> rust("#T($input.as_ref()).as_ref()", RuntimeType.Base64Encode)
is TimestampShape -> {
val timestampFormat =
httpBindingResolver.timestampFormat(
Expand Down
1 change: 1 addition & 0 deletions rust-runtime/aws-smithy-json/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ repository = "https://github.com/awslabs/smithy-rs"

[dependencies]
aws-smithy-types = { path = "../aws-smithy-types" }
base64 = "0.13"

[dev-dependencies]
proptest = "1"
Expand Down
2 changes: 1 addition & 1 deletion rust-runtime/aws-smithy-json/src/deserialize/token.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
use crate::deserialize::error::{Error, ErrorReason};
use crate::escape::unescape_string;
use aws_smithy_types::date_time::Format;
use aws_smithy_types::{base64, Blob, DateTime, Document, Number};
use aws_smithy_types::{Blob, DateTime, Document, Number};
use std::borrow::Cow;

use crate::deserialize::must_not_be_finite;
Expand Down
1 change: 0 additions & 1 deletion rust-runtime/aws-smithy-types/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ ryu = "1.0.5"
time = { version = "0.3.4", features = ["parsing"] }

[dev-dependencies]
base64 = "0.13.0"
lazy_static = "1.4"
proptest = "1"
serde = { version = "1", features = ["derive"] }
Expand Down