Contents
- Introduction
- Spec Test Format
- Use as integration tests
- Using
crypt_shared
- Prose Tests
- 1. Custom Key Material Test
- 2. Data Key and Double Encryption
- 3. External Key Vault Test
- 4. BSON Size Limits and Batch Splitting
- 5. Views Are Prohibited
- 6. Corpus Test
- 7. Custom Endpoint Test
- 8. Bypass Spawning mongocryptd
- 9. Deadlock Tests
- 10. KMS TLS Tests
- 11. KMS TLS Options Tests
- 12. Explicit Encryption
- 13. Unique Index on keyAltNames
- 14. Decryption Events
- 15. On-demand AWS Credentials
- 16. Rewrap
This document describes the format of the driver spec tests included in the
JSON and YAML files included in the legacy
sub-directory. Tests in the
unified
directory are written using the Unified Test Format.
The timeoutMS.yml
/timeoutMS.json
files in this directory contain tests
for the timeoutMS
option and its application to the client-side encryption
feature. Drivers MUST only run these tests after implementing the
Client Side Operations Timeout
specification.
Additional prose tests, that are not represented in the spec tests, are described and MUST be implemented by all drivers.
The spec tests format is an extension of transactions spec tests with some additions:
- A
json_schema
to set on the collection used for operations. - An
encrypted_fields
to set on the collection used for operations. - A
key_vault_data
of data that should be inserted in the key vault collection before each test. - Introduction
autoEncryptOpts
to clientOptions - Addition of $db to command in command_started_event
- Addition of $$type to command_started_event and outcome.
The semantics of $$type is that any actual value matching one of the types indicated by either a BSON type string or an array of BSON type strings is considered a match.
For example, the following matches a command_started_event for an insert of a document where random must be of type binData
:
- command_started_event: command: insert: *collection_name documents: - { random: { $$type: "binData" } } ordered: true command_name: insert
The following matches a command_started_event for an insert of a document where random
must be of type
binData
or string
:
- command_started_event: command: insert: *collection_name documents: - { random: { $$type: ["binData", "string"] } } ordered: true command_name: insert
The values of $$type correspond to these documented string representations of BSON types.
Each YAML file has the following keys:
runOn
Unchanged from Transactions spec tests.database_name
Unchanged from Transactions spec tests.collection_name
Unchanged from Transactions spec tests.data
Unchanged from Transactions spec tests.json_schema
A JSON Schema that should be set on the collection (usingcreateCollection
) before each test run.encrypted_fields
An encryptedFields option that should be set on the collection (usingcreateCollection
) before each test run.key_vault_data
The data that should exist in the key vault collection under test before each test run.tests
: An array of tests that are to be run independently of each other. Each test will have some or all of the following fields:description
: Unchanged from Transactions spec tests.skipReason
: Unchanged from Transactions spec tests.useMultipleMongoses
: Unchanged from Transactions spec tests.failPoint
: Unchanged from Transactions spec tests.clientOptions
: Optional, parameters to pass to MongoClient().autoEncryptOpts
: OptionalkmsProviders
A dictionary of KMS providers to set on the key vault ("aws" or "local")aws
The AWS KMS provider. An empty object. Drivers MUST fill in AWS credentials (accessKeyId, secretAccessKey) from the environment.azure
The Azure KMS provider credentials. An empty object. Drivers MUST fill in Azure credentials (tenantId, clientId, and clientSecret) from the environment.gcp
The GCP KMS provider credentials. An empty object. Drivers MUST fill in GCP credentials (email, privateKey) from the environment.local
The local KMS provider.key
A 96 byte local key.
kmip
The KMIP KMS provider credentials. An empty object. Drivers MUST fill in KMIP credentials (endpoint, and TLS options).
schemaMap
: Optional, a map from namespaces to local JSON schemas.keyVaultNamespace
: Optional, a namespace to the key vault collection. Defaults to "keyvault.datakeys".bypassAutoEncryption
: Optional, a boolean to indicate whether or not auto encryption should be bypassed. Defaults tofalse
.encryptedFieldsMap
An optional document. The document maps collection namespace toEncryptedFields
documents.
operations
: Array of documents, each describing an operation to be executed. Each document has the following fields:name
: Unchanged from Transactions spec tests.object
: Unchanged from Transactions spec tests.. Defaults to "collection" if omitted.collectionOptions
: Unchanged from Transactions spec tests.command_name
: Unchanged from Transactions spec tests.arguments
: Unchanged from Transactions spec tests.result
: Same as the Transactions spec test format with one addition: if the operation is expected to return an error, theresult
document may contain anisTimeoutError
boolean field. Iftrue
, the test runner MUST assert that the error represents a timeout due to the use of thetimeoutMS
option. Iffalse
, the test runner MUST assert that the error does not represent a timeout.
expectations
: Unchanged from Transactions spec tests.outcome
: Unchanged from Transactions spec tests.
Do the following before running spec tests:
- If available for the platform under test, obtain a crypt_shared binary and place it in a location accessible to the tests. Refer to: Using crypt_shared
- Start the mongocryptd process.
- Start a mongod process with server version 4.1.9 or later.
- Place credentials to an AWS IAM user (access key ID + secret access key) somewhere in the environment outside of tracked code. (If testing on evergreen, project variables are a good place).
- Start a KMIP test server on port 5698 by running drivers-evergreen-tools/.evergreen/csfle/kms_kmip_server.py.
Load each YAML (or JSON) file using a Canonical Extended JSON parser.
Then for each element in tests
:
If the
skipReason
field is present, skip this test completely.If the
key_vault_data
field is present:- Drop the
keyvault.datakeys
collection using writeConcern "majority". - Insert the data specified into the
keyvault.datakeys
with write concern "majority".
- Drop the
Create a MongoClient.
Create a collection object from the MongoClient, using the
database_name
andcollection_name
fields from the YAML file. Drop the collection with writeConcern "majority". If ajson_schema
is defined in the test, use thecreateCollection
command to explicitly create the collection:{"create": <collection>, "validator": {"$jsonSchema": <json_schema>}}
If
encrypted_fields
is defined in the test, the required collections and index described in Create and Drop Collection Helpers must be created:- Use the
dropCollection
helper withencrypted_fields
as an option and writeConcern "majority". - Use the
createCollection
helper withencrypted_fields
as an option.
- Use the
If the YAML file contains a
data
array, insert the documents indata
into the test collection, using writeConcern "majority".Create a new MongoClient using
clientOptions
.If
autoEncryptOpts
includesaws
,awsTemporary
,awsTemporaryNoSessionToken
,azure
,gcp
, and/orkmip
as a KMS provider, pass in credentials from the environment.awsTemporary
, andawsTemporaryNoSessionToken
require temporary AWS credentials. These can be retrieved using the csfle set-temp-creds.sh script.aws
,awsTemporary
, andawsTemporaryNoSessionToken
are mutually exclusive.aws
should be substituted with:"aws": { "accessKeyId": <set from environment>, "secretAccessKey": <set from environment> }
awsTemporary
should be substituted with:"aws": { "accessKeyId": <set from environment>, "secretAccessKey": <set from environment> "sessionToken": <set from environment> }
awsTemporaryNoSessionToken
should be substituted with:"aws": { "accessKeyId": <set from environment>, "secretAccessKey": <set from environment> }
gcp
should be substituted with:"gcp": { "email": <set from environment>, "privateKey": <set from environment>, }
azure
should be substituted with:"azure": { "tenantId": <set from environment>, "clientId": <set from environment>, "clientSecret": <set from environment>, }
local
should be substituted with:"local": { "key": <base64 decoding of LOCAL_MASTERKEY> }
kmip
should be substituted with:"kmip": { "endpoint": "localhost:5698" }
Configure KMIP TLS connections to use the following options:
tlsCAFile
(or equivalent) set to drivers-evergreen-tools/.evergreen/x509gen/ca.pem. This MAY be configured system-wide.tlsCertificateKeyFile
(or equivalent) set to drivers-evergreen-tools/.evergreen/x509gen/client.pem.
The method of passing TLS options for KMIP TLS connections is driver dependent.
If
autoEncryptOpts
does not includekeyVaultNamespace
, default it tokeyvault.datakeys
.
For each element in
operations
:Enter a "try" block or your programming language's closest equivalent.
Create a Database object from the MongoClient, using the
database_name
field at the top level of the test file.Create a Collection object from the Database, using the
collection_name
field at the top level of the test file. IfcollectionOptions
is present create the Collection object with the provided options. Otherwise create the object with the default options.Execute the named method on the provided
object
, passing the arguments listed.If the driver throws an exception / returns an error while executing this series of operations, store the error message and server error code.
If the result document has an "errorContains" field, verify that the method threw an exception or returned an error, and that the value of the "errorContains" field matches the error string. "errorContains" is a substring (case-insensitive) of the actual error message.
If the result document has an "errorCodeName" field, verify that the method threw a command failed exception or returned an error, and that the value of the "errorCodeName" field matches the "codeName" in the server error response.
If the result document has an "errorLabelsContain" field, verify that the method threw an exception or returned an error. Verify that all of the error labels in "errorLabelsContain" are present in the error or exception using the
hasErrorLabel
method.If the result document has an "errorLabelsOmit" field, verify that the method threw an exception or returned an error. Verify that none of the error labels in "errorLabelsOmit" are present in the error or exception using the
hasErrorLabel
method.If the operation returns a raw command response, eg from
runCommand
, then compare only the fields present in the expected result document. Otherwise, compare the method's return value toresult
using the same logic as the CRUD Spec Tests runner.
If the test includes a list of command-started events in
expectations
, compare them to the actual command-started events using the same logic as the Command Monitoring Spec Tests runner.For each element in
outcome
:- If
name
is "collection", create a new MongoClient without encryption and verify that the test collection contains exactly the documents in thedata
array. Ensure this find reads the latest data by using primary read preference with local read concern even when the MongoClient is configured with another read preference or read concern.
- If
The spec test MUST be run with and without auth.
On platforms where crypt_shared is available, drivers should prefer to test
with the crypt_shared
library instead of spawning mongocryptd.
crypt_shared is released alongside the server. crypt_shared is only available in versions 6.0 and above. Drivers SHOULD prefer testing a version of crypt_shared that matches the server version being tested. Driver tests on server versions less than 6.0 SHOULD use mongocryptd.
Drivers MUST continue to run all tests with mongocryptd on at least one platform for all tested server versions.
Note that some tests assert on mongocryptd-related behaviors (e.g. the
mongocryptdBypassSpawn
test).
Drivers under test should load the crypt_shared library using either the
cryptSharedLibPath
public API option (as part of the AutoEncryption
extraOptions
), or by setting a special search path instead.
Some tests will require not using crypt_shared. For such tests, one should
ensure that crypt_shared
will not be loaded. Refer to the
client-side-encryption documentation for information on "disabling"
crypt_shared
and setting library search paths.
Note
The crypt_shared dynamic library can be obtained using the mongodl Python script from drivers-evergreen-tools:
$ python3 mongodl.py --component=crypt_shared --version=<VERSION> --out=./crypt_shared/
Other versions of crypt_shared
are also available. Please use the
--list
option to see versions.
Tests for the ClientEncryption type are not included as part of the YAML tests.
In the prose tests LOCAL_MASTERKEY refers to the following base64:
Mng0NCt4ZHVUYUJCa1kxNkVyNUR1QURhZ2h2UzR2d2RrZzh0cFBwM3R6NmdWMDFBMUN3YkQ5aXRRMkhGRGdQV09wOGVNYUMxT2k3NjZKelhaQmRCZGJkTXVyZG9uSjFk
Perform all applicable operations on key vault collections (e.g. inserting an example data key, or running a find command) with readConcern/writeConcern "majority".
- Create a
MongoClient
object (referred to asclient
). - Using
client
, drop the collectionkeyvault.datakeys
. - Create a
ClientEncryption
object (referred to asclient_encryption
) withclient
set as thekeyVaultClient
. - Using
client_encryption
, create a data key with alocal
KMS provider and the following custom key material (given as base64):
xPTAjBRG5JiPm+d3fj6XLi2q5DMXUS/f1f+SMAlhhwkhDRL0kr8r9GDLIGTAGlvC+HVjSIgdL+RKwZCvpXSyxTICWSXTUYsWYPyu3IoHbuBZdmw2faM3WhcRIgbMReU5
- Find the resulting key document in
keyvault.datakeys
, save a copy of the key document, then remove the key document from the collection. - Replace the
_id
field in the copied key document with a UUID with base64 valueAAAAAAAAAAAAAAAAAAAAAA==
(16 bytes all equal to0x00
) and insert the modified key document intokeyvault.datakeys
with majority write concern. - Using
client_encryption
, encrypt the string"test"
with the modified data key using theAEAD_AES_256_CBC_HMAC_SHA_512-Deterministic
algorithm and assert the resulting value is equal to the following (given as base64):
AQAAAAAAAAAAAAAAAAAAAAACz0ZOLuuhEYi807ZXTdhbqhLaS2/t9wLifJnnNYwiw79d75QYIZ6M/aYC1h9nCzCjZ7pGUpAuNnkUhnIXM3PjrA==
First, perform the setup.
Create a MongoClient without encryption enabled (referred to as
client
). Enable command monitoring to listen for command_started events.Using
client
, drop the collectionskeyvault.datakeys
anddb.coll
.Create the following:
- A MongoClient configured with auto encryption (referred to as
client_encrypted
) - A
ClientEncryption
object (referred to asclient_encryption
)
Configure both objects with the following KMS providers:
{ "aws": { "accessKeyId": <set from environment>, "secretAccessKey": <set from environment> }, "azure": { "tenantId": <set from environment>, "clientId": <set from environment>, "clientSecret": <set from environment>, }, "gcp": { "email": <set from environment>, "privateKey": <set from environment>, } "local": { "key": <base64 decoding of LOCAL_MASTERKEY> }, "kmip": { "endpoint": "localhost:5698" } }
Configure KMIP TLS connections to use the following options:
tlsCAFile
(or equivalent) set to drivers-evergreen-tools/.evergreen/x509gen/ca.pem. This MAY be configured system-wide.tlsCertificateKeyFile
(or equivalent) set to drivers-evergreen-tools/.evergreen/x509gen/client.pem.
The method of passing TLS options for KMIP TLS connections is driver dependent.
Configure both objects with
keyVaultNamespace
set tokeyvault.datakeys
.Configure the
MongoClient
with the followingschema_map
:{ "db.coll": { "bsonType": "object", "properties": { "encrypted_placeholder": { "encrypt": { "keyId": "/placeholder", "bsonType": "string", "algorithm": "AEAD_AES_256_CBC_HMAC_SHA_512-Random" } } } } }
Configure
client_encryption
with thekeyVaultClient
of the previously createdclient
.- A MongoClient configured with auto encryption (referred to as
For each KMS provider (aws
, azure
, gcp
, local
, and kmip
), referred to as provider_name
, run the following test.
Call
client_encryption.createDataKey()
.Set keyAltNames to
["<provider_name>_altname"]
.Set the masterKey document based on
provider_name
.For "aws":
{ region: "us-east-1", key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0" }
For "azure":
{ "keyVaultEndpoint": "key-vault-csfle.vault.azure.net", "keyName": "key-name-csfle" }
For "gcp":
{ "projectId": "devprod-drivers", "location": "global", "keyRing": "key-ring-csfle", "keyName": "key-name-csfle" }
For "kmip":
{}
For "local", do not set a masterKey document.
Expect a BSON binary with subtype 4 to be returned, referred to as
datakey_id
.Use
client
to run afind
onkeyvault.datakeys
by querying with the_id
set to thedatakey_id
.Expect that exactly one document is returned with the "masterKey.provider" equal to
provider_name
.Check that
client
captured a command_started event for theinsert
command containing a majority writeConcern.
Call
client_encryption.encrypt()
with the value "hello <provider_name>", the algorithmAEAD_AES_256_CBC_HMAC_SHA_512-Deterministic
, and thekey_id
ofdatakey_id
.- Expect the return value to be a BSON binary subtype 6, referred to as
encrypted
. - Use
client_encrypted
to insert{ _id: "<provider_name>", "value": <encrypted> }
intodb.coll
. - Use
client_encrypted
to run a find querying with_id
of "<provider_name>" and expectvalue
to be "hello <provider_name>".
- Expect the return value to be a BSON binary subtype 6, referred to as
Call
client_encryption.encrypt()
with the value "hello <provider_name>", the algorithmAEAD_AES_256_CBC_HMAC_SHA_512-Deterministic
, and thekey_alt_name
of<provider_name>_altname
.- Expect the return value to be a BSON binary subtype 6. Expect the value to exactly match the value of
encrypted
.
- Expect the return value to be a BSON binary subtype 6. Expect the value to exactly match the value of
Test explicit encrypting an auto encrypted field.
- Use
client_encrypted
to attempt to insert{ "encrypted_placeholder": <encrypted> }
- Expect an exception to be thrown, since this is an attempt to auto encrypt an already encrypted value.
- Use
Run the following tests twice, parameterized by a boolean withExternalKeyVault
.
Create a MongoClient without encryption enabled (referred to as
client
).Using
client
, drop the collectionskeyvault.datakeys
anddb.coll
. Insert the document external/external-key.json intokeyvault.datakeys
.Create the following:
- A MongoClient configured with auto encryption (referred to as
client_encrypted
) - A
ClientEncryption
object (referred to asclient_encryption
)
Configure both objects with the
local
KMS providers as follows:{ "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
Configure both objects with
keyVaultNamespace
set tokeyvault.datakeys
.Configure
client_encrypted
to use the schema external/external-schema.json fordb.coll
by setting a schema map like:{ "db.coll": <contents of external-schema.json>}
If
withExternalKeyVault == true
, configure both objects with an external key vault client. The external client MUST connect to the same MongoDB cluster that is being tested against, except it MUST use the usernamefake-user
and passwordfake-pwd
.- A MongoClient configured with auto encryption (referred to as
Use
client_encrypted
to insert the document{"encrypted": "test"}
intodb.coll
. IfwithExternalKeyVault == true
, expect an authentication exception to be thrown. Otherwise, expect the insert to succeed.Use
client_encryption
to explicitly encrypt the string"test"
with key IDLOCALAAAAAAAAAAAAAAAAA==
and deterministic algorithm. IfwithExternalKeyVault == true
, expect an authentication exception to be thrown. Otherwise, expect the insert to succeed.
First, perform the setup.
Create a MongoClient without encryption enabled (referred to as
client
).Using
client
, drop and create the collectiondb.coll
configured with the included JSON schema limits/limits-schema.json.Using
client
, drop the collectionkeyvault.datakeys
. Insert the document limits/limits-key.jsonCreate a MongoClient configured with auto encryption (referred to as
client_encrypted
)Configure with the
local
KMS provider as follows:{ "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
Configure with the
keyVaultNamespace
set tokeyvault.datakeys
.
Using client_encrypted
perform the following operations:
Insert
{ "_id": "over_2mib_under_16mib", "unencrypted": <the string "a" repeated 2097152 times> }
.Expect this to succeed since this is still under the
maxBsonObjectSize
limit.Insert the document limits/limits-doc.json concatenated with
{ "_id": "encryption_exceeds_2mib", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }
Note: limits-doc.json is a 1005 byte BSON document that encrypts to a ~10,000 byte document.Expect this to succeed since after encryption this still is below the normal maximum BSON document size. Note, before auto encryption this document is under the 2 MiB limit. After encryption it exceeds the 2 MiB limit, but does NOT exceed the 16 MiB limit.
Bulk insert the following:
{ "_id": "over_2mib_1", "unencrypted": <the string "a" repeated (2097152) times> }
{ "_id": "over_2mib_2", "unencrypted": <the string "a" repeated (2097152) times> }
Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using command monitoring.
Bulk insert the following:
- The document limits/limits-doc.json concatenated with
{ "_id": "encryption_exceeds_2mib_1", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }
- The document limits/limits-doc.json concatenated with
{ "_id": "encryption_exceeds_2mib_2", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }
Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using command monitoring.
- The document limits/limits-doc.json concatenated with
Insert
{ "_id": "under_16mib", "unencrypted": <the string "a" repeated 16777216 - 2000 times>
.Expect this to succeed since this is still (just) under the
maxBsonObjectSize
limit.Insert the document limits/limits-doc.json concatenated with
{ "_id": "encryption_exceeds_16mib", "unencrypted": < the string "a" repeated (16777216 - 2000) times > }
Expect this to fail since encryption results in a document exceeding the
maxBsonObjectSize
limit.
Optionally, if it is possible to mock the maxWriteBatchSize (i.e. the maximum number of documents in a batch) test that setting maxWriteBatchSize=1 and inserting the two documents { "_id": "a" }, { "_id": "b" }
with client_encrypted
splits the operation into two inserts.
Create a MongoClient without encryption enabled (referred to as
client
).Using
client
, drop and create a view nameddb.view
with an empty pipeline. E.g. using the command{ "create": "view", "viewOn": "coll" }
.Create a MongoClient configured with auto encryption (referred to as
client_encrypted
)Configure with the
local
KMS provider as follows:{ "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
Configure with the
keyVaultNamespace
set tokeyvault.datakeys
.Using
client_encrypted
, attempt to insert a document intodb.view
. Expect an exception to be thrown containing the message: "cannot auto encrypt a view".
The corpus test exhaustively enumerates all ways to encrypt all BSON value types. Note, the test data includes BSON binary subtype 4 (or standard UUID), which MUST be decoded and encoded as subtype 4. Run the test as follows.
Create a MongoClient without encryption enabled (referred to as
client
).Using
client
, drop and create the collectiondb.coll
configured with the included JSON schema corpus/corpus-schema.json.Using
client
, drop the collectionkeyvault.datakeys
. Insert the documents corpus/corpus-key-local.json, corpus/corpus-key-aws.json, corpus/corpus-key-azure.json, corpus/corpus-key-gcp.json, and corpus/corpus-key-kmip.json.Create the following:
- A MongoClient configured with auto encryption (referred to as
client_encrypted
) - A
ClientEncryption
object (referred to asclient_encryption
)
Configure both objects with
aws
,azure
,gcp
,local
, andkmip
KMS providers as follows:{ "aws": { <AWS credentials> }, "azure": { <Azure credentials> }, "gcp": { <GCP credentials> }, "local": { "key": <base64 decoding of LOCAL_MASTERKEY> }, "kmip": { "endpoint": "localhost:5698" } } }
Configure KMIP TLS connections to use the following options:
tlsCAFile
(or equivalent) set to drivers-evergreen-tools/.evergreen/x509gen/ca.pem. This MAY be configured system-wide.tlsCertificateKeyFile
(or equivalent) set to drivers-evergreen-tools/.evergreen/x509gen/client.pem.
The method of passing TLS options for KMIP TLS connections is driver dependent.
Where LOCAL_MASTERKEY is the following base64:
Mng0NCt4ZHVUYUJCa1kxNkVyNUR1QURhZ2h2UzR2d2RrZzh0cFBwM3R6NmdWMDFBMUN3YkQ5aXRRMkhGRGdQV09wOGVNYUMxT2k3NjZKelhaQmRCZGJkTXVyZG9uSjFk
Configure both objects with
keyVaultNamespace
set tokeyvault.datakeys
.- A MongoClient configured with auto encryption (referred to as
Load corpus/corpus.json to a variable named
corpus
. The corpus contains subdocuments with the following fields:kms
isaws
,azure
,gcp
,local
, orkmip
type
is a BSON type string names coming from here)algo
is eitherrand
ordet
for random or deterministic encryptionmethod
is eitherauto
, for automatic encryption orexplicit
for explicit encryptionidentifier
is eitherid
oraltname
for the key identifierallowed
is a boolean indicating whether the encryption for the given parameters is permitted.value
is the value to be tested.
Create a new BSON document, named
corpus_copied
. Iterate over each field ofcorpus
.If the field name is
_id
,altname_aws
,altname_local
,altname_azure
,altname_gcp
, oraltname_kmip
copy the field tocorpus_copied
.If
method
isauto
, copy the field tocorpus_copied
.If
method
isexplicit
, useclient_encryption
to explicitly encrypt the value.- Encrypt with the algorithm described by
algo
. - If
identifier
isid
- If
kms
islocal
set the key_id to the UUID with base64 valueLOCALAAAAAAAAAAAAAAAAA==
. - If
kms
isaws
set the key_id to the UUID with base64 valueAWSAAAAAAAAAAAAAAAAAAA==
. - If
kms
isazure
set the key_id to the UUID with base64 valueAZUREAAAAAAAAAAAAAAAAA==
. - If
kms
isgcp
set the key_id to the UUID with base64 valueGCPAAAAAAAAAAAAAAAAAAA==
. - If
kms
iskmip
set the key_id to the UUID with base64 valueKMIPAAAAAAAAAAAAAAAAAA==
.
- If
- If
identifier
isaltname
- If
kms
islocal
set the key_alt_name to "local". - If
kms
isaws
set the key_alt_name to "aws". - If
kms
isazure
set the key_alt_name to "azure". - If
kms
isgcp
set the key_alt_name to "gcp". - If
kms
iskmip
set the key_alt_name to "kmip".
- If
If
allowed
is true, copy the field and encrypted value tocorpus_copied
. Ifallowed
is false. verify that an exception is thrown. Copy the unencrypted value to tocorpus_copied
.- Encrypt with the algorithm described by
Using
client_encrypted
, insertcorpus_copied
intodb.coll
.Using
client_encrypted
, find the inserted document fromdb.coll
to a variable namedcorpus_decrypted
. Since it should have been automatically decrypted, assert the document exactly matchescorpus
.Load corpus/corpus_encrypted.json to a variable named
corpus_encrypted_expected
. Usingclient
find the inserted document fromdb.coll
to a variable namedcorpus_encrypted_actual
.Iterate over each field of
corpus_encrypted_expected
and check the following:- If the
algo
isdet
, that the value equals the value of the corresponding field incorpus_encrypted_actual
. - If the
algo
isrand
andallowed
is true, that the value does not equal the value of the corresponding field incorpus_encrypted_actual
. - If
allowed
is true, decrypt the value withclient_encryption
. Decrypt the value of the corresponding field ofcorpus_encrypted
and validate that they are both equal. - If
allowed
is false, validate the value exactly equals the value of the corresponding field ofcorpus
(neither was encrypted).
- If the
Repeat steps 1-8 with a local JSON schema. I.e. amend step 4 to configure the schema on
client_encrypted
with theschema_map
option.
For each test cases, start by creating two ClientEncryption
objects. Recreate the ClientEncryption
objects for each test case.
Create a ClientEncryption
object (referred to as client_encryption
)
Configure with keyVaultNamespace
set to keyvault.datakeys
, and a default MongoClient as the keyVaultClient
.
Configure with KMS providers as follows:
{
"aws": {
"accessKeyId": <set from environment>,
"secretAccessKey": <set from environment>
},
"azure": {
"tenantId": <set from environment>,
"clientId": <set from environment>,
"clientSecret": <set from environment>,
"identityPlatformEndpoint": "login.microsoftonline.com:443"
},
"gcp": {
"email": <set from environment>,
"privateKey": <set from environment>,
"endpoint": "oauth2.googleapis.com:443"
},
"kmip" {
"endpoint": "localhost:5698"
}
}
Create a ClientEncryption
object (referred to as client_encryption_invalid
)
Configure with keyVaultNamespace
set to keyvault.datakeys
, and a default MongoClient as the keyVaultClient
.
Configure with KMS providers as follows:
{
"azure": {
"tenantId": <set from environment>,
"clientId": <set from environment>,
"clientSecret": <set from environment>,
"identityPlatformEndpoint": "doesnotexist.invalid:443"
},
"gcp": {
"email": <set from environment>,
"privateKey": <set from environment>,
"endpoint": "doesnotexist.invalid:443"
},
"kmip": {
"endpoint": "doesnotexist.local:5698"
}
}
Configure KMIP TLS connections to use the following options:
tlsCAFile
(or equivalent) set to drivers-evergreen-tools/.evergreen/x509gen/ca.pem. This MAY be configured system-wide.tlsCertificateKeyFile
(or equivalent) set to drivers-evergreen-tools/.evergreen/x509gen/client.pem.
The method of passing TLS options for KMIP TLS connections is driver dependent.
Call client_encryption.createDataKey() with "aws" as the provider and the following masterKey:
{ region: "us-east-1", key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0" }
Expect this to succeed. Use the returned UUID of the key to explicitly encrypt and decrypt the string "test" to validate it works.
Call client_encryption.createDataKey() with "aws" as the provider and the following masterKey:
{ region: "us-east-1", key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0", endpoint: "kms.us-east-1.amazonaws.com" }
Expect this to succeed. Use the returned UUID of the key to explicitly encrypt and decrypt the string "test" to validate it works.
Call client_encryption.createDataKey() with "aws" as the provider and the following masterKey:
{ region: "us-east-1", key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0", endpoint: "kms.us-east-1.amazonaws.com:443" }
Expect this to succeed. Use the returned UUID of the key to explicitly encrypt and decrypt the string "test" to validate it works.
Call client_encryption.createDataKey() with "aws" as the provider and the following masterKey:
{ region: "us-east-1", key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0", endpoint: "kms.us-east-1.amazonaws.com:12345" }
Expect this to fail with a socket connection error.
Call client_encryption.createDataKey() with "aws" as the provider and the following masterKey:
{ region: "us-east-1", key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0", endpoint: "kms.us-east-2.amazonaws.com" }
Expect this to fail with an exception.
Call client_encryption.createDataKey() with "aws" as the provider and the following masterKey:
{ region: "us-east-1", key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0", endpoint: "doesnotexist.invalid" }
Expect this to fail with a network exception indicating failure to resolve "doesnotexist.invalid".
Call client_encryption.createDataKey() with "azure" as the provider and the following masterKey:
{ "keyVaultEndpoint": "key-vault-csfle.vault.azure.net", "keyName": "key-name-csfle" }
Expect this to succeed. Use the returned UUID of the key to explicitly encrypt and decrypt the string "test" to validate it works.
Call
client_encryption_invalid.createDataKey()
with the same masterKey. Expect this to fail with a network exception indicating failure to resolve "doesnotexist.invalid".Call client_encryption.createDataKey() with "gcp" as the provider and the following masterKey:
{ "projectId": "devprod-drivers", "location": "global", "keyRing": "key-ring-csfle", "keyName": "key-name-csfle", "endpoint": "cloudkms.googleapis.com:443" }
Expect this to succeed. Use the returned UUID of the key to explicitly encrypt and decrypt the string "test" to validate it works.
Call
client_encryption_invalid.createDataKey()
with the same masterKey. Expect this to fail with a network exception indicating failure to resolve "doesnotexist.invalid".Call client_encryption.createDataKey() with "gcp" as the provider and the following masterKey:
{ "projectId": "devprod-drivers", "location": "global", "keyRing": "key-ring-csfle", "keyName": "key-name-csfle", "endpoint": "doesnotexist.invalid:443" }
Expect this to fail with an exception with a message containing the string: "Invalid KMS response".
Call client_encryption.createDataKey() with "kmip" as the provider and the following masterKey:
{ "keyId": "1" }
Expect this to succeed. Use the returned UUID of the key to explicitly encrypt and decrypt the string "test" to validate it works.
Call
client_encryption_invalid.createDataKey()
with the same masterKey. Expect this to fail with a network exception indicating failure to resolve "doesnotexist.local".Call
client_encryption.createDataKey()
with "kmip" as the provider and the following masterKey:{ "keyId": "1", "endpoint": "localhost:5698" }
Expect this to succeed. Use the returned UUID of the key to explicitly encrypt and decrypt the string "test" to validate it works.
Call
client_encryption.createDataKey()
with "kmip" as the provider and the following masterKey:{ "keyId": "1", "endpoint": "doesnotexist.local:5698" }
Expect this to fail with a network exception indicating failure to resolve "doesnotexist.local".
Note
IMPORTANT: If crypt_shared is visible to the operating system's library
search mechanism, the expected server error generated by these
mongocryptdBypassSpawn
tests will not appear because libmongocrypt will
load the crypt_shared
library instead of consulting mongocryptd. For
these tests, it is required that libmongocrypt not load crypt_shared
.
Refer to the client-side-encryption document for more information on
"disabling" crypt_shared
.
The following tests that setting mongocryptdBypassSpawn=true
really does bypass spawning mongocryptd.
Create a MongoClient configured with auto encryption (referred to as
client_encrypted
)Configure the required options. Use the
local
KMS provider as follows:{ "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
Configure with the
keyVaultNamespace
set tokeyvault.datakeys
.Configure
client_encrypted
to use the schema external/external-schema.json fordb.coll
by setting a schema map like:{ "db.coll": <contents of external-schema.json>}
Configure the following
extraOptions
:{ "mongocryptdBypassSpawn": true "mongocryptdURI": "mongodb://localhost:27021/db?serverSelectionTimeoutMS=1000", "mongocryptdSpawnArgs": [ "--pidfilepath=bypass-spawning-mongocryptd.pid", "--port=27021"] }
Drivers MAY pass a different port if they expect their testing infrastructure to be using port 27021. Pass a port that should be free.
Use
client_encrypted
to insert the document{"encrypted": "test"}
intodb.coll
. Expect a server selection error propagated from the internal MongoClient failing to connect to mongocryptd on port 27021.
The following tests that setting bypassAutoEncryption=true
really does bypass spawning mongocryptd.
Create a MongoClient configured with auto encryption (referred to as
client_encrypted
)Configure the required options. Use the
local
KMS provider as follows:{ "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
Configure with the
keyVaultNamespace
set tokeyvault.datakeys
.Configure with
bypassAutoEncryption=true
.Configure the following
extraOptions
:{ "mongocryptdSpawnArgs": [ "--pidfilepath=bypass-spawning-mongocryptd.pid", "--port=27021"] }
Drivers MAY pass a different value to
--port
if they expect their testing infrastructure to be using port 27021. Pass a port that should be free.Use
client_encrypted
to insert the document{"unencrypted": "test"}
intodb.coll
. Expect this to succeed.Validate that mongocryptd was not spawned. Create a MongoClient to localhost:27021 (or whatever was passed via
--port
) with serverSelectionTimeoutMS=1000. Run a handshake command and ensure it fails with a server selection timeout.
Repeat the steps from the "Via bypassAutoEncryption" test, replacing "bypassAutoEncryption=true" with "bypassQueryAnalysis=true".
The following tests only apply to drivers that have implemented a connection pool (see the Connection Monitoring and Pooling specification).
There are multiple parameterized test cases. Before each test case, perform the setup.
Create a MongoClient
for setup operations named client_test
.
Create a MongoClient
for key vault operations with maxPoolSize=1
named client_keyvault
. Capture command started events.
Using client_test
, drop the collections keyvault.datakeys
and db.coll
.
Insert the document external/external-key.json into keyvault.datakeys
with majority write concern.
Create a collection db.coll
configured with a JSON schema external/external-schema.json as the validator, like so:
{"create": "coll", "validator": {"$jsonSchema": <json_schema>}}
Create a ClientEncryption
object, named client_encryption
configured with:
- keyVaultClient``=``client_test
- keyVaultNamespace``="keyvault.datakeys"
- ``kmsProviders``=``{ "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
Use client_encryption
to encrypt the value "string0" with algorithm``="AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic" and ``keyAltName``="local". Store the result in a variable named ``ciphertext
.
Proceed to run the test case.
Each test case configures a MongoClient
with automatic encryption (named client_encrypted
).
Each test must assert the number of unique MongoClient``s created. This can be accomplished by capturing ``TopologyOpeningEvent
, or by checking command started events for a client identifier (not possible in all drivers).
- Create a
MongoClient
namedclient_encrypted
configured as follows: - Set
AutoEncryptionOpts
: keyVaultNamespace="keyvault.datakeys"
kmsProviders``=``{ "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
- Append
TestCase.AutoEncryptionOpts
(defined below)
- Set
- Capture command started events.
- Set
maxPoolSize=TestCase.MaxPoolSize
- Create a
- If the testcase sets
AutoEncryptionOpts.bypassAutoEncryption=true
: - Use
client_test
to insert{ "_id": 0, "encrypted": <ciphertext> }
intodb.coll
.
- Use
- If the testcase sets
- Otherwise:
- Use
client_encrypted
to insert{ "_id": 0, "encrypted": "string0" }
.
- Use
- Use
client_encrypted
to run afindOne
operation ondb.coll
, with the filter{ "_id": 0 }
. - Expect the result to be
{ "_id": 0, "encrypted": "string0" }
. - Check captured events against
TestCase.Expectations
. - Check the number of unique
MongoClient``s created is equal to ``TestCase.ExpectedNumberOfClients
.
- MaxPoolSize: 1
- AutoEncryptionOpts:
- bypassAutoEncryption=false
- keyVaultClient=unset
- Expectations:
- Expect
client_encrypted
to have captured fourCommandStartedEvent
: - a listCollections to "db".
- a find on "keyvault".
- an insert on "db".
- a find on "db"
- Expect
- ExpectedNumberOfClients: 2
- MaxPoolSize: 1
- AutoEncryptionOpts:
- bypassAutoEncryption=false
- keyVaultClient=client_keyvault
- Expectations:
- Expect
client_encrypted
to have captured threeCommandStartedEvent
: - a listCollections to "db".
- an insert on "db".
- a find on "db"
- Expect
- Expect
client_keyvault
to have captured oneCommandStartedEvent
: - a find on "keyvault".
- Expect
- ExpectedNumberOfClients: 2
- MaxPoolSize: 1
- AutoEncryptionOpts:
- bypassAutoEncryption=true
- keyVaultClient=unset
- Expectations:
- Expect
client_encrypted
to have captured threeCommandStartedEvent
: - a find on "db"
- a find on "keyvault".
- Expect
- ExpectedNumberOfClients: 2
- MaxPoolSize: 1
- AutoEncryptionOpts:
- bypassAutoEncryption=true
- keyVaultClient=client_keyvault
- Expectations:
- Expect
client_encrypted
to have captured twoCommandStartedEvent
: - a find on "db"
- Expect
- Expect
client_keyvault
to have captured oneCommandStartedEvent
: - a find on "keyvault".
- Expect
- ExpectedNumberOfClients: 1
Drivers that do not support an unlimited maximum pool size MUST skip this test.
- MaxPoolSize: 0
- AutoEncryptionOpts:
- bypassAutoEncryption=false
- keyVaultClient=unset
- Expectations:
- Expect
client_encrypted
to have captured fiveCommandStartedEvent
: - a listCollections to "db".
- a listCollections to "keyvault".
- a find on "keyvault".
- an insert on "db".
- a find on "db"
- Expect
- ExpectedNumberOfClients: 1
Drivers that do not support an unlimited maximum pool size MUST skip this test.
- MaxPoolSize: 0
- AutoEncryptionOpts:
- bypassAutoEncryption=false
- keyVaultClient=client_keyvault
- Expectations:
- Expect
client_encrypted
to have captured threeCommandStartedEvent
: - a listCollections to "db".
- an insert on "db".
- a find on "db"
- Expect
- Expect
client_keyvault
to have captured oneCommandStartedEvent
: - a find on "keyvault".
- Expect
- ExpectedNumberOfClients: 1
Drivers that do not support an unlimited maximum pool size MUST skip this test.
- MaxPoolSize: 0
- AutoEncryptionOpts:
- bypassAutoEncryption=true
- keyVaultClient=unset
- Expectations:
- Expect
client_encrypted
to have captured threeCommandStartedEvent
: - a find on "db"
- a find on "keyvault".
- Expect
- ExpectedNumberOfClients: 1
Drivers that do not support an unlimited maximum pool size MUST skip this test.
- MaxPoolSize: 0
- AutoEncryptionOpts:
- bypassAutoEncryption=true
- keyVaultClient=client_keyvault
- Expectations:
- Expect
client_encrypted
to have captured twoCommandStartedEvent
: - a find on "db"
- Expect
- Expect
client_keyvault
to have captured oneCommandStartedEvent
: - a find on "keyvault".
- Expect
- ExpectedNumberOfClients: 1
The following tests that connections to KMS servers with TLS verify peer certificates.
The two tests below make use of mock KMS servers which can be run on Evergreen using the mock KMS server script. Drivers can set up their local Python enviroment for the mock KMS server by running the virtualenv activation script.
To start two mock KMS servers, one on port 9000 with ca.pem as a CA file and expired.pem as a cert file, and one on port 9001 with ca.pem as a CA file and wrong-host.pem as a cert file,
run the following commands from the .evergreen/csfle
directory:
. ./activate_venv.sh python -u kms_http_server.py --ca_file ../x509gen/ca.pem --cert_file ../x509gen/expired.pem --port 9000 & python -u kms_http_server.py --ca_file ../x509gen/ca.pem --cert_file ../x509gen/wrong-host.pem --port 9001 &
For both tests, do the following:
- Start a
mongod
process with server version 4.1.9 or later. - Create a
MongoClient
for key vault operations. - Create a
ClientEncryption
object (referred to asclient_encryption
) withkeyVaultNamespace
set tokeyvault.datakeys
.
Start a mock KMS server on port 9000 with ca.pem as a CA file and expired.pem as a cert file.
Call
client_encryption.createDataKey()
with "aws" as the provider and the following masterKey:{ "region": "us-east-1", "key": "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0", "endpoint": "127.0.0.1:9000", }
Expect this to fail with an exception with a message referencing an expired certificate. This message will be language dependent. In Python, this message is "certificate verify failed: certificate has expired". In Go, this message is "certificate has expired or is not yet valid". If the language of implementation has a single, generic error message for all certificate validation errors, drivers may inspect other fields of the error to verify its meaning.
Start a mock KMS server on port 9001 with ca.pem as a CA file and wrong-host.pem as a cert file.
Call
client_encryption.createDataKey()
with "aws" as the provider and the following masterKey:{ "region": "us-east-1", "key": "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0", "endpoint": "127.0.0.1:9001", }
Expect this to fail with an exception with a message referencing an incorrect or unexpected host. This message will be language dependent. In Python, this message is "certificate verify failed: IP address mismatch, certificate is not valid for '127.0.0.1'". In Go, this message is "cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs". If the language of implementation has a single, generic error message for all certificate validation errors, drivers may inspect other fields of the error to verify its meaning.
Start a mongod
process with server version 4.1.9 or later.
Four mock KMS server processes must be running:
The mock KMS HTTP server.
Run on port 9000 with ca.pem as a CA file and expired.pem as a cert file.
Example:
python -u kms_http_server.py --ca_file ../x509gen/ca.pem --cert_file ../x509gen/expired.pem --port 9000
The mock KMS HTTP server.
Run on port 9001 with ca.pem as a CA file and wrong-host.pem as a cert file.
Example:
python -u kms_http_server.py --ca_file ../x509gen/ca.pem --cert_file ../x509gen/wrong-host.pem --port 9001
The mock KMS HTTP server.
Run on port 9002 with ca.pem as a CA file and server.pem as a cert file.
Run with the
--require_client_cert
option.Example:
python -u kms_http_server.py --ca_file ../x509gen/ca.pem --cert_file ../x509gen/server.pem --port 9002 --require_client_cert
The mock KMS KMIP server.
Create the following four ClientEncryption
objects.
Configure each with keyVaultNamespace
set to keyvault.datakeys
, and a default MongoClient as the keyVaultClient
.
Create a
ClientEncryption
object namedclient_encryption_no_client_cert
with the following KMS providers:{ "aws": { "accessKeyId": <set from environment>, "secretAccessKey": <set from environment> }, "azure": { "tenantId": <set from environment>, "clientId": <set from environment>, "clientSecret": <set from environment>, "identityPlatformEndpoint": "127.0.0.1:9002" }, "gcp": { "email": <set from environment>, "privateKey": <set from environment>, "endpoint": "127.0.0.1:9002" }, "kmip" { "endpoint": "127.0.0.1:5698" } }
Add TLS options for the
aws
,azure
,gcp
, andkmip
providers to use the following options:tlsCAFile
(or equivalent) set to ca.pem. This MAY be configured system-wide.
Create a
ClientEncryption
object namedclient_encryption_with_tls
with the following KMS providers:{ "aws": { "accessKeyId": <set from environment>, "secretAccessKey": <set from environment> }, "azure": { "tenantId": <set from environment>, "clientId": <set from environment>, "clientSecret": <set from environment>, "identityPlatformEndpoint": "127.0.0.1:9002" }, "gcp": { "email": <set from environment>, "privateKey": <set from environment>, "endpoint": "127.0.0.1:9002" }, "kmip" { "endpoint": "127.0.0.1:5698" } }
Add TLS options for the
aws
,azure
,gcp
, andkmip
providers to use the following options:tlsCAFile
(or equivalent) set to ca.pem. This MAY be configured system-wide.tlsCertificateKeyFile
(or equivalent) set to client.pem
Create a
ClientEncryption
object namedclient_encryption_expired
with the following KMS providers:{ "aws": { "accessKeyId": <set from environment>, "secretAccessKey": <set from environment> }, "azure": { "tenantId": <set from environment>, "clientId": <set from environment>, "clientSecret": <set from environment>, "identityPlatformEndpoint": "127.0.0.1:9000" }, "gcp": { "email": <set from environment>, "privateKey": <set from environment>, "endpoint": "127.0.0.1:9000" }, "kmip" { "endpoint": "127.0.0.1:9000" } }
Add TLS options for the
aws
,azure
,gcp
, andkmip
providers to use the following options:tlsCAFile
(or equivalent) set to ca.pem. This MAY be configured system-wide.
Create a
ClientEncryption
object namedclient_encryption_invalid_hostname
with the following KMS providers:{ "aws": { "accessKeyId": <set from environment>, "secretAccessKey": <set from environment> }, "azure": { "tenantId": <set from environment>, "clientId": <set from environment>, "clientSecret": <set from environment>, "identityPlatformEndpoint": "127.0.0.1:9001" }, "gcp": { "email": <set from environment>, "privateKey": <set from environment>, "endpoint": "127.0.0.1:9001" }, "kmip" { "endpoint": "127.0.0.1:9001" } }
Add TLS options for the
aws
,azure
,gcp
, andkmip
providers to use the following options:tlsCAFile
(or equivalent) set to ca.pem. This MAY be configured system-wide.
Call client_encryption_no_client_cert.createDataKey() with "aws" as the provider and the following masterKey:
{
region: "us-east-1",
key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0"
endpoint: "127.0.0.1:9002"
}
Expect an error indicating TLS handshake failed.
Call client_encryption_with_tls.createDataKey() with "aws" as the provider and the following masterKey:
{
region: "us-east-1",
key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0"
endpoint: "127.0.0.1:9002"
}
Expect an error from libmongocrypt with a message containing the string: "parse error". This implies TLS handshake succeeded.
Call client_encryption_expired.createDataKey() with "aws" as the provider and the following masterKey:
{
region: "us-east-1",
key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0"
endpoint: "127.0.0.1:9000"
}
Expect an error indicating TLS handshake failed due to an expired certificate.
Call client_encryption_invalid_hostname.createDataKey() with "aws" as the provider and the following masterKey:
{
region: "us-east-1",
key: "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0"
endpoint: "127.0.0.1:9001"
}
Expect an error indicating TLS handshake failed due to an invalid hostname.
Call client_encryption_no_client_cert.createDataKey() with "azure" as the provider and the following masterKey:
{ 'keyVaultEndpoint': 'doesnotexist.local', 'keyName': 'foo' }
Expect an error indicating TLS handshake failed.
Call client_encryption_with_tls.createDataKey() with "azure" as the provider and the same masterKey.
Expect an error from libmongocrypt with a message containing the string: "HTTP status=404". This implies TLS handshake succeeded.
Call client_encryption_expired.createDataKey() with "azure" as the provider and the same masterKey.
Expect an error indicating TLS handshake failed due to an expired certificate.
Call client_encryption_invalid_hostname.createDataKey() with "azure" as the provider and the same masterKey.
Expect an error indicating TLS handshake failed due to an invalid hostname.
Call client_encryption_no_client_cert.createDataKey() with "gcp" as the provider and the following masterKey:
{ 'projectId': 'foo', 'location': 'bar', 'keyRing': 'baz', 'keyName': 'foo' }
Expect an error indicating TLS handshake failed.
Call client_encryption_with_tls.createDataKey() with "gcp" as the provider and the same masterKey.
Expect an error from libmongocrypt with a message containing the string: "HTTP status=404". This implies TLS handshake succeeded.
Call client_encryption_expired.createDataKey() with "gcp" as the provider and the same masterKey.
Expect an error indicating TLS handshake failed due to an expired certificate.
Call client_encryption_invalid_hostname.createDataKey() with "gcp" as the provider and the same masterKey.
Expect an error indicating TLS handshake failed due to an invalid hostname.
Call client_encryption_no_client_cert.createDataKey() with "kmip" as the provider and the following masterKey:
{ }
Expect an error indicating TLS handshake failed.
Call client_encryption_with_tls.createDataKey() with "kmip" as the provider and the same masterKey.
Expect success.
Call client_encryption_expired.createDataKey() with "kmip" as the provider and the same masterKey.
Expect an error indicating TLS handshake failed due to an expired certificate.
Call client_encryption_invalid_hostname.createDataKey() with "kmip" as the provider and the same masterKey.
Expect an error indicating TLS handshake failed due to an invalid hostname.
The Explicit Encryption tests require MongoDB server 6.0+. The tests must not run against a standalone.
Before running each of the following test cases, perform the following Test Setup.
Load the file encryptedFields.json as encryptedFields
.
Load the file key1-document.json as key1Document
.
Read the "_id"
field of key1Document
as key1ID
.
Drop and create the collection db.explicit_encryption
using encryptedFields
as an option. See FLE 2 CreateCollection() and Collection.Drop().
Drop and create the collection keyvault.datakeys
.
Insert key1Document
in keyvault.datakeys
with majority write concern.
Create a MongoClient named keyVaultClient
.
Create a ClientEncryption object named clientEncryption
with these options:
ClientEncryptionOpts {
keyVaultClient: <keyVaultClient>;
keyVaultNamespace: "keyvault.datakeys";
kmsProviders: { "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
}
Create a MongoClient named encryptedClient
with these AutoEncryptionOpts
:
AutoEncryptionOpts {
keyVaultNamespace: "keyvault.datakeys";
kmsProviders: { "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
bypassQueryAnalysis: true
}
Use clientEncryption
to encrypt the value "encrypted indexed value" with these EncryptOpts
:
class EncryptOpts {
keyId : <key1ID>
algorithm: "Indexed",
contentionFactor: 0
}
Store the result in insertPayload
.
Use encryptedClient
to insert the document { "encryptedIndexed": <insertPayload> }
into db.explicit_encryption
.
Use clientEncryption
to encrypt the value "encrypted indexed value" with these EncryptOpts
:
class EncryptOpts {
keyId : <key1ID>
algorithm: "Indexed",
queryType: "equality",
contentionFactor: 0
}
Store the result in findPayload
.
Use encryptedClient
to run a "find" operation on the db.explicit_encryption
collection with the filter { "encryptedIndexed": <findPayload> }
.
Assert one document is returned containing the field { "encryptedIndexed": "encrypted indexed value" }
.
Use clientEncryption
to encrypt the value "encrypted indexed value" with these EncryptOpts
:
class EncryptOpts {
keyId : <key1ID>
algorithm: "Indexed",
contentionFactor: 10
}
Store the result in insertPayload
.
Use encryptedClient
to insert the document { "encryptedIndexed": <insertPayload> }
into db.explicit_encryption
.
Repeat the above steps 10 times to insert 10 total documents. The insertPayload
must be regenerated each iteration.
Use clientEncryption
to encrypt the value "encrypted indexed value" with these EncryptOpts
:
class EncryptOpts {
keyId : <key1ID>
algorithm: "Indexed",
queryType: "equality",
contentionFactor: 0
}
Store the result in findPayload
.
Use encryptedClient
to run a "find" operation on the db.explicit_encryption
collection with the filter { "encryptedIndexed": <findPayload> }
.
Assert less than 10 documents are returned. 0 documents may be returned. Assert each returned document contains the field { "encryptedIndexed": "encrypted indexed value" }
.
Use clientEncryption
to encrypt the value "encrypted indexed value" with these EncryptOpts
:
class EncryptOpts {
keyId : <key1ID>
algorithm: "Indexed",
queryType: "equality",
contentionFactor: 10
}
Store the result in findPayload2
.
Use encryptedClient
to run a "find" operation on the db.explicit_encryption
collection with the filter { "encryptedIndexed": <findPayload2> }
.
Assert 10 documents are returned. Assert each returned document contains the field { "encryptedIndexed": "encrypted indexed value" }
.
Use clientEncryption
to encrypt the value "encrypted unindexed value" with these EncryptOpts
:
class EncryptOpts {
keyId : <key1ID>
algorithm: "Unindexed"
}
Store the result in insertPayload
.
Use encryptedClient
to insert the document { "_id": 1, "encryptedUnindexed": <insertPayload> }
into db.explicit_encryption
.
Use encryptedClient
to run a "find" operation on the db.explicit_encryption
collection with the filter { "_id": 1 }
.
Assert one document is returned containing the field { "encryptedUnindexed": "encrypted unindexed value" }
.
Use clientEncryption
to encrypt the value "encrypted indexed value" with these EncryptOpts
:
class EncryptOpts {
keyId : <key1ID>
algorithm: "Indexed",
contentionFactor: 0
}
Store the result in payload
.
Use clientEncryption
to decrypt payload
. Assert the returned value equals "encrypted indexed value".
Use clientEncryption
to encrypt the value "encrypted unindexed value" with these EncryptOpts
:
class EncryptOpts {
keyId : <key1ID>
algorithm: "Unindexed",
}
Store the result in payload
.
Use clientEncryption
to decrypt payload
. Assert the returned value equals "encrypted unindexed value".
The following setup must occur before running each of the following test cases.
- Create a
MongoClient
object (referred to asclient
). - Using
client
, drop the collectionkeyvault.datakeys
. - Using
client
, create a unique index onkeyAltNames
with a partial index filter for only documents wherekeyAltNames
exists using writeConcern "majority".
The command should be equivalent to:
db.runCommand(
{
createIndexes: "datakeys",
indexes: [
{
name: "keyAltNames_1",
key: { "keyAltNames": 1 },
unique: true,
partialFilterExpression: { keyAltNames: { $exists: true } }
}
],
writeConcern: { w: "majority" }
}
)
- Create a
ClientEncryption
object (referred to asclient_encryption
) withclient
set as thekeyVaultClient
. - Using
client_encryption
, create a data key with alocal
KMS provider and the keyAltName "def".
- Use
client_encryption
to create a new local data key with a keyAltName "abc" and assert the operation does not fail. - Repeat Step 1 and assert the operation fails due to a duplicate key server error (error code 11000).
- Use
client_encryption
to create a new local data key with a keyAltName "def" and assert the operation fails due to a duplicate key server error (error code 11000).
- Use
client_encryption
to create a new local data key and assert the operation does not fail. - Use
client_encryption
to add a keyAltName "abc" to the key created in Step 1 and assert the operation does not fail. - Repeat Step 2, assert the operation does not fail, and assert the returned key document contains the keyAltName "abc" added in Step 2.
- Use
client_encryption
to add a keyAltName "def" to the key created in Step 1 and assert the operation fails due to a duplicate key server error (error code 11000). - Use
client_encryption
to add a keyAltName "def" to the existing key, assert the operation does not fail, and assert the returned key document contains the keyAltName "def" added during Setup.
Before running each of the following test cases, perform the following Test Setup.
Create a MongoClient named setupClient
.
Drop and create the collection db.decryption_events
.
Create a ClientEncryption object named clientEncryption
with these options:
ClientEncryptionOpts {
keyVaultClient: <setupClient>,
keyVaultNamespace: "keyvault.datakeys",
kmsProviders: { "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
}
Create a data key with the "local" KMS provider. Storing the result in a variable named keyID
.
Use clientEncryption
to encrypt the string "hello" with the following EncryptOpts
:
EncryptOpts {
keyId: <keyID>,
algorithm: "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic"
}
Store the result in a variable named ciphertext
.
Copy ciphertext
into a variable named malformedCiphertext
. Change the
last byte to a different value. This will produce an invalid HMAC tag.
Create a MongoClient named encryptedClient
with these AutoEncryptionOpts
:
AutoEncryptionOpts {
keyVaultNamespace: "keyvault.datakeys";
kmsProviders: { "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
}
Configure encryptedClient
with "retryReads=false".
Register a listener for CommandSucceeded events on encryptedClient
.
The listener must store the most recent CommandSucceededEvent
reply for the "aggregate" command.
The listener must store the most recent CommandFailedEvent
error for the "aggregate" command.
Use setupClient
to configure the following failpoint:
{
"configureFailPoint": "failCommand",
"mode": {
"times": 1
},
"data": {
"errorCode": 123,
"failCommands": [
"aggregate"
]
}
}
Use encryptedClient
to run an aggregate on db.decryption_events
.
Expect an exception to be thrown from the command error. Expect a CommandFailedEvent
.
Use setupClient
to configure the following failpoint:
{
"configureFailPoint": "failCommand",
"mode": {
"times": 1
},
"data": {
"errorCode": 123,
"closeConnection": true,
"failCommands": [
"aggregate"
]
}
}
Use encryptedClient
to run an aggregate on db.decryption_events
.
Expect an exception to be thrown from the network error. Expect a CommandFailedEvent
.
Use encryptedClient
to insert the document { "encrypted": <malformedCiphertext> }
into db.decryption_events
.
Use encryptedClient
to run an aggregate on db.decryption_events
with an empty pipeline.
Expect an exception to be thrown from the decryption error.
Expect a CommandSucceededEvent
. Expect the CommandSucceededEvent.reply
to contain BSON binary for the field cursor.firstBatch.encrypted
.
Use encryptedClient
to insert the document { "encrypted": <ciphertext> }
into db.decryption_events
.
Use encryptedClient
to run an aggregate on db.decryption_events
with an empty pipeline.
Expect no exception.
Expect a CommandSucceededEvent
. Expect the CommandSucceededEvent.reply
to contain BSON binary for the field cursor.firstBatch.encrypted
.
These tests require valid AWS credentials. Refer: Automatic AWS Credentials.
For these cases, create a ClientEncryption object C with the following options:
ClientEncryptionOpts {
keyVaultClient: <setupClient>,
keyVaultNamespace: "keyvault.datakeys",
kmsProviders: { "aws": {} },
}
Do not run this test case in an environment where AWS credentials are available (e.g. via environment variables or a metadata URL). (Refer: Obtaining credentials for AWS)
Attempt to create a datakey with C using the "aws"
KMS provider.
Expect this to fail due to a lack of KMS provider credentials.
For this test case, the environment variables AWS_ACCESS_KEY_ID
and
AWS_SECRET_ACCESS_KEY
must be defined and set to a valid set of AWS
credentials.
Use the client encryption to create a datakey using the "aws"
KMS provider.
This should successfully load and use the AWS credentials that were defined in
the environment.
When the following test case requests setting masterKey
, use the following values based on the KMS provider:
For "aws":
{
"region": "us-east-1",
"key": "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0"
}
For "azure":
{
"keyVaultEndpoint": "key-vault-csfle.vault.azure.net",
"keyName": "key-name-csfle"
}
For "gcp":
{
"projectId": "devprod-drivers",
"location": "global",
"keyRing": "key-ring-csfle",
"keyName": "key-name-csfle"
}
For "kmip":
{}
For "local", do not set a masterKey document.
Run the following test case for each pair of KMS providers (referred to as srcProvider
and dstProvider
).
Include pairs where srcProvider
equals dstProvider
.
Drop the collection
keyvault.datakeys
.Create a
ClientEncryption
object namedclientEncryption1
with these options: .. code:: typescript- ClientEncryptionOpts {
keyVaultClient: <new MongoClient>; keyVaultNamespace: "keyvault.datakeys"; kmsProviders: <all KMS providers>
}
Call
clientEncryption1.createDataKey
withsrcProvider
and these options: .. code:: typescript- class DataKeyOpts {
masterKey: <depends on srcProvider>
}
Store the return value in
keyID
.Call
clientEncryption1.encrypt
with the value "test" and these options: .. code:: typescript- class EncryptOpts {
keyId : keyID, algorithm: "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic"
}
Store the return value in
ciphertext
.Create a
ClientEncryption
object namedclientEncryption2
with these options: .. code:: typescript- ClientEncryptionOpts {
keyVaultClient: <new MongoClient>; keyVaultNamespace: "keyvault.datakeys"; kmsProviders: <all KMS providers>
}
Call
clientEncryption2.rewrapManyDataKey
with an emptyfilter
and these options:class RewrapManyDataKeyOpts { provider: dstProvider masterKey: <depends on dstProvider> }
Assert that the returned
RewrapManyDataKeyResult.bulkWriteResult.modifiedCount
is 1.Call
clientEncryption1.decrypt
with theciphertext
. Assert the return value is "test".Call
clientEncryption2.decrypt
with theciphertext
. Assert the return value is "test".