Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MOTOR-689: Add async wrapper for pymongo.encryption.ClientEncryption #103

Merged
merged 44 commits into from
Mar 31, 2021
Merged
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
ee7b490
added wrapper for Asyncio explicit encryption
guanlinzhou Mar 25, 2021
e252b38
add docs
guanlinzhou Mar 25, 2021
8fed6c1
syntax mistake
guanlinzhou Mar 25, 2021
1ef5c2a
add docstring
guanlinzhou Mar 25, 2021
3e11eb7
add tornado support
guanlinzhou Mar 25, 2021
dea8025
shorten docstring
guanlinzhou Mar 25, 2021
9a60728
expose close api
guanlinzhou Mar 26, 2021
75d2fdc
checkout
guanlinzhou Mar 29, 2021
9a58140
checkout
guanlinzhou Mar 29, 2021
a3575c0
get tests working
guanlinzhou Mar 29, 2021
bf05409
remove .eggs
guanlinzhou Mar 29, 2021
13f4cdb
newline
guanlinzhou Mar 29, 2021
4254aae
eof
guanlinzhou Mar 29, 2021
efddbe8
eof
guanlinzhou Mar 29, 2021
d3a73ab
cleanup calls back
guanlinzhou Mar 29, 2021
0c0e70b
cleanup calls back
guanlinzhou Mar 29, 2021
c6c7edd
updated travis config for installing pymongocrypt
guanlinzhou Mar 29, 2021
7f9b84a
add setup
guanlinzhou Mar 29, 2021
aea3442
augment evergreen testing
guanlinzhou Mar 29, 2021
0b2ca3b
fix config
guanlinzhou Mar 29, 2021
05050b1
fix test script
guanlinzhou Mar 30, 2021
180509b
virtualenv eg
guanlinzhou Mar 30, 2021
68bc359
fix
guanlinzhou Mar 30, 2021
b4849e0
fix python binary
guanlinzhou Mar 30, 2021
f2615ed
replace with pip install
guanlinzhou Mar 30, 2021
bb4cbe8
test if extra commands are needed
guanlinzhou Mar 30, 2021
310ff1c
test fix
guanlinzhou Mar 30, 2021
bcff10c
use createvenv
guanlinzhou Mar 30, 2021
29b0880
fix
guanlinzhou Mar 30, 2021
e0337ca
add testing lines
guanlinzhou Mar 30, 2021
d569bd0
install through tox
guanlinzhou Mar 30, 2021
ef043bb
restore dsi files
guanlinzhou Mar 30, 2021
e093948
replace config.yml
guanlinzhou Mar 30, 2021
64f68d7
remove extra newline
guanlinzhou Mar 30, 2021
a9b976b
nits
guanlinzhou Mar 30, 2021
7011887
Merge branch 'master' of github.com:mongodb/motor into PYTHON-689
guanlinzhou Mar 30, 2021
c4e81a3
update installation info
guanlinzhou Mar 30, 2021
9a03574
link to pymongocrypt
guanlinzhou Mar 30, 2021
171becc
single source python dependency
guanlinzhou Mar 31, 2021
26cf82f
extra
guanlinzhou Mar 31, 2021
81f87cd
revert changes
guanlinzhou Mar 31, 2021
3f5f880
fix failure with setUp
guanlinzhou Mar 31, 2021
87e711f
fix failure with setUp
guanlinzhou Mar 31, 2021
8f6026e
update older pip version
guanlinzhou Mar 31, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ services: mongodb

install:
- pip install tornado
- pip install -e '.[encryption]'

script: "python setup.py test"

Expand Down
7 changes: 7 additions & 0 deletions doc/api-asyncio/asyncio_motor_client_encryption.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
:class:`~motor.motor_asyncio.AsyncIOMotorClientEncryption`
==========================================================

.. currentmodule:: motor.motor_asyncio

.. autoclass:: AsyncIOMotorClientEncryption
:members:
1 change: 1 addition & 0 deletions doc/api-asyncio/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Motor asyncio API
asyncio_motor_database
asyncio_motor_collection
asyncio_motor_change_stream
asyncio_motor_client_encryption
cursors
asyncio_gridfs
aiohttp
Expand Down
1 change: 1 addition & 0 deletions doc/api-tornado/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Motor Tornado API
motor_database
motor_collection
motor_change_stream
motor_client_encryption
cursors
gridfs
web
Expand Down
7 changes: 7 additions & 0 deletions doc/api-tornado/motor_client_encryption.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
:class:`~motor.motor_tornado.MotorClientEncryption`
===================================================

.. currentmodule:: motor.motor_tornado

.. autoclass:: MotorClientEncryption
:members:
2 changes: 1 addition & 1 deletion doc/examples/encryption.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ you will need to install the
as well as the driver itself. Install both the driver and a compatible
version of pymongocrypt like this::

$ python -m pip install 'pymongo[encryption]'
$ python -m pip install 'motor[encryption]'

Note that installing on Linux requires pip 19 or later for manylinux2010 wheel
support. For more information about installing pymongocrypt see
Expand Down
49 changes: 48 additions & 1 deletion motor/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
from pymongo.cursor import Cursor, RawBatchCursor, _QUERY_OPTIONS
from pymongo.database import Database
from pymongo.driver_info import DriverInfo
from pymongo.encryption import ClientEncryption

from . import version as motor_version
from .metaprogramming import (AsyncCommand,
Expand Down Expand Up @@ -140,7 +141,7 @@ def __init__(self, *args, **kwargs):

:Parameters:
- `io_loop` (optional): Special event loop
instance to use instead of default
instance to use instead of default.
"""
if 'io_loop' in kwargs:
io_loop = kwargs.pop('io_loop')
Expand Down Expand Up @@ -1795,3 +1796,49 @@ def __enter__(self):

def __exit__(self, exc_type, exc_val, exc_tb):
pass

class AgnosticClientEncryption(AgnosticBase):
guanlinzhou marked this conversation as resolved.
Show resolved Hide resolved
"""Explicit client-side field level encryption."""

__motor_class_name__ = 'MotorClientEncryption'
__delegate_class__ = ClientEncryption

create_data_key = AsyncCommand(doc=create_data_key_doc)
encrypt = AsyncCommand()
decrypt = AsyncCommand()
close = AsyncCommand(doc=close_doc)

def __init__(self, kms_providers, key_vault_namespace, key_vault_client, codec_options, io_loop=None):
guanlinzhou marked this conversation as resolved.
Show resolved Hide resolved
"""Explicit client-side field level encryption.

Takes the same constructor arguments as
:class:`pymongo.encryption.ClientEncryption`, as well as:

:Parameters:
- `io_loop` (optional): Special event loop
instance to use instead of default.
"""
if io_loop:
self._framework.check_event_loop(io_loop)
else:
io_loop = self._framework.get_event_loop()
sync_client = key_vault_client.delegate
delegate = self.__delegate_class__(kms_providers, key_vault_namespace, sync_client, codec_options)
super().__init__(delegate)
self.io_loop = io_loop

def get_io_loop(self):
return self.io_loop

async def __aenter__(self):
return self

async def __aexit__(self, exc_type, exc_val, exc_tb):
if self.delegate:
await self.close()

def __enter__(self):
raise RuntimeError('Use {} in "async with", not "with"'.format(self.__class__.__name__))

def __exit__(self, exc_type, exc_val, exc_tb):
pass
24 changes: 24 additions & 0 deletions motor/docstrings.py
Original file line number Diff line number Diff line change
Expand Up @@ -1268,3 +1268,27 @@ async def coro():
.. _$expr: https://docs.mongodb.com/manual/reference/operator/query/expr/
.. _$where: https://docs.mongodb.com/manual/reference/operator/query/where/
"""

create_data_key_doc = """Create and insert a new data key into the key vault collection.

Takes the same arguments as
:class:`pymongo.encryption.ClientEncryption.create_data_key`,
with only the following slight difference using async syntax.
The following example shows creating and referring to a data
key by alternate name::

await client_encryption.create_data_key("local", keyAltNames=["name1"])
# reference the key with the alternate name
await client_encryption.encrypt("457-55-5462", keyAltName="name1",
algorithm=Algorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Random)
"""

close_doc = """Release resources.

Note that using this class in a with-statement will automatically call
:meth:`close`::

async with AsyncIOMotorClientEncryption(...) as client_encryption:
encrypted = await client_encryption.encrypt(value, ...)
decrypted = await client_encryption.decrypt(encrypted)
"""
6 changes: 5 additions & 1 deletion motor/motor_asyncio.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
from .frameworks import asyncio as asyncio_framework
guanlinzhou marked this conversation as resolved.
Show resolved Hide resolved
from .metaprogramming import create_class_with_framework

__all__ = ['AsyncIOMotorClient']
__all__ = ['AsyncIOMotorClient','AsyncIOMotorClientEncryption']


def create_asyncio_class(cls):
Expand Down Expand Up @@ -70,3 +70,7 @@ def create_asyncio_class(cls):

AsyncIOMotorGridOutCursor = create_asyncio_class(
motor_gridfs.AgnosticGridOutCursor)


AsyncIOMotorClientEncryption = create_asyncio_class(
core.AgnosticClientEncryption)
5 changes: 4 additions & 1 deletion motor/motor_tornado.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
from .frameworks import tornado as tornado_framework
from .metaprogramming import create_class_with_framework

__all__ = ['MotorClient']
__all__ = ['MotorClient', 'MotorClientEncryption']


def create_motor_class(cls):
Expand Down Expand Up @@ -60,3 +60,6 @@ def create_motor_class(cls):


MotorGridOutCursor = create_motor_class(motor_gridfs.AgnosticGridOutCursor)


MotorClientEncryption = create_motor_class(core.AgnosticClientEncryption)
5 changes: 5 additions & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@

tests_require = ['mockupdb>=1.4.0']

extras_require = {
'encryption': ['pymongo[encryption]>=3.11,<4'],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add the >=3.11,<4 here? I would hope that the version reqs are added transitively from pymongo in install_requires.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@prashantmital thoughts on this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think extras are applied on top of (i.e. after) install_requiries so we would probably need this. I'll run a quick test and get back to you.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a setup.py that looks like:

setup(
    name='mypackage',
    version='0.1.0',
    packages=find_packages(),
    ext_modules=get_extension_modules(),
    install_requires=['pymongo==3.10'],
    extras_require={'foo': ['pymongo[srv]']},
)

installing the package has a rather strange output log:

➜   pip install -e '.[foo]'
Obtaining file:///Users/pmital/Developer/playgrounds/python-packaging
Collecting pymongo==3.10
  Downloading pymongo-3.10.0-cp37-cp37m-macosx_10_9_x86_64.whl (350 kB)
     |████████████████████████████████| 350 kB 1.6 MB/s 
Collecting pymongo[srv]
  Using cached pymongo-3.11.3-cp37-cp37m-macosx_10_6_intel.whl (414 kB)
  Downloading pymongo-3.11.2-cp37-cp37m-macosx_10_6_intel.whl (414 kB)
     |████████████████████████████████| 414 kB 1.7 MB/s 
  Downloading pymongo-3.11.1-cp37-cp37m-macosx_10_6_intel.whl (414 kB)
     |████████████████████████████████| 414 kB 2.6 MB/s 
  Downloading pymongo-3.11.0-cp37-cp37m-macosx_10_9_x86_64.whl (378 kB)
     |████████████████████████████████| 378 kB 3.2 MB/s 
  Downloading pymongo-3.10.1-cp37-cp37m-macosx_10_9_x86_64.whl (350 kB)
     |████████████████████████████████| 350 kB 3.8 MB/s 
Collecting dnspython<2.0.0,>=1.16.0
  Using cached dnspython-1.16.0-py2.py3-none-any.whl (188 kB)
Installing collected packages: pymongo, dnspython, mypackage
  Attempting uninstall: mypackage
    Found existing installation: mypackage 0.1.0
    Uninstalling mypackage-0.1.0:
      Successfully uninstalled mypackage-0.1.0
  Running setup.py develop for mypackage
Successfully installed dnspython-1.16.0 mypackage pymongo-3.10.0
➜   pip list 
Package    Version Location
---------- ------- ----------------------------------------------------
Cython     0.29.22
dnspython  1.16.0
mypackage  0.1.0   /Users/pmital/Developer/playgrounds/python-packaging
pip        21.0.1
pymongo    3.10.0
setuptools 47.1.0
wheel      0.36.2

for some reason, it downloads all the versions between 3.10 and latest but ends up only installing the one in install_requires. So @ShaneHarvey does seem to be correct in that pip honors the install_requires version. That being said, it might result in longer install times if there are a lot versions to download.

Specifying the version in extras_require eliminates the extra downloads:

➜  pip --no-cache-dir install -e '.[foo]'
Obtaining file:///Users/pmital/Developer/playgrounds/python-packaging
Collecting pymongo==3.10
  Downloading pymongo-3.10.0-cp37-cp37m-macosx_10_9_x86_64.whl (350 kB)
     |████████████████████████████████| 350 kB 2.1 MB/s 
Requirement already satisfied: dnspython<2.0.0,>=1.16.0 in /Users/pmital/.pyenv/versions/3.7.10/envs/packaging/lib/python3.7/site-packages (from pymongo==3.10->mypackage==0.1.0) (1.16.0)
Installing collected packages: pymongo, mypackage
  Running setup.py develop for mypackage
Successfully installed mypackage pymongo-3.10.0

@ShaneHarvey how would you like to proceed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh well. Let's keep >=3.11,<4. We'll just need to ensure those two reqs never drift apart.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see your concern. Let's single-source the PyMongo dependency. Maybe something like:

pymongo_ver = ">=3.11,<4"
install_requires = ["pymongo" + pymongo_ver]
extras_require = {'encryption': ["pymongo" + "[encryption]" + pymongo_ver]}

Thoughts @ShaneHarvey ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes let's do your single-source option.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guanlinzhou can you implement this?

}

class test(Command):
description = "run the tests"

Expand Down Expand Up @@ -138,6 +142,7 @@ def run(self):
url='https://github.com/mongodb/motor/',
python_requires='>=3.5.2',
install_requires=install_requires,
extras_require=extras_require,
license='http://www.apache.org/licenses/LICENSE-2.0',
classifiers=[c for c in classifiers.split('\n') if c],
keywords=[
Expand Down