Move master to python 3 (Recidiviz/recidiviz-data#432)

* Remove urlfetch monkey patch (Recidiviz/recidiviz-data#239)

* migrate from users to cloud iap (Recidiviz/recidiviz-data#241)

* move sessions to new datastore api (Recidiviz/recidiviz-data#236)

Part of Recidiviz/recidiviz-data#176
* move sessions to new datastore api
* Use emulator to test writing and reading sessions
* Move emulator environment vars into a fixture

* remove unnecessary pytest emulator mark (Recidiviz/recidiviz-data#277)

* Move secrets to new datastore api (Recidiviz/recidiviz-data#276)

* Removed local functionality, added new fields. (Recidiviz/recidiviz-data#282)

* add test only annotation (Recidiviz/recidiviz-data#281)

* Migrated push queues from TaskQueues to Cloud Tasks. (Recidiviz/recidiviz-data#323)

* Remove urlfetch monkey patch (Recidiviz/recidiviz-data#239)

* migrate from users to cloud iap (Recidiviz/recidiviz-data#241)

* move sessions to new datastore api (Recidiviz/recidiviz-data#236)

Part of Recidiviz/recidiviz-data#176
* move sessions to new datastore api
* Use emulator to test writing and reading sessions
* Move emulator environment vars into a fixture

* remove unnecessary pytest emulator mark (Recidiviz/recidiviz-data#277)

* Move secrets to new datastore api (Recidiviz/recidiviz-data#276)

* Removed local functionality, added new fields. (Recidiviz/recidiviz-data#282)

* Migrated scraper.py and worker.py from TaskQueues to Cloud Tasks.

* Added google-cloud-tasks to Pipenv.

* Fixed Worker tests, removed App Engine testbed.

* Migrated scraper_tests from TaskQueues to Cloud Tasks.

* Use same queue_path as scrapers.

* Use same queue_path as scrapers for remaining calls.

* Added APPLICATION_ID to Travis env.

* Migrated us_vt_scraper_test from TaskQueue to Cloud Tasks.

* Removed extra newline.

* Migrated us_ny_scraper_test from TaskQueue to Cloud Tasks.

* Commenting out unused tests to satisfy pylint.

* Deleting unused code to please pylint.

* Moved push queue operations to their own module.

* Removed taskqueue testbed stuff.

* Moved queues_test.py to recidiviz/tests/ingest/

* Switched from camelCase to lower_with_under naming for tests.

* Refactor docket to use pubsub (Recidiviz/recidiviz-data#335)

* move docket to pubsub

* remove snapshot logic from docket

* Fix python3 build (Recidiviz/recidiviz-data#352)

* fix infer release usage of sessions

* remove app identity usage from tests

* fix get closed only sessions

* remove entity kinds from regions.yaml (Recidiviz/recidiviz-data#360)

also removes related code that uses the entity kinds, most notable the
test data populator and `compare_and_set_snapshot` from scraper.

* remove us-ny specific models (Recidiviz/recidiviz-data#374)

* Fix python build

* Remove deferred (Recidiviz/recidiviz-data#369)

* Remove uses of deferred throughout

* adding to docket should use pubsub batching

* add comment about name query

* add another test

* Remove appengine library use (Recidiviz/recidiviz-data#413)

* Remove lib directory (Recidiviz/recidiviz-data#411)

* Switch to python 3 interpreter (Recidiviz/recidiviz-data#421)

* Switch to python 3 interpreter

* don't cache packages for now

* pin pyyaml to 3.13 to fix breakage

yaml/pyyaml#126

* fix pylint

* remove object inheritance

* read names file in text mode

* fix lint errors

* update requirements.txt

* re-enable pip caching on travis

* fix queue test

* fix more

* Add pipfile to gcloudignore

* Fix pipfile gcloudignore

* Remove `base.yaml`

* Remove defunct datastore indices

* Manually setup cloud logging client

* Update `in_prod` to check custom env var

* Add google-cloud-logging to pipfile/requirements

* Fix environment test

* Rewrite docket to avoid repeated get calls

* fix docket test

* remove parens from class defs

* decode bytes

* update metaclass syntax for scraper

* update persistence imports

* fix auth test

* pylint disable unused __new__ buildable_attr

* remove app id env var

* weird compression encode/decode

* fix compression encoding / decoding

* last fixes (Recidiviz/recidiviz-data#431)

* Add pymysql to pipenv & requirements

* Remove missed empty parens on classes

* Add note about pubsub emulator for tests

* Fix comment in sessions

* Fix some comments

GitOrigin-RevId: 6c57a9dc530fbc3df3fafafa31158c0cf5b30682

.gcloudignore

@@ -0,0 +1,2 @@
+local.yaml
+Pipfile*

.pylintrc

@@ -263,7 +263,7 @@ max-public-methods=20
 
 # List of interface methods to ignore, separated by a comma. This is used for
 # instance to not check methods defines in Zope's Interface base class.
-#ignore-iface-methods=isImplementedBy,deferred,extends,names,namesAndDescriptions,queryDescriptionFor,getBases,getDescriptionFor,getDoc,getName,getTaggedValue,getTaggedValueTags,isEqualOrExtendedBy,setTaggedValue,isImplementedByInstancesOf,adaptWith,is_implemented_by
+#ignore-iface-methods=isImplementedBy,extends,names,namesAndDescriptions,queryDescriptionFor,getBases,getDescriptionFor,getDoc,getName,getTaggedValue,getTaggedValueTags,isEqualOrExtendedBy,setTaggedValue,isImplementedByInstancesOf,adaptWith,is_implemented_by
 
 # List of method names used to declare (i.e. assign) instance attributes.
 defining-attr-methods=__init__,__new__,setUp

.travis.yml

@@ -1,12 +1,23 @@
 language: python
 cache: pip
 python:
-- '2.7'
+- '3.7-dev'
+before_install:
+- export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && 
+    echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && 
+    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && 
+    sudo apt-get update -y && sudo apt-get install google-cloud-sdk -y &&
+    sudo apt-get install google-cloud-sdk-datastore-emulator -y &&
+    sudo apt-get install google-cloud-sdk-pubsub-emulator -y
 install:
+- pip install --upgrade setuptools
 - pip install -r requirements.txt
+before_script: 
+- gcloud beta emulators datastore start --no-store-on-disk --consistency 1.0 --project test-project > ds_emulator.out 2> ds_emulator.err &
+- gcloud beta emulators pubsub start --project test-project > ps_emulator.out 2> ps_emulator.err &
 script:
 - pylint *.py recidiviz
-- pytest recidiviz/tests/
+- pytest recidiviz/tests/ --with-emulator
 after_success:
 - coveralls
 notifications:

Pipfile

@@ -5,8 +5,7 @@ name = "pypi"
 
 [packages]
 altgraph = "==0.10.2"
-appengine-sdk = "==1.9.75.post0"
-astroid = "==1.6.3"
+astroid = "==2.1.0"
 attrs = "~=18.2.0"
 beautifulsoup4 = "==4.6.0"
 callee = "==0.3"
@@ -18,7 +17,6 @@ coveralls = "==1.3.0"
 cssselect = "==1.0.3"
 enum34 = "==1.1.6"
 funcsigs = "==1.0.2"
-futures = "==3.2.0"
 idna = "==2.6"
 isort = "==4.3.4"
 lazy-object-proxy = "==1.3.1"
@@ -33,15 +31,14 @@ pbr = "==4.0.2"
 pluggy = "==0.6.0"
 py = "==1.5.3"
 py2app = "==0.7.3"
-pylint = "==1.8.4"
+pylint = "==2.2.2"
 pyparsing = "==2.0.1"
 pytest = "==3.5.0"
 pytest-cov = "==2.5.1"
 pytest-mock = "==1.10.0"
 python-dateutil = "==2.7.5"
 pytz = "==2013.7"
 requests = "==2.20.0"
-requests-toolbelt = "==0.8.0"
 responses = "==0.9.0"
 singledispatch = "==3.4.0.3"
 six = "==1.11.0"
@@ -52,12 +49,12 @@ urllib3 = {extras = ["secure"],version = "==1.23"}
 virtualenv = "==15.2.0"
 waitress = "==1.1.0"
 wrapt = "==1.10.11"
-xattr = "==0.6.4"
+xattr = "*"
 backports_abc = "==0.5"
 "backports.functools_lru_cache" = "==1.5"
 bdist_mpkg = "==0.5.0"
 pyOpenSSL = "==18.0.0"
-PyYAML = "==3.12"
+pyyaml = "==3.13"
 WebOb = "==1.8.1"
 "zope.interface" = "==4.1.1"
 flask = "==1.0.2"
@@ -82,10 +79,19 @@ pycparser = "==2.19"
 regex = "==2018.11.22"
 tzlocal = "==1.5.1"
 werkzeug = "==0.14.1"
+pyjwt = "*"
+google-cloud-datastore = "*"
+google-cloud-tasks = "*"
+google-cloud-pubsub = "*"
+google-api-core = "*"
+grpcio = "*"
+gunicorn = "*"
+google-cloud-logging = "*"
+pymysql = "*"
 
 [dev-packages]
 rope = "*"
 "autopep8" = "*"
 
 [requires]
-python_version = "2.7"
+python_version = "3.7"