[CUMULUS-3056] Setup for testing Postgres LIST queries (#3127)

* initial commit * add postgres query to LIST endpoints * add tests for granules and collections * Release 13.3.x with master (#3121) * Release 13.3.1 (#3106) * merge CUMULUS-2557 * merge CUMULUS-2971 * CUMULUS-3021: Support Collections with a collection.version containing slashes (#3096) * Support Collections with a collection.version containing slashes * merge CUMULUS-3021 * merge CUMULUS-3024 * bump version to 13.3.1 * update documentation * update changelog * remove duplicate api user * Sort results in unit test before comparison Co-authored-by: Charles Huang <charleshuang80@users.noreply.github.com> Co-authored-by: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> * CUMULUS-3027 -- Tightly constrain typescript version due to knex typing (#3095) * Tightly constrain typescript verseion due to knex typing Unpinned dependency resulted in typescript 4.8 being installed, however knex (all versions) has an outstanding bug: knex/knex#5279 * Add CHANGELOG 🔔 * Update generate-ts-build-cache to always 'npm install' * Update CHANGELOG * Release 13.3.2 (#3117) * Update CNM lambdas to utilize newer releases (#3094) * 13.3.2 version bump * update changelog Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> * update changelog * re-add comma Co-authored-by: Charles Huang <charleshuang80@users.noreply.github.com> Co-authored-by: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> * fix lint error * CUMULUS-3104:Update async operation image Dockerfile (#3130) * CUMULUS-3104:Update async operation image Dockerfile to use node:14.19.3-buster Upgraded saml2-js 4.0.0, rewire to 6.0.0 to address security vulnerabilities * add xmldom to audit allowlist * update default async_operation_image * update changelog skip-integration-tests * CUMULUS-2944: granules/bulkDelete endpoint has PayloadTooLargeError for 4k granules (#3122) * Update body parser configuration to increase limit * Add unit test, update changelog * Update unit test and fix lint * Fix lint errors * Fix lint error * Relocate granulesList.js and update reference in unit test * Update helper function to dynamically generate IDs and move to different file. Update unit test * Fix lint errors * Increase limit to match lambda limit * CUMULUS-3102: Update broken link (#3125) * Fix docs that has bad link * Update url to reference markdown file directly * CUMULUS-3104: Fixed TS compilation error caused by @aws-sdk/client-s3 upgrade (#3132) * Fixed TS compilation error caused by @aws-sdk/client-s3 3.190->3.193 upgrade * fix cumulus/package.json * update lerna to fix security vulnerability * fix lzards-api-client package * temporarily skip deleteS3Objects test * remove @Cumulus from root, skip test deleteObjects, add missing package * fix ci post build failure * put xmldom back to allow list * add comments for skipped test Co-authored-by: Charles Huang <charleshuang80@users.noreply.github.com> Co-authored-by: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Co-authored-by: Jonathan Kovarik <kovarik@nsidc.org> Co-authored-by: Jennifer Tran <12633533+botanical@users.noreply.github.com>
nasa · Oct 27, 2022 · 2cf0559 · 2cf0559
1 parent 27616bc
commit 2cf0559
Show file tree

Hide file tree

Showing 92 changed files with 623 additions and 406 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ## Unreleased
 
+### Notable changes
+
+- Published new tag [`43` of `cumuluss/async-operation` to Docker Hub](https://hub.docker.com/layers/cumuluss/async-operation/43/images/sha256-5f989c7d45db3dde87c88c553182d1e4e250a1e09af691a84ff6aa683088b948?context=explore) which was built with node:14.19.3-buster.
+
 ### Added
 
 - **CUMULUS-2998**
@@ -14,7 +18,6 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
     - files_to_granules_task_timeout and files_to_granule_task_memory_size
     - hello_world_task_timeout and hello_world_task_memory_size
     - sf_sqs_report_task_timeout and sf_sqs_report_task_memory_size
-
 - **CUMULUS-2986**
   - Adds Terraform memory_size configurations to lambda functions with customizable timeouts enabled (the minimum default size has also been raised from 256 MB to 512 MB)
     allowed properties include:
@@ -34,28 +37,16 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
       - update_cmr_access_constraints_task_memory_size
       - update_granules_cmr_task_memory_size
   - Initializes the lambda_memory_size(s) variable in the Terraform variable list
-
 - **CUMULUS-2631**
   - Added 'Bearer token' support to s3credentials endpoint
 - **CUMULUS-2787**
   - Added `lzards-api-client` package to Cumulus with `submitQueryToLzards` method
+- **CUMULUS-2944**
+  - Added configuration to increase the limit for body-parser's JSON and URL encoded parsers to allow for larger input payloads
 
 ### Changed
 
 - Updated `example/cumulus-tf/variables.tf` to have `cmr_oauth_provider` default to `launchpad`
-
-- **CUMULUS-3024**
-  - Update PUT /granules endpoint to operate consistently across datastores
-    (PostgreSQL, ElasticSearch, DynamoDB). Previously it was possible, given a
-    partial Granule payload to have different data in Dynamo/ElasticSearch and PostgreSQL
-  - Given a partial Granule object, the /granules update endpoint now operates
-    with behavior more consistent with a PATCH operation where fields not provided
-    in the payload will not be updated in the datastores.
-  - Granule translation (db/src/granules.ts) now supports removing null/undefined fields when converting from API to Postgres
-    granule formats.
-  - Update granule write logic: if a `null` files key is provided in an update payload (e.g. `files: null`),
-    an error will be thrown. `null` files were not previously supported and would throw potentially unclear errors. This makes the error clearer and more explicit.
-  - Update granule write logic: If an empty array is provided for the `files` key, all files will be removed in all datastores
 - **CUMULUS-2787**
   - Updated `lzards-backup-task` to send Cumulus provider and granule createdAt values as metadata in LZARDS backup request to support querying LZARDS for reconciliation reports
 - **CUMULUS-2913**
@@ -68,23 +59,41 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
   - Pinned typescript to ~4.7.x to address typing incompatibility issues
     discussed in https://github.com/knex/knex/pull/5279
   - Update generate-ts-build-cache script to always install root project dependencies
+- **CUMULUS-3104**
+  - Updated Dockerfile of async operation docker image to build from node:14.19.3-buster
+  - Sets default async_operation_image version to 43.
+  - Upgraded saml2-js 4.0.0, rewire to 6.0.0 to address security vulnerabilities
+  - Fixed TS compilation error caused by @aws-sdk/client-s3 3.190->3.193 upgrade
+
+## [v13.3.2] 2022-10-10 [BACKPORT]
+
+**Please note** changes in 13.3.2 may not yet be released in future versions, as
+this is a backport and patch release on the 13.3.x series of releases. Updates that
+are included in the future will have a corresponding CHANGELOG entry in future
+releases.
 
 ### Fixed
 
 - **CUMULUS-2557**
   - Updated `@cumulus/aws-client/S3/moveObject` to handle zero byte files (0 byte files).
-- **CUMULUS-2969**
-  - Updated `@cumulus/api/models/rules.buildPayload` to only include
-    stepFunction name and arn in for the `definition` return value, excluding
-    step function definition and other extraneous step function object
-    key/values that are not used downstream, but were causing rules to exceed internal AWS limits.
 - **CUMULUS-2971**
   - Updated `@cumulus/aws-client/S3ObjectStore` class to take string query parameters and
     its methods `signGetObject` and `signHeadObject` to take parameter presignOptions
-
 - **CUMULUS-3021**
   - Updated `@cumulus/api-client/collections` and `@cumulus/integration-tests/api` to encode
     collection version in the URI path
+- **CUMULUS-3024**
+  - Update PUT /granules endpoint to operate consistently across datastores
+    (PostgreSQL, ElasticSearch, DynamoDB). Previously it was possible, given a
+    partial Granule payload to have different data in Dynamo/ElasticSearch and PostgreSQL
+  - Given a partial Granule object, the /granules update endpoint now operates
+    with behavior more consistent with a PATCH operation where fields not provided
+    in the payload will not be updated in the datastores.
+  - Granule translation (db/src/granules.ts) now supports removing null/undefined fields when converting from API to Postgres
+    granule formats.
+  - Update granule write logic: if a `null` files key is provided in an update payload (e.g. `files: null`),
+    an error will be thrown. `null` files were not previously supported and would throw potentially unclear errors. This makes the error clearer and more explicit.
+  - Update granule write logic: If an empty array is provided for the `files` key, all files will be removed in all datastores
 
 ## [v13.3.0] 2022-8-19
 
@@ -1258,7 +1267,7 @@ aws lambda invoke --function-name $PREFIX-data-migration1 \
 ### Changed
 
 - **CUMULUS-NONE**
-  - Adds logging to ecs/async-operation Docker conatiner that launches async
+  - Adds logging to ecs/async-operation Docker container that launches async
     tasks on ECS. Sets default async_operation_image_version to 39.
 
 - **CUMULUS-2845**
@@ -6561,7 +6570,8 @@ Note: There was an issue publishing 1.12.0. Upgrade to 1.12.1.
 
 ## [v1.0.0] - 2018-02-23
 
-[unreleased]: https://github.com/nasa/cumulus/compare/v13.3.0...HEAD
+[unreleased]: https://github.com/nasa/cumulus/compare/v13.3.2...HEAD
+[v13.3.2]: https://github.com/nasa/cumulus/compare/v13.3.0...v13.3.2
 [v13.3.0]: https://github.com/nasa/cumulus/compare/v13.2.1...v13.3.0
 [v13.2.1]: https://github.com/nasa/cumulus/compare/v13.2.0...v13.2.1
 [v13.2.0]: https://github.com/nasa/cumulus/compare/v13.1.0...v13.2.0

diff --git a/audit-ci.json b/audit-ci.json
@@ -2,5 +2,5 @@
     "high": true,
     "pass-enoaudit": true,
     "retry-count": 20,
-    "allowlist": []
+    "allowlist": ["xmldom"]
 }
diff --git a/bamboo/cleanup-integration-tests.sh b/bamboo/cleanup-integration-tests.sh
@@ -20,6 +20,6 @@ npm install
 npm --version
 
 ## This is needed to ensure lock-stack has the expected dependencies
-npx lerna bootstrap --scope @cumulus/cumulus-integration-tests --scope @cumulus/aws-client --scope @cumulus/common --scope @cumulus/errors --scope @cumulus/logger
+npx lerna bootstrap --scope @cumulus/cumulus-integration-tests --scope @cumulus/aws-client --scope @cumulus/checksum --scope @cumulus/common --scope @cumulus/errors --scope @cumulus/logger
 
 cd example && node ./scripts/lock-stack.js lock "$GIT_SHA" "$DEPLOYMENT" false
diff --git a/docs/deployment/README.md b/docs/deployment/README.md
@@ -229,7 +229,7 @@ $ aws dynamodb create-table \
 
 Cumulus requires a PostgreSQL 10.2 compatible database cluster deployed to AWS. We suggest utilizing [RDS](https://docs.aws.amazon.com/rds/index.html). For further guidance about what type of RDS database to use, please [see the guide on choosing and configuring your RDS database](./choosing_configuring_rds.md).
 
-Cumulus provides a default [template and RDS cluster module](PostgreSQL_database_deployment) utilizing Aurora Serverless.
+Cumulus provides a default [template and RDS cluster module](./postgres-database-deployment.md) utilizing Aurora Serverless.
 
 However, Core intentionally provides a "bring your own" approach, and any well-planned cluster setup should work, given the following:
 

diff --git a/example/cumulus-tf/variables.tf b/example/cumulus-tf/variables.tf
@@ -350,7 +350,7 @@ variable "rds_admin_access_secret_arn" {
 variable "async_operation_image_version" {
   description = "docker image version to use for Cumulus async operations tasks"
   type = string
-  default = "42"
+  default = "43"
 }
 
 variable "cumulus_process_activity_version" {

diff --git a/example/lambdas/asyncOperations/package.json b/example/lambdas/asyncOperations/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-async-operations",
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "AsyncOperations Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/ftpPopulateTestLambda/package.json b/example/lambdas/ftpPopulateTestLambda/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/ftp-populate-test-lambda",
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "FTP Population Utility Lambda",
   "main": "index.js",
   "private": true,
@@ -19,12 +19,12 @@
     "access": "private"
   },
   "dependencies": {
-    "@cumulus/api": "13.3.0",
-    "@cumulus/api-client": "13.3.0",
-    "@cumulus/common": "13.3.0",
-    "@cumulus/integration-tests": "13.3.0",
-    "@cumulus/logger": "13.3.0",
-    "@cumulus/test-data": "13.3.0",
+    "@cumulus/api": "13.3.2",
+    "@cumulus/api-client": "13.3.2",
+    "@cumulus/common": "13.3.2",
+    "@cumulus/integration-tests": "13.3.2",
+    "@cumulus/logger": "13.3.2",
+    "@cumulus/test-data": "13.3.2",
     "aws-sdk": "^2.585.0",
     "fs-extra": "^9.0.0",
     "js-yaml": "^3.13.1",

diff --git a/example/lambdas/lzardsClientTest/package.json b/example/lambdas/lzardsClientTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-lzards-api-lambda",
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "LZARDS API Client Test Lambda",
   "private": true,
   "engines": {
@@ -20,7 +20,7 @@
   "author": "Cumulus Authors",
   "license": "Apache-2.0",
   "dependencies": {
-    "@cumulus/lzards-api-client": "13.2.2-alpha.2",
-    "@cumulus/logger": "13.3.0"
+    "@cumulus/lzards-api-client": "13.3.2-alpha.0",
+    "@cumulus/logger": "13.3.2"
   }
 }
diff --git a/example/lambdas/python-processing/package.json b/example/lambdas/python-processing/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-process-activity",
   "private": true,
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "Python reference activity",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-activity",
   "repository": {

diff --git a/example/lambdas/python-reference-activity/package.json b/example/lambdas/python-reference-activity/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-reference-activity",
   "private": true,
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "Python reference activity",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-activity",
   "repository": {

diff --git a/example/lambdas/python-reference-task/package.json b/example/lambdas/python-reference-task/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-reference-task",
   "private": true,
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "Python reference task",
   "main": "index.js",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-task",

diff --git a/example/lambdas/s3AccessTest/package.json b/example/lambdas/s3AccessTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-s3-access",
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "S3 Access Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/snsS3Test/package.json b/example/lambdas/snsS3Test/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-sns-s3",
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "SNS to S3 Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/versionUpTest/package.json b/example/lambdas/versionUpTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-version-up",
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "Version Up Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/package.json b/example/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/cumulus-integration-tests",
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "Cumulus Integration Test Deployment",
   "private": true,
   "main": "index.js",
@@ -45,32 +45,32 @@
     ]
   },
   "dependencies": {
-    "@cumulus/api": "13.3.0",
-    "@cumulus/api-client": "13.3.0",
-    "@cumulus/async-operations": "13.3.0",
-    "@cumulus/aws-client": "13.3.0",
-    "@cumulus/checksum": "13.3.0",
-    "@cumulus/cmr-client": "13.3.0",
-    "@cumulus/cmrjs": "13.3.0",
-    "@cumulus/common": "13.3.0",
-    "@cumulus/discover-granules": "13.3.0",
-    "@cumulus/discover-pdrs": "13.3.0",
-    "@cumulus/files-to-granules": "13.3.0",
-    "@cumulus/hello-world": "13.3.0",
-    "@cumulus/ingest": "13.3.0",
-    "@cumulus/integration-tests": "13.3.0",
-    "@cumulus/message": "13.3.0",
-    "@cumulus/move-granules": "13.3.0",
-    "@cumulus/parse-pdr": "13.3.0",
-    "@cumulus/pdr-status-check": "13.3.0",
-    "@cumulus/post-to-cmr": "13.3.0",
-    "@cumulus/queue-granules": "13.3.0",
-    "@cumulus/queue-pdrs": "13.3.0",
-    "@cumulus/sf-sqs-report": "13.3.0",
-    "@cumulus/sync-granule": "13.3.0",
-    "@cumulus/test-processing": "13.3.0"
+    "@cumulus/api": "13.3.2",
+    "@cumulus/api-client": "13.3.2",
+    "@cumulus/async-operations": "13.3.2",
+    "@cumulus/aws-client": "13.3.2",
+    "@cumulus/checksum": "13.3.2",
+    "@cumulus/cmr-client": "13.3.2",
+    "@cumulus/cmrjs": "13.3.2",
+    "@cumulus/common": "13.3.2",
+    "@cumulus/discover-granules": "13.3.2",
+    "@cumulus/discover-pdrs": "13.3.2",
+    "@cumulus/files-to-granules": "13.3.2",
+    "@cumulus/hello-world": "13.3.2",
+    "@cumulus/ingest": "13.3.2",
+    "@cumulus/integration-tests": "13.3.2",
+    "@cumulus/message": "13.3.2",
+    "@cumulus/move-granules": "13.3.2",
+    "@cumulus/parse-pdr": "13.3.2",
+    "@cumulus/pdr-status-check": "13.3.2",
+    "@cumulus/post-to-cmr": "13.3.2",
+    "@cumulus/queue-granules": "13.3.2",
+    "@cumulus/queue-pdrs": "13.3.2",
+    "@cumulus/sf-sqs-report": "13.3.2",
+    "@cumulus/sync-granule": "13.3.2",
+    "@cumulus/test-processing": "13.3.2"
   },
   "devDependencies": {
-    "@cumulus/test-data": "13.3.0"
+    "@cumulus/test-data": "13.3.2"
   }
 }
diff --git a/example/scripts/generate_ingest/package.json b/example/scripts/generate_ingest/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/generate_ingest",
   "private": true,
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "Script to generate test data for scaled ingest",
   "keywords": [
     "GIBS",
@@ -22,8 +22,8 @@
     "directory": "packages/types"
   },
   "dependencies": {
-    "@cumulus/aws-client": "13.3.0",
-    "@cumulus/common": "13.3.0"
+    "@cumulus/aws-client": "13.3.2",
+    "@cumulus/common": "13.3.2"
   },
   "author": "Cumulus Authors",
   "license": "Apache-2.0"

diff --git a/example/scripts/lib/package.json b/example/scripts/lib/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/example-lib",
   "private": true,
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "example project libs",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/scripts/lib",
   "engines": {

diff --git a/lambdas/data-migration1/package.json b/lambdas/data-migration1/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/data-migration1",
-  "version": "13.3.0",
+  "version": "13.3.2",
   "description": "A Lambda function used for doing data migrations",
   "license": "Apache-2.0",
   "engines": {
@@ -25,18 +25,18 @@
     "timeout": "15m"
   },
   "dependencies": {
-    "@cumulus/api": "13.3.0",
-    "@cumulus/aws-client": "13.3.0",
-    "@cumulus/common": "13.3.0",
-    "@cumulus/db": "13.3.0",
-    "@cumulus/errors": "13.3.0",
-    "@cumulus/logger": "13.3.0",
-    "@cumulus/types": "13.3.0",
+    "@cumulus/api": "13.3.2",
+    "@cumulus/aws-client": "13.3.2",
+    "@cumulus/common": "13.3.2",
+    "@cumulus/db": "13.3.2",
+    "@cumulus/errors": "13.3.2",
+    "@cumulus/logger": "13.3.2",
+    "@cumulus/types": "13.3.2",
     "knex": "0.95.15",
     "lodash": "^4.17.21",
     "pg": "^8.3.0"
   },
   "devDependencies": {
-    "@cumulus/test-data": "13.3.0"
+    "@cumulus/test-data": "13.3.2"
   }
 }