Merge branch 'develop' into pr/5861

* develop: [FEATURE] ZEP - Dynamically define `add_<ASSET_TYPE>_asset()` methods if needed (great-expectations#7121) [FEATURE] Update zep to use sqlalchemy_data_splitter.py (great-expectations#7151) [RELEASE] 0.15.49 (great-expectations#7168) [BUGFIX] ID/PK Spark and Sql fall back when `unexpected_index_column_names` have not been defined (great-expectations#7150) [CONTRIB] Update the rendered text for min and max values to be clearer. (great-expectations#7166) [MAINTENANCE] Clean up additional deprecation warnings from outdated CRUD API (great-expectations#7156) [MAINTENANCE] Bump minimum version of `pytest` (great-expectations#7164) [BUGFIX] pydantic>=1.10.4 - ImportError: cannot import name dataclass_transform (great-expectations#7163) [MAINTENANCE] ZEP - update asset factories method signatures from asset models (great-expectations#7096)
jmoskovc · Feb 16, 2023 · 9773649 · 9773649
2 parents 38584b7 + 3182508
commit 9773649
Show file tree

Hide file tree

Showing 72 changed files with 1,341 additions and 1,015 deletions.
diff --git a/docs/changelog.md b/docs/changelog.md
@@ -2,6 +2,53 @@
 title: Changelog
 ---
 
+### 0.15.49
+* [FEATURE] Enable customization of candidate Regular Expression patterns when running OnboardingDataAssistant ([#7104](https://github.com/great-expectations/great_expectations/pull/7104))
+* [FEATURE] Enable `gx.get_context()` to work without any inputs ([#7074](https://github.com/great-expectations/great_expectations/pull/7074))
+* [FEATURE] Add `datasource` arg to `DataContext` `Datasource` CRUD ([#7070](https://github.com/great-expectations/great_expectations/pull/7070))
+* [BUGFIX] Rename experimental `get_batch_request` to `build_batch_request` ([#7107](https://github.com/great-expectations/great_expectations/pull/7107))
+* [BUGFIX] Remove version from versions.json ([#7109](https://github.com/great-expectations/great_expectations/pull/7109))
+* [BUGFIX] Properly Enable/Disable Spark Integration Tests Depending on pyspark Installation for New Datasources ([#7132](https://github.com/great-expectations/great_expectations/pull/7132))
+* [BUGFIX] Copy previous versions after checking out the the current commit ([#7142](https://github.com/great-expectations/great_expectations/pull/7142))
+* [BUGFIX] `TupleAzureBlobStoreBackend` no longer gives warning when obfuscating connection string ([#7139](https://github.com/great-expectations/great_expectations/pull/7139))
+* [BUGFIX] Patch inconsistent ordering within GCP test asserts ([#7130](https://github.com/great-expectations/great_expectations/pull/7130))
+* [BUGFIX] Parse pandas version correctly for development builds ([#7147](https://github.com/great-expectations/great_expectations/pull/7147)) (thanks @jtilly)
+* [BUGFIX] Patch broken rendered content Cloud tests ([#7155](https://github.com/great-expectations/great_expectations/pull/7155))
+* [BUGFIX] pydantic>=1.10.4 - ImportError: cannot import name dataclass_transform ([#7163](https://github.com/great-expectations/great_expectations/pull/7163))
+* [BUGFIX] ID/PK Spark and Sql fall back when `unexpected_index_column_names` have not been defined ([#7150](https://github.com/great-expectations/great_expectations/pull/7150))
+* [DOCS] Add CRUD API matrix to `AbstractDataContext` docstring ([#7079](https://github.com/great-expectations/great_expectations/pull/7079))
+* [DOCS] Build API docs using latest released version ([#7067](https://github.com/great-expectations/great_expectations/pull/7067))
+* [DOCS] Add displayHTML method to view Data Docs ([#7125](https://github.com/great-expectations/great_expectations/pull/7125)) (thanks @swittchawa)
+* [DOCS] Use named snippets part 1 ([#7131](https://github.com/great-expectations/great_expectations/pull/7131))
+* [DOCS] : fix capitalization of Slack ([#7136](https://github.com/great-expectations/great_expectations/pull/7136)) (thanks @JoelGritter)
+* [DOCS] Remove sitemap.xml ([#7141](https://github.com/great-expectations/great_expectations/pull/7141))
+* [DOCS] doc-464 consolidating and standardizing snippets ([#7154](https://github.com/great-expectations/great_expectations/pull/7154))
+* [MAINTENANCE] Remove Extra Character from ID/PK Example README ([#7098](https://github.com/great-expectations/great_expectations/pull/7098))
+* [MAINTENANCE] Rename experimental `get_batch_request` to `build_batch_request` ([#7095](https://github.com/great-expectations/great_expectations/pull/7095))
+* [MAINTENANCE] Fix incorrect label on "How to configure a SQL Datasource" docs page ([#7106](https://github.com/great-expectations/great_expectations/pull/7106))
+* [MAINTENANCE] Update dependency on pydantic ([#7111](https://github.com/great-expectations/great_expectations/pull/7111))
+* [MAINTENANCE] Move experimental `base_directory` from `_FilesystemDataAsset` to `PandasDatasource` and `SparkDatasource` ([#7078](https://github.com/great-expectations/great_expectations/pull/7078))
+* [MAINTENANCE] Use secret to store algolia api key ([#7115](https://github.com/great-expectations/great_expectations/pull/7115))
+* [MAINTENANCE] Fluent Datasources - don't register "private" Datasource classes ([#7124](https://github.com/great-expectations/great_expectations/pull/7124))
+* [MAINTENANCE] ZEP - Realign pandas asset args for `Datasource` level `base_directory` ([#7123](https://github.com/great-expectations/great_expectations/pull/7123))
+* [MAINTENANCE] format notebooks with `black` ([#7054](https://github.com/great-expectations/great_expectations/pull/7054))
+* [MAINTENANCE] mypy `v1.0.0` ([#7138](https://github.com/great-expectations/great_expectations/pull/7138))
+* [MAINTENANCE] Output Consistent Data Format from "table.head" Metric for every ExecutionEngine ([#7134](https://github.com/great-expectations/great_expectations/pull/7134))
+* [MAINTENANCE] ruff 0.0.246 update ([#7137](https://github.com/great-expectations/great_expectations/pull/7137))
+* [MAINTENANCE] Refactor sql splitter to take selectable instead of str. ([#7133](https://github.com/great-expectations/great_expectations/pull/7133))
+* [MAINTENANCE] Update V3 DataConnector utilities to support New Datasources (ZEP) ([#7144](https://github.com/great-expectations/great_expectations/pull/7144))
+* [MAINTENANCE] Change all instances of `create_expectation_suite` to `add_expectation_suite` in tests, docs, and source code ([#7117](https://github.com/great-expectations/great_expectations/pull/7117))
+* [MAINTENANCE] Clean up pathlib.Path() usage in DataConnector utilities and restore tighter formatting in great_expectations/util.py ([#7149](https://github.com/great-expectations/great_expectations/pull/7149))
+* [MAINTENANCE] Clean up `mypy` violations in `CardinalityChecker` ([#7146](https://github.com/great-expectations/great_expectations/pull/7146))
+* [MAINTENANCE] Remove unused dockerfile ([#7152](https://github.com/great-expectations/great_expectations/pull/7152))
+* [MAINTENANCE] Delete cli v012 tests. ([#7159](https://github.com/great-expectations/great_expectations/pull/7159))
+* [MAINTENANCE] ZEP - update asset factories method signatures from asset models ([#7096](https://github.com/great-expectations/great_expectations/pull/7096))
+* [MAINTENANCE] Bump minimum version of `pytest` ([#7164](https://github.com/great-expectations/great_expectations/pull/7164))
+* [MAINTENANCE] Clean up additional deprecation warnings from outdated CRUD API ([#7156](https://github.com/great-expectations/great_expectations/pull/7156))
+* [CONTRIB] Add experimental expectation to check column values after split ([#7120](https://github.com/great-expectations/great_expectations/pull/7120)) (thanks @ace-racer)
+* [CONTRIB] added new Expectations  - India_zip_code expectation and not_to_be_future_date expectation ([#6086](https://github.com/great-expectations/great_expectations/pull/6086)) (thanks @prachijain136)
+* [CONTRIB] Update the rendered text for min and max values to be clearer. ([#7166](https://github.com/great-expectations/great_expectations/pull/7166))
+
 ### 0.15.48
 * [FEATURE] Place FilesystemDataAsset into separate module (its functionality is used by both PandasDatasource and SparkDatasource) ([#7025](https://github.com/great-expectations/great_expectations/pull/7025))
 * [FEATURE] Add SQL query data asset for new experimental datasources ([#6999](https://github.com/great-expectations/great_expectations/pull/6999))

diff --git a/docs/components/_data.jsx b/docs/components/_data.jsx
@@ -1,5 +1,5 @@
 export default {
-  release_version: 'great_expectations, version 0.15.48',
+  release_version: 'great_expectations, version 0.15.49',
   min_python: 'Python 3.7',
   max_python: 'Python 3.10'
 }
diff --git a/...ed/how_to_create_a_new_expectation_suite_by_profiling_from_a_jsonschema_file.md b/...ed/how_to_create_a_new_expectation_suite_by_profiling_from_a_jsonschema_file.md
@@ -54,7 +54,7 @@ suite = profiler.profile(schema, suite_name)
 ### 6. Save the suite
 
 ```python
-context.save_expectation_suite(suite)
+context.add_expectation_suite(expectation_suite=suite)
 ```
 
 ## 7. (Optional) Generate Data Docs and review the results
@@ -94,6 +94,6 @@ with open(jsonschema_file, "r") as f:
 print("Generating suite...")
 profiler = JsonSchemaProfiler()
 suite = profiler.profile(schema, suite_name)
-context.save_expectation_suite(suite)
+context.add_expectation_suite(expectation_suite=suite)
 ```
 
diff --git a/docs/reference/expectations/result_format.md b/docs/reference/expectations/result_format.md
@@ -46,7 +46,8 @@ rendered in DataDocs when `COMPLETE` is selected.
 
 The `unexpected_index_query`, which for `SQL` and `Spark` is a query that allows you to retrieve the full set of 
 unexpected values from the dataset, is also rendered by default when `COMPLETE` is selected. For `Pandas`, this parameter 
-returns the full set of unexpected indices, which can also be used to retrieve the full set of unexpected values. 
+returns the full set of unexpected indices, which can also be used to retrieve the full set of unexpected values. This is returned
+whether or not the `unexpected_index_column_names` are defined. 
 
 To suppress this output, the `return_unexpected_index_query` parameter can be set to `False`. 
 

diff --git a/docs_rtd/changelog.rst b/docs_rtd/changelog.rst
@@ -4,6 +4,54 @@
 Changelog
 #########
 
+0.15.49
+-----------------
+* [FEATURE] Enable customization of candidate Regular Expression patterns when running OnboardingDataAssistant ([#7104](https://github.com/great-expectations/great_expectations/pull/7104))
+* [FEATURE] Enable `gx.get_context()` to work without any inputs ([#7074](https://github.com/great-expectations/great_expectations/pull/7074))
+* [FEATURE] Add `datasource` arg to `DataContext` `Datasource` CRUD ([#7070](https://github.com/great-expectations/great_expectations/pull/7070))
+* [BUGFIX] Rename experimental `get_batch_request` to `build_batch_request` ([#7107](https://github.com/great-expectations/great_expectations/pull/7107))
+* [BUGFIX] Remove version from versions.json ([#7109](https://github.com/great-expectations/great_expectations/pull/7109))
+* [BUGFIX] Properly Enable/Disable Spark Integration Tests Depending on pyspark Installation for New Datasources ([#7132](https://github.com/great-expectations/great_expectations/pull/7132))
+* [BUGFIX] Copy previous versions after checking out the the current commit ([#7142](https://github.com/great-expectations/great_expectations/pull/7142))
+* [BUGFIX] `TupleAzureBlobStoreBackend` no longer gives warning when obfuscating connection string ([#7139](https://github.com/great-expectations/great_expectations/pull/7139))
+* [BUGFIX] Patch inconsistent ordering within GCP test asserts ([#7130](https://github.com/great-expectations/great_expectations/pull/7130))
+* [BUGFIX] Parse pandas version correctly for development builds ([#7147](https://github.com/great-expectations/great_expectations/pull/7147)) (thanks @jtilly)
+* [BUGFIX] Patch broken rendered content Cloud tests ([#7155](https://github.com/great-expectations/great_expectations/pull/7155))
+* [BUGFIX] pydantic>=1.10.4 - ImportError: cannot import name dataclass_transform ([#7163](https://github.com/great-expectations/great_expectations/pull/7163))
+* [BUGFIX] ID/PK Spark and Sql fall back when `unexpected_index_column_names` have not been defined ([#7150](https://github.com/great-expectations/great_expectations/pull/7150))
+* [DOCS] Add CRUD API matrix to `AbstractDataContext` docstring ([#7079](https://github.com/great-expectations/great_expectations/pull/7079))
+* [DOCS] Build API docs using latest released version ([#7067](https://github.com/great-expectations/great_expectations/pull/7067))
+* [DOCS] Add displayHTML method to view Data Docs ([#7125](https://github.com/great-expectations/great_expectations/pull/7125)) (thanks @swittchawa)
+* [DOCS] Use named snippets part 1 ([#7131](https://github.com/great-expectations/great_expectations/pull/7131))
+* [DOCS] : fix capitalization of Slack ([#7136](https://github.com/great-expectations/great_expectations/pull/7136)) (thanks @JoelGritter)
+* [DOCS] Remove sitemap.xml ([#7141](https://github.com/great-expectations/great_expectations/pull/7141))
+* [DOCS] doc-464 consolidating and standardizing snippets ([#7154](https://github.com/great-expectations/great_expectations/pull/7154))
+* [MAINTENANCE] Remove Extra Character from ID/PK Example README ([#7098](https://github.com/great-expectations/great_expectations/pull/7098))
+* [MAINTENANCE] Rename experimental `get_batch_request` to `build_batch_request` ([#7095](https://github.com/great-expectations/great_expectations/pull/7095))
+* [MAINTENANCE] Fix incorrect label on "How to configure a SQL Datasource" docs page ([#7106](https://github.com/great-expectations/great_expectations/pull/7106))
+* [MAINTENANCE] Update dependency on pydantic ([#7111](https://github.com/great-expectations/great_expectations/pull/7111))
+* [MAINTENANCE] Move experimental `base_directory` from `_FilesystemDataAsset` to `PandasDatasource` and `SparkDatasource` ([#7078](https://github.com/great-expectations/great_expectations/pull/7078))
+* [MAINTENANCE] Use secret to store algolia api key ([#7115](https://github.com/great-expectations/great_expectations/pull/7115))
+* [MAINTENANCE] Fluent Datasources - don't register "private" Datasource classes ([#7124](https://github.com/great-expectations/great_expectations/pull/7124))
+* [MAINTENANCE] ZEP - Realign pandas asset args for `Datasource` level `base_directory` ([#7123](https://github.com/great-expectations/great_expectations/pull/7123))
+* [MAINTENANCE] format notebooks with `black` ([#7054](https://github.com/great-expectations/great_expectations/pull/7054))
+* [MAINTENANCE] mypy `v1.0.0` ([#7138](https://github.com/great-expectations/great_expectations/pull/7138))
+* [MAINTENANCE] Output Consistent Data Format from "table.head" Metric for every ExecutionEngine ([#7134](https://github.com/great-expectations/great_expectations/pull/7134))
+* [MAINTENANCE] ruff 0.0.246 update ([#7137](https://github.com/great-expectations/great_expectations/pull/7137))
+* [MAINTENANCE] Refactor sql splitter to take selectable instead of str. ([#7133](https://github.com/great-expectations/great_expectations/pull/7133))
+* [MAINTENANCE] Update V3 DataConnector utilities to support New Datasources (ZEP) ([#7144](https://github.com/great-expectations/great_expectations/pull/7144))
+* [MAINTENANCE] Change all instances of `create_expectation_suite` to `add_expectation_suite` in tests, docs, and source code ([#7117](https://github.com/great-expectations/great_expectations/pull/7117))
+* [MAINTENANCE] Clean up pathlib.Path() usage in DataConnector utilities and restore tighter formatting in great_expectations/util.py ([#7149](https://github.com/great-expectations/great_expectations/pull/7149))
+* [MAINTENANCE] Clean up `mypy` violations in `CardinalityChecker` ([#7146](https://github.com/great-expectations/great_expectations/pull/7146))
+* [MAINTENANCE] Remove unused dockerfile ([#7152](https://github.com/great-expectations/great_expectations/pull/7152))
+* [MAINTENANCE] Delete cli v012 tests. ([#7159](https://github.com/great-expectations/great_expectations/pull/7159))
+* [MAINTENANCE] ZEP - update asset factories method signatures from asset models ([#7096](https://github.com/great-expectations/great_expectations/pull/7096))
+* [MAINTENANCE] Bump minimum version of `pytest` ([#7164](https://github.com/great-expectations/great_expectations/pull/7164))
+* [MAINTENANCE] Clean up additional deprecation warnings from outdated CRUD API ([#7156](https://github.com/great-expectations/great_expectations/pull/7156))
+* [CONTRIB] Add experimental expectation to check column values after split ([#7120](https://github.com/great-expectations/great_expectations/pull/7120)) (thanks @ace-racer)
+* [CONTRIB] added new Expectations  - India_zip_code expectation and not_to_be_future_date expectation ([#6086](https://github.com/great-expectations/great_expectations/pull/6086)) (thanks @prachijain136)
+* [CONTRIB] Update the rendered text for min and max values to be clearer. ([#7166](https://github.com/great-expectations/great_expectations/pull/7166))
+
 0.15.48
 -----------------
 * [FEATURE] Place FilesystemDataAsset into separate module (its functionality is used by both PandasDatasource and SparkDatasource) ([#7025](https://github.com/great-expectations/great_expectations/pull/7025))

diff --git a/great_expectations/cli/toolkit.py b/great_expectations/cli/toolkit.py
@@ -1041,7 +1041,7 @@ def add_citation_with_batch_request(
             comment="Created suite added via CLI",
             batch_request=batch_request,  # type: ignore[arg-type] # values union
         )
-        data_context.save_expectation_suite(expectation_suite=expectation_suite)
+        data_context.update_expectation_suite(expectation_suite=expectation_suite)
 
 
 def get_batch_request_from_json_file(

diff --git a/great_expectations/cli/v012/toolkit.py b/great_expectations/cli/v012/toolkit.py
@@ -295,7 +295,7 @@ def create_empty_suite(
     )
     suite = context.create_expectation_suite(expectation_suite_name)
     suite.add_citation(comment="New suite added via CLI", batch_kwargs=batch_kwargs)
-    context.save_expectation_suite(suite, expectation_suite_name)
+    context.update_expectation_suite(expectation_suite=suite)
 
 
 def launch_jupyter_notebook(notebook_path: str) -> None:

diff --git a/great_expectations/core/usage_statistics/usage_statistics.py b/great_expectations/core/usage_statistics/usage_statistics.py
@@ -39,9 +39,7 @@
     from great_expectations.core.usage_statistics.anonymizers.types.base import (
         CLISuiteInteractiveFlagCombinations,
     )
-    from great_expectations.core.usage_statistics.events import (
-        UsageStatsEvents,
-    )
+    from great_expectations.core.usage_statistics.events import UsageStatsEvents
     from great_expectations.data_context import AbstractDataContext
     from great_expectations.datasource import LegacyDatasource
     from great_expectations.datasource.new_datasource import BaseDatasource
@@ -398,8 +396,8 @@ def run_validation_operator_usage_statistics(
 # noinspection PyUnusedLocal
 def save_expectation_suite_usage_statistics(
     data_context: AbstractDataContext,
-    expectation_suite: ExpectationSuite,
-    expectation_suite_name: Optional[str] = None,
+    expectation_suite: ExpectationSuite | None = None,
+    expectation_suite_name: str | None = None,
     **kwargs: dict,
 ) -> dict:
     """
@@ -657,9 +655,9 @@ def send_usage_message_from_handler(
 def _handle_expectation_suite_usage_statistics(
     data_context: AbstractDataContext,
     event_arguments_payload_handler_name: str,
-    expectation_suite: Optional[ExpectationSuite] = None,
-    expectation_suite_name: Optional[str] = None,
-    interactive_mode: Optional[CLISuiteInteractiveFlagCombinations] = None,
+    expectation_suite: ExpectationSuite | None = None,
+    expectation_suite_name: str | None = None,
+    interactive_mode: CLISuiteInteractiveFlagCombinations | None = None,
     **kwargs,
 ) -> dict:
     """
@@ -683,6 +681,9 @@ def _handle_expectation_suite_usage_statistics(
     else:
         payload = copy.deepcopy(interactive_mode.value)
 
+    assert (
+        expectation_suite or expectation_suite_name
+    ), "Guaranteed to have at least one of these values from context CRUD"
     if expectation_suite_name is None:
         if isinstance(expectation_suite, ExpectationSuite):
             expectation_suite_name = expectation_suite.expectation_suite_name

diff --git a/great_expectations/data_asset/data_asset.py b/great_expectations/data_asset/data_asset.py
@@ -699,7 +699,9 @@ def save_expectation_suite(
             suppress_warnings,
         )
         if filepath is None and self._data_context is not None:
-            self._data_context.save_expectation_suite(expectation_suite)
+            self._data_context.add_or_update_expectation_suite(
+                expectation_suite=expectation_suite
+            )
         elif filepath is not None:
             with open(filepath, "w") as outfile:
                 json.dump(

diff --git a/great_expectations/data_context/data_context/abstract_data_context.py b/great_expectations/data_context/data_context/abstract_data_context.py
@@ -2874,6 +2874,12 @@ def _persist_suite_with_store(
 
     @public_api
     @new_method_or_class(version="0.15.48")
+    # 20230216 - Chetan - Decorating with save to ensure no gaps in usage stats collection
+    # A future effort will add more granular event names
+    @usage_statistics_enabled_method(
+        event_name=UsageStatsEvents.DATA_CONTEXT_SAVE_EXPECTATION_SUITE,
+        args_payload_fn=save_expectation_suite_usage_statistics,
+    )
     def update_expectation_suite(
         self,
         expectation_suite: ExpectationSuite,
@@ -2931,6 +2937,12 @@ def add_or_update_expectation_suite(
 
     @public_api
     @new_method_or_class(version="0.15.48")
+    # 20230216 - Chetan - Decorating with save to ensure no gaps in usage stats collection
+    # A future effort will add more granular event names
+    @usage_statistics_enabled_method(
+        event_name=UsageStatsEvents.DATA_CONTEXT_SAVE_EXPECTATION_SUITE,
+        args_payload_fn=save_expectation_suite_usage_statistics,
+    )
     def add_or_update_expectation_suite(
         self,
         expectation_suite_name: str | None = None,

diff --git a/great_expectations/deployment_version b/great_expectations/deployment_version
@@ -1 +1 @@
-0.15.48
+0.15.49