Resolve deprecation warnings on Kedro CLI

[yetudada]

Description

I spotted that we have stacks of deprecation warnings when I was looking at Amanda's completion of the Spaceflights tutorial. So this ask is either to change the logging level that all of these warnings appear or reduce the number of them.

Here's what deprecation warnings show when I:

• Create and activate a new conda environment with Python 3.9
• Install Kedro 0.18.4
• Create a new project with the pandas-iris starter
• Install requirements with pip install -r src/requirements.txt
• And hit kedro run

This will happen every time I use a project-specific command in Kedro and I'll only see information related to what I wanted to see at the end of all of these warnings.

I also see this issue has been flagged on StackOverflow in different ways (issue 1 and issue 2) and I imagine this will be highly intimidating for new users of Kedro.

(deprecation-warnings) yetunde_dada@computer deprecation-warnings % kedro run
[01/13/23 13:50:24] WARNING  /Users/yetunde_dada/opt/anaconda3/e warnings.py:109
                             nvs/deprecation-warnings/lib/python                
                             3.9/site-packages/plotly/express/im                
                             show_utils.py:24:                                  
                             DeprecationWarning: `np.bool8` is a                
                             deprecated alias for `np.bool_`.                   
                             (Deprecated NumPy 1.24)                            
                               np.bool8: (False, True),                         
                                                                                
                    WARNING  /Users/yetunde_dada/opt/anaconda3/e warnings.py:109
                             nvs/deprecation-warnings/lib/python                
                             3.9/site-packages/kedro_viz/models/                
                             experiment_tracking.py:16:                         
                             MovedIn20Warning: [31mDeprecated                   
                             API features detected! These                       
                             feature(s) are not compatible with                 
                             SQLAlchemy 2.0. [32mTo prevent                     
                             incompatible upgrades prior to                     
                             updating applications, ensure                      
                             requirements files are pinned to                   
                             "sqlalchemy<2.0". [36mSet                          
                             environment variable                               
                             SQLALCHEMY_WARN_20=1 to show all                   
                             deprecation warnings.  Set                         
                             environment variable                               
                             SQLALCHEMY_SILENCE_UBER_WARNING=1                  
                             to silence this message.[0m                        
                             (Background on SQLAlchemy 2.0 at:                  
                             https://sqlalche.me/e/b8d9)                        
                               Base = declarative_base()                        
                                                                                
[01/13/23 13:50:28] WARNING  /Users/yetunde_dada/opt/anaconda3/e warnings.py:109
                             nvs/deprecation-warnings/lib/python                
                             3.9/site-packages/jupyter_client/co                
                             nnect.py:27: DeprecationWarning:                   
                             Jupyter is migrating its paths to                  
                             use standard platformdirs                          
                             given by the platformdirs library.                 
                             To remove this warning and                         
                             see the appropriate new                            
                             directories, set the environment                   
                             variable                                           
                             `JUPYTER_PLATFORM_DIRS=1` and then                 
                             run `jupyter --paths`.                             
                             The use of platformdirs will be the                
                             default in `jupyter_core` v6                       
                               from jupyter_core.paths import                   
                             jupyter_data_dir                                   
                                                                                
[01/13/23 13:50:29] WARNING  /Users/yetunde_dada/opt/anaconda3/e warnings.py:109
                             nvs/deprecation-warnings/lib/python                
                             3.9/site-packages/plotly/graph_obje                
                             cts/__init__.py:288:                               
                             DeprecationWarning: distutils                      
                             Version classes are deprecated. Use                
                             packaging.version instead.                         
                               if                                               
                             LooseVersion(ipywidgets.__version__                
                             ) >= LooseVersion("7.0.0"):       

[merelcht]

Ideally we would do this before the next kedro-viz release, because some warnings come from dependencies in viz (SQLAlchemy)

[merelcht]

Resolved the warning about SQLAlchemy in kedro-org/kedro-viz#1222

[merelcht]

To get rid of the following warning:

 /Users/merel_theisen/anaconda3/envs/clean/lib/python3.9/site-packages/jupyter_client/connect.py:27: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs               warnings.py:109
                             given by the platformdirs library.  To remove this warning and
                             see the appropriate new directories, set the environment variable
                             `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
                             The use of platformdirs will be the default in `jupyter_core` v6
                               from jupyter_core.paths import jupyter_data_dir

Users will just have to follow the steps described there:

• set the environment variable JUPYTER_PLATFORM_DIRS=1
• run jupyter --paths

[merelcht]

Resolved warnings coming from the visualisation tutorial:

 WARNING  /Users/merel_theisen/Projects/Testing/deprecations-spaceflights/src/deprecations_spaceflights/pipelines/reporting/nodes.py:13: FutureWarning:                                                     warnings.py:109

 The default value of numeric_only in DataFrameGroupBy.mean is deprecated. In a future version, numeric_only will default to False. Either specify numeric_only or select only columns which
 should be valid for the function.

with #2220

[merelcht]

Resolved pandas futurewarning in spaceflights:

 WARNING  /Users/merel_theisen/Projects/Testing/deprecations-spaceflights/src/deprecations_spaceflights/pipelines/data_processing/nodes.py:15: FutureWarning: The default value of regex will change from   warnings.py:109
                             True to False in a future version. In addition, single character regular expressions will *not* be treated as literal strings when regex=True.
                               x = x.str.replace("$", "").str.replace(",", "")

with kedro-org/kedro-starters#112

[merelcht]

When using the pyspark-iris starter I'm seeing:

WARNING  /Users/merel_theisen/anaconda3/envs/clean/lib/python3.9/site-packages/hdfs/config.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation  warnings.py:109
                             for alternative uses
                               from imp import load_source

there's a fix PR for this, but hasn't been merged yet: mtth/hdfs#192

[merelcht]

These warnings are both caused by a change in distutils:

/Users/merel_theisen/anaconda3/envs/clean/lib/python3.9/site-packages/plotly/graph_objects/__init__.py:288: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version   warnings.py:109
                            instead.
                              if LooseVersion(ipywidgets.__version__) >= LooseVersion("7.0.0"):

/Users/merel_theisen/anaconda3/envs/testviz38/lib/python3.8/site-packages/pyarrow/pandas_compat.py:354: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version       warnings.py:109
                             instead.
                               if _pandas_api.is_sparse(col):

• The change is described here: https://peps.python.org/pep-0632/
• This answer on stackoverflow explains why these warnings are shown when doing a kedro run: https://stackoverflow.com/a/72220260/14169842
• For the warning in plotly someone already opened a PR: Updated distutils.Version to packaging.Version plotly/plotly.py#3897

Unfortunately, there's nothing we can do about this warning. If users find it very annoying they could follow this advice to turn it off:
https://stackoverflow.com/a/70971815/14169842

[merelcht]

The following warning I only managed to see in an old environment. In any new environment I created with Python 3.7, 3.8 or 3.9 it didn't appear.

/Users/merel_theisen/anaconda3/envs/testviz38/lib/python3.8/site-packages/plotly/express/imshow_utils.py:24: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated    warnings.py:109
                             NumPy 1.24)
                               np.bool8: (False, True),

As explained here https://stackoverflow.com/a/67779455/14169842 this is a harmless warning and you can just ignore it. Again, if users find it very annoying they could follow this advice to turn it off:
https://stackoverflow.com/a/70971815/14169842

[merelcht]

All known warnings and issues that could be resolved are now resolved. For all the other ones see the above comments about how to "mute" them.

[astrojuanlu]

xref kedro-org/kedro-viz#1244

[astrojuanlu]

After a gentle nudge, the Plotly change has been merged :) It will see the light in the next release, hopefully