Proper support of nullable dtypes as the Categorical dtype

[Dr-Irv]

Now that Categorical depends on ExtensionArray, it makes more sense to return and output pd.NA as a missing value instead of np.nan.

Propose that we announce in 2.0 release that this will change in a future release. Not clear if/how we create a deprecation message here.

Current behavior:

>>> c = pd.Categorical( ["a", "a", "b", "c", "c"], ["a", "b", "c"])
>>> c
['a', 'a', 'b', 'c', 'c']
Categories (3, object): ['a', 'b', 'c']
>>> s = pd.Series(c)
>>> s
0    a
1    a
2    b
3    c
4    c
dtype: category
Categories (3, object): ['a', 'b', 'c']
>>> s.iloc[2] = pd.NA
>>> s.iloc[2]
nan

[Dr-Irv]

Not clear how to track this in #50578 since we should decide that we want to do this.

[jbrockmendel]

id be more inclined to #29962, which would mean getting pd.NA in a targeted subset of cases.

[Dr-Irv]

id be more inclined to #29962, which would mean getting pd.NA in a targeted subset of cases.

The idea in #29962 is to make the NA value dependent on the underlying dtype of the categorical. But there is also a point made in a comment there (#29962 (comment)) that we should only use pd.NA independent of the underlying dtype. (I agree with this)

In either case, I think we need to make a decision and figure out how to do a deprecation notice. Or as you (@jbrockmendel ) suggested in another comment in that issue (#29962 (comment)), just bite the bullet and make the change now for 2.0.

[jbrockmendel]

that we should only use pd.NA independent of the underlying dtype

-1. As long as we distinguish between pd.NA and nan etc (xref #32265), this is a semantic change. Besides which getting pd.NA is a PITA.

Another alternative would be #37930 which would let users specify. That would likely be the biggest breaking change.

[Dr-Irv]

Besides which getting pd.NA is a PITA.

Can you explain why getting pd.NA is a PITA in this context?

And why would we need to distinguish between np.nan and pd.NA for a categorical?

[jorisvandenbossche]

The idea in #29962 is to make the NA value dependent on the underlying dtype of the categorical. But there is also a point made in a comment there (#29962 (comment)) that we should only use pd.NA independent of the underlying dtype. (I agree with this)

I also agree with that (that we should move to only use pd.NA for all dtypes), but as long as we still have dtypes for now that don't use pd.NA (which are actually the default), I think the most logical thing to do is to let Categorical follow the dtype of its categories. If it represents numpy-dtype based categories, use np.nan (and maybe np.NaT for datetime), and if it represents nullable data (a "nullable categorical dtype"), then we should use pd.NA as missing value scalar.

That makes that people who start using the nullable dtypes are ensured they keep nullable dtypes (and matching missing value scalars), while people who didn't opt in to nullable dtypes just keep the current behaviour.
(in theory, a user that just uses the defaults should currently never see pd.NA, as that is still fully opt-in)

(for a long time, a first step was actually support having nullable categories, but I suppose that was "fixed" now we can support EAs in the Index?)

[Dr-Irv]

I changed the title of the issue, and summarize here the discussion on 2/8/2023:
Joris: The way forward is to first properly support “nullable categorical dtypes” (categorical dtype with nullable categories): comparison operations, missing value scalar, convert_dtypes (and everywhere use_nullable_dtypes param exists)

[jorisvandenbossche]

So while a Categorical can now store categories using a nullable dtype, there are still a variety of aspects that don't follow the expected behaviour for "nullable dtypes" (see comment above). Just as a quick illustration of the comparison case:

In [25]: s = pd.Series([1, 2, pd.NA], dtype="Int64")

In [26]: s_cat = s.astype("category")

In [27]: s == 1
Out[27]: 
0     True
1    False
2     <NA>
dtype: boolean

In [28]: s_cat == 1
Out[28]: 
0     True
1    False
2    False
dtype: bool

I expect a "nullable" categorical column to give the same result as for the non-categorical s == 1.