why are TfidfVectorizer and CountVectorizers not TransformerMixIn?

[shgoren]

why are TfidfVectorizer and CountVectorizer not TransformerMixIn?
I would expect anything with transform and fit_transform to be a TransformerMixIn
I'm using this for typing and was surprised to see that TfidfVectorizer and CountVectorizer do not have the TransformerMixIn
perhaps this should be added?

Thanks!

[glemaitre]

They are vectorizers that are relatively different from normal transformers because of the supplied X. For instance, the TfidfTransformer is such a transformer taking an input 2D matrix X. A vectorizer is taking a list of document instead.

[shgoren]

In what sense are they conceptually different?

I am using pandas so was thinking of the list of documents as a 1d vector of sentences. Then it's just a transformer with 1d to many type of relationship.

[glemaitre]

The 2D matrix acknowledges that we deal with structure data while the 1D vector is unstructured and one wants to vectorize to get this structure dataset.

A scikit-learn transformer using the mixin acknowledge that it requires structured data. Our common tests and the check_estimator provided to develop scikit-learn custom estimator rely on this to check the compatibility. Indeed, the vectorizers will not pass these checks.

[aoot]

A TfidfVectorizer = (CountVectorizer, TfidfTransformer) according to sklearn documentation on TfidfVectorizer.

One way to think of them is in terms of the interface of these classes (sklearn tends to use duck-typing). A CountVectorizer: strings -> 1D vector, and TfidfTransformer: 1D vector -> 1D vector. By piping CountVectorizer to TfidfTransformer we get a net transformation of string -> 1D vector, which is the interface of a vectorizer instead of a transformer.

This is why TfidfVectorizer is not using the TransformerMixin.