BUG: groupby.agg with as_index=False and duplicate columns in by fails with pd.NamedAgg.

[sfc-gh-rdurrani]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df = pd.DataFrame(
           {
               "col1": [2, 1, 1, 0, 2, 0],
               "col2": [4, 5, 36, 7, 4, 5],
               "col3": [3.1, 8.0, 12, 10, 4, 1.1],
               "col4": [17, 3, 16, 15, 5, 6],
               "col5": [-1, 3, -1, 3, -2, -1],
           }
)

df.groupby(by=["col1", "col1", "col2"], as_index=False).agg({"col1": [min, max], "col2": "count"}) # works
df.groupby(by=["col1", "col1", "col2"], as_index=False).agg(new_col=pd.NamedAgg('col1', min), new_col1=pd.NamedAgg('col1', max), new_col2=('col2', 'count')) # Fails

Issue Description

df.groupby with duplicate columns in by works with agg when a dictionary is passed in for aggregation function, but fails when pd.NamedAgg is used to provide aggregation functions.

Expected Behavior

Should work, and have a similar output as passing in a dictionary.

Installed Versions

INSTALLED VERSIONS

commit : bdc79c1
python : 3.10.14.final.0
python-bits : 64
OS : Darwin
OS-release : 23.4.0
Version : Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:49 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6020
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.1
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 68.2.2
pip : 23.3.1
Cython : None
pytest : 7.4.4
hypothesis : None
sphinx : 5.0.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.23.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.3.1
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 16.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.13.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report. You checked the box I have confirmed this bug exists on the main branch of pandas. I am seeing the following output on main:

print(df.groupby(by=["col1", "col1", "col2"], as_index=False).agg(new_col=pd.NamedAgg('col1', min), new_col1=pd.NamedAgg('col1', max), new_col2=('col2', 'count')))
#    col1  col2  new_col  new_col1  new_col2
# 0     0     5        0         0         1
# 1     0     7        0         0         1
# 2     1     5        1         1         1
# 3     1    36        1         1         1
# 4     2     4        2         2         2

Can you confirm if you are seeing an error or the above output on main?

[sfc-gh-rdurrani]

I can't seem to install pandas from source - I keep getting the following error:

ModuleNotFoundError: No module named 'pandas._libs.pandas_parser'

but I did notice that the bug is happening on the latest version of pandas - 2.2.2.

[VISWESWARAN1998]

@sfc-gh-rdurrani Your code runs fine without any errors in main branch for me.

I can't seem to install pandas from source - I keep getting the following error:

ModuleNotFoundError: No module named 'pandas._libs.pandas_parser'

but I did notice that the bug is happening on the latest version of pandas - 2.2.2.

Try creating an isolated environment using mamba (recommended) or pip, install the requirements in requirements-dev.txt and try installing pandas from source. More information here: https://pandas.pydata.org/docs/dev/development/contributing_environment.html

[rhshadrach]

Thanks @sfc-gh-rdurrani - there isn't a requirement to try it out on main (but you're certainly welcome to!). I just wanted to make sure I wasn't making a mistake when I see this is fixed on the main branch.

[jasonmokk]

take