BUG: GroupBy.apply reruns the full loop if an exception is raised

[Kodiologist]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

Code Sample

import pandas as pd

d = pd.DataFrame(dict(a = range(10)))

def f(x):
    print(x)
    if x.iloc[0,0] == 2:
       raise Exception('problem')

print(d.groupby('a').apply(f))

The result is

   a
0  0
   a
1  1
   a
2  2
   a
0  0
   a
1  1
   a
2  2
Traceback (most recent call last):
  File "/tmp/ex.py", line 10, in <module>
    print(d.groupby('a').apply(f))
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/groupby/groupby.py", line 894, in apply
    result = self._python_apply_general(f, self._selected_obj)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/groupby/groupby.py", line 928, in _python_apply_general
    keys, values, mutated = self.grouper.apply(f, data, self.axis)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/groupby/ops.py", line 238, in apply
    res = f(group)
  File "/tmp/ex.py", line 8, in f
    raise Exception('problem')
Exception: problem

Problem description

If the func given to GroupBy.apply(func) raises an exception, the exception is suppressed and the application loop restarts from the beginning. My guess is that this is related to the behavior of one of pandas's apply functions that's supposed to ignore exceptions only for the first iteration, or only when an empty argument is provided, to guess the appropriate shape for the output.

#17326 may be related.

Expected Output

The loop shouldn't catch the exception, terminating as soon as it hits the raise.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 7d32926
python : 3.8.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.8.0-44-generic
Version : #50-Ubuntu SMP Tue Feb 9 06:29:41 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.2
numpy : 1.18.4
pytz : 2020.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 51.3.3
Cython : None
pytest : 6.2.1
hypothesis : None
sphinx : 3.4.3
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.5.1
html5lib : 1.1
pymysql : None
psycopg2 : 2.8.5 (dt dec pq3 ext lo64)
jinja2 : 2.11.2
IPython : None
pandas_datareader: None
bs4 : 4.9.1
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[mroeschke]

Looks like on master this doesn't run twice now

In [6]: import pandas as pd
   ...:
   ...: d = pd.DataFrame(dict(a = range(10)))
   ...:
   ...: def f(x):
   ...:     print(x)
   ...:     if x.iloc[0,0] == 2:
   ...:        raise Exception('problem')
   ...:
   ...: print(d.groupby('a').apply(f))
   a
0  0
   a
1  1
   a
2  2
Exception: problem

I believed this was fixed recently as apply use to have a fall back path which would make this run twice. This should either be fixed in 1.3 or the next release of pandas. Closing.