Issue about "Can't perform this operation for unregistered loader type"

[jfzsl]

My script runs correctly and output what I wanted.
But I packed my python scripts to exe file, it raised "Can't perform this operation for unregistered loader type" when I run the exe file.

my nuitka command:
python -m nuitka --mingw64 --standalone --show-progress --recurse-all --enable-plugin=numpy --include-package=openpyxl --include-package=markupsafe --include-package=jinja2 --follow-imports --output-dir=exeFiles mytest.py

my envirionmnet:
Windows 7
I installed all the packages(nuitka, jinja2,...) by pip

python -m nuitka --version
0.6.8.4
Python: 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 23 2018, 23:31:17) [MSC v.1916 32 bit
(Intel)]
Executable: C:\Program Files (x86)\Python36-32\python.exe
OS: Windows
Arch: x86

my source code:
#===============mytest.py=========================

import pandas as pd
import numpy as np



def highlight(df, color = "yellow"):

    attr = 'background-color: {}'.format(color)
    df_bool = pd.DataFrame(df.apply(lambda x: [True if x.iloc[0] < v else False for v in x],axis=1).apply(pd.Series),
                      index=df.index)
    df_bool.columns =df.columns
    return pd.DataFrame(np.where(df_bool, attr, ""),
                       index= df.index, columns=df.columns)
                       
if __name__ == "__main__":
    print('writing....')
    dict = {'A': [1, 1, 1, 1, 1], 'B':[2,1,2,1,2], 'C':[1,2,1,2,1]}
    df = pd.DataFrame(dict)
    try:
        df.style. \
            apply(highlight, axis=None).\
            to_excel("teststyled.xlsx", engine="openpyxl")
        print('Done!')
    except Exception as e:
        print('write error')
        print(str(e))

[jfzsl]

Does any one know this problem? Is it nuitka issue or my problem?

Thank you!

[kayhayen]

I didn't find where the error message comes from by looking ath CPython source code. The example code is good enough though, I will eventually try and reproduce it. Doesn't look like an error of yours, probably something pkgutil or pandas would do. Nuitka has its own loader type, and registers it in some places, but potentially there are more.

[jfzsl]

Thank you very much Kay.
The error message come from "myPython\site-packages\pkg_resources_init_.py "
and it related with Excel style setting "df.style.apply(highlight, axis=None)"

If i did not set the Excel file style, every thing is OK.

[kayhayen]

Ok, great. Yes, pkgutil supports loading data files for packages. It would be your responsibility to make sure they are there, but the error seems to indicate that it's not accepting the nuitka loader to do it. We have implemented get_data and that was working at one time, but maybe this is another trick needed. I will look into that now.

[jfzsl]

@kayhayen
Hello Kay:
Do you have a release plan and the bug fix list?
I had a dream yesterday night, a real dream :), Nuitka had a new release and all the bugs I have met have been fixed. So today I logged in again. But the real work is disappointed :). haha

[fleimgruber]

I can reproduce this with Python 3.8.5, Nuitka 0.6.10.5, pandas 1.1.3 on Windows 10.

The MRE is (note that the pandas._libs.* imports are needed because of e.g. #837) with the relevant line from pandas.io.formats.style import Styler, command and traceback at the very end:

import pandas._libs.window
import pandas._libs.tslibs.base
import pandas as pd
from pandas.io.formats.style import Styler


def main():
    inp = input('enter')


if __name__ == '__main__':
    main()

I would be happy to help with further debugging and testing. Where would be the best place to look into the Nuitka loader?

python -m nuitka --follow-imports --show-progress --show-scons .\example_loader_type.py
...
Nuitka-Scons:INFO: Compiled 389 C files using ccache.
Nuitka-Scons:INFO: Cached C files (using ccache) with result 'cache miss': 389
Nuitka:INFO: Successfully created 'example_loader_type.exe'.
(precert_38) PS C:\Users\user\lg\python\nuitka> .\example_loader_type.exe
Traceback (most recent call last):
  File "C:\Users\user\lg\python\nuitka\example_loader_type.py", line 4, in <module>
    from pandas.io.formats.style import Styler
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\Users\user\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\pandas\io\formats\style.py", line 62, in <module pandas.io.formats.style>
    class Styler:
  File "C:\Users\user\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\pandas\io\formats\style.py", line 135, in Styler
    template = env.get_template("html.tpl")
  File "C:\Users\user\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\jinja2\environment.py", line 883, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "C:\Users\user\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\jinja2\environment.py", line 857, in _load_template
    template = self.loader.load(self, name, globals)
  File "C:\Users\user\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\jinja2\loaders.py", line 115, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "C:\Users\user\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\jinja2\loaders.py", line 248, in get_source
    if not self.provider.has_resource(p):
  File "C:\Users\user\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\pkg_resources\__init__.py", line 1389, in has_resource
    return self._has(self._fn(self.module_path, resource_name))
  File "C:\Users\user\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\pkg_resources\__init__.py", line 1456, in _has
    raise NotImplementedError(
NotImplementedError: Can't perform this operation for unregistered loader type

[kayhayen]

This is very helpful @fleimgruber as it doesn't involve standalone mode, this is likely more workable. This should be as easy as having a plugin that post load of pkg_resources registers the Nuitka loader class with register_loader_type, except I am uncertain what that adapter object needs to be there.

[fleimgruber]

This should be as easy as having a plugin that post load of pkg_resources registers the Nuitka loader class with register_loader_type

@kayhayen glad to help. Could you give me a rough idea on where in the code registering should be done? Or do we need to create the mentioned plugin (still need to get familiar with the Nuitka concepts)? Also, what do you suggest would be a good dev / test setup for this? I would work on Nuitka master, but building and then executing the binary to see the effects is not really practical. Being able to reproduce this without building would be a good first step. WDYT?

I would then look into the relevant docs and see what I can come up with:

https://setuptools.readthedocs.io/en/latest/pkg_resources.html#supporting-custom-importers
https://www.python.org/dev/peps/pep-0302/
https://setuptools.readthedocs.io/en/latest/pkg_resources.html#iresourceprovider
https://setuptools.readthedocs.io/en/latest/pkg_resources.html#resourcemanager-api

[fleimgruber]

I found

Nuitka/nuitka/plugins/standard/NumpyPlugin.py

Line 176 in 0083a93

class NumpyPlugin(NuitkaPluginBase):

and from the docstring I would assume that the Nuitka loader should be registered in the Numpy plugin?

[kayhayen]

I don't think it has anything to do with numpy @fleimgruber but pkg_resources.

I think a better starting point would be the enum-compat plugin. That one is does this kind of thing:

    @staticmethod
    def createPostModuleLoadCode(module):
        full_name = module.getFullName()

        if full_name == "enum":
            code = """\
from __future__ import absolute_import
import enum
try:
    enum.Enum.__new__ = staticmethod(enum.Enum.__new__.__func__)
    enum.IntEnum.__new__ = staticmethod(enum.IntEnum.__new__.__func__)
except AttributeError:
    pass
"""
            return (
                code,
                """\
Monkey patching "enum" for compiled '__new__' methods.""",
            )

        return None, None

[kayhayen]

You would have to mimic that, and handle pkg_resources, and the __loader__ of Python3 will be there to help you and point to a Nuitka loader instance, which has a class, which probably is what needs to be registered. Use --debug and in the build directory, a Python file will show up with your source code in case you need that.

I would appreciate if you could take this on. I think it's as simple as making sure to execute a bit of code right after pkg_resources is loaded. I think I remember that Jinja2 is actually the one to blame on one of the stacks I saw, with it asking that module for a resource, probably the data with a jinja2 template.

Do not use standalone mode while developing it, that leaves the resources at where __file__ would have them expected and generally be fast. For a minimal test case in development, just call has_resource yourself.

[fleimgruber]

Thanks for the pointers and thoughts, I will try to look into this.

[fleimgruber]

I tried to reproduce this only using jinja2 (copied the relevant lines from Pandas) as you suggested via

import jinja2


def main():
    loader = jinja2.PackageLoader("pandas", "io/formats/templates")
    env = jinja2.Environment(loader=loader, trim_blocks=True)
    template = env.get_template("html.tpl")


if __name__ == '__main__':
    main()

and the command

python -m nuitka --debug --follow-imports --show-progress --show-scons .\test_pkg_resources.py

but that errors with a C compilation warning:

scons: building `module.jinja2.utils.o' because it doesn't exist
C:\Users\user\AppData\Local\Nuitka\Nuitka\ccache\v3.7.12\ccache.exe C:\Users\user\AppData\Local\Nuitka\Nuitka\gcc\x86_64\10.2.0-11.0.0-8.0.0-r5\mingw64\bin\gcc.exe -o module.jinja2.utils.o -c -std=c11 -fwrapv -fpartial-inlining -Wunused-but-set-variable -Wno-deprecated-declarations -fno-var-tracking -Wall -Werror -Wno-error=strict-aliasing -Wno-strict-aliasing -Wno-error=format -Wno-format -Wno-misleading-indentation -g -feliminate-unused-debug-types -O2 -pipe -D_WIN32_WINNT=0x0501 -DMS_WIN64 -D_NUITKA_CONSTANTS_FROM_RESOURCE -D_NUITKA_FROZEN=0 -D_NUITKA_EXE -IC:\Users\user\AppData\Local\Continuum\miniconda3\envs\nuitka_dev_38\include -I. -IC:\Users\USER\dev\nuitka\nuitka\build\include -IC:\Users\USER\dev\nuitka\nuitka\build\static_src module.jinja2.utils.c
Nuitka-Scons:INFO: Slow C compilation detected, used 60s so far, this might indicate scalability problems.
module.jinja2.runtime.c:22041:18: error: 'MAKE_GENERATOR_jinja2$runtime$$$function__59___iter__$$$genobj__1___iter__' defined but not used [-Werror=unused-function]
22041 | static PyObject *MAKE_GENERATOR_jinja2$runtime$$$function__59___iter__$$$genobj__1___iter__(struct Nuitka_CellObject **closure) {
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1.exe: all warnings being treated as errors
scons: *** [module.jinja2.runtime.o] Error 1
scons: internal stack trace:
  File "c:\users\USER\dev\nuitka\nuitka\build\INLINE~1\lib\scons-3.1.2\SCons\Job.py", line 258, in run
    task.execute()
  File "c:\users\USER\dev\nuitka\nuitka\build\INLINE~1\lib\scons-3.1.2\SCons\Script\Main.py", line 206, in execute
    SCons.Taskmaster.OutOfDateTask.execute(self)
  File "c:\users\USER\dev\nuitka\nuitka\build\INLINE~1\lib\scons-3.1.2\SCons\Taskmaster.py", line 255, in execute
    self.targets[0].build()
  File "c:\users\USER\dev\nuitka\nuitka\build\INLINE~1\lib\scons-3.1.2\SCons\Node\__init__.py", line 766, in build
    self.get_executor()(self, **kw)
  File "c:\users\USER\dev\nuitka\nuitka\build\INLINE~1\lib\scons-3.1.2\SCons\Executor.py", line 397, in __call__
    return _do_execute_map[self._do_execute](self, target, kw)
  File "c:\users\USER\dev\nuitka\nuitka\build\INLINE~1\lib\scons-3.1.2\SCons\Executor.py", line 131, in execute_action_list
    raise status    # TODO pylint E0702: raising int not allowed
Nuitka-Scons:INFO: Running 'C:\\Users\\user\\AppData\\Local\\Nuitka\\Nuitka\\ccache\\v3.7.12\\ccache.exe C:\\Users\\user\\AppData\\Local\\Nuitka\\Nuitka\\gcc\\x86_64\\10.2.0-11.0.0-8.0.0-r5\\mingw64\\bin\\gcc.exe -o "module.jinja2.nodes.o" -c -std=c11 -fwrapv -fpartial-inlining -Wunused-but-set-variable -Wno-deprecated-declarations -fno-var-tracking -Wall -Werror -Wno-error=strict-aliasing -Wno-strict-aliasing -Wno-error=format -Wno-format -Wno-misleading-indentation -g -feliminate-unused-debug-types -O2 -pipe -D_WIN32_WINNT=0x0501 -DMS_WIN64 -D_NUITKA_CONSTANTS_FROM_RESOURCE -D_NUITKA_FROZEN=0 -D_NUITKA_EXE -IC:\\Users\\user\\AppData\\Local\\Continuum\\miniconda3\\envs\\nuitka_dev_38\\include -I. -IC:\\Users\\USER\\dev\\nuitka\\nuitka\\build\\include -IC:\\Users\\USER\\dev\\nuitka\\nuitka\\build\\static_src "module.jinja2.nodes.c"' took 96.43 seconds
scons: building terminated because of errors.

[fleimgruber]

Putting that aside for now and working with your actual suggestion (has_resource), I can't reproduce with the minimal:

def main():
    from pkg_resources import get_provider
    package_name = "pandas"
    provider = get_provider(package_name)
    p = 'io/formats/templates/html.tpl'
    provider.has_resource(p)


if __name__ == '__main__':
    main()

which should essentially be what jinja2 (v2.11.2, note pallets/jinja#1182) would be doing. Also, jinja will stop using pkg_resources from 3.0 onwards.

[fleimgruber]

For reference, pkg_resources support for pyinstaller report pyinstaller/pyinstaller#4881 and PR pyinstaller/pyinstaller#5284.

[kayhayen]

That debug mode error message is something I am looking into. As for the reproducer, maybe too much guessing from my side. I will also look at that PR code, maybe it's another method needed.

[fleimgruber]

Just a quick follow-up. Regarding the error in #752 (comment) and your #752 (comment), I tried again today with Nuitka 0.6.13.3 and I don't see the debug mode error anymore.

But after

python -m nuitka --debug --follow-imports --show-progress --show-scons .\example_loader_type.py

I see essentially the same error as in #752 (comment):

Traceback (most recent call last):
  File "C:\Users\USER\lg\python\nuitka\example_loader_type.py", line 4, in <module>
    from pandas.io.formats.style import Styler
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\Users\USER\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\pandas\io\formats\style.py", line 62, in <module pandas.io.formats.style>
    class Styler:
  File "C:\Users\USER\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\pandas\io\formats\style.py", line 135, in Styler
    template = env.get_template("html.tpl")
  File "C:\Users\USER\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\jinja2\environment.py", line 883, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "C:\Users\USER\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\jinja2\environment.py", line 857, in _load_template
    template = self.loader.load(self, name, globals)
  File "C:\Users\USER\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\jinja2\loaders.py", line 115, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "C:\Users\USER\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\jinja2\loaders.py", line 248, in get_source
    if not self.provider.has_resource(p):
  File "C:\Users\USER\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\pkg_resources\__init__.py", line 1389, in has_resource
    return self._has(self._fn(self.module_path, resource_name))
  File "C:\Users\USER\AppData\Local\Continuum\Miniconda3\envs\precert_38\lib\site-packages\pkg_resources\__init__.py", line 1456, in _has
    raise NotImplementedError(
NotImplementedError: Can't perform this operation for unregistered loader type

[fleimgruber]

I did another test with jinja2 loader using bokeh, the MRE:

from bokeh.embed import components

def main():
    inp = input('enter')

if __name__ == '__main__':
    main()

with the command

python -m nuitka --standalone example_bokeh_jinja2.py --plugin-enable=numpy

and got a different error, this time directly from jinja2. It seems that bokeh uses jinja2 in a way that does not involve pkg_resource, maybe this helps as additional information.

> .\example_bokeh_jinja2.dist\example_bokeh_jinja2.exe
Traceback (most recent call last):
  File "C:\Users\LEIMGR~1\lg\python\nuitka\EXAMPL~1.DIS\example_bokeh_jinja2.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\Users\LEIMGR~1\lg\python\nuitka\EXAMPL~1.DIS\bokeh\embed\__init__.py", line 23, in <module bokeh.embed>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\Users\LEIMGR~1\lg\python\nuitka\EXAMPL~1.DIS\bokeh\embed\server.py", line 25, in <module bokeh.embed.server>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\Users\LEIMGR~1\lg\python\nuitka\EXAMPL~1.DIS\bokeh\core\templates.py", line 93, in <module bokeh.core.templates>
  File "C:\Users\LEIMGR~1\lg\python\nuitka\EXAMPL~1.DIS\jinja2\environment.py", line 883, in get_template
  File "C:\Users\LEIMGR~1\lg\python\nuitka\EXAMPL~1.DIS\jinja2\environment.py", line 857, in _load_template
  File "C:\Users\LEIMGR~1\lg\python\nuitka\EXAMPL~1.DIS\jinja2\loaders.py", line 115, in load
  File "C:\Users\LEIMGR~1\lg\python\nuitka\EXAMPL~1.DIS\jinja2\loaders.py", line 197, in get_source
jinja2.exceptions.TemplateNotFound: js_resources.html

[royccg]

jinja2. It seems that bokeh uses

I have the same problem about the Jinja2.
jinja2.exceptions.TemplateNotFound: js_resources.html
Is there a solution now？Thanks a lot.

[kayhayen]

I have made successful deployments with Jinja2 for all kinds of loaders now. I think the has_resource should not longer be a problem, in fact the loader now has more methods. And usually Jinja2 doesn't go this way. This must be a strange one.

[kayhayen]

Trying out the reproduce right now, fodder for anti-bloat tuning definitely, lets see if this still happens. I have plans on specializing os.path.exists for Nuitka commercial, and this ties in with that clearly.

[royccg]

I have made successful deployments with Jinja2 for all kinds of loaders now. I think the has_resource should not longer be a problem, in fact the loader now has more methods. And usually Jinja2 doesn't go this way. This must be a strange one.

I add the command --include-package-data=bokeh, now the bug is fixed.
thanks a lot.

[kayhayen]

This pulls in a lot of numba modules, most of which are of course very irrelevant, e.g. analysing elf binaries, this has taken a few iterations so far already, but will shapen anti-bloat for this side of numba. I don't think I usually see it used from Pandas imports, but maybe I didn't notice yet.

[kayhayen]

We try to have required package data automatically included through our Yaml files. Not sure if absent data files show with the symptom described, but possibly they do, which would be ugly, but what can we do. There is a bigger changing coming, where some sort of interactive mode, will trace files that didn't exist, DLLs that failed to be found, etc. and give proper warnings at runtime, but that's probably months away still.

[fleimgruber]

When I just tried with the option mentioned in #752 (comment) I get this on executing the resulting binary:

ValueError: The 'pandas' package was not installed in a way that PackageLoader understands.

where PackageLoader is from jinja2 I guess. Also, I am using Nuitka 0.9.6 and Python 3.8.11.

[kayhayen]

This will probably be reproducible, the loader has gained more compatibility, I am now trying your reproducer from earlier @fleimgruber which seems like it might point out, what to do, if anything still.

[fleimgruber]

Additional information:

> nuitka --version

0.9.6
Commercial: None
Python: 3.8.11 (default, Aug  6 2021, 09:57:55) [MSC v.1916 64 bit (AMD64)]
Flavor: Anaconda Python
Executable: C:\Users\LeimgruberF\opt\Miniconda3\envs\app_nuitka\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 10

[kayhayen]

So, new Jinja2 package loader doesn't use pkg_resources anymore in its package loader, but instead expects a registered loader class, for which now the pkg_resources plugin (always active) create a post-load code to provide that adapter. With that now, not only jinja2.FileSystemLoader but also jinja2.PackageLoader is working, and the later properly allows file embedding for Nuitka commercial.

[kayhayen]

https://nuitka.net/doc/factory.html

[fleimgruber]

Using latest factory, I don't see the jinja2.PackageLoader error anymore. Will test more after #1710.

[fleimgruber]

Somehow now this is failing again, even with 1.1rc (factory). I will test more and come up with a MRE.

Just to be sure: it is enough to remove main.build and main.dist directories for a --standalone build? Do I need to invalidate caches manually as well, for ccache for example?

[fleimgruber]

Just tested with this MRE:

import pandas as pd
from pandas.io.formats.style import Styler


def main():
    inp = input('enter')


if __name__ == '__main__':
    main()

> poetry run python -m nuitka --standalone --mingw --plugin-enable=numpy --include-package-data=bokeh --show-scons main.py
Nuitka-Options:INFO: Used command line options: --standalone --mingw --plugin-enable=numpy --include-package-data=bokeh --show-scons main.py
Nuitka:INFO: Starting Python compilation with Nuitka '1.1rc1' on Python '3.9' commercial grade 'not installed'.
Nuitka-Plugins:INFO: pkg-resources: Injecting post-module load code for module 'pkg_resources':
Nuitka-Plugins:INFO: pkg-resources:     Registering Nuitka loader with "pkg_resources".
Nuitka-Plugins:INFO: multiprocessing: Injecting pre-module load code for module 'multiprocessing':
Nuitka-Plugins:INFO: multiprocessing:     Monkey patching "multiprocessing" load environment.
Nuitka-Plugins:INFO: multiprocessing: Injecting post-module load code for module 'multiprocessing':
Nuitka-Plugins:INFO: multiprocessing:     Monkey patching "multiprocessing" for compiled methods.
Nuitka:INFO: Completed Python level compilation and optimization.
Nuitka:INFO: Generating source code for C backend compiler.
Nuitka:INFO: Running data composer tool for optimal constant value handling.
Nuitka:INFO: Running C compilation via Scons.
...

> .\main.dist\main.exe
Traceback (most recent call last):
  File "C:\Users\LEIMGR~1\dev\PRECER~3\JINJA-~1\MAIN~1.DIS\main.py", line 2, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "C:\Users\LEIMGR~1\dev\PRECER~3\JINJA-~1\MAIN~1.DIS\pandas\io\formats\style.py", line 54, in <module pandas.io.formats.style>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "C:\Users\LEIMGR~1\dev\PRECER~3\JINJA-~1\MAIN~1.DIS\pandas\io\formats\style_render.py", line 71, in <module pandas.io.formats.style_render>
  File "C:\Users\LEIMGR~1\dev\PRECER~3\JINJA-~1\MAIN~1.DIS\pandas\io\formats\style_render.py", line 76, in StylerRenderer
  File "C:\Users\LEIMGR~1\dev\PRECER~3\JINJA-~1\MAIN~1.DIS\jinja2\loaders.py", line 323, in __init__
ValueError: The 'pandas' package was not installed in a way that PackageLoader understands.

[fleimgruber]

Since I am using the factory branch from a local git clone right now, I might as well set up a proper development environment for Nuitka. I am using VS Code anyways for the project to be compiled with Nuitka. Next I would try to mirror the workflow you have when testing such issues yourself. Are you using a Nuitka virtualenv for development and do you debug MREs in VS Code?

[kayhayen]

Running Nuitka inside Visual Code is an open issue, that is not solved yet. It insists on a clean environment, and Visual Code will not work yet.

I suspect, that Jinja2 package loader got changed in a later release. The MRE you gave does have a fix, that but it doesn't affect the Jinja2 anymore, because they decided, they are going to do their own file system based stuff.

They insist on the package folder and the data files to exist. If you include the files it expects, it will not give this error, which if course, can be very misleading. I managed to get commercial file embedding to work without the folder, and non-commercial will have the folder, well, for as long, as a package data file is included. I am going to use your MRE there, seems that renderer there is missing some data files, that we might want to add by default.

[kayhayen]

Your question regarding caching @fleimgruber currently I am not aware of any issues that are caused by caching, I never disable them myself and try to work with them being on.

[kayhayen]

But essentially, what happened is that this issue morphed from a pkg_resources specific problem to a jinaj2 one, that is only semi-real.

[fleimgruber]

Running Nuitka inside Visual Code is an open issue, that is not solved yet. It insists on a clean environment, and Visual Code will not work yet.

What I am doing right now is installing the MRE dependencies into a virtualenv (managed by Poetry) and specifying that virtualenv for VS Code. Then I use e.g. this in VS Code launch.json run configurations:

    "configurations": [
        {
            "name": "Issue 752 standalone",
            "type": "python",
            "request": "launch",
            "program": "nuitka/__main__.py",
            "console": "integratedTerminal",
            "justMyCode": true,
            "args": ["--standalone", "issue/752.py", "--mingw", "--show-progress", "--plugin-enable=numpy", "--include-package-data=bokeh"]
        },

which allows me to to set breakpoints and debug Nuitka using the VS Code visual debugger.

[fleimgruber]

I suspect, that Jinja2 package loader got changed in a later release. The MRE you gave does have a fix, that but it doesn't affect the Jinja2 anymore, because they decided, they are going to do their own file system based stuff.

Mhm, supporting 3rd party libs when they significantly change sounds like a lot of headaches.

[fleimgruber]

But essentially, what happened is that this issue morphed from a pkg_resources specific problem to a jinaj2 one, that is only semi-real.

By "semi-real" you are referring to your comment about package folder and including data files in #752 (comment)? I guess you are at it already using the MRE and including some data files by default. Let me know if I can help there now that my dev env is set up. Otherwise I would continue testing any fixes.

[kayhayen]

Basically it's not only about finding out what files are required there, nothing too difficult. We will have to describe the process at some point. In the case of Jinja2, the misleading code will become subject to changes through anti-bloat. I invite you to check out the yaml files in Nuitka plugins folder, they modify some modules already (called anti-bloat) or specify data files to be included for packages that require them absolutely.

[kayhayen]

One thing, about standalone that is bad, is now it doesn't include the traceback referenced source code, not even as an option, such that one gets to manually locate from a standalone traceback the source code in question. That makes the task a bit harder than necessary.

[fleimgruber]

Thanks for the pointers. I do have a lead, will try this some more tomorrow.

[fleimgruber]

If I read https://nuitka.net/doc/developer-manual.html#the-git-flow-model correctly, I should branch off of develop to try my hands at this? Or from factory in this case?

My plan so far and if I understand things correctly is to create nuitka/plugins/standard/PandasPlugin.py where a new NuitkaPluginPandas gets a method considerDataFiles (similar to the one in DataFileCollectorPlugin.py) which looks at the value for data-files given in the yaml file you mentioned standard.nuitka-package.config.yml under the pandas section. Do you recommend something different?

[kayhayen]

Well, the data files plugin is intended for that, actually you only would need to add content to the yaml, and code is very unlikely for data files to be included. Also there is an option for a user yaml file, so you do not immediately have to edit the yaml file mentioned, but can provide the proper snippet yourself. Visual Code will have a schema that it can apply, but I think it does not do it for user yaml files, we do not use patterns there, maybe Visual Code does not support that, not sure.

[fleimgruber]

Thanks for the explanation. So looking at the relevant lines in the pandas module in question IMHO the yaml should look something like:

- module-name: "pandas"
  dirs:
    - "io/formats/templates"
  data-files:
    patterns:
      - "io/formats/templates/*.tpl"

which when used as issue752.nuitka-package.config.yml in

poetry run python -m nuitka --standalone --mingw --user-package-configuration-file=issue752.nuitka-package.config.yml issue/752.py

Leads to an assertion error here which becomes clear when looking at the comment above it. So is the plan to use parsePackageYaml (in the same way it is used to merge Nuitka built-in package configs) also for the user provided yaml files?

[kayhayen]

Modules can only have a single configuration, merging is not yet supported, this hurts here. I will need to implement that. I am pushing a hotfix currently, that adds the config as follows:

- module-name: "pandas.io.formats"
  data-files:
    dirs:
      - "templates"

Currently on factory, but will be in 1.0.1. Also, the pkg_resources loader registry stuff is going to be in 1.0.1, so this will be fixed with the immanent release of 1.0.1 hotfix.

Sorry for overtaking you here, not a good thing to encourage contributions. Would you be willing to try and examine code and config for examples of how to do things, and add that to the documentation in a PR? I think, this should have been very straightforward for you by now, and that's the missing bit.

[fleimgruber]

@kayhayen sorry for not responding sooner to your question! I just had a look again and found some docs on:

https://nuitka.net/doc/nuitka-package-config.html

and also some here:

https://github.com/Nuitka/Nuitka/blob/12150ee1712b518f433978ee3a7f52535b2878d4/Standard-Plugins-Documentation.rst

I skimmed the docs and could not find anything related to config merging (it may have been implemented in the meantime?). I saw some placeholders for examples, so I could try to fill those, but the examples concerning my use cases are already included in the shipped standard.nuitka-package.config.yaml. Please let me know if you see a good place to contribute.

[kayhayen]

The first doc was online at that point for only a few hours. :-)

You mean Yaml config file merging for the same module? No, unfortunately that is not done yet, it's an omission that I need to work on. Actually I ought to change my personal workflow here from creating a user provided file, testing that, and then merging it with a tool automatically into the official file. Merging in the sense that multiple files work, that is done.

[fleimgruber]

:-) always surprised about those coincidences.

Yes, I was specifically referring to merging for the same module - but I can see that it's tricky. For the common packages, PRs to Nuitka are the way to go and user modules won't be conflicting with Nuitka module configs. So I guess it's a pragmatic "no need to implement at this point".

And having the ability to use multiple files is nice, thanks! I will try this some more for a personal project that needs it's template files included in that way.

[kayhayen]

There is also command line options for including data files by directory, by pattern, etc. doesn't have to Yaml for this, only DLLs have no options.