KeyError: 'django.conf.global_settings'

[Naddiseo]

Not sure where to report this, or even how to debug it, but I've been receiving the following error for the past few days that seems to be coming from django-stubs:

$ c&&DJANGO_PROJECT_SETTINGS=project.settings mypy --pdb --show-traceback project/app1/folder/folder2/somefile.py
~/env/lib/python3.7/site-packages/django-stubs/conf/__init__.pyi:10: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.670
Traceback (most recent call last):
  File "~/env/bin/mypy", line 10, in <module>
    sys.exit(console_entry())
  File "mypy/semanal.py", line 3787, in accept
  File "mypy/nodes.py", line 845, in accept__Node_glue
  File "mypy/nodes.py", line 846, in accept
  File "mypy/semanal.py", line 801, in visit_class_def__StatementVisitor_glue
  File "mypy/semanal.py", line 804, in visit_class_def
  File "mypy/semanal.py", line 822, in analyze_class
  File "mypy/semanal.py", line 830, in analyze_class_body_common
  File "mypy/semanal.py", line 906, in apply_class_plugin_hooks
  File "~/env/lib/python3.7/site-packages/mypy_django_plugin/transformers/settings.py", line 91, in __call__
    module = api.modules[module_name]
KeyError: 'django.conf.global_settings'
~/env/lib/python3.7/site-packages/django-stubs/conf/__init__.pyi:10: : note: use --pdb to drop into pdb
(Pdb) p api.modules.keys()
dict_keys(['project.app1.folder.folder2.somefile', 'project', 'django.db.models.query', 'django.core.checks', 'django.apps', 'django.core.checks.registry', 'django.test', 'django.contrib.auth', 'django.test.testcases', 'django.test.utils', 'django.contrib.auth.backends', 'django.db.models.options', 'django.contrib.auth.signals', 'django.core.servers.basehttp', 'django.test.runner', 'django.conf', 'django.contrib.postgres.fields', '_ast', '_importlib_modulespec', 'ast', 'types', 'abc', 'typing', 'mmap', 'codecs', 'os.path', 'importlib.abc', 'collections', 'io', 'collections.abc', 'sys', 'posix', 'os', 'importlib.machinery', 'importlib.util', 'importlib', 'builtins', 'email.errors', 'email.charset', 'wsgiref.headers', 'django.core.management.color', 'wsgiref.types', 'django.contrib.postgres.fields.mixins', 'django.contrib.postgres', 'argparse', 'wsgiref', 'django.core.servers', 'contextlib', 'warnings', 'django.contrib.contenttypes', 'django.core.handlers', 'json.encoder', 'json.decoder', '_markupbase', 'html', 'django.urls.utils', 'django.contrib', 'django.contrib.sessions', 'django.contrib.sessions.backends', 'django.http.cookie', 'django.db.migrations.operations.base', 'functools', 'django.db.backends.base', 'django.db.backends', 'django.db.backends.sqlite3', 'threading', 'string', 'django.template.loaders', 'enum', 'django.db.backends.ddl_references', 'django.template.backends', 'tempfile', 'django.core.checks.messages', 'numbers', 'django.db.utils', 'django.core.files.utils', 'time', 'itertools', 'uuid', 'unittest.mock', 'django.utils.safestring', 'django.core', 'django.utils', 'django.utils.version', 'mypy_extensions', 'typing_extensions', '__future__', 'project.office', 'project.office.bankimport', 'project.office.bankimport.ml', 'email.header', 'http', 'wsgiref.util', 'socket', 'json', 'django.urls.converters', 'html.parser', 'logging', 're', 'django.core.files.temp', 'django.utils.datastructures', 'decimal', 'django.core.files.base', 'datetime', 'django', 'email.contentmanager', 'email.policy', 'email.message', 'wsgiref.handlers', 'socketserver', 'django.urls.resolvers', 'django.db.backends.utils', 'sqlite3.dbapi2', 'unittest', 'django.core.files', 'django.forms.utils', 'django.core.exceptions', 'email', 'django.urls.conf', 'django.urls.base', 'sqlite3', 'django.core.files.images', 'django.core.validators', 'django.core.files.uploadedfile', 'http.server', 'django.contrib.auth.validators', 'wsgiref.simple_server', 'django.contrib.auth.base_user', 'django.db.models.fields.proxy', 'django.db', 'django.db.models.deletion', 'django.core.management.base', 'django.core.serializers.base', 'django.core.serializers.python', 'django.contrib.contenttypes.models', 'django.core.serializers.json', 'django.core.handlers.base', 'django.contrib.auth.models', 'django.test.client', 'django.core.handlers.wsgi', 'django.urls.exceptions', 'django.http.response', 'django.template.smartif', 'django.urls', 'django.contrib.sessions.backends.base', 'django.http', 'django.apps.config', 'django.db.backends.base.base', 'django.template.loader_tags', 'django.template.defaulttags', 'django.http.request', 'django.db.models.sql.subqueries', 'django.utils.functional', 'django.db.backends.base.schema', 'django.db.backends.sqlite3.base', 'django.db.models.fields.reverse_related', 'django.db.models.fields.related_descriptors', 'django.template.exceptions', 'django.template.context', 'django.template.utils', 'django.template.loaders.base', 'django.template.library', 'django.template.base', 'django.db.models.sql.datastructures', 'django.db.models.sql', 'django.core.files.storage', 'django.db.models.sql.where', 'django.db.models.sql.compiler', 'django.db.models.fields.mixins', 'django.db.models.indexes', 'django.db.models.lookups', 'django.db.models.fields.related', 'django.db.models.fields', 'django.db.models.aggregates', 'django.template', 'django.template.engine', 'django.template.backends.base', 'django.db.models.sql.query', 'django.db.models.expressions', 'django.db.models.fields.files', 'django.forms.formsets', 'django.db.models.query_utils', 'django.db.models.manager', 'django.db.models.base', 'django.db.models', 'django.forms.renderers', 'django.forms.boundfield', 'django.forms.fields', 'django.forms.widgets', 'django.forms.models', 'django.forms.forms', 'django.forms', 'django.core.files.uploadhandler', 'django.dispatch.dispatcher', 'django.db.migrations.operations.models', 'django.db.migrations.operations.fields', 'django.utils.html', 'django.db.migrations.state', 'django.apps.registry', 'django.db.models.fields.related_lookups', 'django.utils.tree', 'django.db.migrations.operations.special', 'django.dispatch', 'django.db.migrations.migration', 'django.template.defaultfilters', 'django.db.migrations.operations', 'django.db.models.signals', 'django.db.migrations', 'django.core.serializers', 'django.core.wsgi', 'django.contrib.postgres.fields.citext', 'django.contrib.postgres.fields.array', 'django.contrib.contenttypes.fields'])
(Pdb) p self.settings_modules
['django.conf.global_settings']

Unfortunately, our codebase is too big to reduce to a test case, but I can do local debugging if you let me know what/where to look.

[ncvc]

I'm also getting this error on Python 3.6.5 with

django==2.1.7
django-stubs==0.10.2
mypy-mypyc==0.670

But uninstalling mypy-mypyc and installing mypy==0.670 works for me.

@Naddiseo can you check if this workaround works for you too?

[mkurnikov]

@ncvc This means that my monkeypatches aren't going to work anymore after python/mypy#6588 lands...

[msullivan]

What does django-stubs need in order to not need to monkeypatch any mypy internals? We can arrange to have the right hooks, within reason.

[mkurnikov]

There are two:

1. https://github.com/mkurnikov/django-stubs/blob/master/mypy_django_plugin/monkeypatch.py#L15

It forces mypy to load modules which never appear in the import graph: django.conf.global_settings, value of DJANGO_SETTINGS_MODULE and (recently added) mypy_extensions to be able to generate TypedDict objects dynamically.

Probable workaround: ask users to seed from those places explicitly, and fallback to Any if they don't to prevent false-positives.

Hook: allow plugin to specify additional modules to seed mypy type checking. python/mypy#5911

2. https://github.com/mkurnikov/django-stubs/blob/master/mypy_django_plugin/monkeypatch.py#L37

Django stores project settings in special DJANGO_SETTINGS_MODULE file, loads it via import_module and allow them to be available from

from django.conf import settings
settings.MY_SETTING

Problem here is that django.conf.settings has no dependencies on the DJANGO_SETTINGS_MODULE module, so it could be processed first, and settings could not be loaded into the django.conf.settings object
https://github.com/mkurnikov/django-stubs/blob/master/mypy_django_plugin/transformers/settings.py#L94.

I've added user settings modules as a dependency for django.conf module, got idea from @ilevkivskyi, so they would load together, as a monkeypatch.

Probable workaround: add catch-all __getattr__ for django.conf.settings object so that get_attribute_hook would always be invoked, wait for mypy refactoring to be over and then process every setting call when respective setting module had been loaded. Assuming I understand correctly how python/mypy#6259 works.

[msullivan]

It seems to me that 1 isn't necessarily needed and should fall out of adding the dependencies (at least, if the dependencies are to stubs, which are always follow=normal?).

With that in mind, it seems like it should be sufficient to add a hook that allows a plugin to add additional dependencies to a module?

[mkurnikov]

So, I will be able to express "make a config.settings.my_local_settings a dependency of django.conf", and mypy will make sure it's loaded before django.conf, right? This should be sufficient.

[msullivan]

Yeah

[ilevkivskyi]

Is this still relevant or can be closed in view of python/mypy#6598 being already merged?

[mkurnikov]

It's going to be relevant when 0.700 will be out, should be closed after a merge of this one https://github.com/mkurnikov/django-stubs/pull/60. Current django-stubs does not support 0.700, I just wait for it to be out to release compatible version.