Django_migrations table not created?

[apenney]

I've spent a while trying to trouble shoot this, but I've given up and I'm here to beg for help. In a nutshell, I'm trying to add pytest-django to an existing legacy Django 1.8 codebase.

I'm in a weird position where I can do:

pytest tests/test_credentials.py - this will pass.
pytest tests - this will fail like:

collected 3851 items                                                                                             [55/47488]

simon/tests/test_alerting.py ..............                                                                         [  0%]
simon/tests/test_audit_log.py ..                                                                                    [  0%]
simon/tests/test_celery_routing.py .....                                                                            [  0%]
simon/tests/test_channel_log_dal.py ...                                                                             [  0%]
simon/tests/test_contactinfo.py .................................                                                   [  1%]
simon/tests/test_credentials.py E

========================================================= ERRORS ==========================================================
_______________________ ERROR at setup of TestCredentials.test_create_new_customer_database_record ________________________

request = <SubRequest '_django_db_marker' for <Function 'test_create_new_customer_database_record'>>

    @pytest.fixture(autouse=True)
    def _django_db_marker(request):
        """Implement the django_db marker, internal to pytest-django.

        This will dynamically request the ``db``, ``transactional_db`` or
        ``django_db_reset_sequences`` fixtures as required by the django_db marker.
        """
        marker = request.node.get_closest_marker("django_db")
        if marker:
            transaction, reset_sequences = validate_django_db(marker)
            if reset_sequences:
                request.getfixturevalue("django_db_reset_sequences")
            elif transaction:
                request.getfixturevalue("transactional_db")
            else:
>               request.getfixturevalue("db")

/root/.local/lib/python2.7/site-packages/pytest_django/plugin.py:485:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/root/.local/lib/python2.7/site-packages/pytest_django/fixtures.py:108: in django_db_setup
    **setup_databases_args
/root/.local/lib/python2.7/site-packages/django/test/runner.py:370: in setup_databases
    serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
/root/.local/lib/python2.7/site-packages/django/db/backends/base/creation.py:368: in create_test_db
    test_flush=not keepdb,
/root/.local/lib/python2.7/site-packages/django/core/management/__init__.py:120: in call_command
    return command.execute(*args, **defaults)
/root/.local/lib/python2.7/site-packages/django/core/management/base.py:445: in execute
    output = self.handle(*args, **options)
/root/.local/lib/python2.7/site-packages/django/core/management/commands/migrate.py:93: in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
/root/.local/lib/python2.7/site-packages/django/db/migrations/executor.py:19: in __init__
    self.loader = MigrationLoader(self.connection)
/root/.local/lib/python2.7/site-packages/django/db/migrations/loader.py:47: in __init__
    self.build_graph()
/root/.local/lib/python2.7/site-packages/django/db/migrations/loader.py:191: in build_graph
    self.applied_migrations = recorder.applied_migrations()
/root/.local/lib/python2.7/site-packages/django/db/migrations/recorder.py:60: in applied_migrations
    return set(tuple(x) for x in self.migration_qs.values_list("app", "name"))
/root/.local/lib/python2.7/site-packages/django/db/models/query.py:162: in __iter__
    self._fetch_all()
/root/.local/lib/python2.7/site-packages/django/db/models/query.py:965: in _fetch_all
    self._result_cache = list(self.iterator())
/root/.local/lib/python2.7/site-packages/django/db/models/query.py:1220: in iterator
    for row in compiler.results_iter():
/root/.local/lib/python2.7/site-packages/django/db/models/sql/compiler.py:794: in results_iter
    results = self.execute_sql(MULTI)
/root/.local/lib/python2.7/site-packages/django/db/models/sql/compiler.py:840: in execute_sql
    cursor.execute(sql, params)
/root/.local/lib/python2.7/site-packages/django/db/backends/utils.py:64: in execute
    return self.cursor.execute(sql, params)
/root/.local/lib/python2.7/site-packages/django/db/utils.py:98: in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
/root/.local/lib/python2.7/site-packages/django/db/backends/utils.py:64: in execute
    return self.cursor.execute(sql, params)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <django.db.backends.sqlite3.base.SQLiteCursorWrapper object at 0x7f0da84fd9d0>
query = 'SELECT "django_migrations"."app", "django_migrations"."name" FROM "django_migrations"', params = ()

    def execute(self, query, params=None):
        if params is None:
            return Database.Cursor.execute(self, query)
        query = self.convert_query(query)
>       return Database.Cursor.execute(self, query, params)
E       OperationalError: no such table: django_migrations

/root/.local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py:318: OperationalError

I can't figure out why it would fail here when I do either pytest or pytest simon/tests, but work if I execute specific tests. I'd really appreciate some help.

I've tried:

--keepdb, --reuse-db, --nomigrations, a mysql test database instead of sqlite..., changing our settings in various ways, landing on just:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(ROOTDIR, 'test.sqlite.db')
    },```

Not sure if any of that is relevant but I'd appreciate any help!

[apenney]

Oh, and just to mention, I tried the current sha out of git, same issue.

[apenney]

I'll also mention that when I do:

pytest simon/tests/test_credentials.py it takes a very long time as it does the migrations. The other failures happen much earlier and faster, as if it doesn't even get through the first ones.

[blueyed]

#725 might help with debugging - use --cli-log-level DEBUG then.

Is a previous test deleting / tearing down the db? Are you transactional_db there?
Are other tests using the db before?
How does test_create_new_customer_database_record look like (e.g. which fixtures does it use)?

I'd also ensure that the used db is the (expected) test db etc (via print or pdb).

[apenney]

False alarm! After 500 attempts at troubleshooting, I found I have a single file that triggers this. In there at the top I found:

import django.db
reload(django.db)

So the issue was that it collects all 4000 tests and that somehow breaks the attempt at the first migration setup on the first test it comes across. I had to effectively bisect the tests to get at the root cause.

I don't know if there's anything that could help in pytest-django, I guess that sql logging would have helped me figure out that a different test was the cause of the pain.

[blueyed]

Thanks for the update.

Would be interesting to know a) why is this done, b) how Django's test runner handles this, and c) why it fails with this then really.

But I can also see that you would rather not look more into this.. ;)

[apenney]

I'd love to help look into it, I'm just not sure how. My python skills are ... basically none, this is the first large python codebase I've ever touched, it all started with me saying "hmm, --parallel would be nice" and then I stumbled into this mess.

I'll chase up on my side to understand what we were doing here and if anyone remembers why we were monkeypatching django.db later in the tests:

monkeypatch.setattr(django.db, 'connections', connections)

[blueyed]

Ah, so the reload is probably done to revert some monkeypatching, which then should be handled via monkeypatch itself probably. Check git-blame, and also if moving this into its separate test (instead of doing it at the module level (as it sounds like)) helps.
But don't bother answering my questions - just let us know if you find out more.

Good luck with the project! :)