-
Notifications
You must be signed in to change notification settings - Fork 340
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mutl databases (multi db) support #924
Comments
Following this! It's a showstopper for us, preventing updates beyond Django 3.0. We currently have an internal monkey patching workaround that works with Django <= 3.0 but I can't get it to work when they remove the multi_db parameter. |
Initial PR in #930. |
@bluetech I'm testing 4.3.0 specifically for the multi db support. It seems to work, kind of. It seems like this old issue describes what I'm seeing: #76 |
Awesome! Thanks for working on this, it looks great. I converted a multidb project over to the experimental API and it seems to be working, including flushing data between test runs. (I was previously using the workaround of It's also working correctly with pytest-xdist, which is very cool.
FWIW I definitely struggled with the lack of a fixture version. Just to spell out the issue you're talking about, with a single-db project I would do something like this:
With multi-db I would imagine wanting to do something like this:
(Not sure if that's possible to implement, but just as an example of how an API could work.) This would be convenient for fixtures in general, because otherwise it's easy to forget to remove
The workaround I found to get my doctests working was to add an autouse fixture so
That's a pretty handy thing to be able to opt into, so it might be nice to have a more official way to do it? But it still leaves the doctests less efficient than they could be otherwise, since they don't actually need access to all the databases, so I think a fixture version would still be useful. One other idea I had while doing the conversion -- it would be cool if there was some flag I could use to be warned about unused databases, so if I removed the Thanks again for pushing this forward! |
@jgb here's what I'm using successfully in case it helps. Packages:
Databases:
I started with adding just a simple test file before converting my actual tests:
Results:
I imagine if you can try an isolated test like that and narrow down the issue it might help bluetech when they get back to working on this. You might also look closely at your fixtures, maybe entirely disable them, and see if the isolated test starts working again -- it wouldn't be that surprising if an old multidb workaround in your fixtures is messing with this. |
Does this mean that if I have a multi-db project, I can not use pytest for tests? I have a legacy DB and I created an app with some models that correlates with tables in that legacy DB, and also I have created some endpoints with Django DRF (managed=False), so no migrations are done. |
Yes, I'm pretty sure we need some integration with fixtures here. Your suggestion should be doable; all we really need is to know the list of databases once the test is to be executed. I think the I'll definitely mull it over. Of course if someone wants to submit a PR with a proposal that would be possible as well.
Right, currently there is no way to add marks directly to doctests. This is pytest-dev/pytest#5794. I can't think of any clear way to support it either.
For the multi-db support, pytest-django depends almost entirely on the django.test code, so such a feature would probably have to go through Django. It might be possible to somehow track whether a connection for a database was used during a test, but I'm not sure. There are also bound to be a lot of false-positives (or rather, cases where you want to keep a DB anyway), so would definitely need to be off by default. |
It's supposed to be the other way around - previously you couldn't, now you can. If you configured the legacy database in your DATABASES then it should work. If you tried it and it didn't work, we'd need to know how it failed. |
Hi, @bluetech In this testapp/settings.py I have 2 databases. One is a legacy db, the one that will be queried by models unmanaged models in API app.
And then I have also a testapp/settings_test.py. In this file, I disable migrations, define sqlite databases and set managed=True for models.
If I run normal unit tests with "production" settings, it fails as expected failing because relations for unmanaged models does not exists. If I run using test settings , it work as expected. BUT, if I run using pytest.
Here is a gist with more code (models, factory, test, settings): https://gist.github.com/gonzaloamadio/14f935d96809299b7f1e9fb88a6e8e94 I put a breakpoint and have inspected DB. And also as expected, when I run with unittest suite, the ingredient table was there? But when run with pytest.. no ingredient table there |
One more comment. I have run unittest with verbose option. This is the output
So I found this solution : https://stackoverflow.com/questions/30973481/django-test-tables-are-not-being-created/50849037#50849037 Basically do in conftest what UnManagedModelTestRunner is doing. This solution worked for me @bluetech
|
@gonzaloamadio Right, pytest doesn't consider |
So I just stumbled upon this section on the docs. We are currently upgrading a multi DB Django project from Django 1.11 to Django 3.2 and also upgrading the pytest and pytest-django packages. I was not aware of all these changes, but for us it worked out of the box without any issues. The tests are passing without problems. So thank you for that! |
Multi DB is supported then? I am facing a strange issue when trying to execute queries using cursor within multiple DBs: Am I doing a bad use of the fixtures? |
Same problem here! Django DB fixtures don't work with the cursors generated by:
??? |
We ran into a problem when we added a second database to the Django settings. When running pytest it tried to create a database with a second We were able to fix it by specifying a specific database name for the test database in the settings (in |
What about accessing a non-default DB within a fixture, how can we do that? I know about the
|
What I did so far - in
However its creating both test database but all of my fixture is executing for Here is my fixture -
Do I need to make any other changes? |
solid multi db support would a huge win for us. we have multi tenant Django app touching a bunch of databases and ancillary databases as well |
Hey guys, thanks to all for all of your work in this project! I found my way to this thread while I was upgrading some packages and running the test suit:
suggestion For now I pinned pytest-django==4.2.0. Thanks again and have a great time! |
my case is even more complicated, I have two databases, one in MySQL, one in Postgresql, not sure how to pytest them |
Is @pytest.fixture(autouse=True)
def enable_db_access_for_all_tests(db):
pass |
I'm unsure how useful this will be for others, but if you're transitioning from a codebase where all of your tests inherit from a standard test class where you set `databases = "all", or if you use that pattern often, you should know:
|
Is the current support meant to include support for the MIRROR setting? Everything except that was working for me, and I also couldn't write the test data directly to the second database without permission errors, e.g. |
Is there any way where I can apply: @pytest.mark.django_db(databases=['default']) To all tests? I have hundreds of tests that are using the db using the "magic" def test_profile(db):
... Edit: Even after removing the 2nd DB from my |
i have same issue |
Is it possible to know which db a test that specifies multiple db's is intended to be running against? i.e. I've got two db's and the test runs twice, does the test know which db this run is for? |
Nice guys, Everything works, is there some way to avoid the database flushing?? |
Everything seems to be working for me! Great feature, IMO. |
Hi there all! Please, could anybody explain how to run a test for specific database from multi databases configuration? I have defined three databases in settings.py: 'default', 'postgres1' and 'postgres2'. The first one is default Django sqlite3 DB. The second one with read-only access and the last one is with read-write access. I need to check that my model is created in 'postgres2' database with read-write access. So, I wrote the test:
If I run the test, I get an error saying that there is no access to 'postgres1' database with read-only access (yep, there is no connection to postgres1, but I expect it will not be used). Thanks in advance! |
Does this work?
https://docs.djangoproject.com/en/2.2/topics/testing/tools/#django.test.TransactionTestCase.databases
https://stackoverflow.com/questions/38307523/test-isolation-broken-with-multiple-databases-in-django-how-to-fix-it
Set databases variable inside the test case
El mié, 7 feb 2024 a las 14:51, kkrasovskii ***@***.***>)
escribió:
… Hi there all!
Please, could anybody explain how to run a test for specific database from
multi databases configuration?
I have defined three databases in settings.py: 'default', 'postgres1' and
'postgres2'. The first one is default Django sqlite3 DB. The second one
with read-only access and the last one is with read-write access. I need to
check that my model is created in 'postgres2' database with read-write
access. So, I wrote the test:
import pytest
from my_app.models import MyModel
@pytest.mark.django_db(databases=['postgres2'])
def test_create_my_model():
MyModel.objects.create()
If I run the test, I get an error saying that there is no access to
'postgres1' database with read-only access.
What is wrong here?
Thanks in advance!
—
Reply to this email directly, view it on GitHub
<#924 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHVLY6YYAFCPS7IXR5UWULYSO5JDAVCNFSM44JQ4JGKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJTGI2TONBZGUYA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
--------
Gonzalo Amadio
|
@kkrasovskii I suspect that you need to tell the ORM which DB to use for this model. Either via a DB router (https://docs.djangoproject.com/en/dev/topics/db/multi-db/#automatic-database-routing) or manually (https://docs.djangoproject.com/en/dev/topics/db/multi-db/#manually-selecting-a-database). Not sure why it is trying |
@gonzaloamadio, @mschoettle, than you so much for reply! The problem doesn't seem to be routing. My apps work fine with the databases: the problem with tests. django version 5.0.2, As I've mentioned earlier, three databases described in settings.py: sqlite3 as 'default' and two postgres databases ('postgres1', 'postgres2'). When I run the test, there is no connection to 'postgres1'. I rewrote the test the way @gonzaloamadio suggested: from django.test import TestCase
from my_app.models import MyModel
class MyTestCase(TestCase):
databases = ['default', 'postgres2']
def setUp(self):
MyModel.objects.create()
def test_something(self):
pass DB routing is done so that MyModel is written to base 'postgres2'. If I run
If I run the same command with 'postgres1' config removed from settings.py, the test is passing. |
I am not entirely sure if this will fix it but we had some issues with the test databases and had to add the following to each database connection setting to force using the correct database in tests: 'TEST': {
'NAME': f'test_{env("DATABASE_NAME")}',
}, |
One more solution in #1113 |
Hi, I ended up here when searching for a solution to use pytest with a memory based sqlite db backend, for tests in an application where the production db is postgres, since the application is largely db engine agnostic. The rationale being in-memory sqlite is blazing fast. The only hiccup being that a couple of my tests do require the postges db due to limitations in the sqlite engine. I naively thought this might be sufficient: # in tests/settings.py (which imports the application settings and overrides test specific config
...
DATABASES = {
"default": {
"ENGINE": "django.db.backends.sqlite3",
"NAME": ":memory:?cache=shared",
},
"postgres": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "...",
...
},
}
...
# in tests that invoke queries not supported by sqlite
@pytest.mark.django_db(databases=["postgres"])
def test_complex_querying(db):
... Unfortunately this results in django.test.testcases.DatabaseOperationForbidden: Database queries to 'default' are not allowed in this test.
Add 'default' to pytest_django.fixtures._django_db_helper.<locals>.PytestDjangoTestCase.databases
to ensure proper test isolation and silence this failure. My assumption here is that this potentially is due to the fact that the model being queried (which actually is constructed from a FactoryBoy factory) is somehow registered with the Now before I go down the rabbit hole to attempting to address that is there something else that I should be aware about w.r.t pytest-django's db setup ? For instance, IIUC, the db setup is session scoped, which means that if I create a Is this correct ? Is there a simpler way that I am not aware of ? TBH, the payoff for this "sqlite db with pg when required" is significant (integration test time down from 14m to 3m bar the 2 failing tests that fail due to the link above). So, I would really like to pursue this. Edit: I just thought of an alternate approach of using pytest Again, is there a simpler way to express this ? |
@lachlancannon any chance you've found a solution to this? I'm using two |
This issue replaces some historical issues: #76, #342, #423, #461, #828, #838, #839 (probably a partial list).
Background
Django supports multi databases. This means defining multiple entries in the
DATABASE
setting, which then allows directly certain queries to certain databases.One case is when an extra database is entirely independent, has its own migrations, setups etc.
Second case is when an extra database is readonly, only used for read queries, not managed by Django.
Third case is a readonly replica, for this Django provides the
MIRROR
settingDjango allows configuring the order in which test databases are set up.
Django's multi-db testing support
pytest-django mostly relies on Django's underlying
TransactionTestCase
andTestCase
classes for dealing with DB setups and such. Each pytest-django test gets run in a dynamically-generatedTestCase
/TransactionTestCase
.The main setting for mutli-db support is
TransactionTestCase.databases
. This tells Django which databases to consider for the test case. By default it's onlydefault
. It's possible to specify__all__
to include all databases.Historical note: The
TransactionTestCase.databases
attribute was added in Django 2.2. Before that amulti_db
attribute was used. pytest-django only supports Django>=2.2 so we happily don't need to concern ourselves with that.Previous attempts
#397 - Adds
multi_db=True
argument topytest.mark.django_db()
, addsdjango_multi_db
fixture. Problem: uses the oldmulti_db
attribute instead of thedatabases
attribute.#416 - Very similar to #397.
#431 - Adds
django_db_testcase
fixture which allows the user to completely customize the test case class, including settingdatabases
. Rejected for being too flexible, I'd prefer direct support for multi-db.#896 - Adds a global per-database setting for whether to add to the
databases
value or not. Rejected because I think it should be possible to customize per-test.Proposed solution
IMO we want something like #397/#416, but modernized to use
databases
instead ofmulti_db
. The fixture part would be a bit problematic because it's no longer just a boolean (fixture enabled/not enabled), but a list of database aliases. So some solution would be needed for that, or maybe only the mark would be supported.I'll try to work on it myself, but if for some reason I don't, PRs are definitely welcome!
The text was updated successfully, but these errors were encountered: