Deprecate Partial Objects

[beberlei]

We plan to remove partial objects in the forseeable future. This doesn't have to be ORM 3, because it depends on a few alternatives being in place.

Partial objects are a bad fix for wanting to avoid loading too much data like text fields /blobs that are not needed in the request. Three solutions are proposed to migrate towards around this deprecation and removal:

• Use data transfer objects with the SELECT new MyDTO()
• Use new fetch=Eager support with Query::setFetchMode to implement multi-step hydration: [GH-1569] Optimize eager fetch for collections to batch query #8391
• Add support for lazy loading some properties or embeddables through a better proxy manager (this is not supported yet) Use ProxyManager #8518

As for use of PARTIAL with array hydration, a potential solution could be the introduction of a dedicated function: PARTIAL(alias, ...) that does the same that PartialExpression previously did. This could encapsulate this logic in ArrayHydrator and a specific function and avoid the polution of the DQL language with this concept.

[gharlan]

I sometimes use the trick from the section "Coding multi-step hydration in Doctrine ORM" from https://ocramius.github.io/blog/doctrine-orm-optimization-hydration/.

What is the best way to migrate from this?

[pszalko]

I have the same concerns as @gharlan. Is there any way to migrate? @beberlei

[stepozer]

I believe the best way for this will be just allow developers to specify what exactly you want to load as we have in ActiveRecord in Laravel - https://laravel.com/docs/8.x/eloquent-relationships#eager-loading. I believe this feature will be useful to avoid this terrible joins with OneToMany hydration. And doctrine should automatically decide is it leftJoin or SELECT FROM IN request.

@beberlei what do you think?

[AlexisFinn]

Hello, just to be clear, do you plan on removing partial arrays, as in

$qb->selec('partial  user.{id, name}')
->getQuery()->getArrayResult();

Because that's actually really usefull for retrieving a nicely formatted array.

[raziel057]

I don't see any answer for the question raised by AlexisFinn

Can you reconsider the use of partial with Array Hydration ? I understand the issues given for Partial Object Hydration but there is no problem at all working with array. it seems better and less time consuming than having to define a Dto or having to use a scalar hydration with the need to specify each property and having all in a flat array.

Ex.:

$qb->select('d, p, partial f.{id, mimetype, name, size}')
            ->from(Document::class, 'd')
            ->join('d.file', 'f')
            ->leftJoin('d.publications', 'p')
            ->getQuery()->getArrayResult();

[raziel057]

From Doctrine page https://www.doctrine-project.org/projects/doctrine-orm/en/2.16/reference/partial-objects.html it's mentioned:

The partial object problem in general does not apply to methods or queries where you do not retrieve the query result as objects. Examples are: Query#getArrayResult(), Query#getScalarResult(), Query#getSingleScalarResult(), etc.

[beberlei]

@gharlan @pszalko for multi step hydration the new way to go will be #8391 this is essentially what @stepozer mentioned laravel also does.

As for array hydration with partial, this is something we haven't considered before, so its a valid use-case and I had the idea maybe this works with a function that can only be used in combination with array hydration:

SELECT PARTIAL(p, id, name, description) FROM product p

[bakugo]

@beberlei #8391 is not an adequate replacement because it does not give the user the same control over when and how the hydration happens. I need to be able to hydrate the associations I need, when I need them, in the queries I specify, so if PARTIAL is removed I'll be forced to just load already loaded entities multiple times for no reason.

Removing partial objects is understandable but there still should be some way to signal to DQL "only load the ID for this alias for association purposes because the relevant entities were already previously loaded", it could even throw an exception if it encounters an ID that isn't already loaded.

[klkvsk]

After reading all comments and links in this issue, it's still unclear how to load entities partially.
Referencing the example by @raziel057

$qb->select('d, p, partial f.{id, mimetype, name, size}')
            ->from(Document::class, 'd')
            ->join('d.file', 'f')
            ->leftJoin('d.publications', 'p')
            ->getQuery()->  getResult() // objects, not array!

So let's assume we want to get documents as objects with all relations, but there is a file.content field, that can be lots of data we won't ever need in that result, and we want to omit just it.

What @beberlei suggested

1. "SELECT new MyDTO()" -- it would not be possible to hydrate $document->file property, which is of class File, with a some other DTO object, isn't it?
2. "fetch=Eager" -- but fetch mode is applied on a related entity(-ies) as a whole, not their separate properties, isn't it?
3. "Add support for lazy loading some properties" -- well, sounds like that's it, but still not implemented.

So what do you actually suggest, except ignoring the deprecation warning until (3) is merged?

[beberlei]

@klkvsk yes, ignore the deprecation and stay on 2.x - you can call Deprecation::ignoreDeprecation (or similar) with the url identifer to silence it for now

[beberlei]

@bakugo you can set the assoc lazy by default and use Query::setFetchMode to change on a per query basis. This is the same level of granularity that PARTIAL offers

[bakugo]

@beberlei FETCH_EAGER is not adequate because it runs an extra query for each association.

The use case in question is basically this: #4762 (comment). I have one "main" entity that has a lot of fields and a lot of associations. If I use EAGER, I end up with more queries than necessary (and it cannot handle nested associations apparently). If I simply JOIN all the associations in one query, there will be too many of them and the query will be extremely slow and inefficient.

The only approach that reliably works is splitting the JOINs into multiple queries, aka multi-step hydration, where you select the "main" entity by ID multiple times and JOIN a different set of associations each time, such that each individual query is efficient, and you end up with all the associations you need hydrated in a low number of queries. The only real problem here is that each query will waste time selecting the fields of the main entity that were already loaded before. SELECT PARTIAL entity.{id} solves this, and does NOT result in partially loaded objects because those objects were already loaded, so there's no reason for it to not be allowed in this case.

[beberlei]

The inefficent way hydration for multiple to many assocs work, i could imagine one query per assoc will be faster most of the time.

[clesquir]

As for array hydration with partial, this is something we haven't considered before, so its a valid use-case and I had the idea maybe this works with a function that can only be used in combination with array hydration:

SELECT PARTIAL(p, id, name, description) FROM product p

We are using partials a lot in our application in order to return nicely formatted arrays. Converting all of them with DTOs will take weeks of work. Also, we need to include OneToMany in there which I was not able to achieve using a DTO.

Would this function be available in Doctrine 3.0 or is it only an idea?

If this is only an idea, can you provide some pointers on how to implement?

Thanks!

[lowwa132]

Same issue as @clesquir here, the DTO alternative is not suitable for OneToMany relations usecase.

[beberlei]

After long deliberation I see no other way than allowing PARTIAL keyword again in 3.0, but only in ArrayHydrator. We will work on adding this again in ORM 3.2 (as 3.0 and 3.1 are already released).

We will also change the deprecation in 2.x to only trigger when the PARTIAL keyword is used in combination with a hydrator that calls UnitOfWork::createEntity.

This will keep the keyword as long as no alternative API for this use-case is available, such as the Entity Graph API from JPA, https://www.baeldung.com/jpa-entity-graph#2-defining-an-entity-graph-with-the-jpa-api

This issue title and description will be updated accordingly once #11365 and #11366 are merged.