Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

universal-lock: implement resolver forking #3358

Open
Tracked by #3350
BurntSushi opened this issue May 3, 2024 · 0 comments
Open
Tracked by #3350

universal-lock: implement resolver forking #3358

BurntSushi opened this issue May 3, 2024 · 0 comments
Labels
preview Experimental behavior

Comments

@BurntSushi
Copy link
Member

This is blocked on #3354 and #3355. It's part of the larger task of #3350.

Implementing forking requires detecting when there are multiple direct dependencies leading to different versions of the same package. In effect, instead of getting just a single list of dependencies, as we do here:

async fn get_dependencies(
&self,
package: &PubGrubPackage,
version: &Version,
priorities: &mut PubGrubPriorities,
request_sink: &tokio::sync::mpsc::Sender<Request>,
) -> Result<Dependencies, ResolveError> {

We will get a list of lists of dependencies. This can be seen in my prototype:

/// Given a candidate package and version, return its dependencies.
#[instrument(skip_all, fields(%package, %version))]
async fn get_dependencies_forking(
&self,
package: &PubGrubPackage,
version: &Version,
priorities: &mut PubGrubPriorities,
request_sink: &tokio::sync::mpsc::Sender<Request>,
) -> Result<Vec<Dependencies>, ResolveError> {
type Dep = (PubGrubPackage, Range<Version>, Option<MarkerTree>);
let deps: Vec<Dep> = match self
.get_dependencies(package, version, priorities, request_sink)
.await?
{
Dependencies::Available(deps) => deps,
Dependencies::Unavailable(err) => return Ok(vec![Dependencies::Unavailable(err)]),
};
// let mut by_grouping: FxHashMap<(&PackageName, &Option<ExtraName>, bool), Vec<&Dep>> =
// FxHashMap::default();
// let mut by_grouping: FxHashMap<&PackageName, Vec<&Dep>> = FxHashMap::default();
// This assumes that the version ranges are non-overlapping. When we do this for real,
// I believe it is true that we do NOT want to fork when there are overlapping version
// ranges.
let mut by_grouping: FxHashMap<&PackageName, FxHashMap<&Range<Version>, Vec<&Dep>>> =
FxHashMap::default();
for dep in deps.iter() {
let (ref pkg, ref range, _) = *dep;
let name = match *pkg {
PubGrubPackage::Root(_) | PubGrubPackage::Python(_) => unreachable!(),
PubGrubPackage::Package { ref name, .. } => name,
};
by_grouping
.entry(name)
.or_insert(FxHashMap::default())
.entry(range)
.or_insert(vec![])
.push(dep);
}
let mut forks: Vec<Vec<Dep>> = vec![vec![]];
for (_, mut groups) in by_grouping {
if groups.len() <= 1 {
for deps in groups.into_values() {
for fork in forks.iter_mut() {
fork.extend(deps.iter().map(|dep| (*dep).clone()));
}
}
} else {
let mut new_forks: Vec<Vec<Dep>> = vec![];
for deps in groups.into_values() {
let mut new_forks_for_group = forks.clone();
for fork in new_forks_for_group.iter_mut() {
fork.extend(deps.iter().map(|dep| (*dep).clone()));
}
new_forks.extend(new_forks_for_group);
}
forks = new_forks;
}
}
Ok(forks.into_iter().map(Dependencies::Available).collect())
}

We should be careful to ensure that we only fork when the conflicting packages have non-overlapping versions and non-overlapping marker expressions.

@BurntSushi BurntSushi added the preview Experimental behavior label May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preview Experimental behavior
Projects
None yet
Development

No branches or pull requests

1 participant