F401 - Distinguish between imports we wish to remove and those we wish to make explicit-exports

[plredmond]

Resolves #10390 and starts to address #10391

Changes to behavior

• In __init__.py we now offer some fixes for unused imports.
  • If the import binding is first-party this PR suggests a fix to turn it into a redundant alias.
  • If the import binding is not first-party, this PR suggests a fix to remove it from the __init__.py.
• The fix-titles are specific to these new suggested fixes.
• checker.settings.ignore_init_module_imports setting is deprecated/ignored. There is probably a documentation change to make that complete which I haven't done.

---

Old description of implementation changes

Changes to the implementation

• In the body of the loop over import statements that contain unused bindings, the bindings are partitioned into to_reexport and to_remove (according to how we want to resolve the fact they're unused) with the following predicate:

  in_init && is_first_party(checker, &import.qualified_name().to_string()) // true means make it a reexport

• Instead of generating a single fix per import statement, we now generate up to two fixes per import statement:

  (fix_by_removing_imports(checker, node_id, &to_remove, in_init).ok(),
   fix_by_reexporting(checker, node_id, &to_reexport, dunder_all).ok())

• The to_remove fixes are unsafe when in_init.
• The to_explicit fixes are safe. Currently, until a future PR, we make them redundant aliases (e.g. import a would become import a as a).

Other changes

• checker.settings.ignore_init_module_imports is deprecated/ignored. Instead, all fixes are gated on checker.settings.preview.is_enabled().
• Got rid of the pattern match on the import-binding bound by the inner loop because it seemed less readable than referencing fields on the binding.
• // FIXME: rename "imports" to "bindings" if reviewer agrees (see code)
• // FIXME: rename "node_id" to "import_statement" if reviewer agrees (see code)

Scope cut until a future PR

• (Not implemented) The to_explicit fixes will be added to __all__ unless it doesn't exist. When __all__ doesn't exist they're resolved by converting to redundant aliases (e.g. import a would become import a as a).

---

Test plan

• crates/ruff_linter/resources/test/fixtures/pyflakes/F401_24 contains an __init__.py without __all__ that exercises the features in this PR, but it doesn't pass.
• crates/ruff_linter/resources/test/fixtures/pyflakes/F401_25_dunder_all contains an __init__.py with __all__ that exercises the features in this PR, but it doesn't pass.
• Write unit tests for the new edit functions in fix::edits::make_redundant_alias.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

ℹ️ ecosystem check detected linter changes. (+0 -0 violations, +104 -0 fixes in 2 projects; 42 projects unchanged)

PostHog/HouseWatch (+0 -0 violations, +4 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview

+ housewatch/models/__init__.py:2:21: F401 [*] `.backup.ScheduledBackup` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- housewatch/models/__init__.py:2:21: F401 `.backup.ScheduledBackup` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ housewatch/models/__init__.py:2:38: F401 [*] `.backup.ScheduledBackupRun` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- housewatch/models/__init__.py:2:38: F401 `.backup.ScheduledBackupRun` imported but unused; consider removing, adding to `__all__`, or using a redundant alias

latchbio/latch (+0 -0 violations, +100 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview

+ latch/__init__.py:10:5: F401 [*] `latch.functions.operators.group_tuple` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:10:5: F401 `latch.functions.operators.group_tuple` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:11:5: F401 [*] `latch.functions.operators.inner_join` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:11:5: F401 `latch.functions.operators.inner_join` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:12:5: F401 [*] `latch.functions.operators.latch_filter` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:12:5: F401 `latch.functions.operators.latch_filter` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:13:5: F401 [*] `latch.functions.operators.left_join` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:13:5: F401 `latch.functions.operators.left_join` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:14:5: F401 [*] `latch.functions.operators.outer_join` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:14:5: F401 `latch.functions.operators.outer_join` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:15:5: F401 [*] `latch.functions.operators.right_join` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:15:5: F401 `latch.functions.operators.right_join` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:17:41: F401 [*] `latch.resources.conditional.create_conditional_section` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:17:41: F401 `latch.resources.conditional.create_conditional_section` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:18:39: F401 [*] `latch.resources.map_tasks.map_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:18:39: F401 `latch.resources.map_tasks.map_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:19:48: F401 [*] `latch.resources.reference_workflow.workflow_reference` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:19:48: F401 `latch.resources.reference_workflow.workflow_reference` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:21:5: F401 [*] `latch.resources.tasks.custom_memory_optimized_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:21:5: F401 `latch.resources.tasks.custom_memory_optimized_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:22:5: F401 [*] `latch.resources.tasks.custom_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:22:5: F401 `latch.resources.tasks.custom_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:23:5: F401 [*] `latch.resources.tasks.large_gpu_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:23:5: F401 `latch.resources.tasks.large_gpu_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:24:5: F401 [*] `latch.resources.tasks.large_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:24:5: F401 `latch.resources.tasks.large_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:25:5: F401 [*] `latch.resources.tasks.medium_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:25:5: F401 `latch.resources.tasks.medium_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:26:5: F401 [*] `latch.resources.tasks.small_gpu_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:26:5: F401 `latch.resources.tasks.small_gpu_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:27:5: F401 [*] `latch.resources.tasks.small_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:27:5: F401 `latch.resources.tasks.small_task` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:29:38: F401 [*] `latch.resources.workflow.workflow` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:29:38: F401 `latch.resources.workflow.workflow` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:7:38: F401 [*] `latch.functions.messages.message` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:7:38: F401 `latch.functions.messages.message` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/__init__.py:9:5: F401 [*] `latch.functions.operators.combine` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/__init__.py:9:5: F401 `latch.functions.operators.combine` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/types/__init__.py:10:5: F401 [*] `latch.types.metadata.LatchMetadata` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/types/__init__.py:10:5: F401 `latch.types.metadata.LatchMetadata` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/types/__init__.py:11:5: F401 [*] `latch.types.metadata.LatchParameter` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/types/__init__.py:11:5: F401 `latch.types.metadata.LatchParameter` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/types/__init__.py:12:5: F401 [*] `latch.types.metadata.LatchRule` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/types/__init__.py:12:5: F401 `latch.types.metadata.LatchRule` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/types/__init__.py:13:5: F401 [*] `latch.types.metadata.Params` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/types/__init__.py:13:5: F401 `latch.types.metadata.Params` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
+ latch/types/__init__.py:14:5: F401 [*] `latch.types.metadata.Section` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
- latch/types/__init__.py:14:5: F401 `latch.types.metadata.Section` imported but unused; consider removing, adding to `__all__`, or using a redundant alias
... 52 additional changes omitted for project

Changes by rule (1 rules affected)

code	total	+ violation	- violation	+ fix	- fix
F401	104	0	0	104	0

[zanieb]

The ecosystem checks look concerning. We're losing fixes in preview? I think there should be no changes in stable?

[plredmond]

Yes, yes those do look concerning. I'll look into it.

[charliermarsh]

Not sure what's up with the Latch changes. I ran it locally and they are indeed classified as first-party (at least by the isort rule).

[charliermarsh]

Just trying to understand what's happening in Latch, at least one thing is that I think isort::categorize expects .module_name() rather than .qualified_name(), so this should probably be binding.import.module_name() (although I wouldn't expect it to change the outcome -- the module name is generally a prefix of the qualified name, and the import categorization just operates on the first segment).

[charliermarsh]

Oh interesting, but crates/ruff_linter/src/rules/flake8_type_checking/rules/typing_only_runtime_import.rs uses the .qualified_name with level=0. Interesting... I mean, on further review, it's possible that level=0 should work here, because the binding name is meant to be the fully-resolved name.

Like, if the user does this from a module named foo:

from . import bar

Then the qualified name on the binding should be ["foo", "bar"] -- and so we should be able to resolve it without taking into account the dots. I'm surprised it's not working for those projects. It's probably worth checking them out, running them locally, and looking at what binding.import.qualified_name is here.

[plredmond]

Done with this round of review. I think the outstanding issues (in order of importance) are:

(1) Incorrectly categorizing many imports as not first-party in ecosystem results.

• Latest change might address this by setting level correctly

(2) Dangling `__all__` code

1. `let dunder_all = None;`
2. `dunder_all: bool` field of `UnusedImportContext::init`
3. `dunder_all: Option` argument of `fix_by_reexporting`
4. `match dunder_all` in `fix_by_reexporting` which has a "not implemented" branch
5. branch in `fix_title` of `impl Violation for UnusedImport` which handles the case for no2

• To complete this feature requires adding a new edit function and calling it at no4.

(3) Some functions still take `impl Iterator<...>` but might be better to take `&'a [...]`

1. `fix::edits::make_redundant_alias`
2. `fix_by_reexporting`
3. `fix_by_removing_imports`

[plredmond]

latch/latchbio is still getting many imports incorrectly categorized, it seems

[plredmond]

I think there might just be a bug in my logic somewhere, since it seems that the only things being miscategorized are bindings in the syntax

from foo.bar.baz import (bind1,bind2)

So I'll write a test for that syntax and see what happens with it.

[charliermarsh]

Cool, I can help take a look at the remaining Latch categorizations tomorrow if it’s helpful!

[charliermarsh]

On second glance, aren't the ecosystem changes correct? We're now offering fixes for first-party imports in __init__.py files, whereas we weren't before?

[charliermarsh]

The behavior here looks correct to me. Would you mind adding a brief summary of the behavior changes to the PR body, so that users that click through in the changelog can understand what changed, rather than being limited to the implementation details?

[charliermarsh]

Congrats on the merge :)