Files are not formatted with latest rustfmt

[bodymindarts]

I'd like to understand the reasoning behind not using rustfmt to format the code (it has been the norm on other rust projects I've worked on).

Incase its not intentional (or perhaps just not formatted with the latest rustfmt version) I have opened a PR #171

[dpc]

If you decide to routinely apply rustfmt (which I recommend myself), please consider adding rustfmt.toml to root directory with the desired settings. There are setting in Vim Rust plugin to detect the presence of rustfmt.toml and auto-format on save. More: https://www.reddit.com/r/rust/comments/9jl6a9/pro_tip_if_you_use_cargo_fmtrustfmt_use_a/

[apoelstra]

@bodymindarts rustfmt is constantly changing their rules. If you can produce a formatted version of this codebase which is idempotent under five consecutive versions of rustfmt I may consider this, but if not, it's just noise.

[dpc]

@apoelstra It's nearing 1.0 so formatting should stay reasonably stable now. https://www.reddit.com/r/rust/comments/97cyfw/rustfmt_10rc/

[stevenroose]

Adding a .rustfmt.toml file to the repo can also make it more stable. Probably once there is 1.0, any changes will be explicit until 2.0 or so and then they might require to explicitly state usage of 2.0 in rustfmt.toml.

I have rustfmt on save and it can be hard to commit changes that way :) I usually end up using vim (instead of nvim) to save the file without rustfmt to make the commit.

[bodymindarts]

Waiting for 1.0, adding a .rustfmt.toml and potentially a githook seem reasonable to me.

[apoelstra]

I'd be happy with a rustfmt.toml (I don't think we should dot-prefix it) that either allowed everything, or only enforced a couple simple uncontroversial things (4-space tabs, etc).

[bodymindarts]

I don't think you can turn individual rules on / off so that they are ignored. You only have the option of setting the behaviour of the individual rules (as documented here). An empty rustfmt.toml leaves all rules at their default values.

Re: Githook
We could add a pre-commit.sh file to the root of the repository:

#!/bin/bash

# Enable this pre-commit hook to auto-format the code via:
# ln -s ../../pre-commit.sh .git/hooks/pre-commit

echo "Pre-commit hook"

which rustfmt 1>/dev/null
if [[ "$?" != "0" ]] ; then
  echo "Please install rustfmt to format code"
  exit 1
fi

echo "Running 'cargo fmt' with $(cargo fmt --version)"

cargo fmt

But for it to get executed individual developers would still have to execute

$ ln -s ../../pre-commit.sh .git/hooks/pre-commit

If we do that we'd have to at least document it.

[apoelstra]

I ran cargo fmt this morning and was happy with all the changes it made. None seemed exotic or likely to change.

It did miss some trailing commas, e.g. doing

+        assert_eq!(
+            u224_res,
+            Uint256([
+                0x7AB6FBBB21524111u64,
+                0xFFFFFFFBA69B4558,
+                0x854904485964BAAA,
+                0xDEADBEEB
+            ])
+        );

(DEADBEEB should have a ,.) I think the reason for this is that the assert_eq! macro could not handle trailing commas in older versions of rustc (including, I believe 1.22.0) so rustfmt disables its trailing-comma code within assert_eq!, which turns out to be too strong a reaction. So probably that will change in future, resulting in another 5-10 line patch.

So I think at this point I'd be willing to accept a format-only PR, especially since several people here have their text editors set to run this on save. cc @TheBlueMatt what do you think? We could add cargo fmt to avoid this situation in the future.

[stevenroose]

I would want to propose one deviation from the default rustfmt setting:

use_small_heuristics = "Off"

Otherwise rustfmt arbitrarily breaks a line even if it doesn't reach the 100-character line limit because it "feels smaller".

edit: fyi, this is my default rustfmt config: https://gist.github.com/stevenroose/468b01df07412b774a38a89453a82038

[vorot93]

@stevenroose if you feel that something is wrong with rustfmt's defaults then it is much better to take it upstream than try to invent custom formatting.

[sanket1729]

I think everyone agrees that rustfmt is now stable, we should aim to enforce rustfmt on the entire codebase, and starting file by file seems the most logical.

If we are concerned about the commit history, we can have multiple commits targetting things file by file in a separate PR and finally create a separate PR with a single commit for cleaner history.

[sgeisler]

I agree that rustfmt is probably stable enough to be introduced. For me the most important part is CI enforcement, otherwise this won't work in the long run. If we do it in one go or file by file I don't care as much, although I have a slight preference for all-at-once since it's mostly mechanical and thus easier to review that way. E.g. a first commit could add attributes to disable fmt for some parts of the code and a second one does the formatting. So only the first one needs review, the second is reproducible.

[apoelstra]

concept ACK. But good luck doing this in practice.

There are no "attributes to disable fmt for some parts of the code", you have to invoke rustfmt on individual files in some script that needs to be added to CI (and I guess everyone has to add to their githooks). If we add a rustfmt.toml that will trigger many peoples' text editors to start rustfmt'ing the entire codebase, producing massive noisy diffs. But this is just as well because there are no rustfmt.toml options that usefully restrict rustfmt, e.g. by telling it not to mess up custom whitespace.

I have tried running rustfmt on this codebase several times over the last year. Every time it makes a mess of things, and I am not even considered the ongoing noise it will produce; even if it is "stable" it still does things like splitting/unsplitting giant iterator chains into several lines based on a line length being changed by 1.

[dpc]

There are no "attributes to disable fmt for some parts of the code", you have to invoke rustfmt on individual files in some script that needs to be added to C

From https://github.com/rust-lang/rustfmt#construction-of-config-options

Rustfmt 2.x merges configuration options from all configuration files in all parent directories, with configuration files nearer the current directory having priority.

Also https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#configuration-file-resolution

Can be combined with https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#ignore

Skip formatting files and directories that match the specified pattern. The pattern format is the same as .gitignore.

From https://github.com/rust-lang/rustfmt#tips

For things you do not want rustfmt to mangle, use #[rustfmt::skip]

[dpc]

I wonder if anyone have built a CI tool that would enforce rustfmt only on files that are new or were already passing rustfmt previously. That seems like a nice way to keep the formatting improve with time, without a big switching event.

Edit: Just asked on user forum.

[apoelstra]

Amazing! Thanks @dpc

[dpc]

So, the approach from rust-analyzer https://users.rust-lang.org/t/enforce-rustfmt-only-on-files-that-were-already-correctly-formatted/53855/3 seems interesting, though it is a bit... elaborated.

One idea that I have that might be simpler and work for rust-bitcoin is:

cargo fmt -- --check --files-with-diff

prints the list of files that are not correctly formatted.

One could check in rustfmt-incorrectly-formatted file into git (generated with an initial cargo fmt -- --check --files-with-diff). Then in CI compare it with cargo fmt -- --check --files-with-diff output and fail if they are not the same. This way during the PR, it would get visible if rustfmt status change for any file and no new incorrectly formatted files can be introduced.

If it changed for the better the author of the PR would have to go ahead and delete it from rustfmt-incorrectly-formatted list.

If it changed for the worse, the author of the PR would have to make it confirm and push the PR again.

This way no big event is needed, and formatting should eventually get fixed everywhere, at which point the list can be removed.

[tcharding]

Perhaps we should consider introducing rustfmt in bitcoin_hashes first since the code base is smaller. Might be an easier test run to see how it goes?

[apoelstra]

I like @dpc's suggested approach. We could start here if we initially had every single file excluded from format-checking.

[TheBlueMatt]

I'd vote somewhat strongly against rustfmt use. In general I find it makes files substantially harder to get context from - preferring (without a lot of config options to change it) to make files 2-3x longer than they should be, which makes it really hard to get context for what you're working on in some cases.

We spent some time trying to use it in rust-lightning and ended up just filing bugs that haven't been touched in 2-3 years now :(.

[dpc]

The problem with formatting is that everyone has their own preferences, habits and setups, text editors, IDEs, screen sizes, font sizes and so on and one can't make everyone happy. Some people like to line up multiple windows side by side horizontally, and like their code narrow, some have their terminal at the bottom, and one window above, and like their code wider.

Rust settled on 4 spaces and breaking code more than less which makes it appear taller. Coming from C embedded/kernel background, I'm historically a 8-wide taber and I like my code wide. Too bad for me. I had to change my habits. Maybe I organize my work setup slightly differently now, maybe my font is different to counterbalance it, I can't really tell now. I guess I do use much more the Gnome's "snap to side border to maximize to half screen" feature routinely. Maybe I even structure my code somewhat differently. Anyway - I'm just used to it.

For one time pain of altering my habits, usually any Rust project I jump into seem superficially familiar and ergonomic, and same goes for other people jumping into my code, I think.

All in all it's a huge win comparing to things were in C/C++ land where every project was it's own little zoo, with stuff going more left, right, parenthesis going here, or there, and especially in C++ with a different subset of the language being blessed/cursed in every organization.

[TheBlueMatt]

Rust settled on 4 spaces and breaking code more than less which makes it appear taller.

Rust the project may have, but rustfmt has some material configurability for some things. eg it supports tabs instead of spaces just fine, or more spaces, or whatever. It also has some settings for how comically it makes your code unreadably tall, but not nearly enough to compensate for its tall-code goals.

My complaint about rustfmt generating unreadable code isn't just about it looking bad to my eye, though, if it were just awkward indentation or breaks in places I don't like, I'd suck it up and move on. But making code impossibly tall has real, very non-zero cost. I've seen many times bugs be missed in review because of interaction with code 10 lines up or down in the file, sometimes 20. Suddenly you 2x the height of your file (rustfmt often does more than that) and instead of it being a question of making sure you have enough lines in your diff, its a question of scrolling through a file, and likely missing the bug anyway cause it isn't visible at the same time as the offending change.

At least personally I generally find any flavor of code pretty readable, I'm just as comfortable reading validation.cpp in core as I am reading rest.cpp or old satoshi code - its all pretty different, but it all has some semblance of reason and none of it gets in my way for being able to scan the code to see what's going on where. Making code obscenely tall, on the other hand, makes things fall off screens and you miss critical context.

Going and editing Core code (in one of its 10 flavors) or kernel code or any other code all is pretty easy, I can look around, see what's going on, and edit things. Nearly every time I try to jump in and edit a rustfmt-formatted project, I spend a bunch of time scrolling up and down to get a feel for what's going on, something I rarely have to do in any other context.

Making less code fit on a terminal, or at least way less code fit on a terminal is about the only reason I'd ever have for thinking a formatter is garbage.

[apoelstra]

In addition to Matt's general dislike for tall code, another very frustrating thing rustfmt does is auto-splitting lines when they exceed some maximum length, so that if you add (or remove) a single parameter from a function definition, this will change the line from 1 line to 10 (or vice-versa) completely obscuring the original diff.

This, and its habit of ruining hand-formatted u8 arrays, are things we need to be able to turn off.

[Kixunil]

While I mostly agree with the sentiment of people against rustfmt in my experience it was occasionally useful to point out code that could've been written better by some other technique - splitting functions into smaller (max 15 lines is pretty good rule of thumb), assigning values to variables etc. I'd agree with making it a rule to prefer these techniques although it's hard to specify exactly.

[apoelstra]

BTW -- I continue to be alright with "enabling" rustfmt on a set of zero files and trying to chip away at them all. It may be that some files (e.g. ones with lots of weirdly-formatted arrays) we'll never be able to rustfmt, but that's fine. On most files we have no big array literals and no long lines.

[Kixunil]

Should this be closed in favor of #788?

[apoelstra]

Yeah, I think so.

[tcharding]

I would want to propose one deviation from the default rustfmt setting:

use_small_heuristics = "Off"

Otherwise rustfmt arbitrarily breaks a line even if it doesn't reach the 100-character line limit because it "feels smaller".

Is this still needed, I saw no difference running rustfmt with and without this set on the current master?