How are lines of code counted?

[jtmr05]

I've noticed that it seems a bit irregular in how it handles line counting. I've tried replicating the behavior using wc and removing blank lines but I can't figure it out! So, what is it?

[spenserblack]

We use tokei for not only detecting languages, but getting lines of code for each language. Tokei identifies not only blank lines, but also comments. So the count you're seeing should be specifically code, excluding blank lines and comments.

https://docs.rs/tokei/12.1.2/tokei/struct.Language.html#structfield.code

onefetch/src/info/langs/mod.rs

Lines 33 to 51 in 6b852c8

	for (language_name, language) in languages.iter() {
	let mut code = language.code;
	let has_children = !language.children.is_empty();
	if has_children {
	for reports in language.children.values() {
	for stats in reports.iter().map(|r| r.stats.summarise()) {
	code += stats.code;
	}
	}
	}
	if code == 0 {
	continue;
	}
	language_distribution.insert(Language::from(*language_name), code as f64);
	}

Also, moving this to the Q&A category 🙂

[jtmr05]

Ah, obviously... Thanks.

[lukehsiao]

I notice there is often a difference between running tokei and onefetch when it comes to lines of code (onefetch 2.19.1, tokei 12.1.2).

Is there a reason that could be the case?

For example, on the onefetch repo itself:

✦ ❯ onefetch
                 R RR RR                  Luke Hsiao ~ git version 2.41.0
              R RRRRRRRR R          R     -------------------------------
 R RR       R RRRRRRRRRRRRR R      RR     Project: onefetch (6 branches, 44 tags)
rR RRR    R RRRRRRRRRRRRRRRRR R   RRR R   Description: Command-line Git information tool
RRR RR   RRRRRRRRRRRRRRRRRRRRRRR  RRRRR   HEAD: 17c1f463454f (main, origin/main)
 RRRRR  RRRRRRRRRRRRRRRRRRRRRRRR  RRRR    Version: 2.18.1
  RRR RRRRRRRRRRRRRRRRRRRRRRRRRRRR RR     Created: 4 years ago
    R  RRRRRRRRRR=  RR = RRRRRRRRRRR      Languages:
     RRRRRRRRRRRR=  RR = RRRRRRRRRR                  ● Rust (97.5 %) ● Shell (2.1 %)
      RRRRRRRRRRR   RR   RRRRRRRRRR                  ● Makefile (0.4 %)
     RR==RRRRRRRRRRRRRRRRRRRRRR===RR      Dependencies: 28 (Cargo)
     RR =  ==RRRRRRR  RRRRRR==  = RR      Authors: 54% Ossama Hjaji 1193
      RR =     ===========     = RR                19% dependabot[bot] 429
       RR                        R                 4% Spenser Black 86
        R                       R         Last change: 4 days ago
         R                                Contributors: 132
                                          URL: ssh://git@github.com/o2sh/onefetch.git
                                          Commits: 2224
                                          Churn (222): Cargo.lock 69
                                                       Cargo.toml 56
                                                       …/vercel/package.json 25
                                          Lines of code: 4954
                                          Size: 2.43 MiB (142 files)
                                          License: MIT




✦ ❯ tokei
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 JSON                    1           25           25            0            0
 Makefile                1           27           21            0            6
 Markdown               18         1913            0         1353          560
 Shell                   5          135          105            6           24
 SVG                     3          264          264            0            0
 Tera                    1          164          157            0            7
 TOML                    5          146          128            2           16
 YAML                    2         2926         2922            0            4
-------------------------------------------------------------------------------
 Rust                   42         5393         4736           43          614
 |- Markdown             5          106           33           59           14
 (Total)                           5499         4769          102          628
===============================================================================
 Total                  78        10993         8358         1404         1231
===============================================================================

Note the 4954 vs 8358.

[spenserblack]

Makefile: 21
Shell: 105
Rust: 4769
Markdown embedded in Rust: 59
$21 + 105 + 4769 + 59 = 4954$

We ignore prose (Markdown) and data (JSON, YAML, TOML) by default. See the -T option in onefetch --help. Some other languages we don't support yet, and would welcome a contribution 😉

Tokei counts Markdown text as "comments." We have an override to count Markdown's "comments" as code, instead, since what tokei counts as comments is in fact significant syntax. This doesn't effect Markdown files, which are prose (again, ignored by default), but it does affect the Markdown doc comments inside Rust files.

[lukehsiao]

Makes sense, thanks!