Huge outputs

[jiri-bocan]

Hello team,
I just came across a peculiar issue.
Imagine you have a repository with an order of magnitude of 10,000-1,000,000 leaks and the desired outcome format is JSON, i.e., an output JSON will have an order of magnitude 10-1000 MB. The question is how to process such a huge file. Conventional ways will usually end up with the OutOfMemory exception. Therefore, I need to write a custom JSON parser.
May I take for granted that the JSON file will always contain an expanded (or "beautified" = one item per line, indented) JSON content or may it happen that the JSON be collapsed (whole JSON in one line, no new-line characters, no indentation, no extra white-spaces)? So far, JSON seems always expanded with depth equal to 1. Would it be possible to provide some switch, that will ensure that both SARIF and JSON be expanded (easier to parse, read) or collapsed (e.g., to save disk space)?
Thank you for your reply beforehand,
jiri

PS: Another option is to tell gitleaks to output only first, e.g., 50,000 leaks which will limit the output size, too.
PPS: After upgrading from Gitleaks 8.8 to 8.15, number of detected leaks of certain type literally skyrocketed. Hope there is no error in corresponding regexes...

[weineran]

Have you considered just outputting the report in CSV format? I believe most CSV readers will read the file line-by-line so you shouldn't run into OutOfMemory issues.

[jiri-bocan]

Yes, I did. But CSV is of no help as there may be new-line characters in, e.g., Match, Secret, Message, or Fingerprint value. And, thus, one leak record may end up being split into many lines.

Another solution would be to set a maximum number of records/leaks per file and to create new files if exceeded.

I am currently looking into methods to deserialize a huge JSON in PowerShell. It seems there are some with help of jq and/or some javascript deserializers.

[weineran]

But CSV is of no help as there may be new-line characters in, e.g., Match, Secret, Message, or Fingerprint value. And, thus, one leak record may end up being split into many lines.

In general, CSVs should be able to support multi-line strings. In gitleaks, we might have to tweak how the CSV is being written. e.g. We might have to enclose multi-line strings in quotes.

Just as a simple PoC, here is a simple CSV:
test.csv

h1,h2,h3
hello2,"hello
world2",goodbye2

And when I open it in Excel, it shows the multi-line string in a single cell. If Excel can do it, we can do it...

[jiri-bocan]

Yes, this (quotes) is exactly, what can be handled via JSON or SARIF without a problem (automatic processing made in PowerShell). However, I still stick with JSON as it contains the richest info of all 3 formats (not mentioning the order of columns/items differs, too). Each format contains different amount of detail...

PS: Remember, due to its simplicity, to process CSV in PowerShell, one does not always need to use ConvertFrom-Csv to deserialize...

[jiri-bocan]

Btw, speaking about CSV, there is a similar issue with commas as they may appear in the above mentioned items, too. In result, there appears to be more separators on one line than what would correspond to the desired data columns. Adding quotes may be helpful here, as well...

[zricethezav]

After upgrading from Gitleaks 8.8 to 8.15, number of detected leaks of certain type literally skyrocketed. Hope there is no error in corresponding regexes...

@jiri-bocan what type? Also give us an example of a secret you are seeing that is a FP (scramble the secret before posting it).

[jiri-bocan]

The "RuleID" is equal to "generic-api-key". Here are all distinct "Match" values as taken from the JSON:

• Authorised = 'A1B2C3D4-A1B2-C3D4-E5F6-ABCD1234EF56'
• ClientCIF_NonACMBRefNO = 'A1B2C3D4-A1B2-C3D4-E5F6-ABCD1234EF56'
• ClientWatchListRefNo ='A1B2C3D4-A1B2-C3D4-E5F6-ABCD1234EF56'
• Key = "ABC12-DE123-FGHIJ-1234"
• KEY = "abcdefgh-1234-ijkl-5678-mnopqrstuvwx"
• key = foo.bar_id_pkey
• key name="FOObarKey123"
• KeyToken=abcd1234efgh5678"
• Password = abcdefghijklmnop1234567890"
• Password=fooBAR
• Token = 'A1B2C3D4-A1B2-C3D4-E5F6-ABCD1234EF56'

Interestingly, there is some 150k files in the given repo, 15 unique matches only(!) but multiplied into ca 750k leaks (resulting JSON has ca 450 MB)! In other words, if I extract all 750k leaks related to the "generic-api-key" RuleID, and keep only distinct Match values, they are only 15 in total. What changes and, thus, makes the leaks unique, is StartColumn, EndConlumn, StartLine, EndLine, and Fingerprint value.

[jiri-bocan]

PS: When running Gitleaks locally (WSL), it produces a 450 MB JSON, when on Azure DevOps build agents, it makes a 560 MB one (same leak count)...

[weineran]

@jiri-bocan Most of those that you posted look like true positives. Do you consider any of them to be false positives?

[jiri-bocan]

I have no problem with that. The original one was how to process said JSONs (considering introduction of additional functionality to alleviate the burden Gitleaks produces) as they are not generated on the fly but only at the very end of the scan, so there is still a room for further processing. I found a way and 560 MB JSON is processed in less than 8 min without any OutOfMemory exception etc. Thanks for your time.