Unable to redact certain information from existing log files

[vinod827]

Hi,

I'm using pino-noir library in some micro services and able to redact sensitive information for e.g. req.headers.authorization in standard output of logs whenever our REST API gets a request. This is working fine as expected. I'm getting the logs where authorization information in the header is redacted.

Now, I have use case where there are already a set of logs generated in JSON format. I want to read those log files and perform the redact by passing the keys in it dynamically. If the key matches anywhere in the JSON log file then redact it else don't perform any action on it and in the end show standard output to console.

I have written a NodeJS program to read these log files but not getting my head around on how to map this log file to the pino object. I know we can pass the pino object to express app like app.use(loggerpdino()); which will start reading the request/response and perform redacting but not sure how to do the similar way using this log file object.

My program looks like this

const noir = require('pino-noir');
const fs = require('fs');

let keys = [];
const label = 'Content Masked';

const redactLogs = (file, obj) => {
    console.log('Object received that needs to be whitelisted/blacklisted or both:::', obj);
    console.log('JSON File Path::', file);
    if(obj !== undefined && obj !== null){
        console.log(Object.keys(obj));
        let jsonData = {}
        if (obj.hasOwnProperty("blacklisted")) {
            console.log(obj['blacklisted']);
            obj['blacklisted'].forEach(element => {
                keys.push(element);
            });
        }
        if (obj.hasOwnProperty("whitelisted")) {
            console.log(obj['whitelisted']);
            keys = keys.filter(function(x) { 
                return obj['whitelisted'].indexOf(x) < 0;
            });
        }
        const redaction = noir(
            paths = keys,
            censor = label
        );
        console.log('Redaction keys:::', redaction);
        if(file !== null){
            fs.readFileSync(file, 'utf-8', (err, data) => {
                if (err) throw err
                    jsonData = JSON.parse(data);
                })
                console.log('JSON data::',jsonData);
                const redactedPino = require('pino')({
                    serializers: redaction
                });
        }
        //perform logging to console
    }
} 

module.exports.redactLogs = redactLogs;

Other app who is going to use this my library will have to import and pass the object like this:-

const { redactLogs } = require('@vinod827/my-logging-framework/my-redacting-logs');

const myobject = {
    blacklisted:['req.headers.user-agent'],
    whitelisted:['req.method']
}
app.use(redactLogs('/Users/vkumar/go/mywork/mydevelopment/my-sample-app/logs/app.log', myobject));

In myobject, I will be passing an array of keys in the blacklisted variable (whitelisted is optional so please ignore that), which eventually be scanning on logs (in JSON format), if a match happens then output its value in masked form else do not mask.

Please let me know how to pass jsonData object to the redactedPino object to perform the scan.

Thank you