Use os-agnostic os.UserConfigDir func #114

dogancanbakir · 2023-05-31T12:45:18Z

Closes #113

tarunKoyalwar

suggesting use of helper functions from utils regarding home,config dirs
also we should implement a migration of old config files to new location

tarunKoyalwar

lgtm !

Added Unit Test for migration and other minor improvements

@Mzack9999 @ehsandeep . once this PR is merged and reflected in all downstream tools like tlsx,nuclei etc. all config files from old config directory will be moved to new config directory by renaming them at once

Just in case we should also test this PR with tools like subfinder , tlsx to avoid any unforseen situations since this is breaking change and cannot be reversed

Mzack9999

As suggested by @tarunKoyalwar, please test the following scenarios via the replace directive:

Individual tool migration (e.g. tlsx with standard settings, specifying a custom directory, a directory with invalid configuration files)
Tools using other tools with goflags as a dependency as libraries (we want to avoid automatic messy migrations with overlapping/misleading names)

dogancanbakir · 2023-06-12T14:04:46Z

Individual tool migration (e.g. tlsx with standard settings, specifying a custom directory, a directory with invalid configuration files)

Nothing happens when HomeDirOrDefault (current user HOME path) GetToolConfigDir returns the same dir. When changing HOME path to something else, for example /mynewhome/fortytwo then migration happens from /home/fortytwo/.config/<toolname> to /mynewhome/fortytwo/.config/<toolname>. When there are invalid config files, it will still move them to /mynewhome/fortytwo/.config/<toolname> but will create the necessary files, for example., config.yaml.

Tools using other tools with goflags as a dependency as libraries (we want to avoid automatic messy migrations with overlapping/misleading names)

I tested similar scenarios using subfinder as a library. The behavior is the same as using the tool from the CLI.

It would be beneficial to have someone else test these cases in order to ensure that nothing is overlooked.

Mzack9999

lgtm - running it with nuclei (which uses multiple tools as libraries) the only moved config files appears to be nuclei one.

Requesting review from @ehsandeep for visibility and too coordinate release since it's a breaking change

ehsandeep

nested directories under ($HOME/.config/$PROJECT_NAME) are not moved to a new location
remove moved / empty directory.

dogancanbakir · 2023-07-04T05:12:14Z

Fixed in projectdiscovery/utils#198. I will update this as soon as it is merged.

tarunKoyalwar · 2023-07-24T15:24:22Z

Bash script to check migration

#!/bin/sh

### Create migration directory
mkdir migration_tests

export WORK_DIR=$(pwd)"/migration_tests"
export PD_GIT="https://github.com/projectdiscovery"
export SEP="\n====================================\n"
export GOFLAGS_PR="https://github.com/dogancanbakir/goflags"

### Clone goflags repo
cd $WORK_DIR && git clone $GOFLAGS_PR 

printf "\nCheckout os_agnostic_configdir branch\n"

cd $WORK_DIR/goflags && git checkout os_agnostic_configdir  && export GOFLAGS_DIR=$(pwd)

printf "\nUsing goflags from $GOFLAGS_DIR\n"

function check_migration() {
    ### Clone repo
    ### $1 is the repo name
    ### $2 is go.mod directory if at root leave it empty
    git clone $PD_GIT/$1.git
    printf $SEP
    cd $1$2 && go mod edit -replace github.com/projectdiscovery/goflags=$GOFLAGS_DIR && go mod tidy
    printf $SEP
    printf "Content of previous config directory\n"
    tree $HOME/.config/$1
    printf $SEP
    printf "Staring migration\n"
    cd $WORK_DIR/$1$2/cmd/$1 && go run . 
    printf $SEP
    printf "Content of new config directory and old config directory\n"
    tree $HOME/Library/Application\ Support/$1 $HOME/.config/$1
    printf $SEP
    rm -rf $WORK_DIR
}


# Check first argument
if [ "$1" == "uncover" ]; then
    # First part of the script
    printf "Starting uncover migration\n"
    cd $WORK_DIR && check_migration uncover ""
    # Add your commands here
elif [ "$1" == "nuclei" ]; then
    # Second part of the script
    printf "Starting nuclei migration\n"
    cd $WORK_DIR &&  check_migration nuclei "/v2"
    # Add your commands here
else
    # Error message if first argument is not "uncover" or "nuclei"
    echo "Error: Please input 'uncover' or 'nuclei' as first argument."
    exit 1
fi

tarunKoyalwar · 2023-07-24T15:25:56Z

Migration of uncover config ✅

$ bash migration.sh uncover | tee uncover.log
Cloning into 'goflags'...
remote: Enumerating objects: 813, done.
remote: Counting objects: 100% (385/385), done.
remote: Compressing objects: 100% (171/171), done.
remote: Total 813 (delta 263), reused 262 (delta 205), pack-reused 428
Receiving objects: 100% (813/813), 6.66 MiB | 18.38 MiB/s, done.
Resolving deltas: 100% (477/477), done.

Checkout os_agnostic_configdir branch
Switched to a new branch 'os_agnostic_configdir'
branch 'os_agnostic_configdir' set up to track 'origin/os_agnostic_configdir'.

Using goflags from /Users/tarun/testing/migration_tests/goflags
Starting uncover migration
Cloning into 'uncover'...
remote: Enumerating objects: 1570, done.
remote: Counting objects: 100% (342/342), done.
remote: Compressing objects: 100% (212/212), done.
remote: Total 1570 (delta 243), reused 182 (delta 128), pack-reused 1228
Receiving objects: 100% (1570/1570), 427.06 KiB | 2.15 MiB/s, done.
Resolving deltas: 100% (1030/1030), done.

====================================
go: finding module for package github.com/kr/pretty
go: found github.com/kr/pretty in github.com/kr/pretty v0.3.1

====================================
Content of previous config directory
/Users/tarun/.config/uncover
├── config.yaml
├── provider-config.yaml
└── testDir
    └── backup.yaml

2 directories, 3 files

====================================
Staring migration

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/

		projectdiscovery.io

[INF] Current uncover version v1.0.5 (latest)
[FTL] Program exiting: no query provided
exit status 1

====================================
Content of new config directory and old config directory
/Users/tarun/Library/Application Support/uncover
├── config.yaml
├── provider-config.yaml
└── testDir
    └── backup.yaml
/Users/tarun/.config/uncover  [error opening dir]

2 directories, 3 files

tarunKoyalwar · 2023-07-24T15:26:51Z

Migration of nuclei config ✅

$  bash migration.sh nuclei | tee nuclei2.log
Cloning into 'goflags'...
remote: Enumerating objects: 813, done.
remote: Counting objects: 100% (371/371), done.
remote: Compressing objects: 100% (167/167), done.
remote: Total 813 (delta 249), reused 251 (delta 195), pack-reused 442
Receiving objects: 100% (813/813), 6.66 MiB | 963.00 KiB/s, done.
Resolving deltas: 100% (477/477), done.

Checkout os_agnostic_configdir branch
Switched to a new branch 'os_agnostic_configdir'
branch 'os_agnostic_configdir' set up to track 'origin/os_agnostic_configdir'.

Using goflags from /Users/tarun/testing/migration_tests/goflags
Starting nuclei migration
Cloning into 'nuclei'...
remote: Enumerating objects: 34429, done.
remote: Counting objects: 100% (562/562), done.
remote: Compressing objects: 100% (257/257), done.
remote: Total 34429 (delta 331), reused 458 (delta 291), pack-reused 33867
Receiving objects: 100% (34429/34429), 19.97 MiB | 16.49 MiB/s, done.
Resolving deltas: 100% (24016/24016), done.

====================================

====================================
Content of previous config directory
/Users/tarun/.config/nuclei
├── config.yaml
├── reporting-config.yaml
├── resume-cip9isbjtoj1gtd04odg.cfg
├── resume-cip9j0rjtoj1h2044oh0.cfg
├── resume-cipe5qjjtojec3ta4kmg.cfg
├── resume-ciqp8tbjtoj157c79s70.cfg
├── resume-ciqplu3jtoj6nag115ug.cfg
├── resume-cirskijjtoj94ibj1gtg.cfg
├── resume-cirvs7bjtoj5mbth5p00.cfg
├── resume-cis0693jtoj01ibq11h0.cfg
├── resume-cis0k8jjtoj49udf3v20.cfg
├── resume-cis0kdrjtoj4a0df7kbg.cfg
├── resume-cis0kfjjtoj4a21ooljg.cfg
├── resume-cis0kmbjtoj4a4m92qa0.cfg
├── resume-cisir3rjtoj2j6hvnimg.cfg
├── resume-cisirn3jtoj2k3d2lg00.cfg
└── resume-cit52sjjtoj0p5rhtlr0.cfg

1 directory, 17 files

====================================
Staring migration

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.9

		projectdiscovery.io

[ERR] Could not read nuclei-ignore file: open /Users/tarun/.config/nuclei/.nuclei-ignore: no such file or directory
[INF] Current nuclei version: v2.9.9 (latest)
[INF] Current nuclei-templates version: v9.5.8 (latest)
[INF] New templates added in latest release: 113
[INF] Templates loaded for current scan: 6466
[INF] No results found. Better luck next time!

====================================
Content of new config directory and old config directory
/Users/tarun/Library/Application Support/nuclei
├── config.yaml
├── reporting-config.yaml
├── resume-cip9isbjtoj1gtd04odg.cfg
├── resume-cip9j0rjtoj1h2044oh0.cfg
├── resume-cipe5qjjtojec3ta4kmg.cfg
├── resume-ciqp8tbjtoj157c79s70.cfg
├── resume-ciqplu3jtoj6nag115ug.cfg
├── resume-cirskijjtoj94ibj1gtg.cfg
├── resume-cirvs7bjtoj5mbth5p00.cfg
├── resume-cis0693jtoj01ibq11h0.cfg
├── resume-cis0k8jjtoj49udf3v20.cfg
├── resume-cis0kdrjtoj4a0df7kbg.cfg
├── resume-cis0kfjjtoj4a21ooljg.cfg
├── resume-cis0kmbjtoj4a4m92qa0.cfg
├── resume-cisir3rjtoj2j6hvnimg.cfg
├── resume-cisirn3jtoj2k3d2lg00.cfg
└── resume-cit52sjjtoj0p5rhtlr0.cfg
/Users/tarun/.config/nuclei
└── reporting-config.yaml

2 directories, 18 files

====================================

tarunKoyalwar

Migration does not happen if destination directory does not exist already exists 🤔

$  bash migration.sh nuclei | tee nuclei2.log
Cloning into 'goflags'...
remote: Enumerating objects: 813, done.
remote: Counting objects: 100% (371/371), done.
remote: Compressing objects: 100% (167/167), done.
remote: Total 813 (delta 249), reused 251 (delta 195), pack-reused 442
Receiving objects: 100% (813/813), 6.66 MiB | 18.38 MiB/s, done.
Resolving deltas: 100% (477/477), done.

Checkout os_agnostic_configdir branch
Switched to a new branch 'os_agnostic_configdir'
branch 'os_agnostic_configdir' set up to track 'origin/os_agnostic_configdir'.

Using goflags from /Users/tarun/testing/migration_tests/goflags
Starting nuclei migration
Cloning into 'nuclei'...
remote: Enumerating objects: 34429, done.
remote: Counting objects: 100% (562/562), done.
remote: Compressing objects: 100% (257/257), done.
remote: Total 34429 (delta 331), reused 458 (delta 291), pack-reused 33867
Receiving objects: 100% (34429/34429), 19.97 MiB | 1.53 MiB/s, done.
Resolving deltas: 100% (24016/24016), done.

====================================

====================================
Content of previous config directory
/Users/tarun/.config/nuclei
├── config.yaml
├── reporting-config.yaml
├── resume-cip9isbjtoj1gtd04odg.cfg
├── resume-cip9j0rjtoj1h2044oh0.cfg
├── resume-cipe5qjjtojec3ta4kmg.cfg
├── resume-ciqp8tbjtoj157c79s70.cfg
├── resume-ciqplu3jtoj6nag115ug.cfg
├── resume-cirskijjtoj94ibj1gtg.cfg
├── resume-cirvs7bjtoj5mbth5p00.cfg
├── resume-cis0693jtoj01ibq11h0.cfg
├── resume-cis0k8jjtoj49udf3v20.cfg
├── resume-cis0kdrjtoj4a0df7kbg.cfg
├── resume-cis0kfjjtoj4a21ooljg.cfg
├── resume-cis0kmbjtoj4a4m92qa0.cfg
├── resume-cisir3rjtoj2j6hvnimg.cfg
├── resume-cisirn3jtoj2k3d2lg00.cfg
└── resume-cit52sjjtoj0p5rhtlr0.cfg

1 directory, 17 files

====================================
Staring migration

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.9

		projectdiscovery.io

[INF] Current nuclei version: v2.9.9 (latest)
[INF] Current nuclei-templates version: v9.5.8 (latest)
[INF] New templates added in latest release: 113
[INF] Templates loaded for current scan: 6424
[INF] No results found. Better luck next time!

====================================
Content of new config directory and old config directory
/Users/tarun/Library/Application Support/nuclei
└── config.yaml
/Users/tarun/.config/nuclei
├── config.yaml
├── reporting-config.yaml
├── resume-cip9isbjtoj1gtd04odg.cfg
├── resume-cip9j0rjtoj1h2044oh0.cfg
├── resume-cipe5qjjtojec3ta4kmg.cfg
├── resume-ciqp8tbjtoj157c79s70.cfg
├── resume-ciqplu3jtoj6nag115ug.cfg
├── resume-cirskijjtoj94ibj1gtg.cfg
├── resume-cirvs7bjtoj5mbth5p00.cfg
├── resume-cis0693jtoj01ibq11h0.cfg
├── resume-cis0k8jjtoj49udf3v20.cfg
├── resume-cis0kdrjtoj4a0df7kbg.cfg
├── resume-cis0kfjjtoj4a21ooljg.cfg
├── resume-cis0kmbjtoj4a4m92qa0.cfg
├── resume-cisir3rjtoj2j6hvnimg.cfg
├── resume-cisirn3jtoj2k3d2lg00.cfg
└── resume-cit52sjjtoj0p5rhtlr0.cfg

2 directories, 18 files

====================================

while this is expected behaviour . what do you think ?? @ehsandeep @Mzack9999

Next Steps

Release Major version ?? (since this is breaking change )
update config directory in tools (nuclei , subfinder ,alterx etc) which use config dir to store some stuff
Add new config path to common healthcheck on all tools (it is not predicatable since new config dir follows convention and is different for most distros and envs)
Release Co-ordination with other tools
Ensure this does not migrate dependency imports of tool (when merged nuclei should not update config dirs of uncover,tlsx etc)

ehsandeep

@dogancanbakir can you please resolve the merge conflict?

use os agnostic config dir

4d6c97d

dogancanbakir requested a review from tarunKoyalwar May 31, 2023 12:45

tarunKoyalwar requested changes May 31, 2023

View reviewed changes

dogancanbakir mentioned this pull request Jun 1, 2023

Add config dir helper funcs projectdiscovery/utils#171

Closed

dogancanbakir and others added 2 commits June 6, 2023 15:40

introduce config dir migration

2de3097

Merge branch 'main' into os_agnostic_configdir

fd3763d

dogancanbakir requested a review from tarunKoyalwar June 6, 2023 12:46

dogancanbakir and others added 2 commits June 6, 2023 15:59

add MigrateConfigFiles func

ac46fb9

add tests+ minor improvements

120aadd

tarunKoyalwar requested a review from Mzack9999 June 9, 2023 16:14

tarunKoyalwar added 5 commits June 9, 2023 21:47

fix flag parse warning

d0aa3a4

fix lint error

94843b1

debug on test failure

493294e

skip test for linux under normal condition

244f946

improve build workflow

b3bc7fa

tarunKoyalwar approved these changes Jun 9, 2023

View reviewed changes

using osutil

236c573

Mzack9999 requested changes Jun 12, 2023

View reviewed changes

dogancanbakir requested a review from Mzack9999 June 19, 2023 12:22

Mzack9999 approved these changes Jun 19, 2023

View reviewed changes

Mzack9999 requested a review from ehsandeep June 19, 2023 12:51

Mzack9999 assigned dogancanbakir Jun 19, 2023

ehsandeep and others added 2 commits June 27, 2023 18:02

Merge branch 'main' into os_agnostic_configdir

dd260e8

go mod tidy

081bbba

ehsandeep requested changes Jun 27, 2023

View reviewed changes

dogancanbakir mentioned this pull request Jul 4, 2023

Fix MigrateDir func projectdiscovery/utils#197

Closed

2 tasks

dogancanbakir added 2 commits July 17, 2023 12:47

use SyncDirectory

fd29b62

create dir w/ tool name under .config

c49e6d6

dogancanbakir requested a review from ehsandeep July 18, 2023 07:04

ehsandeep requested a review from tarunKoyalwar July 21, 2023 14:11

tarunKoyalwar reviewed Jul 24, 2023

View reviewed changes

tarunKoyalwar mentioned this pull request Jul 28, 2023

Read-only filesystem fixes for when ~/.config is not writable projectdiscovery/nuclei#3990

Closed

4 tasks

ehsandeep mentioned this pull request Jul 29, 2023

Add config dir helper funcs projectdiscovery/utils#172

Merged

ehsandeep requested changes Sep 28, 2023

View reviewed changes

Merge branch 'main' into os_agnostic_configdir

4af426c

dogancanbakir requested a review from ehsandeep September 28, 2023 14:23

ehsandeep approved these changes Sep 29, 2023

View reviewed changes

ehsandeep merged commit 065d52b into projectdiscovery:main Sep 29, 2023
5 checks passed

This was referenced Oct 9, 2023

use standard config directories projectdiscovery/uncover#302

Merged

make migration configurable #145

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use os-agnostic os.UserConfigDir func #114

Use os-agnostic os.UserConfigDir func #114

dogancanbakir commented May 31, 2023

tarunKoyalwar left a comment

tarunKoyalwar left a comment

Mzack9999 left a comment •

edited by dogancanbakir

dogancanbakir commented Jun 12, 2023

Mzack9999 left a comment •

edited

ehsandeep left a comment •

edited by dogancanbakir

dogancanbakir commented Jul 4, 2023

tarunKoyalwar commented Jul 24, 2023

tarunKoyalwar commented Jul 24, 2023

tarunKoyalwar commented Jul 24, 2023

tarunKoyalwar left a comment •

edited

ehsandeep left a comment

Use os-agnostic os.UserConfigDir func #114

Use os-agnostic os.UserConfigDir func #114

Conversation

dogancanbakir commented May 31, 2023

tarunKoyalwar left a comment

Choose a reason for hiding this comment

tarunKoyalwar left a comment

Choose a reason for hiding this comment

Mzack9999 left a comment • edited by dogancanbakir

Choose a reason for hiding this comment

dogancanbakir commented Jun 12, 2023

Mzack9999 left a comment • edited

Choose a reason for hiding this comment

ehsandeep left a comment • edited by dogancanbakir

Choose a reason for hiding this comment

dogancanbakir commented Jul 4, 2023

tarunKoyalwar commented Jul 24, 2023

Bash script to check migration

tarunKoyalwar commented Jul 24, 2023

Migration of uncover config ✅

tarunKoyalwar commented Jul 24, 2023

Migration of nuclei config ✅

tarunKoyalwar left a comment • edited

Choose a reason for hiding this comment

Migration does not happen if destination directory does not exist already exists 🤔

Next Steps

ehsandeep left a comment

Choose a reason for hiding this comment

Mzack9999 left a comment •

edited by dogancanbakir

Mzack9999 left a comment •

edited

ehsandeep left a comment •

edited by dogancanbakir

tarunKoyalwar left a comment •

edited