Both --kaniko-dir and KANIKO_DIR needs to be set

[euven]

Actual behavior
I have to set both --kaniko-dir and KANIKO_DIR in order to ensure /kaniko is not used.

Expected behavior
From the help:

--kaniko-dir string                         Path to the kaniko directory, this takes precedence over the KANIKO_DIR environment variable. (default "/kaniko")`

My understanding, reading the above, is that one only has to set --kaniko-dir=/something in order to ensure /kaniko is not being used.

To Reproduce
Steps to reproduce the behavior:

1. I'm using gcr.io/kaniko-project/executor:debug in a CI setting where my CI runners are running on Kubernetes, with limited cluster node storage. In order to work around the storage limitation on the node, I want to use a PVC for kaniko's storage
2. Build an image with --kaniko-dir=/something
3. While the image is building, observe the /something and /kaniko directories and note that /something is used mainly for storing the binaries:

docker-credential-acr-env    docker-credential-ecr-login  docker-credential-gcr        executor                     ssl                          warmer

While the /kaniko directory is still being used for the actual image snapshot storage, or something:

0           1           1188976700  1318558203  1377799484  2368873790  346759186   3952834964  856294469   Dockerfile  stages

4. Now, set the KANIKO_DIR=/something environment var, then build the image once again with --kaniko-dir=/something.
5. Observe that /something is being used exclusively.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ No ]
Please check if the build works in docker but not in kaniko	- [ Yes ]
Please check if this error is seen when you use --cache flag	- [ n/a ]
Please check if your dockerfile is a multistage dockerfile	- [ Yes ]

[mayooot]

When I set --kaniko-dir=foo, during kaniko execution, it still used the config.KanikoDir to get the environment variables(KANIKO_DIR), hance the problem in this issue.

// pkg/executor/build.go:807
dstDir := filepath.Join(config.KanikoDir, strconv.Itoa(index))

When add the --kaniko-dir parameter with corba, the default value is constants.DefaultKanikoPath. I think can remove config.KanikoDir and use the opts.KanikoDir instead.
Because the Kaniko_DIR environment variable and the --kaniko-dir parameter tend to be confusing, don't you think?

// cmd/executor/cmd/root.go:243
RootCmd.PersistentFlags().StringVarP(&opts.KanikoDir, "kaniko-dir", "", constants.DefaultKanikoPath, "Path to the kaniko directory, this takes precedence over the KANIKO_DIR environment variable.")

[prima101112]

when doing copy and snapshooting we called config.KanikoDir which read from envar default from package config
and we are not passing opts.KanikoDir to executeCommand so kaniko options is actually not passing to every command

if err := command.ExecuteCommand(&s.cf.Config, s.args); err != nil {
	return errors.Wrap(err, "failed to execute command")
}

there is no kaniko-dir passsed

fixing all possibilities and pass the opts to every command in executeCommand method will be confusing and make a lot of changes so instead of passing forcing env KANIKO_DIR to values in --kaniko-dir would be better faster and more clear that KANIKO_DIR will be replaced once there is --kaniko-dir flag

with this you no need to specify both of them again to make it work