Writing single file ended with ~9 GCS calls

[kobi-lemberg]

Hi,

we are using HDFS GCS connector to read and write files to GCS.
as part of our RDD flow, we have a code which write to GCS in a custom logic.
below there is a code snippet that create 3 simple file in GCS:

val path = "gs://collection-dev/kobi6/test11"
writeToFile(s"$path/text1.csv", lines = Seq("1", "2", "3"))
writeToFile(s"$path/text2.csv", lines = Seq("1", "2", "3"))
writeToFile(s"$path/text3.csv", lines = Seq("1", "2", "3"))

def writeToFile(fileName: String, lines: Seq[String], withSeparator: Boolean = false): Option[Throwable] = {
  writeContext(fileName) { outputStream =>
    // create index and regular file
    for (currentLine <- lines) {
      outputStream.write((currentLine + '\n').getBytes)
    }
    outputStream.close()
  } match {
    case Success(_) => None
    case Failure(err) =>
      logger.errorP(s"Failed to write to $fileName", err, category)
      Some(err)
  }
}

def writeContext(filename: String)(f: OutputStream => Unit): Try[Unit] = {
  val path = new Path(filename)
  val fileSystem = path.getFileSystem(getConfiguration)
  val out = fileSystem.create(path, true)
  val writeAttempt = Try {
    f(out)
    out.close()
  }
  writeAttempt
}

when running this code, for each file, we noticed that there were made 9 GCS calls (get directories structure, get there metadata and so on)
this is translated to costs

I also tried to put the following configuration:

"fs.gs.status.parallel.enable", "false"
"fs.gs.create.items.conflict.check.enable", "false"

but I still see those calls through stack driver audit logs
below is a picture of the audit logs:

is there any option to eliminate those calls?