Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caused by: java.io.IOException: Found 2 files that match #2104

Closed
wind57 opened this issue Sep 20, 2021 · 10 comments
Closed

Caused by: java.io.IOException: Found 2 files that match #2104

wind57 opened this issue Sep 20, 2021 · 10 comments

Comments

@wind57
Copy link

wind57 commented Sep 20, 2021

Starting Liquibase at 20:48:01 (version 4.4.3 #53 built at 2021-08-05 18:32+0000)

When I try to execute:

liquibase --username XXX --pasword ... --file /liquibase-changelog-file/databaseChangelog.json

I get:

Caused by: liquibase.exception.ChangeLogParseException: Error parsing /liquibase-changelog-file/databaseChangelog.json
        at liquibase.parser.core.yaml.YamlChangeLogParser.parse(YamlChangeLogParser.java:88)
        at liquibase.Liquibase.getDatabaseChangeLog(Liquibase.java:369)
        at liquibase.Liquibase$16.run(Liquibase.java:1981)
        at liquibase.Scope.lambda$child$0(Scope.java:166)
        at liquibase.Scope.child(Scope.java:175)
        at liquibase.Scope.child(Scope.java:165)
        at liquibase.Scope.child(Scope.java:144)
        at liquibase.Liquibase.runInScope(Liquibase.java:2404)
        at liquibase.Liquibase.listUnrunChangeSets(Liquibase.java:1977)
        at liquibase.Liquibase.reportStatus(Liquibase.java:2054)
        at liquibase.integration.commandline.Main.doMigration(Main.java:1841)
        at liquibase.integration.commandline.Main$1.lambda$run$0(Main.java:402)
        at liquibase.Scope.lambda$child$0(Scope.java:166)
        at liquibase.Scope.child(Scope.java:175)
        at liquibase.Scope.child(Scope.java:165)
        at liquibase.Scope.child(Scope.java:144)
        at liquibase.integration.commandline.Main$1.run(Main.java:401)
        ... 20 more
Caused by: java.io.IOException: Found 2 files that match /liquibase-changelog-file/databaseChangelog.json: file:/liquibase-changelog-file/..2021_09_20_20_47_59.020895025/databaseChangelog.json, file:/liquibase-changelog-file/databaseChangelog.json

I can see that there are two files indeed in that directory: liquibase-changelog-file, but I did specify exactly the file to use?

@molivasdat
Copy link
Contributor

Hi @wind57 Thanks for creating this issue. I was not aware of the --file parameter. We will add this to the list of issues to process. What command were you executing when you provided this parameter so that we can recreate the issue.

@wind57
Copy link
Author

wind57 commented Sep 29, 2021

I have to make a correction first, terrible sorry. --file is in our script, that would translate into --changelog-file in liquibase.

this is what I was trying, against cassandra:

liquibase   --username admin --password admin --url "jdbc:cassandra://192.168.0.5:9042/killer_video;DefaultKeyspace=killer_video" --changelog-file /liquibase-changelog-file//databaseChangelog.json --log-level DEBUG update

@catull
Copy link

catull commented Apr 2, 2022

Using Liquibase 4.9.1.

in pom.xml I have the following configuration:

      <plugin>
        <groupId>org.liquibase</groupId>
        <artifactId>liquibase-maven-plugin</artifactId>
        <configuration>
           <changeLogDirectory>/target/classes</changeLogDirectory>
           <propertyFile>/src/main/resources/liquibase.properties</propertyFile>
        </configuration>
        <executions>
          <execution>
            <id>user-schema-database-update</id>
            <phase>process-resources</phase>
            <configuration>
              <changeLogDirectory>/src/main/resources</changeLogDirectory>
              <username>${username}</username>
              <password>${password}</password>
              <changeLogFile>/std/master.xml</changeLogFile>
              <propertyFileWillOverride>false</propertyFileWillOverride>
              <promptOnNonLocalDatabase>false</promptOnNonLocalDatabase>
              <outputDefaultSchema>false</outputDefaultSchema>
              <verbose>true</verbose>
            </configuration>
            <goals>
              <goal>update</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

Although I explicitly configure changeLogDirectory the diagnostic states:

[ERROR] Failed to execute goal org.liquibase:liquibase-maven-plugin:4.9.1:update (user-schema-database-update) on project ######:
[ERROR] Error setting up or running Liquibase:
[ERROR] liquibase.exception.SetupException: java.io.IOException:
Found 2 files that match std/4/24/2/dml/#######_KRIT.sql: file:/###########/target/classes/std/4/24/2/dml/#######_KRIT.sql,
file:/###########/src/main/resources/std/4/24/2/dml/#######_KRIT.sql

If I remove or comment out

<changeLogDirectory>src/main/resources</changeLogDirectory>

the error is worse:

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.liquibase:liquibase-maven-plugin:4.9.1:update (user-schema-database-update) on project bezug-db-init:
[ERROR] Error setting up or running Liquibase:
[ERROR] The file std/master.xml was not found in
[ERROR]     - ${HOME}/.m2/repository/com/oracle/ojdbc8/12.2.0.1.0/ojdbc8-12.2.0.1.0.jar
[ERROR]     - ${HOME}/.m2/repository/junit/junit/4.13.2/junit-4.13.2.jar
ERROR]     - ${HOME}/.m2/repository/org/liquibase/liquibase-core/4.9.1/liquibase-core-4.9.1.jar
[ERROR]     - ${HOME}/.m2/repository/org/liquibase/liquibase-maven-plugin/4.9.1/liquibase-maven-plugin-4.9.1.jar
[ERROR]     - ${HOME}/.m2/repository/org/yaml/snakeyaml/1.27/snakeyaml-1.27.jar
[ERROR]     - [PROJECT}
[ERROR]     - [PROJECT}/target/classes
[ERROR]     - [PROJECT}/target/test-classes
[ERROR] Specifying files by absolute path was removed in Liquibase 4.0. Please use a relative path or add '/' to the classpath parameter.

The last line is very confusing.
Basically absolute paths are discouraged, yet you have to add '/' to the classpath parameter ???

Adding

<classpath>/target/classes</classpath>

I still get the error about 2 conflicting matching files outlined at the top.

How do I resolve this ?

@kataggart
Copy link
Contributor

@catull We are doing some discovery regarding how to clarify some of the classpath related errors and confusion. While we work on that though, I wanted to make sure you were able to resolve your immediate issue. Were you? Thanks!

@kataggart kataggart self-assigned this May 23, 2022
@antoniolucasnobar
Copy link

antoniolucasnobar commented Jun 3, 2022

Hello @kataggart .

I was facing same error as OP. I need some scripts to execute every day. I was using default image (4.4.3) with files (changelog and sql folder) mounted by Kubernetes. The yaml definition:

---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: sigep-jobs-cronjob
#  annotations:
#    argocd.argoproj.io/hook: Sync
spec:
  concurrencyPolicy: Replace
  # horario UTC
  schedule: '5 5 * * *'
  jobTemplate:
    spec:
      template:
        spec:
          nodeSelector:
            app: "true"
          containers:
            - name: sigep-jobs
              image: registry.trt12.jus.br/liquibase/liquibase:4.4.3
              imagePullPolicy: IfNotPresent
              envFrom:
                - configMapRef:
                    name: autoatendimento-comum-config
                - secretRef:
                    name: folhaweb-secret
              args: [
                  "--logLevel=INFO",
                  "--url=$(ENV_DB_CONNECTIONSTRING)",
                  "--username=$(BANCO_USUARIO_SRH2)",
                  "--password=$(BANCO_SENHA_SRH2)",
                  "update"
              ]
              volumeMounts:
                - name: sigep-jobs-config-vol
                  mountPath: /liquibase/liquibase.docker.properties
                  readOnly: true
                  subPath: liquibase.docker.properties
                - name: sigep-jobs-config-vol
                  mountPath: /liquibase/changelog.yaml
                  readOnly: true
                  subPath: changelog.yaml
                - name: sigep-jobs-sql-vol
                  mountPath: /liquibase/trt12/sql
                  readOnly: true
          restartPolicy: OnFailure
          volumes:
            - name: sigep-jobs-config-vol
              configMap:
                defaultMode: 420
                name: sigep-jobs-liquibase-config
            - name: sigep-jobs-sql-vol
              configMap:
                defaultMode: 420
                name: sigep-jobs-sql-config
      backoffLimit: 4

My SQL file has the following header (I tried logicalFilePath with no luck):

--liquibase formatted sql logicalFilePath:trt12/sql/update_logins_vazios.sql
--changeset amandio:DML_UPDATE_LOGINS_VAZIOS stripComments:true splitStatements:true runAlways:true

changelog.yaml:

databaseChangeLog:
  - logical-file-path: trt12/sql
  - includeAll:
      path: trt12/sql
      author: trt12
      relativeToChangelogFile: true

The error I got:

Caused by: liquibase.exception.ChangeLogParseException: Error parsing changelog.yaml
	at liquibase.parser.core.yaml.YamlChangeLogParser.parse(YamlChangeLogParser.java:88)
	at liquibase.Liquibase.getDatabaseChangeLog(Liquibase.java:369)
	at liquibase.Liquibase.lambda$update$1(Liquibase.java:224)
	at liquibase.Scope.lambda$child$0(Scope.java:166)
	at liquibase.Scope.child(Scope.java:175)
	at liquibase.Scope.child(Scope.java:165)
	at liquibase.Scope.child(Scope.java:144)
	at liquibase.Liquibase.runInScope(Liquibase.java:2404)
	at liquibase.Liquibase.update(Liquibase.java:211)
	at liquibase.Liquibase.update(Liquibase.java:197)
	at liquibase.integration.commandline.Main.doMigration(Main.java:1882)
	at liquibase.integration.commandline.Main$1.lambda$run$0(Main.java:402)
	at liquibase.Scope.lambda$child$0(Scope.java:166)
	at liquibase.Scope.child(Scope.java:175)
	at liquibase.Scope.child(Scope.java:165)
	at liquibase.Scope.child(Scope.java:144)
	at liquibase.integration.commandline.Main$1.run(Main.java:401)
	... 20 more
Caused by: liquibase.exception.SetupException: java.io.IOException: Found 2 files that match trt12/sql/..data/update_logins_vazios.sql: file:/liquibase/trt12/sql/..2022_06_03_18_23_02.036673803/update_logins_vazios.sql, file:/liquibase/trt12/sql/..data/update_logins_vazios.sql
	at liquibase.changelog.DatabaseChangeLog.includeAll(DatabaseChangeLog.java:554)
	at liquibase.changelog.DatabaseChangeLog.handleChildNode(DatabaseChangeLog.java:412)
	at liquibase.changelog.DatabaseChangeLog.load(DatabaseChangeLog.java:321)
	at liquibase.parser.core.yaml.YamlChangeLogParser.parse(YamlChangeLogParser.java:82)

After many many tries, I built a custom docker image (based on the same liquibase 4.4.3 image) with all my scripts inside it . Et voilá. it works just fine:

Starting Liquibase at 15:53:11 (version 4.4.3 #53 built at 2021-08-05 18:32+0000)
Liquibase Version: 4.4.3
Liquibase Community 4.4.3 by Datical
[2022-06-03 15:53:13] INFO [liquibase.lockservice] Successfully acquired change log lock
[2022-06-03 15:53:14] INFO [liquibase.resource] Relative path changelog.yml in /liquibase/. does not exist
[2022-06-03 15:53:14] INFO [liquibase.changelog] Reading resource: trt12/sql/update_logins_vazios.sql
[2022-06-03 15:53:14] INFO [liquibase.changelog] Reading from SRH2TRT12.DATABASECHANGELOG
[2022-06-03 15:53:15] INFO [liquibase.lockservice] Successfully released change log lock
[2022-06-03 15:53:15] INFO [liquibase.lockservice] Successfully acquired change log lock
Skipping auto-registration
[2022-06-03 15:53:15] WARNING [liquibase.hub] Skipping auto-registration
[2022-06-03 15:53:15] INFO [liquibase.changelog] Custom SQL executed
[2022-06-03 15:53:15] INFO [liquibase.changelog] ChangeSet trt12/sql/update_logins_vazios.sql::DML_UPDATE_LOGINS_VAZIOS::amandio ran successfully in 198ms
[2022-06-03 15:53:15] INFO [liquibase.lockservice] Successfully released change log lock
Liquibase command 'update' was executed successfully.

I think liquibase is creating and reading aditional files (inside ..data and ..2022_06_03_18_23_02.036673803/ folders) when we use files from outside the image.

I hope it helps in your investigation. For me, the best would be not need to create a new image for every script. But, for now, it is the only way it works.

PS: We are running Liquibase against an Oracle 19c database. But, I think this is not related with any database.

@kataggart
Copy link
Contributor

@nvoxland I think this could be related to #2818

@nvoxland
Copy link
Contributor

Yes it is. The comment #2818 (comment) applies to this one as well and both #3006 and #2917 as possible options for addressing this.

@antoniolucasnobar
Copy link

Dear @kataggart and @nvoxland ,

Thanks for your solution. But I think it is strange in my case (see my comment above), because I use default liquibase docker image and inject the scripts via volume mount. There are no duplicated files in jar or anywhere else.
Should I open a new Issue?

@nvoxland
Copy link
Contributor

nvoxland commented Jul 6, 2022

Yes, @antoniolucasnobar I think the setup is something unique to the docker, so it would be nice to split that off into a separate issue we can look at.

We are wrapping up the 4.13.0 release which includes some improved logging in addition to the PRs mentioned above. Could you try your setup with the new release when it's available in docker and create an issue describing it and including the new log output?

@nvoxland
Copy link
Contributor

Closing this since the original issue is asking about the expected behavior of "if a path matches multiple files in the search-path, we fail with an error". The fix for that is to update your searchpath to not include duplicate files, or to use the liquibase.duplicateFileMode setting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

No branches or pull requests

6 participants