Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update index operator has bug after insert overwrite table by hive engine #339

Open
hfsugar opened this issue Apr 1, 2022 · 0 comments
Open

Comments

@hfsugar
Copy link
Contributor

hfsugar commented Apr 1, 2022

Software Environment:

  • OpenLooKeng version (source or binary):
    1.7
  • OS platform & distribution (eg., Linux Ubuntu 16.04):
    linux
  • Java version:

Describe the current behavior

exp thrown in updateIndex
java.lang.IllegalStateException: Duplicate key /warehouse1/bili_infra.db/index_test2/log_date=20210711/000000_0 (attempted merging values 1648718284093 and 1648786448561)

Describe the expected behavior

FileIndexWriter.persist
needs to check if any old index files should be deleted.

Steps to reproduce the issue

create index idx2 using minmax on hive.bili_infra.index_test2 (mid);

image

executing sql in hive engine as below

insert overwrite table bili_infra.index_test2 partition (log_date) select * from bili_infra.index_test2 where log_date='20210711' and mid<>0;
update index idx2
  1. update again
update index idx2

java.lang.IllegalStateException: Duplicate key /warehouse1/bili_infra.db/index_test2/log_date=20210711/000000_0 (attempted merging values 1648718284093 and 1648786448561)
image

Related log/screenshots

index_test2 created as below.

image

Special notes for this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant