Re-create Elasticsearch schema on startup in tests/dev mode with Hibernate Search and Elasticsearch dev services

docs/src/main/asciidoc/hibernate-search-orm-elasticsearch.adoc

@@ -544,38 +544,57 @@ quarkus.ssl.native=false <1>
 
 quarkus.datasource.db-kind=postgresql <2>
 
-quarkus.hibernate-orm.database.generation=drop-and-create <3>
-quarkus.hibernate-orm.sql-load-script=import.sql <4>
+quarkus.hibernate-orm.sql-load-script=import.sql <3>
 
-quarkus.hibernate-search-orm.elasticsearch.version=7 <5>
-quarkus.hibernate-search-orm.elasticsearch.analysis.configurer=bean:myAnalysisConfigurer <6>
-quarkus.hibernate-search-orm.schema-management.strategy=drop-and-create <7>
-quarkus.hibernate-search-orm.automatic-indexing.synchronization.strategy=sync <8>
+quarkus.hibernate-search-orm.elasticsearch.version=7 <4>
+quarkus.hibernate-search-orm.elasticsearch.analysis.configurer=bean:myAnalysisConfigurer <5>
+quarkus.hibernate-search-orm.automatic-indexing.synchronization.strategy=sync <6>
 
-%prod.quarkus.datasource.jdbc.url=jdbc:postgresql://localhost/quarkus_test <9>
+%prod.quarkus.datasource.jdbc.url=jdbc:postgresql://localhost/quarkus_test <7>
 %prod.quarkus.datasource.username=quarkus_test
 %prod.quarkus.datasource.password=quarkus_test
-%prod.hibernate-search-orm.elasticsearch.hosts=localhost:9200 <9>
+%prod.quarkus.hibernate-orm.database.generation=create
+%prod.hibernate-search-orm.elasticsearch.hosts=localhost:9200 <7>
 ----
 <1> We won't use SSL, so we disable it to have a more compact native executable.
 <2> Let's create a PostgreSQL datasource.
-<3> We will drop and recreate the schema every time we start the application.
-<4> We load some initial data.
-<5> We need to tell Hibernate Search about the version of Elasticsearch we will use.
+<3> We load some initial data on startup.
+<4> We need to tell Hibernate Search about the version of Elasticsearch we will use.
 It is important because there are significant differences between Elasticsearch mapping syntax depending on the version.
 Since the mapping is created at build time to reduce startup time, Hibernate Search cannot connect to the cluster to automatically detect the version.
 Note that, for OpenSearch, you need to prefix the version with `opensearch:`; see <<opensearch>>.
-<6> We point to the custom `AnalysisConfigurer` which defines the configuration of our analyzers and normalizers.
-<7> Obviously, this is not for production: we drop and recreate the index every time we start the application.
-<8> This means that we wait for the entities to be searchable before considering a write complete.
+<5> We point to the custom `AnalysisConfigurer` which defines the configuration of our analyzers and normalizers.
+<6> This means that we wait for the entities to be searchable before considering a write complete.
 On a production setup, the `write-sync` default will provide better performance.
 Using `sync` is especially important when testing as you need the entities to be searchable immediately.
-<9> For development and tests, we rely on <<dev-services,Dev Services>>,
+<7> For development and tests, we rely on <<dev-services,Dev Services>>,
 which means Quarkus will start a PostgreSQL database and Elasticsearch cluster automatically.
 In production mode, however,
 you will want to start a PostgreSQL database and Elasticsearch cluster manually,
 which is why we provide Quarkus with this connection info in the `prod` profile (`%prod.` prefix).
 
+[NOTE]
+====
+Because we rely on <<dev-services,Dev Services>>, the database and Elasticsearch schema
+will automatically be dropped and re-created on each application startup
+in tests and dev mode
+(unless link:#quarkus-hibernate-search-orm-elasticsearch_quarkus.hibernate-search-orm.schema-management.strategy[`quarkus.hibernate-search-orm.schema-management.strategy`] is set explicitly).
+
+If for some reason you cannot use Dev Services,
+you will have to set the following properties to get similar behavior:
+
+[source,properties]
+----
+%dev.quarkus.hibernate-orm.database.generation=drop-and-create
+%test.quarkus.hibernate-orm.database.generation=drop-and-create
+%dev.quarkus.hibernate-search-orm.schema-management.strategy=drop-and-create
+%test.quarkus.hibernate-search-orm.schema-management.strategy=drop-and-create
+----
+
+See also link:#quarkus-hibernate-search-orm-elasticsearch_quarkus.hibernate-search-orm.schema-management.strategy[`quarkus.hibernate-search-orm.schema-management.strategy`].
+====
+
+
 [TIP]
 For more information about the Hibernate Search extension configuration please refer to the <<configuration-reference, Configuration Reference>>.
 
@@ -690,7 +709,6 @@ quarkus.hibernate-orm.dialect=org.hibernate.dialect.H2Dialect
 
 quarkus.hibernate-search-orm.elasticsearch.hosts=es1.mycompany.com:9200
 quarkus.hibernate-search-orm.elasticsearch.version=7
-quarkus.hibernate-search-orm.automatic-indexing.synchronization.strategy=write-sync
 ----
 
 Using a map based approach, it is also possible to configure named persistence units:
@@ -711,11 +729,9 @@ quarkus.hibernate-orm."inventory".packages=org.acme.model.inventory
 
 quarkus.hibernate-search-orm."users".elasticsearch.hosts=es1.mycompany.com:9200 <5>
 quarkus.hibernate-search-orm."users".elasticsearch.version=7
-quarkus.hibernate-search-orm."users".automatic-indexing.synchronization.strategy=write-sync
 
 quarkus.hibernate-search-orm."inventory".elasticsearch.hosts=es2.mycompany.com:9200 <6>
 quarkus.hibernate-search-orm."inventory".elasticsearch.version=7
-quarkus.hibernate-search-orm."inventory".automatic-indexing.synchronization.strategy=write-sync
 ----
 <1> Define a datasource named `users`.
 <2> Define a datasource named `inventory`.

extensions/hibernate-search-orm-elasticsearch/deployment/src/main/java/io/quarkus/hibernate/search/orm/elasticsearch/deployment/HibernateSearchElasticsearchProcessor.java

@@ -1,6 +1,9 @@
 package io.quarkus.hibernate.search.orm.elasticsearch.deployment;
 
 import static io.quarkus.hibernate.search.orm.elasticsearch.deployment.HibernateSearchClasses.INDEXED;
+import static io.quarkus.hibernate.search.orm.elasticsearch.runtime.HibernateSearchElasticsearchRuntimeConfig.backendPropertyKey;
+import static io.quarkus.hibernate.search.orm.elasticsearch.runtime.HibernateSearchElasticsearchRuntimeConfig.elasticsearchVersionPropertyKey;
+import static io.quarkus.hibernate.search.orm.elasticsearch.runtime.HibernateSearchElasticsearchRuntimeConfig.mapperPropertyKey;
 
 import java.nio.file.Files;
 import java.nio.file.Path;
@@ -22,15 +25,18 @@
 import org.hibernate.search.mapper.orm.automaticindexing.session.AutomaticIndexingSynchronizationStrategy;
 import org.jboss.jandex.AnnotationInstance;
 import org.jboss.jandex.IndexView;
+import org.jboss.logging.Logger;
 
 import io.quarkus.arc.deployment.UnremovableBeanBuildItem;
 import io.quarkus.deployment.Feature;
+import io.quarkus.deployment.IsNormal;
 import io.quarkus.deployment.annotations.BuildProducer;
 import io.quarkus.deployment.annotations.BuildStep;
 import io.quarkus.deployment.annotations.ExecutionTime;
 import io.quarkus.deployment.annotations.Record;
 import io.quarkus.deployment.builditem.ApplicationArchivesBuildItem;
 import io.quarkus.deployment.builditem.CombinedIndexBuildItem;
+import io.quarkus.deployment.builditem.DevServicesAdditionalConfigBuildItem;
 import io.quarkus.deployment.builditem.FeatureBuildItem;
 import io.quarkus.deployment.builditem.HotDeploymentWatchedFileBuildItem;
 import io.quarkus.deployment.builditem.NativeImageFeatureBuildItem;
@@ -54,12 +60,15 @@
 import io.quarkus.hibernate.search.orm.elasticsearch.runtime.HibernateSearchElasticsearchRecorder;
 import io.quarkus.hibernate.search.orm.elasticsearch.runtime.HibernateSearchElasticsearchRuntimeConfig;
 import io.quarkus.hibernate.search.orm.elasticsearch.runtime.graal.DisableLoggingFeature;
+import io.quarkus.runtime.configuration.ConfigUtils;
 import io.quarkus.runtime.configuration.ConfigurationException;
 
 class HibernateSearchElasticsearchProcessor {
 
     private static final String HIBERNATE_SEARCH_ELASTICSEARCH = "Hibernate Search ORM + Elasticsearch";
 
+    private static final Logger LOG = Logger.getLogger(HibernateSearchElasticsearchProcessor.class);
+
     HibernateSearchElasticsearchBuildTimeConfig buildTimeConfig;
 
     @BuildStep(onlyIf = NativeOrNativeSourcesBuild.class)
@@ -323,26 +332,6 @@ private static void registerClasspathFileFromConfig(String persistenceUnitName,
         hotDeploymentWatchedFiles.produce(new HotDeploymentWatchedFileBuildItem(classpathFile));
     }
 
-    private static String elasticsearchVersionPropertyKey(String persistenceUnitName, String backendName) {
-        return backendPropertyKey(persistenceUnitName, backendName, null, "version");
-    }
-
-    private static String backendPropertyKey(String persistenceUnitName, String backendName, String indexName, String radical) {
-        StringBuilder keyBuilder = new StringBuilder("quarkus.hibernate-search-orm.");
-        if (!PersistenceUnitUtil.isDefaultPersistenceUnit(persistenceUnitName)) {
-            keyBuilder.append(persistenceUnitName).append(".");
-        }
-        keyBuilder.append("elasticsearch.");
-        if (backendName != null) {
-            keyBuilder.append(backendName).append(".");
-        }
-        if (indexName != null) {
-            keyBuilder.append("indexes.").append(indexName).append(".");
-        }
-        keyBuilder.append(radical);
-        return keyBuilder.toString();
-    }
-
     private void registerReflectionForGson(BuildProducer<ReflectiveClassBuildItem> reflectiveClass) {
         String[] reflectiveClasses = GsonClasses.typesRequiringReflection().toArray(String[]::new);
         reflectiveClass.produce(new ReflectiveClassBuildItem(true, true, reflectiveClasses));
@@ -354,7 +343,9 @@ DevservicesElasticsearchBuildItem devServices(HibernateSearchElasticsearchBuildT
         // If the version is not set, the default backend is not in use.
                 && buildTimeConfig.defaultPersistenceUnit.defaultBackend.version.isPresent()) {
             ElasticsearchVersion version = buildTimeConfig.defaultPersistenceUnit.defaultBackend.version.get();
-            return new DevservicesElasticsearchBuildItem("quarkus.hibernate-search-orm.elasticsearch.hosts",
+            String hostsPropertyKey = backendPropertyKey(PersistenceUnitUtil.DEFAULT_PERSISTENCE_UNIT_NAME, null, null,
+                    "hosts");
+            return new DevservicesElasticsearchBuildItem(hostsPropertyKey,
                     version.versionString(),
                     DevservicesElasticsearchBuildItem.Distribution.valueOf(version.distribution().toString().toUpperCase()));
         } else {
@@ -363,4 +354,27 @@ DevservicesElasticsearchBuildItem devServices(HibernateSearchElasticsearchBuildT
             return null;
         }
     }
+
+    @BuildStep(onlyIfNot = IsNormal.class)
+    void devServicesDropAndCreateAndDropByDefault(
+            List<HibernateSearchElasticsearchPersistenceUnitConfiguredBuildItem> configuredPersistenceUnits,
+            BuildProducer<DevServicesAdditionalConfigBuildItem> devServicesAdditionalConfigProducer) {
+        for (HibernateSearchElasticsearchPersistenceUnitConfiguredBuildItem configuredPersistenceUnit : configuredPersistenceUnits) {
+            String puName = configuredPersistenceUnit.getPersistenceUnitName();
+            String propertyKeyIndicatingHostsConfigured = backendPropertyKey(puName, null, null, "hosts");
+
+            if (!ConfigUtils.isPropertyPresent(propertyKeyIndicatingHostsConfigured)) {
+                String schemaManagementStrategyPropertyKey = mapperPropertyKey(puName, "schema-management.strategy");
+                if (!ConfigUtils.isPropertyPresent(schemaManagementStrategyPropertyKey)) {
+                    String forcedValue = "drop-and-create-and-drop";
+                    devServicesAdditionalConfigProducer
+                            .produce(new DevServicesAdditionalConfigBuildItem(propertyKeyIndicatingHostsConfigured,
+                                    schemaManagementStrategyPropertyKey, forcedValue,
+                                    () -> LOG.infof("Setting %s=%s to initialize Dev Services managed Elasticsearch server",
+                                            schemaManagementStrategyPropertyKey, forcedValue)));
+                }
+            }
+        }
+    }
+
 }

extensions/hibernate-search-orm-elasticsearch/runtime/src/main/java/io/quarkus/hibernate/search/orm/elasticsearch/runtime/HibernateSearchElasticsearchRuntimeConfig.java

@@ -2,6 +2,7 @@
 
 import java.util.Map;
 
+import io.quarkus.hibernate.orm.runtime.PersistenceUnitUtil;
 import io.quarkus.runtime.annotations.ConfigDocMapKey;
 import io.quarkus.runtime.annotations.ConfigDocSection;
 import io.quarkus.runtime.annotations.ConfigItem;
@@ -25,4 +26,32 @@ public class HibernateSearchElasticsearchRuntimeConfig {
     @ConfigItem(name = ConfigItem.PARENT)
     public Map<String, HibernateSearchElasticsearchRuntimeConfigPersistenceUnit> persistenceUnits;
 
+    public static String elasticsearchVersionPropertyKey(String persistenceUnitName, String backendName) {
+        return backendPropertyKey(persistenceUnitName, backendName, null, "version");
+    }
+
+    public static String mapperPropertyKey(String persistenceUnitName, String radical) {
+        StringBuilder keyBuilder = new StringBuilder("quarkus.hibernate-search-orm.");
+        if (!PersistenceUnitUtil.isDefaultPersistenceUnit(persistenceUnitName)) {
+            keyBuilder.append("\"").append(persistenceUnitName).append("\".");
+        }
+        keyBuilder.append(radical);
+        return keyBuilder.toString();
+    }
+
+    public static String backendPropertyKey(String persistenceUnitName, String backendName, String indexName, String radical) {
+        StringBuilder keyBuilder = new StringBuilder("quarkus.hibernate-search-orm.");
+        if (!PersistenceUnitUtil.isDefaultPersistenceUnit(persistenceUnitName)) {
+            keyBuilder.append(persistenceUnitName).append(".");
+        }
+        keyBuilder.append("elasticsearch.");
+        if (backendName != null) {
+            keyBuilder.append("\"").append(backendName).append("\".");
+        }
+        if (indexName != null) {
+            keyBuilder.append("indexes.\"").append(indexName).append("\".");
+        }
+        keyBuilder.append(radical);
+        return keyBuilder.toString();
+    }
 }

extensions/hibernate-search-orm-elasticsearch/runtime/src/main/java/io/quarkus/hibernate/search/orm/elasticsearch/runtime/HibernateSearchElasticsearchRuntimeConfigPersistenceUnit.java

@@ -380,7 +380,7 @@ public static class SchemaManagementConfig {
          *
          * For indexes that already exist, do nothing: assume that their schema matches Hibernate Search's expectations.
          *
-         * !create-or-validate (**default**)
+         * !create-or-validate (**default** unless using Dev Services)
          * !For indexes that do not exist, create them along with their schema.
          *
          * For indexes that already exist, validate that their schema matches Hibernate Search's expectations.
@@ -402,7 +402,7 @@ public static class SchemaManagementConfig {
          *
          * For indexes that already exist, drop them, then create them along with their schema.
          *
-         * !drop-and-create-and-drop
+         * !drop-and-create-and-drop (**default** when using Dev Services)
          * !For indexes that do not exist, create them along with their schema.
          *
          * For indexes that already exist, drop them, then create them along with their schema.
@@ -416,7 +416,7 @@ public static class SchemaManagementConfig {
          * @asciidoclet
          */
         // @formatter:on
-        @ConfigItem(defaultValue = "create-or-validate")
+        @ConfigItem(defaultValue = "create-or-validate", defaultValueDocumentation = "drop-and-create-and-drop when using Dev Services; create-or-validate otherwise")
         SchemaManagementStrategyName strategy;
 
     }

integration-tests/hibernate-search-orm-elasticsearch/src/main/java/io/quarkus/it/hibernate/search/orm/elasticsearch/devservices/HibernateSearchDevServicesTestResource.java

@@ -12,6 +12,7 @@
 
 import org.hibernate.Session;
 import org.hibernate.SessionFactory;
+import org.hibernate.search.mapper.orm.schema.management.SchemaManagementStrategyName;
 import org.hibernate.search.mapper.orm.session.SearchSession;
 
 @Path("/test/dev-services")
@@ -34,6 +35,15 @@ public String hosts() {
         return ((List<String>) sessionFactory.getProperties().get("hibernate.search.backend.hosts")).iterator().next();
     }
 
+    @GET
+    @Path("/schema-management-strategy")
+    @Transactional
+    public String schemaManagementStrategy() {
+        var strategy = ((SchemaManagementStrategyName) sessionFactory.getProperties()
+                .get("hibernate.search.schema_management.strategy"));
+        return strategy == null ? null : strategy.externalRepresentation();
+    }
+
     @PUT
     @Path("/init-data")
     @Transactional

integration-tests/hibernate-search-orm-elasticsearch/src/main/resources/application.properties

@@ -14,8 +14,12 @@ quarkus.hibernate-search-orm.elasticsearch.indexes.Analysis2TestingEntity.analys
 quarkus.hibernate-search-orm.elasticsearch.indexes.Analysis3TestingEntity.analysis.configurer=io.quarkus.it.hibernate.search.orm.elasticsearch.analysis.IndexAnalysis3Configurer
 quarkus.hibernate-search-orm.elasticsearch.indexes.Analysis4TestingEntity.analysis.configurer=index-analysis-4
 quarkus.hibernate-search-orm.elasticsearch.indexes.Analysis5TestingEntity.analysis.configurer=io.quarkus.it.hibernate.search.orm.elasticsearch.analysis.IndexAnalysis5Configurer
-quarkus.hibernate-search-orm.schema-management.strategy=drop-and-create-and-drop
 quarkus.hibernate-search-orm.automatic-indexing.synchronization.strategy=sync
 
+# Use drop-and-create instead of drop-and-create-and-drop
+# so we can differentiate between the value we set here
+# and the value set automatically by the extension when using dev services
+# See io.quarkus.it.hibernate.search.orm.elasticsearch.devservices.HibernateSearchElasticsearchDevServicesEnabledImplicitlyTest.testHibernateSearch
+%test.quarkus.hibernate-search-orm.schema-management.strategy=drop-and-create
 %test.quarkus.hibernate-search-orm.elasticsearch.hosts=${elasticsearch.hosts:localhost:9200}
 %test.quarkus.hibernate-search-orm.elasticsearch.protocol=${elasticsearch.protocol:http}

integration-tests/hibernate-search-orm-elasticsearch/src/test/java/io/quarkus/it/hibernate/search/orm/elasticsearch/devservices/HibernateSearchElasticsearchDevServicesEnabledImplicitlyTest.java

@@ -52,6 +52,12 @@ public void testHibernateSearch() {
                 .statusCode(200)
                 .body(is(context.devServicesProperties().get("quarkus.hibernate-search-orm.elasticsearch.hosts")));
 
+        RestAssured.when().get("/test/dev-services/schema-management-strategy").then()
+                .statusCode(200)
+                // If the value is drop-and-create, this would indicate we're using the %test profile:
+                // that would be a bug in this test (see the Profile class above).
+                .body(is("drop-and-create-and-drop"));
+
         RestAssured.when().get("/test/dev-services/count").then()
                 .statusCode(200)
                 .body(is("0"));