Jetty 9.4.x 2140 infinispan expired sessions (#3457)

Infinispan and hazelcast changes to scavenge zombie expired sessions.

Signed-off-by: Lachlan Roberts <lachlan@webtide.com>
Signed-off-by: Jan Bartel <janb@webtide.com>
Signed-off-by: olivier lamy <oliver.lamy@gmail.com>

jetty-documentation/src/main/asciidoc/administration/sessions/session-configuration-hazelcast.adoc

@@ -84,6 +84,7 @@ Opening the `start.d/session-store-hazelcast-remote.ini` will show a list of all
 #jetty.session.hazelcast.mapName=jetty_sessions
 #jetty.session.hazelcast.onlyClient=true
 #jetty.session.hazelcast.configurationLocation=
+jetty.session.hazelcast.scavengeZombies=false
 #jetty.session.gracePeriod.seconds=3600
 #jetty.session.savePeriod.seconds=0
 ----
@@ -94,6 +95,8 @@ jetty.session.hazelcast.onlyClient::
 Hazelcast instance will be configured in client mode
 jetty.session.hazelcast.configurationLocation::
 Path to an an Hazelcast xml configuration file
+jetty.session.hazelcast.scavengeZombies::
+True/False. `False` by default. If `true`, jetty will use hazelcast queries to find sessions that are no longer being used on any jetty node and whose expiry time has passed. If you enable this option, and your session stores attributes that reference classes from inside your webapp, or jetty classes, you will need to ensure that these classes are available on each of your hazelcast instances.
 jetty.session.gracePeriod.seconds::
 Amount of time, in seconds, to wait for other nodes to be checked to verify an expired session is in fact expired throughout the cluster before closing it.
 jetty.session.savePeriod.seconds=0::
@@ -106,6 +109,8 @@ Configuring `savePeriod` is useful if your persistence technology is very slow/c
 In a clustered environment, there is a risk of the last access time of the session being out-of-date in the shared store for up to `savePeriod` seconds.
 This allows the possibility that a node may prematurely expire the session, even though it is in use by another node.
 Thorough consideration of the `maxIdleTime` of the session when setting the `savePeriod` is imperative - there is no point in setting a `savePeriod` that is larger than the `maxIdleTime`.
+
+Be aware using the `scavengeZombies` option that if your session attributes contain classes from inside your webapp (or jetty classes) then you will need to put these classes onto the classpath of all of your hazelcast instances.
 ____
 
 ==== Configuring Embedded Hazelcast Clustering
@@ -165,15 +170,18 @@ Opening the `start.d/start.d/session-store-hazelcast-embedded.ini` will show a l
 
 #jetty.session.hazelcast.mapName=jetty_sessions
 #jetty.session.hazelcast.configurationLocation=
+jetty.session.hazelcast.scavengeZombies=false
 #jetty.session.gracePeriod.seconds=3600
 #jetty.session.savePeriod.seconds=0
 ----
 jetty.session.hazelcast.mapName::
 Name of the Map in Hazelcast where sessions will be stored.
-jetty.session.gracePeriod.seconds::
-Amount of time, in seconds, to wait for other nodes to be checked to verify an expired session is in fact expired throughout the cluster before closing it.
 jetty.session.hazelcast.configurationLocation::
 Path to an an Hazelcast xml configuration file
+jetty.session.hazelcast.scavengeZombies::
+True/False. `False` by default. If `true`, jetty will use hazelcast queries to find sessions that are no longer being used on any jetty node and whose expiry time has passed. If you enable this option, and your sessions contain attributes that reference classes from inside your webapp (or jetty classes) you will need to ensure that these classes are available on each of your hazelcast instances.
+jetty.session.gracePeriod.seconds::
+Amount of time, in seconds, to wait for other nodes to be checked to verify an expired session is in fact expired throughout the cluster before closing it.
 jetty.session.savePeriod.seconds=0::
 By default whenever the last concurrent request leaves a session, that session is always persisted via the `SessionDataStore`, even if the only thing that changed on the session is its updated last access time.
 A non-zero value means that the `SessionDataStore` will skip persisting the session if only the access time changed, and it has been less than `savePeriod` seconds since the last time the session was written.
@@ -184,4 +192,6 @@ Configuring `savePeriod` is useful if your persistence technology is very slow/c
 In a clustered environment, there is a risk of the last access time of the session being out-of-date in the shared store for up to `savePeriod` seconds.
 This allows the possibility that a node may prematurely expire the session, even though it is in use by another node.
 Thorough consideration of the `maxIdleTime` of the session when setting the `savePeriod` is imperative - there is no point in setting a `savePeriod` that is larger than the `maxIdleTime`.
+
+Be aware using the `scavengeZombies` option that if your session attributes contain classes from inside your webapp (or jetty classes) then you will need to put these classes onto the classpath of all of your hazelcast instances. In the cast of embedded hazelcast, as it is started before your webapp, it will NOT have access to your webapp's classes - you will need to extract these classes and put them onto the jetty server's classpath.
 ____

jetty-documentation/src/main/asciidoc/administration/sessions/session-configuration-infinispan.adoc

@@ -24,10 +24,6 @@
 
 When using the Jetty distribution, you will first need to enable the `session-store-infinispan-remote` link:#startup-modules[module] for your link:#startup-base-and-home[Jetty base] using the `--add-to-start` argument on the command line.
 
-____
-[IMPORTANT]
-If you are running Jetty with JDK 9 or greater, enable `session-store-infinispan-remote-910.mod` instead.
-____
 
 [source, screen, subs="{sub-order}"]
 ----
@@ -52,7 +48,7 @@ INFO  : server          transitively enabled, ini template available with --add-
 INFO  : sessions        transitively enabled, ini template available with --add-to-start=sessions
 INFO  : session-store-infinispan-remote initialized in ${jetty.base}/start.d/session-store-infinispan-remote.ini
 MKDIR : ${jetty.base}/lib/infinispan
-DOWNLD: https://repo1.maven.org/maven2/org/infinispan/infinispan-remote/7.1.1.Final/infinispan-remote-7.1.1.Final.jar to ${jetty.base}/lib/infinispan/infinispan-remote-7.1.1.Final.jar
+DOWNLD: https://repo1.maven.org/maven2/org/infinispan/infinispan-remote-it/9.4.8.Final/infinispan-remote-it-9.4.8.Final.jar to ${jetty.base}/lib/infinispan/infinispan-remote-it-9.4.8.Final.jar
 MKDIR : ${jetty.base}/resources
 COPY  : ${jetty.home}/modules/session-store-infinispan-remote/resources/hotrod-client.properties to ${jetty.base}/resources/hotrod-client.properties
 INFO  : Base directory was modified
@@ -93,13 +89,17 @@ Opening the `start.d/session-store-infinispan-remote.ini` will show a list of al
 jetty.session.infinispan.remoteCacheName::
 Name of the cache in Infinispan where sessions will be stored.
 jetty.session.infinispan.idleTimeout.seconds::
-Amount of time, in seconds, that the system allows the connector to remain idle before closing the connection.
+Amount of time, in seconds, that a session entry in infinispan can be idle (ie not read or written) before infinispan will delete its entry.
+Usually, you do *not* want to set a value for this, as you want jetty to handle all session expiration (and call any SessionListeners).
+However, if there is the possibility that sessions can be left in infinispan but no longer referenced by any jetty node (so called "zombie" or "orphan" sessions), then you might want to use this feature.
+You should make sure that the number of seconds you specify is sufficiently large to avoid the situation where a session is still being referenced by jetty, but is rarely accessed and thus deleted by infinispan.
+Alternatively, you can enable the `infinispan-remote-query` module, which will allow jetty to search the infinispan session cache to proactively find and properly (ie calling any SessionListeners) scavenge defunct sessions.
 jetty.session.gracePeriod.seconds::
 Amount of time, in seconds, to wait for other nodes to be checked to verify an expired session is in fact expired throughout the cluster before closing it.
 jetty.session.savePeriod.seconds=0::
 By default whenever the last concurrent request leaves a session, that session is always persisted via the `SessionDataStore`, even if the only thing that changed on the session is its updated last access time.
 A non-zero value means that the `SessionDataStore` will skip persisting the session if only the access time changed, and it has been less than `savePeriod` seconds since the last time the session was written.
-+
+
 ____
 [NOTE]
 Configuring `savePeriod` is useful if your persistence technology is very slow/costly for writes.
@@ -108,6 +108,19 @@ This allows the possibility that a node may prematurely expire the session, even
 Thorough consideration of the `maxIdleTime` of the session when setting the `savePeriod` is imperative - there is no point in setting a `savePeriod` that is larger than the `maxIdleTime`.
 ____
 
+==== Configuring the Remote Infinispan Query Module
+
+Enabling this module allows jetty to search infinispan for expired sessions that are no longer being referenced by any jetty node.
+Note that this is an *additional* module, to be used in conjuction with the `session-store-infinispan-remote` module.
+
+[source, screen, subs="{sub-order}"]
+----
+java -jar ../start.jar --add-to-start=infinispan-remote-query
+----
+
+There are no configuration properties associated with this module.
+
+
 ==== Configuring Embedded Inifinspan Clustering
 
 During testing, it can be helpful to run an in-process instance of Infinispan.
@@ -137,7 +150,7 @@ Proceed (y/N)? y
 INFO : server          initialised (transitively) in ${jetty.base}/start.d/server.ini
 INFO : sessions        initialised (transitively) in ${jetty.base}/start.d/sessions.ini
 INFO : session-store-infinispan-embedded initialised in ${jetty.base}/start.d/session-store-infinispan-embedded.ini
-DOWNLOAD: https://repo1.maven.org/maven2/org/infinispan/infinispan-embedded/7.1.1.Final/infinispan-embedded-7.1.1.Final.jar to ${jetty.base}/lib/infinispan/infinispan-embedded-7.1.1.Final.jar
+DOWNLOAD: https://repo1.maven.org/maven2/org/infinispan/infinispan-embedded-it/9.4.8.Final/infinispan-embedded-it-9.4.8.Final.jar to ${jetty.base}/lib/infinispan/infinispan-embedded-it-9.4.8.Final.jar
 INFO : Base directory was modified
 ----
 
@@ -180,6 +193,19 @@ This allows the possibility that a node may prematurely expire the session, even
 Thorough consideration of the `maxIdleTime` of the session when setting the `savePeriod` is imperative - there is no point in setting a `savePeriod` that is larger than the `maxIdleTime`.
 ____
 
+
+==== Configuring Inifinspan Embedded Query
+
+Similarly to the `session-store-infinispan-remote` module, the `session-store-infinispan-embedded` module has an adjunct module `infinispan-embedded-query`, which when enabled, will allow jetty to detect and properly scavenge defunct sessions stranded in infinispan.
+
+[source, screen, subs="{sub-order}"]
+----
+java -jar ../start.jar --add-to-start=infinispan-embedded-query
+----
+
+There are no configuration properties associated with this module.
+
+
 ==== Converting session format for jetty-9.4.13
 
 From jetty-9.4.13 onwards, we have changed the format of the serialized session when using a remote cache (ie using hotrod).

jetty-hazelcast/pom.xml

@@ -12,7 +12,7 @@
   <name>Jetty :: Hazelcast Session Manager</name>
 
   <properties>
-    <hazelcast.version>3.9.3</hazelcast.version>
+    <hazelcast.version>3.9.4</hazelcast.version>
     <bundle-symbolic-name>${project.groupId}.hazelcast</bundle-symbolic-name>
   </properties>
 

jetty-hazelcast/src/main/config/etc/sessions/hazelcast/default.xml

@@ -13,6 +13,7 @@
       <New id="sessionDataStoreFactory" class="org.eclipse.jetty.hazelcast.session.HazelcastSessionDataStoreFactory">
         <Set name="mapName"><Property name="jetty.session.hazelcast.mapName" default="jetty-distributed-session-map" /></Set>
         <Set name="hazelcastInstanceName"><Property name="jetty.session.hazelcast.hazelcastInstanceName" default="JETTY_DISTRIBUTED_SESSION_INSTANCE" /></Set>
+        <Set name="scavengeZombies"><Property name="jetty.session.hazelcast.scavengeZombies" default="false"/></Set>
         <Set name="gracePeriodSec"><Property name="jetty.session.gracePeriod.seconds" default="3600" /></Set>
         <Set name="savePeriodSec"><Property name="jetty.session.savePeriod.seconds" default="0" /></Set>
       </New>

jetty-hazelcast/src/main/config/etc/sessions/hazelcast/remote.xml

@@ -12,6 +12,7 @@
       <New id="sessionDataStoreFactory" class="org.eclipse.jetty.hazelcast.session.HazelcastSessionDataStoreFactory">
         <Set name="mapName"><Property name="jetty.session.hazelcast.mapName" default="jetty-distributed-session-map" /></Set>
         <Set name="hazelcastInstanceName"><Property name="jetty.session.hazelcast.hazelcastInstanceName" default="JETTY_DISTRIBUTED_SESSION_INSTANCE" /></Set>
+        <Set name="scavengeZombies"><Property name="jetty.session.hazelcast.scavengeZombies" default="false"/></Set>
         <Set name="gracePeriodSec"><Property name="jetty.session.gracePeriod.seconds" default="3600" /></Set>
         <Set name="savePeriodSec"><Property name="jetty.session.savePeriod.seconds" default="0" /></Set>
         <Set name="onlyClient"><Property name="jetty.session.hazelcast.onlyClient" default="true" /></Set>

jetty-hazelcast/src/main/config/modules/session-store-hazelcast-embedded.mod

@@ -13,7 +13,7 @@ session-store
 sessions
 
 [files]
-maven://com.hazelcast/hazelcast/3.9.3|lib/hazelcast/hazelcast-3.9.3.jar
+maven://com.hazelcast/hazelcast/3.9.4|lib/hazelcast/hazelcast-3.9.4.jar
 
 [xml]
 etc/sessions/hazelcast/default.xml
@@ -32,5 +32,6 @@ http://www.apache.org/licenses/LICENSE-2.0.html
 jetty.session.hazelcast.mapName=jetty-distributed-session-map
 jetty.session.hazelcast.hazelcastInstanceName=JETTY_DISTRIBUTED_SESSION_INSTANCE
 #jetty.session.hazelcast.configurationLocation=
+jetty.session.hazelcast.scavengeZombies=false
 jetty.session.gracePeriod.seconds=3600
 jetty.session.savePeriod.seconds=0

jetty-hazelcast/src/main/config/modules/session-store-hazelcast-remote.mod

@@ -13,8 +13,8 @@ session-store
 sessions
 
 [files]
-maven://com.hazelcast/hazelcast/3.9.3|lib/hazelcast/hazelcast-3.9.3.jar
-maven://com.hazelcast/hazelcast-client/3.9.3|lib/hazelcast/hazelcast-client-3.9.3.jar
+maven://com.hazelcast/hazelcast/3.9.4|lib/hazelcast/hazelcast-3.9.4.jar
+maven://com.hazelcast/hazelcast-client/3.9.4|lib/hazelcast/hazelcast-client-3.9.4.jar
 
 [xml]
 etc/sessions/hazelcast/remote.xml
@@ -33,6 +33,7 @@ http://www.apache.org/licenses/LICENSE-2.0.html
 jetty.session.hazelcast.mapName=jetty-distributed-session-map
 jetty.session.hazelcast.hazelcastInstanceName=JETTY_DISTRIBUTED_SESSION_INSTANCE
 jetty.session.hazelcast.onlyClient=true
+jetty.session.hazelcast.scavengeZombies=false
 #jetty.session.hazelcast.configurationLocation=
 jetty.session.gracePeriod.seconds=3600
 jetty.session.savePeriod.seconds=0

jetty-hazelcast/src/main/java/org/eclipse/jetty/hazelcast/session/HazelcastSessionDataStore.java

@@ -18,7 +18,8 @@
 
 package org.eclipse.jetty.hazelcast.session;
 
-import java.util.Collections;
+import java.util.Collection;
+import java.util.HashSet;
 import java.util.Set;
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.stream.Collectors;
@@ -33,6 +34,9 @@
 import org.eclipse.jetty.util.log.Logger;
 
 import com.hazelcast.core.IMap;
+import com.hazelcast.query.EntryObject;
+import com.hazelcast.query.Predicate;
+import com.hazelcast.query.PredicateBuilder;
 
 /**
  * Session data stored in Hazelcast
@@ -47,9 +51,35 @@ public class HazelcastSessionDataStore
 
     private IMap<String, SessionData> sessionDataMap;
 
+    private boolean _scavengeZombies;
+
     public HazelcastSessionDataStore()
     {
-        // no op
+    }
+
+    /** Control whether or not to execute queries to find
+     * "zombie" sessions - ie sessions that are no longer
+     * actively referenced by any jetty instance and should
+     * be expired.
+     * 
+     * If you use this feature, be aware that if your session
+     * stores any attributes that use classes from within your
+     * webapp, or from within jetty, you will need to make sure
+     * those classes are available to all of your hazelcast 
+     * instances, whether embedded or remote.
+     * 
+     * @param scavengeZombies true means unreferenced sessions
+     * will be actively sought and expired. False means that they
+     * will remain in hazelcast until some other mechanism removes them.
+     */
+    public void setScavengeZombieSessions (boolean scavengeZombies)
+    {
+        _scavengeZombies = scavengeZombies;
+    }
+
+    public boolean isScavengeZombies()
+    {
+        return _scavengeZombies;
     }
 
     @Override
@@ -97,7 +127,9 @@ public void setSessionDataMap( IMap<String, SessionData> sessionDataMap )
     public void initialize( SessionContext context )
         throws Exception
     {
-        _context = context;
+        super.initialize(context);
+        if (isScavengeZombies())
+            sessionDataMap.addIndex("expiry", true);
     }
 
     @Override
@@ -113,16 +145,13 @@ public boolean isPassivating()
         return true;
     }
 
+
     @Override
     public Set<String> doGetExpired( Set<String> candidates )
     {
-        if (candidates == null || candidates.isEmpty())
-        {
-            return Collections.emptySet();
-        }
-
         long now = System.currentTimeMillis();
-        return candidates.stream().filter( candidate -> {
+
+        Set<String> expiredSessionIds = candidates.stream().filter( candidate -> {
 
             if (LOG.isDebugEnabled())
                 LOG.debug( "Checking expiry for candidate {}", candidate );
@@ -184,7 +213,49 @@ public Set<String> doGetExpired( Set<String> candidates )
             }
             return false;
         } ).collect( Collectors.toSet() );
+
+        if (isScavengeZombies())
+        {
+            //Now find other sessions in hazelcast that have expired
+            final AtomicReference<Set<String>> reference = new AtomicReference<>();
+            final AtomicReference<Exception> exception = new AtomicReference<>();
+
+            _context.run(()->
+            {
+                try
+                {
+                    Set<String> ids = new HashSet<>();
+                    EntryObject eo = new PredicateBuilder().getEntryObject();
+                    Predicate<?, ?> predicate = eo.get("expiry").greaterThan(0).and(eo.get("expiry").lessEqual(now));
+                    Collection<SessionData> results = sessionDataMap.values(predicate);
+                    if (results != null)
+                    {
+                        for (SessionData sd: results)
+                            ids.add(sd.getId());
+                    }
+                    reference.set(ids);
+                }
+                catch (Exception e)
+                {
+                    exception.set(e);
+                }
+            });
+
+            if (exception.get() != null)
+            {
+                LOG.warn("Error querying for expired sessions {}", exception.get());
+                return expiredSessionIds;
+            }
+
+            if (reference.get() != null)
+            {
+                expiredSessionIds.addAll(reference.get());
+            }
+        }
+
+        return expiredSessionIds;
     }
+
 
     @Override
     public boolean exists( String id )

jetty-hazelcast/src/main/java/org/eclipse/jetty/hazelcast/session/HazelcastSessionDataStoreFactory.java

@@ -54,8 +54,20 @@ public class HazelcastSessionDataStoreFactory
     private HazelcastInstance hazelcastInstance;
 
     private MapConfig mapConfig;
+
+    private boolean scavengeZombies = false;
 
 
+    public boolean isScavengeZombies()
+    {
+        return scavengeZombies;
+    }
+
+    public void setScavengeZombies(boolean scavengeZombies)
+    {
+        this.scavengeZombies = scavengeZombies;
+    }
+
     @Override
     public SessionDataStore getSessionDataStore( SessionHandler handler )
         throws Exception
@@ -122,9 +134,10 @@ public SessionDataStore getSessionDataStore( SessionHandler handler )
             }
         }
         // initialize the map
-        hazelcastSessionDataStore.setSessionDataMap(hazelcastInstance.getMap( mapName ) );
-        hazelcastSessionDataStore.setGracePeriodSec( getGracePeriodSec() );
-        hazelcastSessionDataStore.setSavePeriodSec( getSavePeriodSec() );
+        hazelcastSessionDataStore.setSessionDataMap(hazelcastInstance.getMap( mapName ));
+        hazelcastSessionDataStore.setGracePeriodSec(getGracePeriodSec());
+        hazelcastSessionDataStore.setSavePeriodSec(getSavePeriodSec());
+        hazelcastSessionDataStore.setScavengeZombieSessions(scavengeZombies);
         return hazelcastSessionDataStore;
     }
 

jetty-infinispan/src/main/config/modules/session-store-infinispan-embedded/infinispan-embedded.xml → jetty-home/etc/infinispan-embedded.xml

@@ -2,4 +2,4 @@
    <cache-container default-cache="jetty-sessions">
       <local-cache name="jetty-sessions"/>
    </cache-container>
-</infinispan>
+</infinispan>