[Broker/Proxy] Update default TLS protocols to TLSv1.3,TLSv1.2 (#10598)

Fixes #10335

### Motivation

See #10335. TLSv1.3 support has been introduced in the JDK since [JDK 8 since 8u261](adoptium/temurin-build#1254 (comment)). TLS protocol TLSv1.1 and TLSv1 aren't considered secure.

### Modifications

- change default TLS protocols to TLSv1.3,TLSv1.2
- update tests, docs & config files to reflect the change

bouncy-castle/bcfips-include-test/src/test/java/org/apache/pulsar/client/TlsProducerConsumerBase.java

@@ -69,6 +69,7 @@ protected void internalSetUpForBroker() throws Exception {
         conf.setClusterName(clusterName);
         conf.setTlsRequireTrustedClientCertOnConnect(true);
         Set<String> tlsProtocols = Sets.newConcurrentHashSet();
+        tlsProtocols.add("TLSv1.3");
         tlsProtocols.add("TLSv1.2");
         conf.setTlsProtocols(tlsProtocols);
         conf.setNumExecutorThreadPoolSize(5);

conf/broker.conf

@@ -537,7 +537,7 @@ tlsAllowInsecureConnection=false
 
 # Specify the tls protocols the broker will use to negotiate during TLS handshake
 # (a comma-separated list of protocol names).
-# Examples:- [TLSv1.2, TLSv1.1, TLSv1]
+# Examples:- [TLSv1.3, TLSv1.2]
 tlsProtocols=
 
 # Specify the tls cipher the broker will use to negotiate during TLS Handshake
@@ -602,7 +602,7 @@ brokerClientTlsCiphers=
 
 # Specify the tls protocols the broker will use to negotiate during TLS handshake
 # (a comma-separated list of protocol names).
-# e.g.  [TLSv1.2, TLSv1.1, TLSv1]
+# e.g.  [TLSv1.3, TLSv1.2]
 # used by the internal client to authenticate with Pulsar brokers
 brokerClientTlsProtocols=
 

conf/proxy.conf

@@ -161,7 +161,7 @@ tlsHostnameVerificationEnabled=false
 
 # Specify the tls protocols the broker will use to negotiate during TLS handshake
 # (a comma-separated list of protocol names).
-# Examples:- [TLSv1.2, TLSv1.1, TLSv1]
+# Examples:- [TLSv1.3, TLSv1.2]
 tlsProtocols=
 
 # Specify the tls cipher the broker will use to negotiate during TLS Handshake

conf/standalone.conf

@@ -316,7 +316,7 @@ tlsAllowInsecureConnection=false
 
 # Specify the tls protocols the broker will use to negotiate during TLS handshake
 # (a comma-separated list of protocol names).
-# Examples:- [TLSv1.2, TLSv1.1, TLSv1]
+# Examples:- [TLSv1.3, TLSv1.2]
 tlsProtocols=
 
 # Specify the tls cipher the broker will use to negotiate during TLS Handshake
@@ -381,7 +381,7 @@ brokerClientTlsCiphers=
 
 # Specify the tls protocols the broker will use to negotiate during TLS handshake
 # (a comma-separated list of protocol names).
-# e.g.  [TLSv1.2, TLSv1.1, TLSv1]
+# e.g.  [TLSv1.3, TLSv1.2]
 # used by the internal client to authenticate with Pulsar brokers
 brokerClientTlsProtocols=
 
@@ -686,12 +686,12 @@ managedLedgerPrometheusStatsLatencyRolloverSeconds=60
 # Whether trace managed ledger task execution time
 managedLedgerTraceTaskExecution=true
 
-# If you want to custom bookie ID or use a dynamic network address for the bookie, 
-# you can set this option. 
-# Bookie advertises itself using bookieId rather than 
+# If you want to custom bookie ID or use a dynamic network address for the bookie,
+# you can set this option.
+# Bookie advertises itself using bookieId rather than
 # BookieSocketAddress (hostname:port or IP:port).
 # bookieId is a non empty string that can contain ASCII digits and letters ([a-zA-Z9-0]),
-# colons, dashes, and dots. 
+# colons, dashes, and dots.
 # For more information about bookieId, see http://bookkeeper.apache.org/bps/BP-41-bookieid/.
 # bookieId=
 

deployment/terraform-ansible/templates/broker.conf

@@ -464,7 +464,7 @@ tlsAllowInsecureConnection=false
 
 # Specify the tls protocols the broker will use to negotiate during TLS handshake
 # (a comma-separated list of protocol names).
-# Examples:- [TLSv1.2, TLSv1.1, TLSv1]
+# Examples:- [TLSv1.3, TLSv1.2]
 tlsProtocols=
 
 # Specify the tls cipher the broker will use to negotiate during TLS Handshake
@@ -529,7 +529,7 @@ brokerClientTlsCiphers=
 
 # Specify the tls protocols the broker will use to negotiate during TLS handshake
 # (a comma-separated list of protocol names).
-# e.g.  [TLSv1.2, TLSv1.1, TLSv1]
+# e.g.  [TLSv1.3, TLSv1.2]
 # used by the internal client to authenticate with Pulsar brokers
 brokerClientTlsProtocols=
 

deployment/terraform-ansible/templates/proxy.conf

@@ -155,7 +155,7 @@ tlsHostnameVerificationEnabled=false
 
 # Specify the tls protocols the broker will use to negotiate during TLS handshake
 # (a comma-separated list of protocol names).
-# Examples:- [TLSv1.2, TLSv1.1, TLSv1]
+# Examples:- [TLSv1.3, TLSv1.2]
 tlsProtocols=
 
 # Specify the tls cipher the broker will use to negotiate during TLS Handshake

pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java

@@ -1020,7 +1020,7 @@ public class ServiceConfiguration implements PulsarConfiguration {
     @FieldContext(
         category = CATEGORY_TLS,
         doc = "Specify the tls protocols the broker will use to negotiate during TLS Handshake.\n\n"
-            + "Example:- [TLSv1.2, TLSv1.1, TLSv1]"
+            + "Example:- [TLSv1.3, TLSv1.2]"
     )
     private Set<String> tlsProtocols = Sets.newTreeSet();
     @FieldContext(
@@ -2149,7 +2149,7 @@ public class ServiceConfiguration implements PulsarConfiguration {
             category = CATEGORY_KEYSTORE_TLS,
             doc = "Specify the tls protocols the broker will use to negotiate during TLS handshake"
                   + " (a comma-separated list of protocol names).\n\n"
-                  + "Examples:- [TLSv1.2, TLSv1.1, TLSv1] \n"
+                  + "Examples:- [TLSv1.3, TLSv1.2] \n"
                   + " used by the internal client to authenticate with Pulsar brokers"
     )
     private Set<String> brokerClientTlsProtocols = Sets.newTreeSet();

pulsar-broker/src/test/java/org/apache/pulsar/client/api/TlsProducerConsumerBase.java

@@ -68,6 +68,7 @@ protected void internalSetUpForBroker() {
         conf.setClusterName(clusterName);
         conf.setTlsRequireTrustedClientCertOnConnect(true);
         Set<String> tlsProtocols = Sets.newConcurrentHashSet();
+        tlsProtocols.add("TLSv1.3");
         tlsProtocols.add("TLSv1.2");
         conf.setTlsProtocols(tlsProtocols);
         conf.setNumExecutorThreadPoolSize(5);

pulsar-broker/src/test/java/org/apache/pulsar/client/impl/AdminApiKeyStoreTlsAuthTest.java

@@ -101,6 +101,7 @@ public void setup() throws Exception {
 
         conf.setClusterName(clusterName);
         conf.setTlsRequireTrustedClientCertOnConnect(true);
+        tlsProtocols.add("TLSv1.3");
         tlsProtocols.add("TLSv1.2");
         conf.setTlsProtocols(tlsProtocols);
 

pulsar-broker/src/test/java/org/apache/pulsar/client/impl/KeyStoreTlsProducerConsumerTestWithAuthTest.java

@@ -119,6 +119,7 @@ protected void internalSetUpForClient(boolean addCertificates, String lookupUrl)
         }
 
         Set<String> tlsProtocols = Sets.newConcurrentHashSet();
+        tlsProtocols.add("TLSv1.3");
         tlsProtocols.add("TLSv1.2");
 
         ClientBuilder clientBuilder = PulsarClient.builder().serviceUrl(lookupUrl)

pulsar-broker/src/test/java/org/apache/pulsar/client/impl/KeyStoreTlsProducerConsumerTestWithoutAuthTest.java

@@ -98,6 +98,7 @@ protected void internalSetUpForBroker() {
 
         conf.setClusterName(clusterName);
         conf.setTlsRequireTrustedClientCertOnConnect(true);
+        tlsProtocols.add("TLSv1.3");
         tlsProtocols.add("TLSv1.2");
         conf.setTlsProtocols(tlsProtocols);
         conf.setNumExecutorThreadPoolSize(5);

pulsar-broker/src/test/java/org/apache/pulsar/client/impl/KeyStoreTlsTest.java

@@ -20,6 +20,7 @@
 
 import static org.apache.pulsar.common.util.SecurityUtility.getProvider;
 import java.security.Provider;
+import java.util.Collections;
 import org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext;
 import org.apache.pulsar.common.util.keystoretls.SSLContextValidatorEngine;
 import org.testng.annotations.Test;
@@ -71,7 +72,8 @@ public void testValidate() throws Exception {
                 CLIENT_TRUSTSTORE_PW,
                 false,
                 null,
-                null);
+                // set client's protocol to TLSv1.2 since SSLContextValidatorEngine.validate doesn't handle TLSv1.3
+                Collections.singleton("TLSv1.2"));
         clientSSLContext.createSSLContext();
 
         SSLContextValidatorEngine.validate(clientSSLContext::createSSLEngine, serverSSLContext::createSSLEngine);

pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/PulsarAdminBuilder.java

@@ -221,7 +221,7 @@ PulsarAdminBuilder authentication(String authPluginClassName, Map<String, String
     /**
      * The SSL protocol used to generate the SSLContext.
      * Default setting is TLS, which is fine for most cases.
-     * Allowed values in recent JVMs are TLS, TLSv1.1 and TLSv1.2. SSL, SSLv2.
+     * Allowed values in recent JVMs are TLS, TLSv1.3, TLSv1.2 and TLSv1.1.
      *
      * @param tlsProtocols
      */

pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ClientBuilder.java

@@ -358,7 +358,7 @@ ClientBuilder authentication(String authPluginClassName, Map<String, String> aut
     /**
      * The SSL protocol used to generate the SSLContext.
      * Default setting is TLS, which is fine for most cases.
-     * Allowed values in recent JVMs are TLS, TLSv1.1 and TLSv1.2. SSL, SSLv2.
+     * Allowed values in recent JVMs are TLS, TLSv1.3, TLSv1.2 and TLSv1.1.
      *
      * @param tlsProtocols
      * @return the client builder instance

pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java

@@ -82,7 +82,6 @@ public class SecurityUtility {
     // also used to get Factories. e.g. CertificateFactory.getInstance("X.509", "BCFIPS")
     public static final String BC_FIPS = "BCFIPS";
     public static final String BC = "BC";
-    private static final String SSLCONTEXT_ALGORITHM = "TLSv1.2";
 
     public static boolean isBCFIPS() {
         return BC_PROVIDER.getClass().getCanonicalName().equals(BC_FIPS_PROVIDER_CLASS);

pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java

@@ -49,7 +49,7 @@
 public class KeyStoreSSLContext {
     public static final String DEFAULT_KEYSTORE_TYPE = "JKS";
     public static final String DEFAULT_SSL_PROTOCOL = "TLS";
-    public static final String DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.1,TLSv1";
+    public static final String DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.3,TLSv1.2";
     public static final String DEFAULT_SSL_KEYMANGER_ALGORITHM = KeyManagerFactory.getDefaultAlgorithm();
     public static final String DEFAULT_SSL_TRUSTMANAGER_ALGORITHM = TrustManagerFactory.getDefaultAlgorithm();
 

pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/SSLContextValidatorEngine.java

@@ -20,6 +20,7 @@
 
 import static javax.net.ssl.SSLEngineResult.HandshakeStatus.FINISHED;
 import java.nio.ByteBuffer;
+import java.util.Arrays;
 import javax.net.ssl.SSLEngine;
 import javax.net.ssl.SSLEngineResult;
 import javax.net.ssl.SSLException;
@@ -42,9 +43,16 @@ public interface SSLEngineProvider {
     private ByteBuffer netBuffer;
     private boolean finished = false;
 
+    /**
+     * Validates TLS handshake up to TLSv1.2.
+     * TLSv1.3 has a differences in TLS handshake as described in https://stackoverflow.com/a/62465859
+     */
     public static void validate(SSLEngineProvider clientSslEngineSupplier, SSLEngineProvider serverSslEngineSupplier)
             throws SSLException {
         SSLContextValidatorEngine clientEngine = new SSLContextValidatorEngine(clientSslEngineSupplier);
+        if (Arrays.stream(clientEngine.sslEngine.getEnabledProtocols()).anyMatch(s -> s.equals("TLSv1.3"))) {
+            throw new IllegalStateException("This validator doesn't support TLSv1.3");
+        }
         SSLContextValidatorEngine serverEngine = new SSLContextValidatorEngine(serverSslEngineSupplier);
         try {
             clientEngine.beginHandshake();

pulsar-discovery-service/src/main/java/org/apache/pulsar/discovery/service/server/ServiceConfig.java

@@ -94,7 +94,7 @@ public class ServiceConfig implements PulsarConfiguration {
     // Accept untrusted TLS certificate from client
     private boolean tlsAllowInsecureConnection = false;
     // Specify the tls protocols the broker will use to negotiate during TLS Handshake.
-    // Example:- [TLSv1.2, TLSv1.1, TLSv1]
+    // Example:- [TLSv1.3, TLSv1.2]
     private Set<String> tlsProtocols = Sets.newTreeSet();
     // Specify the tls cipher the broker will use to negotiate during TLS Handshake.
     // Example:- [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]

pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java

@@ -344,7 +344,7 @@ public class ProxyConfiguration implements PulsarConfiguration {
         category = CATEGORY_TLS,
         doc = "Specify the tls protocols the broker will use to negotiate during TLS handshake"
             + " (a comma-separated list of protocol names).\n\n"
-            + "Examples:- [TLSv1.2, TLSv1.1, TLSv1]"
+            + "Examples:- [TLSv1.3, TLSv1.2]"
     )
     private Set<String> tlsProtocols = Sets.newTreeSet();
     @FieldContext(
@@ -454,7 +454,7 @@ public class ProxyConfiguration implements PulsarConfiguration {
             category = CATEGORY_KEYSTORE_TLS,
             doc = "Specify the tls protocols the broker will use to negotiate during TLS handshake"
                   + " (a comma-separated list of protocol names).\n\n"
-                  + "Examples:- [TLSv1.2, TLSv1.1, TLSv1] \n"
+                  + "Examples:- [TLSv1.3, TLSv1.2] \n"
                   + " used by the Pulsar proxy to authenticate with Pulsar brokers"
     )
     private Set<String> brokerClientTlsProtocols = Sets.newTreeSet();

pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/ProxyWithAuthorizationTest.java

@@ -93,9 +93,8 @@ public Object[][] protocolsCiphersProviderCodecProvider() {
         // Test explicitly specifying protocols defaults
         Set<String> ciphers_2 = Sets.newTreeSet();
         Set<String> protocols_2 = Sets.newTreeSet();
+        protocols_2.add("TLSv1.3");
         protocols_2.add("TLSv1.2");
-        protocols_2.add("TLSv1.1");
-        protocols_2.add("TLSv1");
 
         // Test for invalid ciphers
         Set<String> ciphers_3 = Sets.newTreeSet();

site2/docs/reference-configuration.md

@@ -203,7 +203,7 @@ brokerServiceCompactionThresholdInBytes|If the estimated backlog size is greater
 |tlsKeyFilePath|  Path for the TLS private key file ||
 |tlsTrustCertsFilePath| Path for the trusted TLS certificate file. This cert is used to verify that any certs presented by connecting clients are signed by a certificate authority. If this verification fails, then the certs are untrusted and the connections are dropped. ||
 |tlsAllowInsecureConnection| Accept untrusted TLS certificate from client. If it is set to `true`, a client with a cert which cannot be verified with the 'tlsTrustCertsFilePath' cert will be allowed to connect to the server, though the cert will not be used for client authentication. |false|
-|tlsProtocols|Specify the tls protocols the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- ```TLSv1.2```, ```TLSv1.1```, ```TLSv1``` ||
+|tlsProtocols|Specify the tls protocols the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- ```TLSv1.3```, ```TLSv1.2``` ||
 |tlsCiphers|Specify the tls cipher the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- ```TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256```||
 |tlsEnabledWithKeyStore| Enable TLS with KeyStore type configuration in broker |false|
 |tlsProvider| TLS Provider for KeyStore type ||
@@ -216,7 +216,7 @@ brokerServiceCompactionThresholdInBytes|If the estimated backlog size is greater
 |brokerClientTlsTrustStore| TLS TrustStore path for internal client, used by the internal client to authenticate with Pulsar brokers ||
 |brokerClientTlsTrustStorePassword| TLS TrustStore password for internal client, used by the internal client to authenticate with Pulsar brokers ||
 |brokerClientTlsCiphers| Specify the tls cipher the internal client will use to negotiate during TLS Handshake. (a comma-separated list of ciphers) e.g.  [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]||
-|brokerClientTlsProtocols|Specify the tls protocols the broker will use to negotiate during TLS handshake. (a comma-separated list of protocol names). e.g.  [TLSv1.2, TLSv1.1, TLSv1] ||
+|brokerClientTlsProtocols|Specify the tls protocols the broker will use to negotiate during TLS handshake. (a comma-separated list of protocol names). e.g.  `TLSv1.3`, `TLSv1.2` ||
 |ttlDurationDefaultInSeconds|The default Time to Live (TTL) for namespaces if the TTL is not configured at namespace policies. When the value is set to `0`, TTL is disabled. By default, TTL is disabled. |0|
 |tokenSecretKey| Configure the secret key to be used to validate auth tokens. The key can be specified like: `tokenSecretKey=data:;base64,xxxxxxxxx` or `tokenSecretKey=file:///my/secret.key`.  Note: key file must be DER-encoded.||
 |tokenPublicKey| Configure the public key to be used to validate auth tokens. The key can be specified like: `tokenPublicKey=data:;base64,xxxxxxxxx` or `tokenPublicKey=file:///my/secret.key`.  Note: key file must be DER-encoded.||
@@ -741,7 +741,7 @@ The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) can be config
 |tlsTrustCertsFilePath| Path for the trusted TLS certificate pem file ||
 |tlsHostnameVerificationEnabled|  Whether the hostname is validated when the proxy creates a TLS connection with brokers  |false|
 |tlsRequireTrustedClientCertOnConnect|  Whether client certificates are required for TLS. Connections are rejected if the client certificate isn’t trusted. |false|
-|tlsProtocols|Specify the tls protocols the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- ```TLSv1.2```, ```TLSv1.1```, ```TLSv1``` ||
+|tlsProtocols|Specify the tls protocols the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- ```TLSv1.3```, ```TLSv1.2``` ||
 |tlsCiphers|Specify the tls cipher the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- ```TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256```||
 | httpReverseProxyConfigs | HTTP directs to redirect to non-pulsar services | |
 | httpOutputBufferSize | HTTP output buffer size. The amount of data that will be buffered for HTTP requests before it is flushed to the channel. A larger buffer size may result in higher HTTP throughput though it may take longer for the client to see data. If using HTTP streaming via the reverse proxy, this should be set to the minimum value (1) so that clients see the data as soon as possible. | 32768 |

site2/docs/security-tls-keystore.md

@@ -139,7 +139,7 @@ Optional settings that may worth consider:
     algorithm used to negotiate the security settings for a network connection using TLS network protocol. By default,
     it is null. [OpenSSL Ciphers](https://www.openssl.org/docs/man1.0.2/apps/ciphers.html)
     [JDK Ciphers](http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#ciphersuites)
-3. tlsProtocols=[TLSv1.2,TLSv1.1,TLSv1] (list out the TLS protocols that you are going to accept from clients).
+3. tlsProtocols=[TLSv1.3,TLSv1.2] (list out the TLS protocols that you are going to accept from clients).
     By default, it is not set.
 
 ### Configuring Clients

site2/docs/security-tls-transport.md

@@ -137,15 +137,15 @@ You can configure the broker (and proxy) to require specific TLS protocol versio
 Both the TLS protocol versions and cipher properties can take multiple values, separated by commas. The possible values for protocol version and ciphers depend on the TLS provider that you are using. Pulsar uses OpenSSL if the OpenSSL is available, but if the OpenSSL is not available, Pulsar defaults back to the JDK implementation.
 
 ```properties
-tlsProtocols=TLSv1.2,TLSv1.1
+tlsProtocols=TLSv1.3,TLSv1.2
 tlsCiphers=TLS_DH_RSA_WITH_AES_256_GCM_SHA384,TLS_DH_RSA_WITH_AES_256_CBC_SHA
 ```
 
-OpenSSL currently supports ```SSL2```, ```SSL3```, ```TLSv1```, ```TLSv1.1``` and ```TLSv1.2``` for the protocol version. You can acquire a list of supported cipher from the openssl ciphers command, i.e. ```openssl ciphers -tls_v2```.
+OpenSSL currently supports ```TLSv1.1```, ```TLSv1.2``` and ```TLSv1.3``` for the protocol version. You can acquire a list of supported cipher from the openssl ciphers command, i.e. ```openssl ciphers -tls1_3```.
 
-For JDK 8, you can obtain a list of supported values from the documentation:
-- [TLS protocol](https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#SSLContext)
-- [Ciphers](https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#ciphersuites)
+For JDK 11, you can obtain a list of supported values from the documentation:
+- [TLS protocol](https://docs.oracle.com/en/java/javase/11/security/oracle-providers.html#GUID-7093246A-31A3-4304-AC5F-5FB6400405E2__SUNJSSEPROVIDERPROTOCOLPARAMETERS-BBF75009)
+- [Ciphers](https://docs.oracle.com/en/java/javase/11/security/oracle-providers.html#GUID-7093246A-31A3-4304-AC5F-5FB6400405E2__SUNJSSE_CIPHER_SUITES)
 
 ## Proxy Configuration