RELEASE_NOTES

$Id$

3.41 2019/12/31 (dwd)
    - When a "X-Frontier-Opts: DontCacheErrors" request header is set, 
      abort error responses before they send the zero-length http/1.1
      last chunk.  This prevents proxies from caching the response.

3.40 2018/08/20 (dwd)
    - Ignore any text before the BEGIN line in the private key file.
      This appears to have been done in 3.33 but lost in 3.34.

3.39 2018/01/23 (dwd)
    - When responding with an error condition before headers are sent,
      instead of setting a Last-Modified header to be blank, set it
      to the oldest possible time of midnight on January 1, 1970. 
      With the old behavior, Squid-3 does not clear the cache even
      though Squid-2 does.  With the new behavior, squid-3 always
      clears the cache even when another error sends the same
      Last-Modified time.

3.38 2017/05/24 (dwd)
    - Add log message after DB data transferred including the msecs
      it took to transfer the data.

3.37 2017/03/22 (dwd)
    - Fix bug introduced in 3.36 that caused a null pointer exception in
      in the FilePlugin.

3.36 2017/02/08 (dwd)
    - Support HTTP 1.1 chunked encoding to backend FilePlugin http servers.
      Note that the protocol to the client can now send multiple byte
      arrays per request instead of just one.  An implication is that
      fn-fileget from frontier-client versions older than 2.8.21 will not
      always retrieve all of the data when the backend is an http server
      (although it will when reading directly from files).
    - Support MySQL as a database backend in the SQLPlugin. This required
      changing the Last-Modified handling to use the MySQL table that
      always keeps track of modification times on all tables (it does not
      require an add-on function like oracle does).

3.35 2016/10/09 (dwd)
    - Add new servlet option ExpireEmptyQueriesLikeErrors, default false,
      which when true causes empty queries to be expired at the same shorter
      age as errors (default 5 minutes).  Previously this was always true.
    - When ExpireEmptyQueriesLikeErrors is true, mark the returned payload
      with the shorter max_age so if a cached item age exceeds the time,
      the client will repeat the request with the shorter max age.  This
      can happen because a later refresh with If-Modified-Since when the
      Last-Modified time hasn't changed, the regular long expiration time
      will be returned because the query is not repeated.

3.34 2014/10/29 (dwd)
    - Support host private keys (for digital signatures) that start with
      'BEGIN PRIVATE KEY' in addition to 'BEGIN RSA PRIVATE KEY'.  That
      form has begun to be used at CERN with 4096-bit RSA keys; the
      old form was previously used for 2048-bit keys.
    - Eliminate an infinite loop when there's an '&' in an error message
      as sometimes happens with the FilePlugin.
    - Do one less level of url decoding on the query string put into
      catalina.out logs, because one level is already done and that
      better matches the URL seen by a client using the FilePlugin.

3.33 2013/12/26 (dwd)
    - Added new "forever" time to live, a third cache expiration time
      requested by clients, in addition to "short" and "long" times to
      live.  Added config.properties option ForeverCacheExpireSeconds
      which defaults to a year of seconds.  If a "forever" query returns
      empty results, it is considered to be an error.
    - Reduced the time that empty query results will be cached for
      "short" or "long" time to live queries to a maximum of 5 minutes
      or the "short" time, whichever is smaller.
    - Added a "max_age" value at the end of the response if the http
      header has already been sent with a longer Cache-control: max-age
      than is later determined to be necessary.  The things that can
      cause the http header to be sent are either a keepalive message
      because the request had to wait longer than 5 seconds to read
      from the database, or at least a buffer full of data was already
      sent before an error occurred.  The events that can reduce the
      max age late are some kind of error with the database (which
      used to be handled by only signalling "global" errors to the
      client) or an empty result from a database query.
    - Show a more user friendly error message if the private key file
      cannot be decoded.
    - Added removal of any excess text from before the "BEGIN" line in
      the host certificate.

3.32 2013/10/22 (dwd)
    - Add support to the FilePlugin for reading from http URLs in
      addition to files on the local disk.  If FileBaseDirectory
      begins with http://, this new feature is enabled.  If the http
      query returns a Content-Length header, the response is streamed
      through to the client, otherwise the response is read into a
      memory buffer to determine the length.  Last-Modified headers
      are passed through from the backend http server to the client,
      and If-Modified-Since is passed through from the client to the
      backend http server.
    - Fix bug in the FilePlugin that prevented it from sending the
      Last-Modified header for files.
    - Add queuing and keepalive support to the FilePlugin, similar to
      the SQLPlugin.  The maximum number of simultaneous connections are
      specified in the new config.properties option MaxFileConnections,
      default 5.  The options MaxDbAcquireSeconds and MaxDbExecuteSeconds
      then become relevant for sending keepalive messages while waiting,
      even though it isn't technically a "DB".  When reading files (as
      opposed to http urls) the time to "execute" is instantaneous
      because it just opens the file, so MaxDbExecuteSeconds isn't
      important then, but when reading http urls the execute time is
      significant because the http server can take a while to respond.
    - Improve the accuracy of the active connection count in the log by
      moving the counter decrement to be before the connection release.
    - Remove old unused code that supported reading an "xsd_type" from
      a database table whose name was specified in config.properties
      option XsdTableName, and that executed a plugin based on the name
      read from the database.  Also remove related config.properties
      option UseFdoCache.
    - Remove old unused code related to MonAlisa monitoring, including
      config.properties options MonitorNode and MonitorMillisDelay.

3.31 2013/10/21 (dwd)
    - Fix off-by-one error in FilePlugin that had the effect of
      allowing ".." in the file path.

3.30 2013/06/28 (dwd)
    - Add support for sending digital signatures on responses instead
      of an md5 hash, when the client includes the URL parameter
      "&sec=sig".  Also add support for sending the host certificate
      when the client requests it as type "cert_request".  Requires
      the certificate to be specified in a new parameter 'CertFileName'
      and the key in a new parameter 'KeyFileName'.
    - Change to just print one error message to the log when the
      client drops the connection instead of multiple stack traces.
    - Change to not print a stack trace when there's an error getting
      the last-modified time.
    - Add msecs= on the DB connection acquired message to show how 
      long it took to acquire the connection.
    - Change internal errors to force a restart of the servlet just
      like database close problems do.

3.29 2011/05/17 (dwd)
    - Fixed a bug that corrupts the response when a query for a
      last-modified time timestamp takes more than 5 seconds, causing
      a keepalive to be sent.  The bug results in the client seeing an
      error message:
	  XML parse error 7:mismatched tag at line 7
      Now if the query takes too long the result is discarded, because
      it is too late to insert the Last-Modified header after a
      keepalive is sent.
    - Add the number of active and remaining connections to the DB
      connection acquired and released log messages, respectively.

3.28 2011/03/24 (dwd)
    - Add support for loading locally-accessible files in addition
      to database queries.  Add new FileBaseDirectory property to
      indicate the base directory of the files that the servlet will
      load.  Make the DataSourceName property no longer required; now
      at least one of DataSourceName and FileBaseDirectory are
      required.  Corresponding support in the client will be in
      frontier_client 2.8.1.

3.27 2011/01/20 (dwd)
    - Change the expiration time on the 503 SERVICE_UNAVAILABLE error
      to be the time used for errors (the lesser of 5 minutes and the
      short cache expiration time) instead of the long cache expiration
      time.

3.26 2011/01/12 (dwd)
    - Add new SendStaleIfError property, default true, which will send
      the Cache-control header "stale-if-error" if the client sends in
      Cache-control: max-stale.  This prevents squid from re-using
      expired data when there's a server error (by default it will
      reuse data that's up to a week old).  Setting max-stale on the
      client side doesn't affect squid but it indicates that the
      client is ready to deal with the server errors (frontier_client
      has been doing that since October 2008 but I never enabled the
      server side because I thought there was a problem -- details in
      squid bug #2481).

3.25 2010/12/09 (dwd)
    - Add new MaxThreads property to be the maximum number of threads
      that the servlet will allow to wait for a database connection.
      If a NOT_MODIFIED response can be sent quickly without
      contacting the database, that will still happen but otherwise if
      the database needs to be contacted and there are too many
      threads running the response will be SERVICE_UNAVAILABLE.
      Default 100.
    - Ignore exceptions when writing out the transaction end xml.
      This was causing a MUST NOT HAPPEN HERE exception in cases when
      the connection had already been dropped by the client and
      already been reported as an exception.
    - No longer require the XsdTableName property to be set, as it 
      isn't used by current deployments.

3.24 2010/07/28 (dwd)
    - Clearly identify the license for the package to be the open
      source (BSD) Fermitools license.
    - Support having a slash after the 'Frontier' in the URL in 
      place of the question mark.  A slash is better because 
      by default squid does not cache URLs with question marks
      in them so with slashes a standard squid configuration
      can be used instead of a custom one.
    - Add elapsed time to the DB execute log message

3.23 2010/02/12 (dwd)
    - Fixed race condition in the assignment of thread id numbers.
      This has been happening for a long time but is now more
      important because the thread names are used for cleaning up
      keepalive timer threads.  Also include the servlet name in the
      thread names to ensure no clashes of names between servlets
      (which has not been seen in practice, it is just in case).

3.22 2009/11/25 (dwd)
    - There was a serious flaw with the implementation of sending
      keepalives during DB execute: it was only able to track one at
      a time, and as a result multiple simultaneous connections to the
      DB caused a null pointer exception and soon was unable to
      connect to the DB anymore.  It's now able to track any number of
      simultaneous connections.
    - Added -Xlint:unchecked javac compile option, and cleaned up all
      the warnings it added to unchecked generic types.
    - Made the indentation on keepalive XML elements prettier.

3.21 2009/11/02 (dwd)
    - Changed the parsing for multiple table names.  Now looks for an
      optional table alias followed by a comma to indicate that another
      table name is coming, rather than searching for a specific list of
      keywords.  The keywords were matching portions of some table names
      in CMS.
    - Added support for continuing to send keepalive messages to the
      client while waiting for the database to execute a query.  This
      was needed in ATLAS because it has some queries that can take
      over 10 seconds to execute the first time, until it gets into an
      Oracle cache.  Added config.properties option MaxDbExecuteSeconds
      to control the amount of time allowed, default 10 seconds.  The
      frontier_client's own readtimeoutsecs option (which also defaults
      to 10 seconds) is added to that before it times out.
    - Added an additional config.properties option MaxDbAcquireSeconds
      which is how long to send keepalives while waiting to acquire the
      database connection.  Defaults to 300 seconds which was the value
      previously hardcoded.
    - If a query contains the form "WHERE OWNER=''" (which happened once
      in CMS) then no longer attempt to look up a modification timestamp
      for that owner, just let the query proceed without a timestamp.

3.20 2009/10/02 (dwd)
    - Added a global error to a response where one had inadvertently
      been left out, after a failure acquiring the database.  Global
      errors cause the client to force a refresh to clean the error
      out of the cache, and a case was found after a database overload
      last week where a 'Timed out waiting to acquire the DB' error
      was cached indefinitely because it wasn't marked as a global
      error.

3.19 2009/07/14 (dwd)
    Changes for this release primarily written by David Front.
    - Support queries on multiple tables at once when tracking
      modification times.  This is needed in order for ATLAS to use
      Frontier.  When multiple tables are in a query, the timestamps
      on all the tables are looked up in a single database query and
      the most recent timestamp is cached.
    - Add new config.properties option 'VerbosityLevel'.  When set to
      2 or higher, add source file name and line number to each log
      file line.  When set to 4 or higher, add many debug messages.

3.18 2009/05/21 (dwd)
    - If there's an exception when releaseing the database connection, force
      the servlet to restart by touching WEB-INF/classes/config.properties.
      That's quite ugly, but I couldn't think of a better way to do it.  We
      saw a case where this happened and it ended up hanging the servlet and
      eventually the whole tomcat, so it is important to take of this.
      NOTE: if installed by root but run under a different user id, as
      is often the case when it is installed by rpm, this approach will
      not work!

3.17 2009/02/26 (dwd)
    - Add new configuration variable LastModifiedTableName that
      defines the name of the single table where modification times
      are kept for the whole database, rather than reading them
      from a LAST_MODIFIED_TIMES table in each account.  This is
      now required if ValidateLastModifiedSeconds is set.

3.16 2008/12/19 (dwd)
    - Log 'modified at time' also when the modification time was
      cached
    - Log 'using cached last-modified time of' the account.table
      when a cached modification time was found

3.15 2008/12/17 (dwd)
    - Distinguish in the log when an item has been modified after an
      if-modified-since by noting it as 'modified at time'.
    - When the X-Frontier-Id header is missing, still put in the
      initial log entry as much as is known so we can find out where
      the request is coming from.

3.14 2008/12/04 (dwd)
    - Avoid accumulating threads indefinitely when a database is down,
      by maintaining a lock for getting connections to the database and
      raising an exception immediately if the 5-minute keepalive
      thread had already given up by the time the lock is acquired.
      Previously all the threads would in turn wait 6+ minutes to try
      to open a socket to the database server and would tie up so many
      resources that they began to interfere with connections to other
      databases.
    - Avoid a couple null pointer exceptions that happened when
      required pieces were missing from a request.  Instead, raise
      more helpful exceptions for "no query string" and "no
      X-Frontier-ID".
    - Also avoid a null pointer exception leading to a "MUST NEVER
      HAPPEN HERE" internal error when there's no stack trace
      available in an exception.
    - Remove an extraneous blank in exception log entries that
      sometimes made the log appear as if it had an extra blank line.

3.13 2008/11/05 (dwd)
    - Make sure that a '?' bind variable substitution exists in query
      before looking for colons, because colon is occasionally used in
      queries as part of a table value.

3.12 2008/11/04 (dwd)
    - Add new configuration variable ValidateLastModifiedSeconds which
      defines the number of seconds between querying for modification
      times in a LAST_MODIFIED_TIMES table in the same account as
      each query.  The modified time of each table is cached
      internally for that number of seconds.  The time is used to send
      a Last-Modified header.  Squid in turn uses that value once
      the item expires to send an If-Modified-Since header.  When
      that is received, if the time has not changed send back
      an http '304 NOT MODIFIED' response.  This all combines to
      allow much more frequent object expirations without overloading
      the frontier infrastructure, because most times very little
      data is transferred, only the query and NOT MODIFIED.  This
      requires that every account loaded with this option have
      a LAST_MODIFIED_TIMES table kept up to date.  One way is
      provided to do this in Oracle, using a PL/SQL script called
      setup_modtimes.sql.
    - Use "Cache-Control: max-age" instead of "Expires" headers in
      the response.  Squid handles both and the latter are more
      difficult to calculate and manage.
    - Support bind variables in queries, with a question mark where
      each variable is to be inserted.  The values for the variables
      must be appended to the query, separated by colons (:).  This
      helps performance a bit at least on Oracle, because it is able
      to cache the parsed value of the query and re-use it rather than
      caching many similar queries.
    - Handle global errors more robustly, so they should always be 
      able to be parsed by the client.  Previously some kinds of
      errors would result in invalid xml code in the response.
    - Give more debug information in the log if the command parser
      sees a premature end of command.

3.11 2008/04/14 (dwd)
    - Set the maximum keepalive time (which includes the time it takes
      to allocate a connection to the DB) to 5 minutes

3.10 2008/02/28 (dwd)
    - Set the expiration time for exception messages to 5 minutes.

3.9 2007/10/24 (dwd)
    - When a blob is null, send a null indication rather than empty.
      This is the more correct way to handle null blobs than what was
      put in in release 3.5.

3.8 2007/05/24 (dwd)
    - Back out change from 3.7 because that ends up eliminating the
      Content-Length header from even small responses, causing squid
      to drop the connection after every query instead of keeping
      persistent connections.  Instead, we will rely on squid 2.6's
      "collapsed_forwarding" feature to maximize sharing.  Also fix an
      inadvertent too-early sending of the response header that was
      put in in version 3.5.  Add a log message about the response
      being precommitted whenever the header is committed early,
      resulting in the removal of Content-Length.  This now happens
      only when responses take up more than the default 8K buffer in
      the ServletOutputStream.

3.7 2007/05/01 (dwd)
    - Send response headers as soon as possible to maximize the
      sharing that squid does.

3.6 2007/04/05 (dwd)
    - Get the transaction id (with a -ka suffix) into log messages
      from the keepalive thread

3.5 2007/03/22 (dwd)
    - Cleaned up log messages so those that occur regularly all will
      show timestamp and thread id
    - Greatly improved performance for blobs, for small rows, and for
      zipping
    - Replaced Base64 encode/decode implementation with much simpler/
      cleaner version
    - Eliminate null pointer exception on null blobs
    - Renamed CacheExpireSeconds and CacheExpireHourOfDay to 
      LongCacheExpireSeconds and LongExpireHourOfDay.  Added two new
      parameters ShortCacheExpireSeconds and ShortCacheExpireHourOfDay
      which are the same except apply to all queries that have
      "&ttl=short" (ttl stands for time-to-live).  The default on
      ShortCacheExpireSeconds is 0.
    - Catch all Throwables, not just Exceptions, so that Errors are
      also included and debugged
    - When an Exception or Error occurs pass the function, source file
      number, and line number to the application in addition to the
      name of the throwable
    - The "stop" log message now includes the entire transaction, so
      the number of active threads and the elapsed time are more
      meaningful

3.4 2006/10/26 (dwd)
    - Added support for oracle BLOB (Binary Large OBject) types
    - Added two optional parameters to WEB-INF/classes/config.properties:
	CacheExpireSeconds - number of seconds after queries in which
	  the caching of the query should expire.  Default is 7 days of
	  seconds, 604800.
	CacheExpireHourOfDay - the hour of the day at which all
	  query caching should expire, if less than CacheExpireSeconds.
	  For example, if it is set to "23", all items will expire at
	  11:00 p.m. in the local time of the server.  Default is
	  to use CacheExpireSeconds instead.

3.3 2006/09/05 (dwd)
    - Fixed bug where the keepalive message task wasn't shut down if
      the database connection failed
    - Added a refinement to Luis Ramos' patch of 3.1: don't add the
      parenthesized precision after the NUMBER data type if the
      precision is zero

3.2 2006/07/26 (dwd)
    - Added sending a keepalive message to the client every 5 seconds
      while waiting to acquire a database connection, to avoid client
      timeouts under heavy load
    - Added sending the servlet version to the catalina log and the
      client
    - Reduced non-useful debugging output to the catalina log and
      improved some others

3.1 2006/06/07 (dwd)
    - Added feature to zip the retrieved payload using gzip levels 0-9
      where level 0 is no zipping, level 1 is fastest but least
      compressed, and level 9 is slowest but most compressed
    - Added precision in parentheses to the NUMBER data type.  Requested
      by Luis Ramos of CERN and patch supplied by him
    - Cleaned up some exception message typos

3.0 and earlier releases did not have release notes