changelog.txt

CYLC METASCHEDULER CHANGE LOG, INITIALIZED after version 3.0.3.

Non-trivial changes of relevance to users are listed. All changes are
likely to involve unlisted minor updates and corrections to cylc 
documentation (User Guide, command line help, and GUI help).

Version number X.Y.Z incrementing:
  X - major changes
  Y - significant changes
  Z - minor changes and bug fixes

3.0.4
 + More documentation of error detection in task scripts.
 + 'cylc admin create-cdb' now uses its own test suite.

3.0.5
 + Clarified graph-based control GUI help.

3.1.0
 + Allow use of lockserver in simulation mode.
 + New suite compare/diff functionality added.

3.2.0
 + Big speed up for large suites due to some optimization of the
   scheduling algorithm.
 + Monospaced fonts in GUI output capture windows.

3.3.0
 + Updated the terminal monitor to display held (paused) tasks properly.
 + Entirely eliminated the need for explicit task wrapping:
   - moved 'task started' and 'task succeeded' messages, and automatic
   'task failed' messaging, to the generated task job script.
   - task scripts now only have to call 'task succeeded/failed', and
     only for tasks whose initiating processes detach and exit early.
   - this means the entire task job script is now seen by cylc, not just 
     the (formerly) wrapped command line.
   - task-specific pre-command scripting is no longer needed - instead
     the task command string can contain multiline scripting.
 + New variable $CYLC_SUITE_ENVIRONMENT in the task execution environment
   for use in custom wrappers for tasks with manual completion messaging.
 + Added and documented new example suite to illustrate custom task wrapping.

3.3.1
 + Fixed a typo in writing out the new $CYLC_SUITE_ENVIRONMENT to job
 scripts, which could break new-style custom wrappers for detaching tasks. 

3.3.2
 + Fixed a bug introduced at 3.2.0 that prevented failed tasks from 
   becoming 'unfailed' after retriggering them.

3.3.3
 + Full path configuration for suite state dump and log directories.
 + Fixed recently introduced bug affecting loadleveler job submission: an
   executable statement appeared before the directives in task job scripts.

3.3.4
 + Autogenerated HTML version of the User Guide added.
 + Fixed a bug introduced at 3.2.0 affecting suite warm starts:
   - a misspelled function call in bin/_run caused suite warm starts to fail.
 + Fixed a bug introduced at 3.3.0 affecting detached tasks in simulation mode:
   -  Since eliminating 'cylc wrap' tasks that handle their own
      completion messaging in real mode need automatic completion in sim mode.  
 + Documented local installation of Pyro, Graphviz, and Pygraphviz.
 + Documented how to upgrade to a new version of cylc.

 3.3.5
 + Fix a bug that affected task submission and execution timeouts:
   - if a timeout was set, the hook script was called even if the event
     did occur in time - which should cancel the timeout.
 + More detailed documentation of complete installation by non-root users.

 3.3.6
 + Documentation of graphviz installation without QT.
 + Improved detection of errors in suite.rc graph hours headings.
 + Improved detection of erroneous suite registrations on the command line.
 + Fixed a bug in 'cylc log' command task filtering.
 + At installation, unregister example suites from admin user's private db.
 + Task event hook script argument list changed. Cylc now calls:
     <hook script> EVENT SUITE TASK CTIME MESSAGE
   The new SUITE argument is particularly needed if running multiple suites.
   WARNING: this change will break existing task event hook scripts.
 + Added a task event hook script to 'cylc util' that sends email alerts.

3.3.7
 + Disabled the cylc lockserver by default.
 + Altered environment.sh to work with 'set -u'.
 + Better task error trapping: in addition to trapping processes that
 abort with non-zero exit status (ERR signal) cylc now also registers
 task failure after unexpected exit with normal exit status (EXIT
 signal), process termination by the TERM signal (e.g. by 'kill PID' or
 'llcancel JOBID'), and termination (e.g. by the batch scheduler) due
 to exceeding some system time limit (XCPU signal).

 
3.4.0 (4 August 2011)

  + Fixed bug introduced in 3.3.7: trapping of EXIT signal must be 
  turned off at the end of job scripts for detaching tasks (which
  handle their own completion messaging).

  + Task event hook script argument list changed again. It is now:
     <hook script> EVENT SUITE TASKID MESSAGE
  where TASKID works with both cycling and non-cycling (asynchronous)
  tasks. WARNING: this will break existing task event hook scripts.
  
  + Also, the EVENT argument for job submission failures is now
  "submission_failed" rather than "submit_failed", to be more consistent
  with the associated suite.rc item.

  + Disabled run time graphing by default, in suite.rc [visualization]. 
  It is mainly intended for use in cylc development and debugging.
  WARNING: this breaks existing suite.rc run time graph settings.

  + Initial and final cycles for graph plotting can now be provided in 
  the suite.rc file, and can be overridden on the command line.

  + Initial and final cycle times can now be provided in the suite.rc
  file, and optionally overridden on the command line.

  + Fixed a lockserver bug, introduced in March 2011, that only affected
  owned tasks.

  + Added asynchronous tasks (no cycle time):
    - one-off asynchronous tasks for any non-cycling jobs.
    - repeating asynchronous tasks for processing satellite passes.

  + Made the User Guide accessible from GUI help menus.

3.4.0-a (bug fix release 5 Aug 2011)

  + Fixed a bug affecting detaching tasks in simulation mode only - job
  script EXIT signal trapping was not being cleared before exiting. 
  + Fixed a bug in the FFTask example suite (which is not actually
  referred to in the User Guide yet) - it had not been upgraded for
  compatibility with recent cylc versions.

3.5.0

 + Fixed a bug, introduced in 3.4.0, in the lockserver client program
  for manual lock management.

 + The cylc code base is now a Python library and the environment.sh
 script is no longer needed: you just need the cylc command in $PATH.

 + Task generators: definition of multiple similar tasks all at once.
 Two new examples suites illustrate this new feature.

 3.5.1  bug fix release 15 Aug 2011

  + Fixed a Python indentation error introduced in 3.4.0, which 
  affected the list of valid hours for tasks that appear in multiple 
  suite.rc graph strings (i.e. tasks with different triggers at 
  different cycle times). 

________________________________________________________________________
4.0.0-rc1

+ WARNING cylc-4.0 IS NOT ENTIRELY BACKWARD COMPATIBLE with 3.x
due to significant changes to suite.rc structure and item names. 
 Suggested suite upgrade strategy: 
    1/ read this changelog, then
    2/ use 'cylc validate' to see which items are no longer valid ...
    3/ ... then consult the suite.rc reference, and 
    4/ examine the cylc example suites if necessary
Future changes will be more incremental and we will ensure backward
compatibility or provide a clear (possibly automated) upgrade path.

+ UTC Mode: optionally run a suite on UTC time even if the suite
 host's system clock is not in UTC mode.

+ Fixed a suite graph plotting bug: if the graph initial hour was 
 greater than the section hour for single hour graph sections, the
 section would be skipped even if the graph range encompassed the
 section, due to miss-handling of the 24-hour boundary.

+ Fixed a subtle bug in environment variable overriding: if a variable
 was defined twice, the second definition would override the first as
 it should, but it would retain the position (in the list of defined
 variables) of the first.

+ General parenthesized conditional triggers
  - WARNING: This required a change in intercycle dependency notation
    from ROUND to SQUARE brackets (to avoid clashing with parentheses 
    in conditional expressions):
       foo(T-6) becomes foo[T-6]

+ Embedded list-generating Python expressions can now be used to
 generate related dependencies in the graph (e.g. for the internal
 dependencies of an ensemble); and lists of task names or
 list-generating Python expressions in task config section headings,
 to define multiple tasks at once.

+ Configurable command template for at_now, background and loadleveler.
 Some example use cases might be:
  - a user may want to use batch instead of at now.
  - a user may want to use an option of llsubmit.
  - a user may want to use nice with a background submit.

+ Configurable ssh command template.
  - no longer require scp - use stdin to transfer job file.
  - -oBatchMode=yes prevents any hang ups.
  - (but ssh command is bit more complex.)

+ sudo job submission (for other owner on the suite host) removed. Now
use ssh as for remote hosts.

+ New default directory structure.
  - $HOME/cylc-run/$CYLC_SUITE_REG_NAME/{log/suite,log/job,state}

+ Task logs now go to the proper directory tree on remote hosts (and remote 
directories are created on the fly if necessary).

+ Use microseconds since epoch instead of random string for tagging a job task.
   - this allows "ls" to list things in the right order.
   - stdout and stderr files are now named after their task job scripts.

+ Fixed a bug preventing use of :fail triggers with asynchronous tasks.

 + New "suicide" triggers: graph = "A => !B" means B will take itself out of 
 the suite (after spawning, if it is a cycling task) if A succeeds. This can
 be used for automated recovery from task failures using sub-trees of tasks
 that tag along with a model (say) waiting for it to fail, but which
 automatically remove themselves if the model does not fail.

+ MAJOR CHANGE TO SUITE.RC STRUCTURE: 
  * better organization under new top level sections:
  [cylc] - non task-related suite config. 
  [scheduling] - determines when tasks are ready to run
  [runtime] - determines what runs, and how and where, when a task is ready
  * under [runtime] family and task properties are defined by a
  "namespace inheritance hierarchy" that allows common configuration of
  related tasks without repetition. 
WARNING: there are a number of changes associated with this item that
will break existing suite.rc files.

+ Task families are now defined by the runtime namespace hierarchy.
  * when used in the graph, they are suite definition shorthand only
   (no pseudo-task representing the family appears in the task pool). 
  * can be individually expanded or collapsed for graph plotting, and in
   the graph-based suite control GUI.

+ 'command =' config item changed to 'command scripting =' for consistency
  with 'pre-' and 'post-command scripting', and because it has long since
  ceased to be a single command.

+ New command 'cylc get-config' to retrieve any suite configuration data.

+ gcylc entry labels made consistent with command help, and gcylc Help
 buttons now call command help on the fly.

+ MAJOR CHANGE TO SUITE REGISTRATION SYNTAX:
   - changed from GROUP:NAME to an arbitrarily nested path (foo,
     foo.bar, foo.bar.baz, etc.)  Delimiter changed from colon to dot to
     avoid problems with use of suite identity in PATH variables.

+ suite identity variables changed for new registration names:
    CYLC_SUITE_REG_NAME    # foo.bar.baz
    CYLC_SUITE_REG_PATH    # foo/bar/baz
    CYLC_SUITE_DEF_PATH    # suite definition directory
+ task identity variables made consistent with suite identity variables:
    CYLC_TASK_ID
    CYLC_TASK_NAME
    CYLC_TASK_CYCLE_TIME
    CYLC_TASK_NAMESPACE_HIERARCHY  # <-- new

+ 'cylc admin test-suite' now detects failures properly.

+ Updated the User Guide for all the above.

+ suite hold is now implemented by holding all waiting tasks instead of
just not running them - this is more transparent to the user.

+ RUNAHEAD LIMIT:
    - now implemented by putting tasks into a special 'runahead' held
      state, rather than just delaying spawning. This is more
      transparent to the user.
    - now applies if (Ttask-Toldest) >= limit (was '>'). This generally 
    makes the limit shorter by one cycle, but is more intuitive.

+ Default start and stop times, if defined, are now displayed in the 
 GUI start panel (and the same for suite graph plotting).

+ Inlined editing of suite.rc files that use include-files: the 
 temporarily inlined file is now the live file during the editing
 session - i.e. changes take effect, for graphing or running the
 suite etc., whenever the inlined file is saved rather than on
 exiting the edit session.

________________________________________________________________________
 4.0.0-rc2

+ Bug fix: suite validation output aborted if an error occurred 
during task proxy instantiation.

+ 'cylc validate' now returns exit status 1 if validation fails.

+ restored full syntax highlighting to the User Guide.

+ corrected 'cylc email-alert' command help.

+ completed and documented the new task work and share directories

________________________________________________________________________
 4.0.0-rc3

+ documented a subtlety of namespace environment variable override.

+ vertical task name labelling in the GUI "LED light panel".

+ interactive family grouping controls for the graph viewer.

________________________________________________________________________
4.0.0-rc4

+ bug fix 'cylc graph' - it was failing if no initial or final cycle
times were provided by command line or in the suite.rc file.

+ better error messages for any failed attempt to create a directory,
if the user gives an illegal $TMPDIR or suite copy target directory.

+ JOB-SUBMISSION:
   - added support for PBS and SGE (qsub)
   - provided a simple mechanism for user-defined job submission methods
   - removed NIWA-specific job submission methods from the source tree

+ fixed a recently introduced fatal bug in the 'cylc [admin] test-db'
script (it referred to an example suite that has since been renamed).

________________________________________________________________________
4.1.0

+ A new "initial scripting" suite.rc item, for scripting inserted into
the very top of task job scripts. This can be used for environment setup
prior to configuring cylc access, e.g. for getting remote tasks to
source your login scripts (ssh remote command invocation doesn't do this).

+ Added support for the Jinja2 template engine, with documentation and
example suites. This allows powerful programmatic generation of suite
definitions.

+ Replaced the 'inline' command with 'view', which can show a temporary
Jinja2-processed suite in addition to inlining include-files.

+ Recently introduced bugs fixed: 
   - a background thread hung if the 'cylc graph' window was killed via
     the window manager.
   - a critical background thread died in the graph suite control GUI if
     the graph key was toggled on.

+ Unlock the suite database after any error other than "already locked".

+ Allow quotes around include-file filenames.

+ Bug fix: GUI suite restart: if an initial cycle was defined it was
passed through to the command line even if greyed out for a restart.

+ The graph command now handles suite parsing errors gracefully - if you
save an invalid suite.rc file while graphing, the graph window will
retry until the error is corrected, rather than aborting immediately.

+ Documented how to handle line continuation in graph strings.

+ Removed the ability to parse "Python:" expressions in the suite.rc
file, and to ignore items starting with underscores for use in configobj
string interpolation.  These have been made redundant by the more
powerful and general Jinja2 template processor (above).

+ A new --stdout option for 'cylc view' prints suite definitions to
stdout.

+ Fixed a bug in the unregister command - unregistering a group 'foo'
would result in group 'foo-bar' being unregistered too (this only
affected hyphenated names as '-' is now legal in suite names but is not
a regular expression "word character", and name matching was done up to 
a word boundary).

+ Simulation mode now works for tasks with internal outputs, and for
remotely hosted tasks without having to comment out some remote config.

+ Added trigger time offset and valid cycles to retrieved task state info.

+ Changed suite.rc special placeholder variables to avoid confusion with
environment variables:
   $(CYLC_TASK_CYCLE_TIME) => <CYLC_TASK_CYCLE_TIME>
   $(TASK) => <TASK>
   $(NAMESPACE) => <NAMESPACE>
   $(ASYNCID) => <ASYNCID>
See the updated User Guide, Section 8.7 "Special Placeholder Variables".

+ Many minor improvements to the User Guide.

________________________________________________________________________
4.1.1

+ Fixed a bug affecting home directory replacement in the share and work
directory paths for remote tasks that do not explicity override the
default share and work paths.

+ Defining the 'cylc directory' is now optional for remote tasks, for
sites that install cylc in the default $PATH.

+ A new variable CYLC_TASK_LOG_ROOT records the task stdout/stderr file
path prefix (everything prior to .out and .err) to allow use of the same
for related files (e.g. stats files).

+ Directives (e.g. for qsub or llsubmit) are now ignored unless the
chosen job submission method actually uses them. Previously they would
be written to the task job script regardless of the job submission
method (which was harmless - they are shell comments - but unnecessary).

+ Fixed two bugs affecting the runahead limit:
1/ runahead-held tasks were recorded in the final suite state dump, just
prior to shutdown, as normal "held" tasks and so were erroneously
released on restarting the suiteing the suiteing the suite.
2/ ability to change the runahead limit at run time was broken.

+ Updated the gcylc GUI Quick Guide help window.

+ Access to the cylc internet homepage from the gcylc Help menu.

+ New 'cylc browse' command for quick access to cylc documentation.

+ Validate command: 1/ bug fix: error messages during task proxy
instantiation were not being printed properly when multiple errors
occured. 2/ detect erroneous use of synchronous special tasks in
asynchronous graphs.

________________________________________________________________________
4.2.0

+ Cross-version suite compatibility. Put "#!cylc-x.y.z" as the first 
line of your suite.rc file, and cylc-x.y.z will be used to access the 
suite regardless of which cylc version is initially invoked.

+ Remote task stdout and stderr logs can now be accessed by gcylc, 
by the same passwordless-ssh based method used to submit remote tasks.

+ Runtime namespace inheritance hierarchy graphing.

+ gcylc now spawns ssh commands to retrieve remote task output via the
shell, so that task owner name can be defined by environment variable.

+ Task event handling:
  - restored the 'warning' event hook.
  - made timeout handling consistent with the other events:
    'submission_timeout' and/or 'execution_timeout' should now be 
    specified instead of just 'timeout'.

+ Added suite event handling, with one event: shutdown (have your suite
send an email before it shuts itself after completing the final cycle).
WARNING, this required changing the name of the existing ready-made
event handler:
   cylc [util] email-alert
has been replaced by:
   cylc [hook] email-task   (for task events)
   cylc [hook] email-suite  (for suite events)

+ Cycling tasks now spawn successors when they enter the submitted 
state, rather than the running state. This is better for external 
queueing systems (jobs get queued sooner) but otherwise makes no
difference.

+ Documented how multiple cycling graph sections combine to make the
complete graph. 

+ Family graph triggers with a cycle time offset are now handled
correctly, e.g. "foo[T-6] => bar" where foo is a task family defined
by the runtime namespace hierarchy.

+ Asynchronous (non-cycling) suites can now be stopped and restarted
mid-run.

+ Clock-triggered tasks can now be triggered manually before their
normal trigger time is up.

+ Improved graph string validation: dangling arrows are now caught;
the bad graph line is printed just before the error message.

+ Illegal namespace environment variable names are now detected; this
catches some beginner suite.rc section nesting errors such as thinking 
that indentation amount allows you to put higher level items such as 
'command scripting' below an environment section.

+ Documented how to omit tasks at runtime, completely or for later
insertion.

+ Graph GUI layout stability has been improved by redisplaying the graph
only if displayed state information (suite task content and task
state/color) changes.  The layout may still jump when a task changes
state/color even if nothing else has changed.

+ PBS job names are now truncated to 15 characters max (this is a
requirement for pre version 11 PBS).

+ Internal queues - the number of active tasks (i.e. those submitted or
running) at any one time can be limited, in one or more queues.

+ New User Guide chapter "Running Suites" which so far documents 
task states and internal queues.

________________________________________________________________________
4.2.1

+ Fixed an error in the housekeeping utility program, affecting
operations with no reference destination defined.

+ Fixed a serious bug affecting manual re-triggering of failed tasks: if
such a task failed again the failure would be logged but the task
state would remain 'running'; and if it succeeded the the previous
failure message would be erroneously identified as an "incomplete
output" resulting in cylc setting the task failed again (introduced
sometime after 3.5.1).

+ Exception handling and sensible error messages in case of suite state
dump corruption.

+ Restored remote host access to the ping, dump, show, and monitor
commands.  The '--host' and '--owner' command line arguments hard been
rendered ineffective by registration name validation assuming localhost.

________________________________________________________________________
4.2.2

+ Implemented start triggers (Issue #27):
     graph = "A:start => B"
B triggers when A starts executing; previously this required use of 
special internal task outputs.

+ Implemented finish trigger shorthand:
     graph = "A:finish => B"
is equivalent to:
     graph = "A | A:fail => B"
B triggers once A is finished, whether by success or failure.

+ Disallowed non-standard argument order in the register command.

+ Bug fix: the register command was aborting without unlocking the 
suite database if the suite.rc file was unreadable (broken since
cross-version compatibility introduced in 4.2.0).

+ Bug fix: runtime inheritance of boolean config items was broken - 
"False" values were inadvertantly caught in by a test designed to avoid
undefined values and empty lists.

+ New standard task execution environment variables:
  $CYLC_SUITE_INITIAL_CYCLE_TIME
  $CYLC_SUITE_FINAL_CYCLE_TIME
This allows tasks to determine if they are in the first or final cycle.

+ Simulation mode now works for any suite without any modification of
the suite.rc file. Previously some config items such as initial
scripting and user environment could break simulation mode by
referencing scripts or variables that are missing or undefined outside
of the suite's normal operating setting (these, along with any other
config items that are not needed in simulation mode are now omitted
from simulation mode job scripts).

+ Duplicate prerequisites are no longer treated as an error. This change
also fixes a recently introduced simulation mode bug: internal outputs
are ignored in simulation mode (because the dummy tasks do not report
internal outputs) so foo:fail, in simulation mode, was treated as
foo:succeeded, resulting in "duplicate prerequisite" errors from
expressions like "foo | foo:fail => bar".

+ Bug fix: Override of $CYLC_DIR and $CYLC_SUITE_DEF_PATH for remotely
hosted tasks used a common data structure instead of a task-specific
one, which resulted in subsequent local tasks getting the overridden
values too.

+ Bug fix: restored the ability to scan, and display port number in
gcylc, your own running secure suites (i.e. suites using secure
passphrases).

+ Bug fix: in dummy tasks (i.e. tasks with no runtime config, which are
defined only by their presence in the graph) '<TASK>'  was being
replaced by "root" because dummy tasks were inheriting the root
namespace after the interpolation of <TASK> in non-dummy namespaces.

+ Validation now detects invalid conditional trigger expressions, such 
as unbalanced parentheses, in the graph string.

+ Restored the ability to submit single tasks that have defined runtime
config but do not currently appear in the suite graph. Task type
(cycling or asynchronous) and validity (valid hours of the day for the
task) are normally defined by the graph. In the absence of that
information we just assume that the cycle time given on the command line
is valid for the task.

+ Added CommsTest simple Pyro client/server programs to demonstrate how
network communication is handled in cylc.

+ Validation now detects possible task name case errors (myFoo vs MyFoo).

+ A "task failed" output is now recorded when a task is manually reset
to the failed state, to allow downstream tasks to trigger off the failure.

+ Tasks now spawn on failure as well as success, if they haven't done so
already. So, for instance, if a waiting task is forced to the failed
state, it will spawn a successor.

________________________________________________________________________
4.5.0

+ Bug fix: disabling the runahead limit now works (but is not advisable!)

+ Bug fix: multiple start-up tasks on the left of a dependency arrow are
now handled correctly. Previously, only one was handled as a start-up
task (in that dependence on it should disappear after the first cycle).

+ Validation now fails use of start-up tasks and normal cycling tasks in
the same conditional trigger expression (the meaning of such a trigger
is undefined beyond the first cycle). 

+ Suite visualization node groups are now automatically defined for
all task families so that styling can be applied to family and member
nodes at once using the family name.

+ Bug fix: "release suite" now does not release tasks to run if they are
on hold because they have passed the suite stop time.

+ Sequential tasks now do not spawn on failure, only success, reversing
a change in 4.2.2 (otherwise restarting a suite with a failed sequential
task would result in the failed task and its newly spawned successor
running in parallel, which is what sequential tasks are designed to
avoid). 

+ A new '--spawn' option for the "reset" command forces a task to spawn
a successor if it has not done so already. This may be useful in the case 
of a failed sequential task that cannot be re-run successfully, to force 
it to carry on into the next cycle (see previous bullet point for why 
this should not be done automatically).

+ An alternative task-to-suite communication mechansim: optional use of
ssh to re-invoke messaging commands on the suite host. This may be
useful at sites in which one or more remote task hosts do not allow
access to network ports for cylc's normal Pyro TCP/IP communication
mechanisim.

+ Time offset syntax in explicit internal output messages has been changed
from <CYLC_TASK_CYCLE_TIME+n> to [T+n]. The new syntax is the same as
for cycle time offsets in the suite graph. Angle brackets are now
reserved for a few "place holder" variables in suite.rc (see User Guide).

+ If multiple retries are configured for a task, the task failure event
hook is now activated only for the final definitive failure.

+ The temporary directory required by some cylc commands is now chosen 
automatically, by default, using Python's tempfile.mkdtemp(), but users
can optionally override by setting CYLC_TMPDIR. In the default case
the temporary directory is also cleaned up automatically after use.

+ GENERALIZED CYCLING: cylc's original NWP-style hours-of-the-day
cycling mechanism, e.g. [[0,6,12,18]], has been replaced with an
extensible mechanism that currently supports hours-of-the-day, and
stepped daily, monthly, and yearly cycling - for climate modeling. 

+ The "cylc cycletime" utility command has been extended to handle
daily, monthly, and yearly cycle time offsets, as well as hours.
WARNING: this change will break any previous computation of cycle time
offsets with this utility command, because extending the functionality
required changing the existing command line options.

+ The "cylc template" filename templating utility has been removed;
templating is now handled by the "cylc cycletime" utility.

+ Flexible specification of cycle times on the command line (in
combination with generalized cycling, above): YYYY[MM[DD[HH[mm[ss]]]]]
(brackets indicate optional digits).

+ Added an emacs mode for syntax highlighting suite.rc files.

+ Task job scripting now prints out suite, task, and host ID info.

+ New graphing option: use node outline color for node labels.

+ The cylc scan command now reports simple space-delimited output,
    "suite owner host port"
for easier parsing.

+ Changes to the cylc ping command:
  1/ "host not found" errors are handled properly,
  2/ a successful ping just exits with success status (no output) ...
  3/ ... unless verbose mode is chosen, the it prints as scan above.

+ Bug fix: the "cylc util housekeeping" script was not correctly
interpolating month MM in the archive destination path.

+ Jinja2 processing leaves extra blank lines in the suite definition
corresponding to lines that originally had only Jinja2 code. This could
result in a non-validating suite if the line preceding the blank one
ended in a line continuation marker. 'cylc view -j' (formerly '-p) was
removing blank lines, purely (it was thought) for display purposes,
which made such non-validating suites appear to be correct. All blank
lines are now removed following Jinja2 processing, to prevent this.

+ Complete remote control of suites has been enabled, with compulsory
use of suite passphrases for authentication.  Accordingly the "use
secure passphrase" suite configuration item has been removed.  A
random passphrase is generated automatically in the suite definition
directory when a suite is registered; it must be distributed to any
remote task host accounts and any remote accounts from which you want 
suite control access.

+ Tasks ready to run at the same time are now be submitted in the order
in which they are first encountered in the suite graph (at least in the
first cycle...).

+ Task retry mechanism upgraded: "command scripting" is now a single
item, not a list, and a new config item "retry delays" lists successive
delays in minutes, if any, after which to resubmit the job with the same
command scripting. When delayed prior to a retry tasks go into a new
"retry delayed" state, coloured pink, and a new task execution
environment variable $CYLC_TASK_TRY_NUMBER is incremented (starting from
one in the original run); this variable can be used to modify task
behavior in each try.

+ A new config item:
    [runtime][[NAME]]dummy task run length = <integer> (seconds)
allows users to change the default dummy task run length. This sets 
the value of a new task execution environment variable 
$CYLC_TASK_DUMMY_RUN_LENGTH, which is used as an argument to the
sleep command in default command scripting. 

+ Line numbers are now printed for Jinja2 syntax errors.

+ Major suite control GUI improvements:
  + The two control GUIs have been merged into a single multi-view GUI
  with a cleaner interface (still evolving)
  + Families can now be expand and collapsed in the text view
  + Family state coloring in the graph text views

+ The suite host environment can now be accessed via Jinja2, e.g.:
   #!Jinja2
   #...
   [[[environment]]]
       MY_HOME_DIR_ON_SUITE_HOST = {{environ['HOME']}}

+ The grep command now supports only cylc-style (not grep-style)
argument order: 'cylc grep SUITE pattern' 

+ Moved the lockserver commands to the 'utility' command category.

+ Removed central suite database functionality, but extended user
database functionality to allow viewing (etc.) of suites registered in
other databases, including suites owned by other users (see the new
'--db=' command option. Copying suites between databases is also 
supported, although currently by the command line interface only.
Accordingly, the default suite database location has been changed to
$HOME/.cylc/DB. 

+ Bug fix "cylc log -r N" (for viewing old rotations)

+ Support for Jinja2 custom filters, by which you can modify Jinja2
variable values with custom Python functions.

+ Cylc version re-invocation (pseudo backward compatibility) now assumes 
parallel cylc installations - you can no longer specify a full path to a
required cylc version at the top of a suite definition.

+ New network connection timeout option for commands that connect to
running suites. Previously, with no timeout, a Ctrl-Z-stopped suite
(or similar) could hold up its port and cause port scans to hang
indefinitely.

+ The "gcylc" command is now an alias for "cylc gui" and it only invokes
the main cylc GUI - i.e. the suite database viewer from which you can
operate on any of your suites. The suite control GUI, which has always
been a separate application anyway, is now invoked by the "gcontrol
commands, which is an alias for "cylc gcontrol".  This was necessitated
by the above remote control changes (the two commands must be handled
differently because gcylc is not suite-connecting but gcontrol is).
This change also resolves the ambiguity of the former "gcylc" name, which 
could refer to either the main db-viewer or a suite control GUI. 

+ Bug fix: explicit outputs now work for one-off asynchronous tasks.

+ (missed above: In gcontrol views - graphical and text based - the task
content of the suite is now determined and updated dynamically by
interrogating the running suite. Previously the task content was only
loaded once, at GUI start up, and thus did not update if the suite was
altered while the GUI was still running. 

________________________________________________________________________
4.5.1

+ Task retry delays now work with clock-triggered tasks.

+ Cleaned up use of Pyro connection timeouts. Prior to 4.5.0 there was
no timeout. At 4.5.0 we set a 1 seconds default, but depending on
circumstances (network, load, etc.) this proved to be too short
(resulting in messaging failures when tasks timed out attempting to
communicate with their suite). So now:
  * The timeout value passed to tasks by their parent suite can be set
    by '[cylc]-> pyro connection timeout' and defaults to no timeout.
  * A value specified on the command line of the run, restart, gcylc,
    and gcontrol commands will be passed on the started suites,
    overriding the above default.
  * Suite-connecting commands ("cylc dump" etc.) also take a timeout
    value on the command line but default to no timeout.

+ Fixed a bug in the passphrase location checking code (this would not 
have affected running suites with the passphrase installed in the suite
definition directory on the suite host or remote task hosts).

+ Stopped profilic suite logging caused by gcontrol continually checking
for the current list of task names (as of 4.5.0 the displayed task list
is determined dynamically).

+ Two new job submission methods for test purposes: background_slow
(takes 10 seconds to complete) and fail (always fails).

+ Submission of tasks that become ready to run at the same time
is now done in batches (size configurable): batch members
are submitted in parallel, and cylc waits on all member job submissions
(but not the submitted jobs, note) to return before continuing to the
next batch. Cylc cannot respond to network requests while submitting 
tasks so this should significantly improve performance for large
ensemble suites.

+ gcontrol "View -> Suite Log" now works even if the suite was not
running when the GUI started up.

+ the "outputs" example suite was not validating due to use of old-style 
cycle time variable in output messages. The new release generator
script now aborts in case of non-validating examples.

+ The remote task execution mechanism now automatically sources
/etc/profile and $HOME/.profile on the remote host, if they exist.
This may make most previous use of the "initial scripting" config item
obsolete.

________________________________________________________________________
4.6.0

+ The new delayed retry feature now works properly for cycling tasks
beyond the first cycle.

+ For ssh messaging, where remote messaging commands re-invoke
themselves on the suite host, the path to cylc on the suite host is now
transferred explicitly via the remote task job script - we no longer
rely on the user's .profile on the suite host for this.

+ New 'retry' event hook - calls the event handler if a task fails when
it has a retry lined up.

+ New User Guide sections under "Running Suites" to document "Automatic
Retry On Task Failure" and "Suite And Task Event Handling".

+ The task environment variables $CYLC_SUITE_INITIAL_CYCLE_TIME and
$CYLC_SUITE_FINAL_CYCLE_TIME now persist across restarts, and are 
always defined but may have the value "None" if no value is set on the
command line, state dump, or suite.rc file. The initial cycle time 
variable is automatically set in a cold start, but only optionally by a
warm start - which is really an implicit restart that does not
reference a previous state.

+ Tasks that specify "enable resurrection = True" will be revived from
the dead if they start running again (by external non-cylc means) after
reporting failure. This can be used to handle HPC-style job preemption.

+ Moved the User Guide documentation of cylc version re-invocation from
the installation section to the running suites section. 

+ Correct Quick Start Guide suite names (the initial component of the
registration name hierarchy had been omitted).

+ Do remote request handling before shutdown check (reverses the
original order) so suites can shutdown if requested even when a
failing task has retries lined up with a very short delay period.

+ New config item "[cylc] abort if any task fails = False"; if True,
the suite will abort with exit status 1 if any task false. The main
intended use for this is for pre-release automated test suites (but 
see also "reference test runs" below).

+ Suites can now be told to abort with error status if a shutdown handler 
fails - to allow use of shutdown handlers as suite run diagnosis tools
in cylc test suites (but see also "reference test runs" below).

+ Initial GUI text view task state filtering now shows all tasks, as
users often forget that some task states are filtered out if they did
not activate the filters themselves.

+ Removed the requirement to preface the integer TAG of an asynchronous
task, on the command line, with 'a:'.

+ Suites can now be restarted after task definitions have been deleted, 
without manually altering the suite state dump file. Undefined tasks in
the state dump are just be ignored at start up.

+ WARNING for anyone overriding the default task log, work, and share
directory config items, they have been moved up out of the "job
submission" runtime sub-section, because they are not really properties 
of the job submission process. 

+ The main cylc command interface no longer checks that Pyro is
installed, but the commands that it do. Consequently Pyro does not have
to be installed on a remote task host if you choose to use ssh
messaging, and you can still work with suites - but not run them - if
Pyro is not installed on the suite host.

+ Suites with intercycle connections like the following now run correctly:
    [scheduling]
        initial cycle time = 20110101
        final cycle time = 20110102
        [[dependencies]]
            [[[ 0,12 ]]]
                graph = "A"
            [[[ 6,18 ]]]
                graph  =  "A[T-6] => B"
(previously A would run in 6,18 as well as 0,12, which it shouldn't have).

+ Restored runtime "resolved dependency" graphing (disabled since
internal queues were implemented); and added optional logging of
resolved dependencies.

+ The gcapture log window (used to display suite stdout) now only color-highlights 
lines containing the strings CRITICAL, WARNING, or ERROR (not "failed"
or "error" anymore).

+ Added a suite startup event hook.

+ Replaced original simulation mode, which submitted real dummy tasks to
run locally after nobbling the task job script, with: (1) a proper
simulation mode that submits not real dummy tasks, and (2) a dummy mode
that simply dummies out command scripting but leaves job submission,
hosting, etc. intact.

+ Added a simple means of creating self-diagnosing cylc test suites:
running a suite in any run mode with '--reference-log' creates a special
reference suite log in the suite definition directory containing
task triggering information. Later, running with '--reference-test'
automatically switches on "abort if any task fails", "abort if suite
times out" (configured in [cylc][[reference test]]), and runs "cylc hook
check-triggering" on the shutdown hook with "abort if shutdown event
handler fails" switched on. The event handler compares triggering
behaviour in the test run with that of the reference run, and aborts the
test with non-zero exit status if any differences, disregarding timing
and order, are detected (i.e. tasks triggering off different upstream
tasks or running in different cycles).
The intention is to create a battery of automatic tests to make it
harder to break cylc in the future; and to make it easy for users with
complex suites to quickly verify, in dummy mode, that changes to cylc
will not break their suites. 

+ Suite definitions can now be reparsed at run time with the new "cylc
reload" command, even in tasks have been removed or added (in the later
case a specific instance of an added task will still have to be inserted
manually after the reload operation). This allows task settings
(environment variables etc.) to be changed without restarting the suite.

+ Higher-level family triggers (families of families) can now be used in
the graph.

+ Fixed a minor bug that could, in usual circumstances, cause cylc not
to recognize the need for another round of task processing and requiring
a manual nudge to get the suite to continue.

+ Documented stepped daily, monthly, and yearly cycling.

+ Added a simple utility command that generates random integers on a
given interval; used for computing random task run lengths in simulation
and dummy mode. 

+ Allow running simulation and dummy mode on the real time clock.

+ Bug fix: Sub-suites now load their own passphrase on start-up, not
that of their parent suite.

+ The gcontrol dot view now has hover-over task state information, and
right-click menus.

+ Added a new "environment scripting" task runtime configuration item.
In the task job script it fits between the cylc-defined environment 
and the task runtime environment.

+ External host selection officially supported (host = selector.sh).

+ The runahead limit base is now the oldest non-finished task (i.e. not
succeeded or failed); it defaults to the twice the smallest cycling
interval in the suite; and it will not stall a suite even if set less
than the smallest cycling.

+ A new command 'cylc broadcast NAME VALUE' allows tasks to broadcast
simple information to environment variables made available to other
tasks downstream of them. 

+ Fixed a problem with remote command invocation that caused the new ssh
messaging feature to fail in some cases.

+ Suite info, view, log, and graphing from gcontrol View menu.

+ A new task run time environment variable to allow a task to determine
whether or not it is a cold-start task: $CYLC_TASK_IS_COLDSTART ('True'
or 'False')