diff --git a/docs/readme-docs.md b/docs/readme-docs.md index 54e5602d..116d5ab3 100644 --- a/docs/readme-docs.md +++ b/docs/readme-docs.md @@ -1,47 +1,38 @@ # Documenting traitlets -[Documentation for `traitlets`](https://traitlets.readthedocs.io/en/latest/) -is hosted on ReadTheDocs. +[Documentation for `traitlets`](https://traitlets.readthedocs.io/en/latest/) is hosted on ReadTheDocs. ## Build documentation locally -#### Change directory to documentation root: +With [`hatch`](https://hatch.pypa.io/), one can build environments, docs, and serve it in one command: ``` -cd docs +pip install hatch +hatch run docs:build ``` -#### Create environment +#### Build documentation manually -- \[**conda**\] Create conda env (and install relevant dependencies): +Otherwise to build docs manually, - ``` - conda env create -f environment.yml - ``` +``` +cd docs +``` -- \[**pip**\] Create virtual environment (and install relevant dependencies): +Create virtual environment (and install relevant dependencies): - ``` +``` virtualenv traitlets_docs -p python3 - pip install -r requirements.txt - ``` - -#### Activate the newly built environment `traitlets_docs` - -- \[**conda**\] Activate conda env: - - ``` - source activate traitlets_docs - ``` + pip install -r traitlets[docs] +``` -- \[**pip**\] The virtualenv should have been automatically activated. If - not: +The virtualenv should have been automatically activated. If not: - ``` - source activate - ``` +``` +source activate +``` -#### Build documentation using: +##### Build documentation using: - Makefile for Linux and OS X: @@ -55,7 +46,7 @@ cd docs make.bat html ``` -#### Display the documentation locally +##### Display the documentation locally - Navigate to `build/html/index.html` in your browser. @@ -77,5 +68,3 @@ cd docs - `source/conf.py` - Sphinx build configuration file - `source` directory - source for documentation - `source/index.rst` - Main landing page of the Sphinx documentation -- `requirements.txt` - list of packages to install when using pip -- `environment.yml` - list of packages to install when using conda diff --git a/docs/source/config.rst b/docs/source/config.rst index 72dc0089..22468617 100644 --- a/docs/source/config.rst +++ b/docs/source/config.rst @@ -24,11 +24,12 @@ Configuration object: :class:`~traitlets.config.Config` Application: :class:`~traitlets.config.Application` An application is a process that does a specific job. The most obvious application is the :command:`ipython` command line program. Each - application reads *one or more* configuration files and a single set of + application may read configuration files and a single set of command line options and then produces a master configuration object for the application. This configuration object is then passed to the configurable objects that the - application creates. These configurable objects implement the actual logic + application creates, usually either via the `config` or `parent` constructor + arguments. These configurable objects implement the actual logic of the application and know how to configure themselves given the configuration object. @@ -50,14 +51,16 @@ Configurable: :class:`~traitlets.config.Configurable` Developers create :class:`~traitlets.config.Configurable` subclasses that implement all of the logic in the application. Each of these subclasses has its own configuration information that controls how - instances are created. + instances are created. When constructing a :class:`~traitlets.config.Configurable`, + the `config` or `parent` arguments can be passed to the constructor (respectively + a :class:`~traitlets.config.Config` and a Configurable object with a ``.config``). Singletons: :class:`~traitlets.config.SingletonConfigurable` Any object for which there is a single canonical instance. These are just like Configurables, except they have a class method :meth:`~traitlets.config.SingletonConfigurable.instance`, that returns the current active instance (or creates one if it - does not exist). :class:`~traitlets.config.Application`s is a singleton. + does not exist). :class:`~traitlets.config.Application` is a singleton. This lets objects easily connect to the current running Application without passing objects around everywhere. For instance, to get the current running @@ -87,15 +90,20 @@ Configuration objects and files A configuration object is little more than a wrapper around a dictionary. A configuration *file* is simply a mechanism for producing that object. -The main IPython configuration file is a plain Python script, -which can perform extensive logic to populate the config object. -IPython 2.0 introduces a JSON configuration file, -which is just a direct JSON serialization of the config dictionary, -which is easily processed by external software. - +Configuration files currently can be a plain Python script or a JSON file. +The former has the benefit that it can perform extensive logic to populate +the config object, while the latter is just a direct JSON serialization of +the config dictionary and can be easily processed by external software. When both Python and JSON configuration file are present, both will be loaded, with JSON configuration having higher priority. +The configuration files can be loaded by calling :meth:`Application.load_config_file()`, +which takes the relative path to the config file (with or without file extension) +and the directories in which to search for the config file. All found configuration +files will be loaded in reverse order, so that configs in earlier directories will +have higher priority. + + Python configuration Files -------------------------- @@ -120,13 +128,13 @@ subclass from traitlets import Int, Float, Unicode, Bool class MyClass(Configurable): - name = Unicode('defaultname' - help="the name of the object" - ).tag(config=True) + name = Unicode('defaultname', help="the name of the object").tag(config=True) ranking = Integer(0, help="the class's ranking").tag(config=True) value = Float(99.0) # The rest of the class implementation would go here.. + # Construct from config via MyClass(config=..) + In this example, we see that :class:`MyClass` has three attributes, two of which (``name``, ``ranking``) can be configured. All of the attributes are given types and default values. If a :class:`MyClass` is instantiated, @@ -180,7 +188,7 @@ but with a .json extension. Configuration described in previous section could be written as follows in a JSON configuration file: -.. sourcecode:: json +.. code-block:: json { "MyClass": { @@ -204,20 +212,25 @@ Let's say you want to have different configuration files for various purposes. Our configuration system makes it easy for one configuration file to inherit the information in another configuration file. The :func:`load_subconfig` command can be used in a configuration file for this purpose. Here is a simple -example that loads all of the values from the file :file:`base_config.py`:: +example that loads all of the values from the file :file:`base_config.py`: - # base_config.py - c = get_config() +.. code-block:: python + :caption: examples/docs/configs/base_config.py + + c = get_config() # noqa c.MyClass.name = 'coolname' c.MyClass.ranking = 100 -into the configuration file :file:`main_config.py`:: +into the configuration file :file:`main_config.py`: + +.. code-block:: python + :caption: examples/docs/configs/main_config.py + :emphasize-lines: 4 - # main_config.py - c = get_config() + c = get_config() # noqa # Load everything from base_config.py - load_subconfig('base_config.py') + load_subconfig('base_config.py') # noqa # Now override one of the values c.MyClass.name = 'bettername' @@ -225,17 +238,19 @@ into the configuration file :file:`main_config.py`:: In a situation like this the :func:`load_subconfig` makes sure that the search path for sub-configuration files is inherited from that of the parent. Thus, you can typically put the two in the same directory and everything will -just work. - +just work. An example app using these configuration files can be found at +`examples/docs/load_config_app.py `__. Class based configuration inheritance ===================================== There is another aspect of configuration where inheritance comes into play. Sometimes, your classes will have an inheritance hierarchy that you want -to be reflected in the configuration system. Here is a simple example:: +to be reflected in the configuration system. Here is a simple example: - from traitlets.config.configurable import Configurable +.. code-block:: python + + from traitlets.config import Application, Configurable from traitlets import Integer, Float, Unicode, Bool class Foo(Configurable): @@ -246,11 +261,15 @@ to be reflected in the configuration system. Here is a simple example:: name = Unicode('barname', config=True) othervalue = Int(0, config=True) + # construct Bar(config=..) + Now, we can create a configuration file to configure instances of :class:`Foo` -and :class:`Bar`:: +and :class:`Bar`: + +.. code-block:: python # config file - c = get_config() + c = get_config() # noqa c.Foo.name = 'bestname' c.Bar.othervalue = 10 @@ -258,7 +277,7 @@ and :class:`Bar`:: This class hierarchy and configuration file accomplishes the following: * The default value for :attr:`Foo.name` and :attr:`Bar.name` will be - 'bestname'. Because :class:`Bar` is a :class:`Foo` subclass it also + ``'bestname'``. Because :class:`Bar` is a :class:`Foo` subclass it also picks up the configuration information for :class:`Foo`. * The default value for :attr:`Foo.value` and :attr:`Bar.value` will be ``100.0``, which is the value specified as the class default. @@ -273,11 +292,29 @@ Command-line arguments ====================== All configurable options can also be supplied at the command line when launching -the application. Applications use a parser called -:class:`~traitlets.config.loader.KVArgParseConfigLoader` to load values into a Config -object. +the application. Internally, when :meth:`Application.initialize()` is called, +a :class:`~traitlets.config.loader.KVArgParseConfigLoader` instance is constructed +to load values into a :class:`~traitlets.config.Config` object. (For advanced users, +this can be overridden in the helper method :meth:`Application._create_loader()`.) + +Most command-line scripts simply need to call :meth:`Application.launch_instance()`, +which will create the Application singleton, parse the command-line arguments, and +start the application: -By default, values are assigned in much the same way as in a config file: +.. code-block:: python + :emphasize-lines: 8 + + from traitlets.config import Application + + class MyApp(Application): + def start(self): + pass # app logic goes here + + if __name__ == "__main__": + MyApp.launch_instance() + +By default, config values are assigned from command-line arguments in much the +same way as in a config file: .. code-block:: bash @@ -285,12 +322,42 @@ By default, values are assigned in much the same way as in a config file: is the same as adding: -.. sourcecode:: python +.. code-block:: python c.InteractiveShell.autoindent = False c.BaseIPythonApplication.profile = 'myprofile' -to your configuration file. +to your configuration file. Command-line arguments take precedence over +values read from configuration files. (This is done in +:meth:`Application.load_config_file()` by merging `Application.cli_config` +over values read from configuration files.) + +Note that even though :class:`Application` is a :class:`SingletonConfigurable`, multiple +applications could still be started and called from each other by constructing +them as one would with any other :class:`Configurable`: + +.. code-block:: python + :caption: examples/docs/multiple_apps.py + :emphasize-lines: 11,12,13 + + from traitlets.config import Application + + class OtherApp(Application): + def start(self): + print("other") + + class MyApp(Application): + classes = [OtherApp] + def start(self): + # similar to OtherApp.launch_instance(), but without singleton + self.other_app = OtherApp(config=self.config) + self.other_app.initialize(["--OtherApp.log_level", "INFO"]) + self.other_app.start() + + if __name__ == "__main__": + MyApp.launch_instance() + + .. versionchanged:: 5.0 @@ -329,10 +396,9 @@ to your configuration file. .. note:: - Any error in configuration files which lead to this configuration - file will be ignored by default. Application subclasses may specify - `raise_config_file_errors = True` to exit on failure to load config files, - instead of the default of logging the failures. + By default, an error in a configuration file will cause the configuration + file to be ignored and a warning logged. Application subclasses may specify + `raise_config_file_errors = True` to exit on failure to load config files instead. .. versionadded:: 4.3 @@ -376,6 +442,31 @@ to specify the whole class name: When specifying ``alias`` dictionary in code, the values might be the strings like ``'Class.trait'`` or two-tuples like ``('Class.trait', "Some help message")``. +For example: + +.. code-block:: python + :caption: examples/docs/aliases.py + :emphasize-lines: 11,12 + + from traitlets import Bool + from traitlets.config import Application, Configurable + + class Foo(Configurable): + enabled = Bool(False, help="whether enabled").tag(config=True) + + class App(Application): + classes = [Foo] + dry_run = Bool(False, help="dry run test").tag(config=True) + aliases = { + "dry-run": "App.dry_run", + ("f", "foo-enabled"): ("Foo.enabled", "whether foo is enabled"), + } + + if __name__ == "__main__": + App.launch_instance() + +By default, the ``--log-level`` alias will be set up for ``Application.log_level``. + Flags ***** @@ -400,6 +491,41 @@ For instance: # is equivalent to $ ipython --TerminalIPythonApp.display_banner=False +And a runnable code example: + +.. code-block:: python + :caption: examples/docs/flags.py + :emphasize-lines: 11,12,13,14,15,16,17 + + from traitlets import Bool + from traitlets.config import Application, Configurable + + class Foo(Configurable): + enabled = Bool(False, help="whether enabled").tag(config=True) + + class App(Application): + classes = [Foo] + dry_run = Bool(False, help="dry run test").tag(config=True) + flags = { + "dry-run": ({"App": {"dry_run": True}}, dry_run.help), + ("f", "enable-foo"): ({ + "Foo": {"enabled": True}, + }, "Enable foo"), + ("disable-foo"): ({ + "Foo": {"enabled": False}, + }, "Disable foo"), + } + + if __name__ == "__main__": + App.launch_instance() + +Since flags are a bit more complicated to set up, there are a couple of common patterns +implemented in helper methods. For example, :func:`traitlets.config.boolean_flag()` sets +up the flags ``--x`` and ``--no-x``. By default, the following few flags are set up: +``--debug`` (setting ``log_level=DEBUG``), ``--show-config``, and ``--show-config-json`` +(print config to stdout and exit). + + Subcommands ----------- @@ -414,8 +540,9 @@ after :command:`git`, and are called with the form :command:`command subcommand .. currentmodule:: traitlets.config -Subcommands are specified as a dictionary on :class:`~traitlets.config.Application` -instances, mapping *subcommand names* to two-tuples containing these: +Subcommands are specified as a dictionary assigned to a ``subcommands`` class member +of :class:`~traitlets.config.Application` instances. This dictionary maps +*subcommand names* to two-tuples containing these: 1. A subclass of :class:`Application` to handle the subcommand. This can be specified as: @@ -429,18 +556,86 @@ instances, mapping *subcommand names* to two-tuples containing these: .. note:: The return value of the factory above is an *instance*, not a class, - son the :meth:`SingletonConfigurable.instance()` is not invoked + so the :meth:`SingletonConfigurable.instance()` is not invoked in this case. - In all cases, the instanciated app is stored in :attr:`Application.subapp` + In all cases, the instantiated app is stored in :attr:`Application.subapp` and its :meth:`Application.initialize()` is invoked. 2. A short description of the subcommand for use in help output. +For example (refer to ``examples/subcommands_app.py`` for a more complete example): + +.. code-block:: python + :caption: examples/docs/subcommands.py + :emphasize-lines: 14,15 + + from traitlets.config import Application + + class SubApp1(Application): + pass + + class SubApp2(Application): + @classmethod + def get_subapp_instance(cls, app: Application) -> Application: + app.clear_instance() # since Application is singleton, need to clear main app + return cls.instance(parent=app) + + class MainApp(Application): + subcommands = { + "subapp1": (SubApp1, "First subapp"), + "subapp2": (SubApp2.get_subapp_instance, "Second subapp"), + } + + if __name__ == "__main__": + MainApp.launch_instance() + + To see a list of the available aliases, flags, and subcommands for a configurable -application, simply pass ``-h`` or ``--help``. And to see the full list of +application, simply pass ``-h`` or ``--help``. To see the full list of configurable options (*very* long), pass ``--help-all``. +For more complete examples +of setting up :class:`~traitlets.config.Application`, refer to the +`application examples `__. + + +Other `Application` members +--------------------------- + +The following are typically set as class variables of :class:`~traitlets.config.Application` +subclasses, but can also be set as instance variables. + +* ``.classes``: A list of :class:`~traitlets.config.Configurable` classes. Similar to configs, + any class name can be used in ``--Class.trait=value`` arguments, including classes that the + :class:`~traitlets.config.Application` might not know about. However, the + ``--help-all`` menu will only enumerate ``config`` traits of classes in ``Application.classes``. + Similarly, ``.classes`` is used in other places where an application wants to list all + configurable traits; examples include :meth:`Application.generate_config_file()` + and the :ref:`argcomplete` handling. + +* ``.name``, ``.description``, ``.option_description``, ``.keyvalue_description``, + ``.subcommand_description``, ``.examples``, ``.version``: Various strings used in the ``--help`` + menu and other messages + +* ``.log_level``, ``.log_datefmt``, ``.log_format``, ``.logging_config``: Configurable options + to control application logging, which is emitted via the logger `Application.log`. For more + information about these, refer to their respective traits' ``.help``. + +* ``.show_config``, ``.show_config_json``: Configurable boolean options, which if set to ``True``, + will cause the application to print the config to stdout instead of calling + :meth:`Application.start()` + +Additionally, the following are set by :class:`~traitlets.config.Application`: + +* ``.cli_config``: The :class:`~traitlets.config.Config` created from the command-line arguments. + This is saved to override any config values loaded from configuration files called by + :meth:`Application.load_config_file()`. + +* ``.extra_args``: This is a list holding any positional arguments remaining from + the command-line arguments parsed during :meth:`Application.initialize()`. + As noted earlier, these must be contiguous in the command-line. + .. _cli_strings: @@ -496,26 +691,25 @@ but does not any more with traitlets 5, please `let us know `__. To use this, +follow the instructions for setting up argcomplete; +you will likely want to +`activate global completion `__ +by doing something alone the lines of: + +.. code-block:: bash + + # pip install argcomplete + mkdir -p ~/.bash_completion.d/ + activate-global-python-argcomplete --dest=~/.bash_completion.d/argcomplete + # source ~/.bash_completion.d/argcomplete from your ~/.bashrc + +(Follow relevant instructions for your shell.) For any script you want tab-completion +to work on, include the line: + +.. code-block:: python + + # PYTHON_ARGCOMPLETE_OK + +in the first 1024 bytes of the script. + +The following options can be tab-completed: + +* Flags and aliases + +* The classes in ``Application.classes``, which can be initially completed as ``--Class.`` + + * Once a completion is narrows to a single class, the individual ``config`` traits + of the class will be tab-completable, as ``--Class.trait``. + +* The available values for :class:`traitlets.Bool` and :class:`traitlets.Enum` will be completable, + as well as any other custom :class:`traitlets.TraitType` which defines a ``argcompleter()`` method + returning a list of available string completions. + +* Custom completer methods can be assigned to a trait by tagging an ``argcompleter`` metadata tag. + Refer to `argcomplete's documentation `__ + for examples of creating custom completer methods. + +Detailed examples of these can be found in the docstring of +`examples/argcomplete_app.py `__. + + +Caveats with `argcomplete` handling +*********************************** + +The support for `argcomplete` is still relatively new and may not work with all ways in +which an :class:`~traitlets.config.Application` is used. Some known caveats: + +* `argcomplete` is called when any `Application` first constructs and uses a + :class:`~traitlets.config.KVArgParseConfigLoader` instance, which constructs + a `argparse.ArgumentParser` instance. + We assume that this is usually first done in scripts when parsing the command-line arguments, + but technically a script can first call ``Application.initialize(["other", "args"])`` for + some other reason. + +* `traitlets` does not actually add ``"--Class.trait"`` options to the `ArgumentParser`, + but instead directly parses them from ``argv``. In order to complete these, a custom + :class:`~traitlets.config.argcomplete_config.CompletionFinder` is subclassed from + ``argcomplete.CompletionFinder``, which dynamically inserts the ``"--Class.""`` and ``"--Class.trait"`` + completions when it thinks suitable. However, this handling may be a bit fragile. + +* Because `traitlets` initializes configs from `argv` and not from `ArgumentParser`, it may be + more difficult to write custom completers which dynamically provide completions based on the + state of other parsed arguments. + +* Subcommand handling is especially tricky. `argcomplete`'s strategy is to call the python script + with no arguments e.g. ``len(sys.argv) == 1``, run until `argcomplete` is called on an `ArgumentParser` + and determine what completions are available. On the other hand, `traitlet`'s subcommand-handling + strategy is to check ``sys.argv[1]`` and see if it matches a subcommand, and if so then dynamically + load the subcommand app and initialize it with ``sys.argv[1:]``. To reconcile these two different + approaches, some hacking was done to get `traitlets` to recognize the current command-line as seen + by `argcomplete`, and to get `argcomplete` to start parsing command-line arguments after subcommands + have been evaluated. + + * Currently, completing subcommands themselves is not yet supported. + + * Some applications like `Jupyter` have custom ways of constructing subcommands or parsing ``argv`` + which complicates matters even further. + +More details about these caveats can be found in the `original pull request `__. + Design requirements =================== diff --git a/examples/argcomplete_app.py b/examples/argcomplete_app.py new file mode 100755 index 00000000..dbb7c662 --- /dev/null +++ b/examples/argcomplete_app.py @@ -0,0 +1,152 @@ +#!/usr/bin/env python +# PYTHON_ARGCOMPLETE_OK +"""A simple example to test CLI completion with `traitlets.Application` + +Follow the installation instructions in +https://github.com/kislyuk/argcomplete#installation +to install argcomplete. For example for bash, you can set up global completion +via something like:: + + $ activate-global-python-argcomplete --dest=~/.bash_completion.d + +and ``source ~/.bash_completion.d`` in your ``~/.bashrc``. To use the +`global-python-argcomplete`, your `traitlets.Application`-based script should +have the string ``PYTHON_ARGCOMPLETE_OK`` in the first few lines of the script. + +Afterwards, try tab completing options to this script:: + + # Option completion, show flags, aliases and --Class. + $ examples/argcomplete_app.py --[TAB] + --Application. --JsonPrinter. --env-vars --s + --ArgcompleteApp. --e --help --skip-if-missing + --EnvironPrinter. --env-var --json-indent --style + + $ examples/argcomplete_app.py --A[TAB] + --Application. --ArgcompleteApp. + + # Complete class config traits + $ examples/argcomplete_app.py --EnvironPrinter.[TAB] + --EnvironPrinter.no_complete --EnvironPrinter.style + --EnvironPrinter.skip_if_missing --EnvironPrinter.vars + + # Using argcomplete's provided EnvironCompleter + $ examples/argcomplete_app.py --EnvironPrinter.vars=[TAB] + APPDATA LS_COLORS + COMP_LINE NAME + COMP_POINT OLDPWD + COMP_TYPE PATH + ... + + $ examples/argcomplete_app.py --EnvironPrinter.vars USER [TAB] + APPDATA LS_COLORS + COMP_LINE NAME + COMP_POINT OLDPWD + COMP_TYPE PATH + ... + + # Alias for --EnvironPrinter.vars + $ examples/argcomplete_app.py --env-vars P[TAB] + PATH PWD PYTHONPATH + + # Custom completer example + $ examples/argcomplete_app.py --env-vars PWD --json-indent [TAB] + 2 4 8 + + # Enum completer example + $ examples/argcomplete_app.py --style [TAB] + ndjson posix verbose + + # Bool completer example + $ examples/argcomplete_app.py --Application.show_config_json [TAB] + 0 1 false true + +If completions are not showing, you can set the environment variable ``_ARC_DEBUG=1`` +to assist in debugging argcomplete. This was last checked with ``argcomplete==1.12.3``. +""" +import json +import os + +try: + from argcomplete.completers import EnvironCompleter, SuppressCompleter # type: ignore[import] +except ImportError: + EnvironCompleter = SuppressCompleter = None +from traitlets import Bool, Enum, Int, List, Unicode +from traitlets.config.application import Application +from traitlets.config.configurable import Configurable + + +def _indent_completions(**kwargs): + """Example of a custom completer, which could be dynamic""" + return ["2", "4", "8"] + + +class JsonPrinter(Configurable): + indent = Int(None, allow_none=True).tag(config=True, argcompleter=_indent_completions) + + def print(self, obj): + print(json.dumps(obj, indent=self.indent)) + + +class EnvironPrinter(Configurable): + """A class that has configurable, typed attributes.""" + + vars = List(trait=Unicode(), help="Environment variable").tag( + # NOTE: currently multiplicity is ignored by the traitlets CLI. + # Refer to issue GH#690 for discussion + config=True, + multiplicity="+", + argcompleter=EnvironCompleter, + ) + no_complete = Unicode().tag(config=True, argcompleter=SuppressCompleter) + style = Enum(values=["posix", "ndjson", "verbose"], default_value="posix").tag(config=True) + skip_if_missing = Bool(False, help="Skip variable if not set").tag(config=True) + + def print(self): + for env_var in self.vars: + if env_var not in os.environ: + if self.skip_if_missing: + continue + else: + raise KeyError(f"Environment variable not set: {env_var}") + + value = os.environ[env_var] + if self.style == "posix": + print(f"{env_var}={value}") + elif self.style == "verbose": + print(f">> key: {env_var} value:\n{value}\n") + elif self.style == "ndjson": + JsonPrinter(parent=self).print({"key": env_var, "value": value}) + + +def bool_flag(trait, value=True): + return ({trait.this_class.__name__: {trait.name: value}}, trait.help) + + +class ArgcompleteApp(Application): + name = Unicode("argcomplete-example-app") + description = Unicode("prints requested environment variables") + classes = [JsonPrinter, EnvironPrinter] + + config_file = Unicode("", help="Load this config file").tag(config=True) + + aliases = { + ("e", "env-var", "env-vars"): "EnvironPrinter.vars", + ("s", "style"): "EnvironPrinter.style", + ("json-indent"): "JsonPrinter.indent", + } + + flags = { + "skip-if-missing": bool_flag(EnvironPrinter.skip_if_missing), + } + + def initialize(self, argv=None): + self.parse_command_line(argv) + if self.config_file: + self.load_config_file(self.config_file) + + def start(self): + EnvironPrinter(parent=self).print() + + +if __name__ == "__main__": + ArgcompleteApp.launch_instance() diff --git a/examples/docs/aliases.py b/examples/docs/aliases.py new file mode 100755 index 00000000..2a6953fb --- /dev/null +++ b/examples/docs/aliases.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python +# PYTHON_ARGCOMPLETE_OK +"""A simple example of using Application aliases, for docs""" + +from traitlets import Bool +from traitlets.config import Application, Configurable + + +class Foo(Configurable): + enabled = Bool(False, help="whether enabled").tag(config=True) + + +class App(Application): + classes = [Foo] + dry_run = Bool(False, help="dry run test").tag(config=True) + aliases = { + "dry-run": "App.dry_run", + ("f", "foo-enabled"): ("Foo.enabled", "whether foo is enabled"), + } + + +if __name__ == "__main__": + App.launch_instance() diff --git a/examples/docs/configs/base_config.py b/examples/docs/configs/base_config.py new file mode 100644 index 00000000..ed81f110 --- /dev/null +++ b/examples/docs/configs/base_config.py @@ -0,0 +1,5 @@ +# Example config used by load_config_app.py + +c = get_config() # noqa +c.MyClass.name = 'coolname' +c.MyClass.ranking = 100 diff --git a/examples/docs/configs/main_config.py b/examples/docs/configs/main_config.py new file mode 100644 index 00000000..26dfb0ce --- /dev/null +++ b/examples/docs/configs/main_config.py @@ -0,0 +1,9 @@ +# Example config used by load_config_app.py + +c = get_config() # noqa + +# Load everything from base_config.py +load_subconfig('base_config.py') # noqa + +# Now override one of the values +c.MyClass.name = 'bettername' diff --git a/examples/docs/container.py b/examples/docs/container.py new file mode 100755 index 00000000..9d43bd5e --- /dev/null +++ b/examples/docs/container.py @@ -0,0 +1,20 @@ +#!/usr/bin/env python +# PYTHON_ARGCOMPLETE_OK +"""A simple example of using container traits in Application command-line""" + +from traitlets import Dict, Integer, List, Unicode +from traitlets.config import Application + + +class App(Application): + aliases = {"x": "App.x", "y": "App.y"} + x = List(Unicode(), config=True) + y = Dict(Integer(), config=True) + + def start(self): + print(f"x={self.x}") + print(f"y={self.y}") + + +if __name__ == "__main__": + App.launch_instance() diff --git a/examples/docs/flags.py b/examples/docs/flags.py new file mode 100755 index 00000000..940553ec --- /dev/null +++ b/examples/docs/flags.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python +# PYTHON_ARGCOMPLETE_OK +"""A simple example of using Application flags, for docs""" + +from traitlets import Bool +from traitlets.config import Application, Configurable + + +class Foo(Configurable): + enabled = Bool(False, help="whether enabled").tag(config=True) + + +class App(Application): + classes = [Foo] + dry_run = Bool(False, help="dry run test").tag(config=True) + flags = { + "dry-run": ({"App": {"dry_run": True}}, dry_run.help), + ("f", "enable-foo"): ( + { + "Foo": {"enabled": True}, + }, + "Enable foo", + ), + ("disable-foo"): ( + { + "Foo": {"enabled": False}, + }, + "Disable foo", + ), + } + + +if __name__ == "__main__": + App.launch_instance() diff --git a/examples/docs/from_string.py b/examples/docs/from_string.py new file mode 100755 index 00000000..eb38cb78 --- /dev/null +++ b/examples/docs/from_string.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python +# PYTHON_ARGCOMPLETE_OK +"""A simple example of using TraitType.from_string, for docs""" + +from binascii import a2b_hex + +from traitlets import Bytes +from traitlets.config import Application + + +class HexBytes(Bytes): + def from_string(self, s): + return a2b_hex(s) + + +class App(Application): + aliases = {"key": "App.key"} + key = HexBytes( + help=""" + Key to be used. + + Specify as hex on the command-line. + """, + config=True, + ) + + def start(self): + print(f"key={self.key}") + + +if __name__ == "__main__": + App.launch_instance() diff --git a/examples/docs/load_config_app.py b/examples/docs/load_config_app.py new file mode 100755 index 00000000..1c73c096 --- /dev/null +++ b/examples/docs/load_config_app.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python +# PYTHON_ARGCOMPLETE_OK +"""A simple example of loading configs and overriding + +Example: + + $ ./examples/docs/load_config_app.py + bettername ranking:100 + + $ ./examples/docs/load_config_app.py --name cli_name + cli_name ranking:100 + + $ ./examples/docs/load_config_app.py --name cli_name --MyApp.MyClass.ranking=99 + cli_name ranking:99 + + $ ./examples/docs/load_config_app.py -c "" + default ranking:0 +""" + +from pathlib import Path + +from traitlets import Int, Unicode +from traitlets.config import Application, Configurable + + +class MyClass(Configurable): + name = Unicode(default_value="default").tag(config=True) + ranking = Int().tag(config=True) + + def __str__(self): + return f"{self.name} ranking:{self.ranking}" + + +class MyApp(Application): + classes = [MyClass] + config_file = Unicode(default_value="main_config", help="base name of config file").tag( + config=True + ) + aliases = { + "name": "MyClass.name", + "ranking": "MyClass.ranking", + ("c", "config-file"): "MyApp.config_file", + } + + def initialize(self, argv=None): + super().initialize(argv=argv) + if self.config_file: + self.load_config_file(self.config_file, [Path(__file__).parent / "configs"]) + + def start(self): + print(MyClass(parent=self)) + + +if __name__ == "__main__": + MyApp.launch_instance() diff --git a/examples/docs/multiple_apps.py b/examples/docs/multiple_apps.py new file mode 100755 index 00000000..71a3ce1f --- /dev/null +++ b/examples/docs/multiple_apps.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python +# PYTHON_ARGCOMPLETE_OK +"""A simple example of one application calling another""" + +from traitlets.config import Application + + +class OtherApp(Application): + def start(self): + print("other") + + +class MyApp(Application): + classes = [OtherApp] + + def start(self): + # similar to OtherApp.launch_instance(), but without singleton + self.other_app = OtherApp(config=self.config) + self.other_app.initialize(["--OtherApp.log_level", "INFO"]) + self.other_app.start() + + +if __name__ == "__main__": + MyApp.launch_instance() diff --git a/examples/docs/subcommands.py b/examples/docs/subcommands.py new file mode 100755 index 00000000..954d8bfe --- /dev/null +++ b/examples/docs/subcommands.py @@ -0,0 +1,27 @@ +#!/usr/bin/env python +# PYTHON_ARGCOMPLETE_OK +"""A simple example of using Application subcommands, for docs""" + +from traitlets.config import Application + + +class SubApp1(Application): + pass + + +class SubApp2(Application): + @classmethod + def get_subapp_instance(cls, app: Application) -> Application: + app.clear_instance() # since Application is singleton, need to clear main app + return cls.instance(parent=app) # type: ignore[no-any-return] + + +class MainApp(Application): + subcommands = { + "subapp1": (SubApp1, "First subapp"), + "subapp2": (SubApp2.get_subapp_instance, "Second subapp"), + } + + +if __name__ == "__main__": + MainApp.launch_instance() diff --git a/examples/myapp.py b/examples/myapp.py old mode 100644 new mode 100755 index 77135ac6..e54c2bf9 --- a/examples/myapp.py +++ b/examples/myapp.py @@ -1,3 +1,5 @@ +#!/usr/bin/env python +# PYTHON_ARGCOMPLETE_OK """A simple example of how to use traitlets.config.application.Application. This should serve as a simple example that shows how the traitlets config @@ -29,8 +31,7 @@ When the config attribute of an Application is updated, it will fire all of the trait's events for all of the config=True attributes. """ - -from traitlets import Bool, Dict, Int, List, Unicode +from traitlets import Bool, Dict, Enum, Int, List, Unicode from traitlets.config.application import Application from traitlets.config.configurable import Configurable @@ -41,6 +42,7 @@ class Foo(Configurable): i = Int(0, help="The integer i.").tag(config=True) j = Int(1, help="The integer j.").tag(config=True) name = Unicode("Brian", help="First name.").tag(config=True, shortname="B") + mode = Enum(values=["on", "off", "other"], default_value="on").tag(config=True) class Bar(Configurable): @@ -60,6 +62,7 @@ class MyApp(Application): i="Foo.i", j="Foo.j", name="Foo.name", + mode="Foo.mode", running="MyApp.running", enabled="Bar.enabled", log_level="MyApp.log_level", @@ -93,10 +96,10 @@ def start(self): print("app.config:") print(self.config) print("try running with --help-all to see all available flags") - self.log.info("Info Mesage") - self.log.debug("DebugMessage") - self.log.critical("Warning") - self.log.critical("Critical mesage") + self.log.debug("Debug Message") + self.log.info("Info Message") + self.log.warning("Warning Message") + self.log.critical("Critical Message") def main(): diff --git a/examples/subcommands_app.py b/examples/subcommands_app.py new file mode 100755 index 00000000..d3a2033c --- /dev/null +++ b/examples/subcommands_app.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python +# PYTHON_ARGCOMPLETE_OK +"""A simple example to demonstrate subcommands with traitlets.Application + +Example: + + $ examples/subcommands_app.py foo --print-name alice + foo + hello alice + + $ examples/subcommands_app.py bar --print-name bob + bar + hello bob +""" + +from traitlets import Enum, Unicode +from traitlets.config.application import Application +from traitlets.config.configurable import Configurable + + +class PrintHello(Configurable): + greet_name = Unicode("world").tag(config=True) + greeting = Enum(values=["hello", "hi", "bye"], default_value="hello").tag(config=True) + + def run(self): + print(f"{self.greeting} {self.greet_name}") + + +class FooApp(Application): + name = Unicode("foo") + classes = [PrintHello] + aliases = { + "print-name": "PrintHello.greet_name", + } + + config_file = Unicode("", help="Load this config file").tag(config=True) + + def start(self): + print(self.name) + PrintHello(parent=self).run() + + +class BarApp(Application): + name = Unicode("bar") + classes = [PrintHello] + aliases = { + "print-name": "PrintHello.greet_name", + } + + config_file = Unicode("", help="Load this config file").tag(config=True) + + def start(self): + print(self.name) + PrintHello(parent=self).run() + + @classmethod + def get_subapp(cls, main_app: Application) -> Application: + main_app.clear_instance() + return cls.instance(parent=main_app) # type: ignore[no-any-return] + + +class MainApp(Application): + name = Unicode("subcommand-example-app") + description = Unicode("demonstrates app with subcommands") + subcommands = { + # Subcommands should be a dictionary mapping from the subcommand name + # to one of the following: + # 1. The Application class to be instantiated e.g. FooApp + # 2. A string e.g. "traitlets.examples.subcommands_app.FooApp" + # which will be lazily evaluated + # 3. A callable which takes this Application and returns an instance + # (not class) of the subcommmand Application + "foo": (FooApp, "run foo"), + "bar": (BarApp.get_subapp, "run bar"), + } + + +if __name__ == "__main__": + MainApp.launch_instance() diff --git a/pyproject.toml b/pyproject.toml index 06230a49..3296418b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -22,7 +22,7 @@ requires-python = ">=3.7" dynamic = ["version"] [project.optional-dependencies] -test = ["pytest", "pre-commit"] +test = ["pytest", "pytest-mock", "pre-commit", "argcomplete>=2.0"] docs = [ "myst-parser", "pydata-sphinx-theme", @@ -89,9 +89,10 @@ warn_unused_configs = true warn_redundant_casts = true warn_return_any = true warn_unused_ignores = true +exclude = ["examples/docs/configs"] [tool.pytest.ini_options] -addopts = "--durations=10 -ra --showlocals --doctest-modules --color yes" +addopts = "--durations=10 -ra --showlocals --doctest-modules --color yes --ignore examples/docs/configs" testpaths = [ "traitlets", "examples", @@ -199,6 +200,8 @@ unfixable = [ # N802 Function name `assertIn` should be lowercase # F841 Local variable `t` is assigned to but never used "traitlets/tests/*" = ["B011", "F841", "C408", "E402", "T201", "B007", "N802", "F841"] +# B003 Assigning to os.environ doesn't clear the environment +"traitlets/config/tests/*" = ["B003"] # F401 `_version.__version__` imported but unused # F403 `from .traitlets import *` used; unable to detect undefined names "traitlets/*__init__.py" = ["F401", "F403"] diff --git a/traitlets/config/application.py b/traitlets/config/application.py index 66f56f72..d1660106 100644 --- a/traitlets/config/application.py +++ b/traitlets/config/application.py @@ -400,7 +400,7 @@ def _log_default(self): # this must be a dict of two-tuples, # the first element being the application class/import string # and the second being the help string for the subcommand - subcommands: t.Union[t.Dict[str, t.Tuple[str, str]], Dict] = Dict() + subcommands: t.Union[t.Dict[str, t.Tuple[t.Any, str]], Dict] = Dict() # parse_command_line will initialize a subapp, if requested subapp = Instance("traitlets.config.application.Application", allow_none=True) @@ -525,6 +525,7 @@ def emit_alias_help(self): for c in cls.mro()[:-3]: classdict[c.__name__] = c + fhelp: t.Optional[str] for alias, longname in self.aliases.items(): try: if isinstance(longname, tuple): @@ -786,11 +787,58 @@ def flatten_flags(self): def _create_loader(self, argv, aliases, flags, classes): return KVArgParseConfigLoader(argv, aliases, flags, classes=classes, log=self.log) + @classmethod + def _get_sys_argv(cls, check_argcomplete: bool = False) -> t.List[str]: + """Get `sys.argv` or equivalent from `argcomplete` + + `argcomplete`'s strategy is to call the python script with no arguments, + so ``len(sys.argv) == 1``, and run until the `ArgumentParser` is constructed + and determine what completions are available. + + On the other hand, `traitlet`'s subcommand-handling strategy is to check + ``sys.argv[1]`` and see if it matches a subcommand, and if so then dynamically + load the subcommand app and initialize it with ``sys.argv[1:]``. + + This helper method helps to take the current tokens for `argcomplete` and pass + them through as `argv`. + """ + if check_argcomplete and "_ARGCOMPLETE" in os.environ: + try: + from traitlets.config.argcomplete_config import get_argcomplete_cwords + + cwords = get_argcomplete_cwords() + assert cwords is not None + return cwords + except (ImportError, ModuleNotFoundError): + pass + return sys.argv + + @classmethod + def _handle_argcomplete_for_subcommand(cls): + """Helper for `argcomplete` to recognize `traitlets` subcommands + + `argcomplete` does not know that `traitlets` has already consumed subcommands, + as it only "sees" the final `argparse.ArgumentParser` that is constructed. + (Indeed `KVArgParseConfigLoader` does not get passed subcommands at all currently.) + We explicitly manipulate the environment variables used internally by `argcomplete` + to get it to skip over the subcommand tokens. + """ + if "_ARGCOMPLETE" not in os.environ: + return + + try: + from traitlets.config.argcomplete_config import increment_argcomplete_index + + increment_argcomplete_index() + except (ImportError, ModuleNotFoundError): + pass + @catch_config_error def parse_command_line(self, argv=None): """Parse the command line arguments.""" assert not isinstance(argv, str) - argv = sys.argv[1:] if argv is None else argv + if argv is None: + argv = self._get_sys_argv(check_argcomplete=bool(self.subcommands))[1:] self.argv = [cast_unicode(arg) for arg in argv] if argv and argv[0] == "help": @@ -802,6 +850,7 @@ def parse_command_line(self, argv=None): subc, subargv = argv[0], argv[1:] if re.match(r"^\w(\-?\w)*$", subc) and subc in self.subcommands: # it's a subcommand, and *not* a flag or class parameter + self._handle_argcomplete_for_subcommand() return self.initialize_subcommand(subc, subargv) # Arguments after a '--' argument are for the script IPython may be diff --git a/traitlets/config/argcomplete_config.py b/traitlets/config/argcomplete_config.py new file mode 100644 index 00000000..054debda --- /dev/null +++ b/traitlets/config/argcomplete_config.py @@ -0,0 +1,203 @@ +"""Helper utilities for integrating argcomplete with traitlets""" +import argparse +import os +import typing as t + +try: + import argcomplete # type: ignore[import] + from argcomplete import CompletionFinder +except ImportError: + # This module and its utility methods are written to not crash even + # if argcomplete is not installed. + class StubModule: + def __getattr__(self, attr): + if not attr.startswith("__"): + raise ModuleNotFoundError("No module named 'argcomplete'") + raise AttributeError(f"argcomplete stub module has no attribute '{attr}'") + + argcomplete = StubModule() + CompletionFinder = object + + +def get_argcomplete_cwords() -> t.Optional[t.List[str]]: + """Get current words prior to completion point + + This is normally done in the `argcomplete.CompletionFinder` constructor, + but is exposed here to allow `traitlets` to follow dynamic code-paths such + as determining whether to evaluate a subcommand. + """ + if "_ARGCOMPLETE" not in os.environ: + return None + + comp_line = os.environ["COMP_LINE"] + comp_point = int(os.environ["COMP_POINT"]) + # argcomplete.debug("splitting COMP_LINE for:", comp_line, comp_point) + comp_words: t.List[str] + try: + ( + cword_prequote, + cword_prefix, + cword_suffix, + comp_words, + last_wordbreak_pos, + ) = argcomplete.split_line(comp_line, comp_point) + except ModuleNotFoundError: + return None + + # _ARGCOMPLETE is set by the shell script to tell us where comp_words + # should start, based on what we're completing. + # 1: