command line option parsing

[dimpase]

Issue created by migration from Trac.

Original creator: @williamstein

Original creation time: 2006-09-12 23:21:18

Assignee: somebody

CC: kini saraedum iandrus

We should improve and/or modernize and/or revise Sage's command-line parsing.

Two ideas, which could be debated endlessly and could also be implemented independently of each other:

• use Python's argparse module to handle the parsing, rather than a shell script
• change Sage's options from flags to subcommands: sage --package ... would be changed to sage package ..., etc. (comment:56 and comment:57 list some possibilities)

(changed by jhpalmieri at 2020-08-11 17:35:51)

[dimpase]

Comment by @williamstein created at 2007-01-13 01:56:18

no -- you can't combine command line options like that. this isn't a bug
but a not implemented yet issue.

[dimpase]

Comment by @williamstein created at 2007-01-13 01:56:18

Changing type from defect to enhancement.

[dimpase]

Comment by mabshoff created at 2007-09-11 02:11:39

This should be fixable, but the long term goal is to do a proper rewrite of the command line options.

Cheers,

Michael

[dimpase]

Comment by gfurnish created at 2008-03-16 07:59:18

Changing assignee from somebody to gfurnish.

[dimpase]

Comment by gfurnish created at 2008-03-16 07:59:18

Changing status from new to assigned.

[dimpase]

Comment by gfurnish created at 2008-04-04 19:55:55

Changing component from basic arithmetic to interfaces.

[dimpase]

Comment by mabshoff created at 2008-09-24 02:59:36

See also #180 for a bunch of related failures due to the option parsing being dumb :o

Cheers,

Michael

[dimpase]

Comment by mabshoff created at 2008-09-24 02:59:36

Changing status from assigned to new.

[dimpase]

Comment by mabshoff created at 2008-09-24 02:59:36

Changing assignee from gfurnish to mabshoff.

[dimpase]

Comment by mabshoff created at 2008-09-24 02:59:46

Changing status from new to assigned.

[dimpase]

Comment by kcrisman created at 2009-12-30 05:15:23

Note that sage -bn now builds, then does notebook, though of course it doesn't fix the issue here.

[dimpase]

Comment by jhpalmieri created at 2010-03-19 15:41:48

the file SAGE_ROOT/makefile

[dimpase]

Attachment makefile by jhpalmieri created at 2010-03-19 15:41:55

[dimpase]

Attachment makefile.diff by jhpalmieri created at 2010-03-19 15:42:11

extcode repo

[dimpase]

Attachment trac_21-extcode.patch by jhpalmieri created at 2010-03-19 15:42:32

sagenb repo

[dimpase]

Attachment trac_21-sagenb.patch by jhpalmieri created at 2010-03-19 15:50:37

Here are patches. After applying "trac_21-scripts.patch", you may need to make "SAGE_ROOT/local/bin/sage-sage.py" executable. The build process works for me with these patches. For the standard packages, the third line in

if [ "$SAGE_LOCAL" = "" ]; then
   echo "SAGE_LOCAL undefined ... exiting";
   echo "Maybe run 'sage -sh'?"
   exit 1
fi

should be changed to "Maybe run 'sage --sh'?", but this doesn't affect the functioning of the packages, and otherwise, they don't need changing. I haven't looked at optional packages.

This approaches uses Python's optparse to parse command-line options. If someone wants to write a version using shflags or some other package, go ahead.

I propose the following approach, whether using these patches or other ones:

• first, we include new command-line options but don't turn them on by default, instead printing a message like this one when you type "sage [...]" with a nonempty argument:

    Note: Using old-style Sage command-line options. 

    To try out Sage's experimental GNU/Posix-style command-line options 
    (for example, 'sage --notebook' instead of 'sage -notebook'), set the 
    environment variable $SAGE_NEW_OPTIONS to something nonempty. 
    To bypass this message, set the environment variable 
    $SAGE_SKIP_OPTIONS_MESSAGE to something nonempty. 

Running "sage" (with no arguments) would not trigger this message. (Perhaps we could only turn this on in prerelease (alpha and rc) versions? Alternatively, a change like this could go with the version 5.0 release.)

• after a while, we switch this to

    Warning: Using old-style Sage command-line options. 

    Sage is changing to use conventional GNU/Posix-style command-line options 
    (for example, 'sage --notebook instead of 'sage -notebook).  This change will 
    become the default soon.  Meanwhile, to use this new style (and therefore 
    to avoid seeing this message), set the environment variable 
    $SAGE_NEW_OPTIONS to something nonempty. 

perhaps with no easy way of disabling this message while using old-style options.

• finally, we turn on the new options, perhaps with an environment variable $SAGE_OLD_OPTIONS to use the old ones, with the understanding that any changes in command-line options may not be maintained for the old version.

See sage-devel for some discussion.

[dimpase]

Comment by jhpalmieri created at 2010-03-19 15:50:37

Changing status from new to needs_review.

[dimpase]

Comment by jhpalmieri created at 2010-03-19 15:50:47

Changing priority from minor to critical.

[dimpase]

Comment by jhpalmieri created at 2010-03-19 20:25:24

I've marked this as "needs review", but it might need work. In the previously cited thread from sage-devel, there was the following suggestion:

Another possibility might be to first check for "--gp", "--gap", etc., 
and do those before doing the general option parsing.   I.e., just do 
what you already planned, but with one optimization to deal with this 
use case. 

This is to speed up access to these programs: do a check like this in a shell script, and then pass the rest of the arguments to Python's optparse using the script included in this patch, or one like it. Then you avoid the slight delay involved in starting up Python if you want to run "gp". It would be nice to have a shell script which had a list of strings "gp", "gap", etc., checked to see if the first(?) argument was "--STR" for STR in this list, and if so, run the appropriate program from SAGE_LOCAL/lib, passing the rest of the line as arguments. Having one list containing all of these strings would make it easy to customize.

[dimpase]

Attachment sage by jhpalmieri created at 2010-03-19 21:15:39

the file SAGE_ROOT/sage

[dimpase]

Attachment sage.diff by jhpalmieri created at 2010-03-19 21:22:45

Replying to [comment:10 jhpalmieri]:

I've marked this as "needs review", but it might need work. In the previously cited thread from sage-devel, there was the following suggestion:

Another possibility might be to first check for "--gp", "--gap", etc., 
and do those before doing the general option parsing.   I.e., just do 
what you already planned, but with one optimization to deal with this 
use case. 

Okay, here's a new version which does this: it adds a file sage-sage-quickstart which gets run first, implementing the above idea. Then if SAGE_NEW_OPTIONS is nonempty, it calls sage-sage.py, the Python/optparse version with GNU/Posix standard command-line options. Otherwise, it calls the old parser sage-sage.

For the record, the commands in sage-sage-quickstart are: axiom, ecl/lisp, gap, gp, hg, ipython, maxima, mwrank, python, R, singular. Are any others particularly sensitive to startup times? (Using python adds something less than .1 second on my two-year old iMac, so we're not talking about a lot of time, in any case.)

[dimpase]

Comment by ohanar created at 2013-01-25 02:33:20

Replying to [comment:45 kini]:

Right, we might have a very thin bash wrapper that loads sage-env (which will be a bash script) before the main Python script. Or, since sage-env ideally should just set up environment variables and do nothing else (right?), we could turn it into a config file that was read independently by sage-sh and by sage. This would also allow us to rely less on environment variables for random things seemingly unrelated to the shell.

Are you saying that a configuration file should store the current environment? And that anytime it is changed (such as if the root directory of sage is moved) that this should be updated?

The problem with having two-pass argument parsing is that it separates the processing of arguments into multiple areas, making the architecture of the startup process needlessly complex. It is also pretty ugly to actually do this in the standard option parsing way because either you start to want to enforce arbitrary argument orders like we currently do (sage -tp works and sage -pt doesn't, sage -br works and sage -rb doesn't, etc.), or now the bash script needs to basically reimplement optparse/argparse in bash in order to correctly read the flags it's looking for.

Also

• argparse/optparse handles help functionality (consistent formatting), so all pre-parsed commands would still need stubs in argparse/optparse (and -1 for code duplication)
• argparse (and maybe optparse) matches subcommands so long as they are not ambiguous. So if (for instance) sage only had the subcommands foo and bar then sage f [args] would be expanded to sage foo [args]. (this is fairly standard for software with subcommands) This functionality would be inconsistent if there were any pre-parsed commands.

In any case, if I as a new Sage developer want to know or modify what option --foo does, there should be one obvious place to look for it. Making sage-sh parse arguments also means that we are shadowing arguments that could be passed on to the shell, etc. etc. Splitting argument parsing into two places is just generally a nasty design IMHO.
+1

Why does part of the startup need to be bash, other than because of sage-env?

For one of two reasons:

• python may not be in PATH because python is not currently a dependency
• even if python is in PATH, sage may not work with the default python

[dimpase]

Comment by kini created at 2013-01-25 07:10:02

Replying to [comment:46 ohanar]:

Replying to [comment:45 kini]:

Right, we might have a very thin bash wrapper that loads sage-env (which will be a bash script) before the main Python script. Or, since sage-env ideally should just set up environment variables and do nothing else (right?), we could turn it into a config file that was read independently by sage-sh and by sage. This would also allow us to rely less on environment variables for random things seemingly unrelated to the shell.

Are you saying that a configuration file should store the current environment? And that anytime it is changed (such as if the root directory of sage is moved) that this should be updated?

It should store the current startup environment. It would change if the root directory of Sage is moved, for example, yes. But if a user decided to change an environment variable in a Sage session with os.environ that wouldn't become reflected in the file, of course.

Why does part of the startup need to be bash, other than because of sage-env?

For one of two reasons:

• python may not be in PATH because python is not currently a dependency
• even if python is in PATH, sage may not work with the default python

Oh, right, of course. So then yes, sage should be bootstrapped in the way you described [comment:43 above].

[dimpase]

Comment by jdemeyer created at 2013-01-25 07:35:11

Replying to [comment:45 kini]:

Or, since sage-env ideally should just set up environment variables and do nothing else
Well, some of these environment variables are conditional, so it's not that easy to have a file which works both from bash and from Python. And I certainly don't see it as a problem that sage-env remains in bash as it is now.

The problem with having two-pass argument parsing is that it separates the processing of arguments into multiple areas, making the architecture of the startup process needlessly complex. It is also pretty ugly to actually do this in the standard option parsing way because either you start to want to enforce arbitrary argument orders like we currently do (sage -tp works and sage -pt doesn't, sage -br works and sage -rb doesn't, etc.), or now the bash script needs to basically reimplement optparse/argparse in bash in order to correctly read the flags it's looking for.
All these arguments are essentially irrelevant if the first pass needs to support just very few options. Things like ./sage -tp or ./sage -pt would be handled anyway by the argparse script.

Of course it's bad design to have two-pass argument parsing, but it would be so nice to keep ./sage -i and ./sage --sh working.

Why does part of the startup need to be bash, other than because of sage-env?
I think sage-env is the main reason.

[dimpase]

Comment by kcrisman created at 2014-11-20 13:33:25

The command line interface continues to evolve; can someone (who cares) give a summary of what still would be needed? comment:24 still hasn't been resolved, and comment:33 (as well as quick, non-Python-starting, use of sage -maxima and friends) seems quite relevant.

[dimpase]

Comment by jdemeyer created at 2015-06-23 13:45:35

Changing component from interfaces to user interface.

[dimpase]

Comment by embray created at 2017-06-01 14:55:00

Still reading up on this ticket, and don't have any comments to add yet to the existing discussion. But one question I have in general: Is anyone opposed at all to the idea of creating a new sub-command based interface, more like git, than the slightly unusual interface that uses single-character flags for subcommands? E.g. replace sage -t with sage test. Yes, it's more to type, but only by two characters, and is much less unusual. The old -t could still be supported very easily for backwards compatibility, but perhaps with a deprecation warning.

[dimpase]

Comment by jhpalmieri created at 2017-06-01 17:32:59

William Stein and I were just talking about this idea yesterday. Something like this?

sage FILE.[sage|py|spyx]

sage help
sage help --advanced?

sage -c <CMD>

sage package config
sage package name <TARBALL>
sage package list
sage package list standard
sage package list optional
sage package list experimental
sage package apropos
sage package download
sage package update
sage package fix-checksum
sage package create

sage install <PKGS>
  options: [-f, --force] [-c, --check] [-d, --download] [-s, --save]
   [-y, --yes] [-n, --no] [--no-dependency] [-i, --info]

sage pip

sage gap
sage gap3
sage gp
sage maxima
sage python
sage python3
sage ipython
sage ipython3 (not yet implemented)
sage R
sage singular
sage git
sage cython
sage cleaner
sage ecl
sage gdb
sage kash
sage lisp
sage M2
sage mwrank
sage polymake
sage scons
sage sqlite3
sage twistd

sage shell

sage notebook=[default|ssagenb|jupyter|export|jupyterlab|ipython]
  options: --log=...

sage notebook rst2ipynb ...
sage notebook rst2txt ...
sage notebook rst2sws ...
sage notebook sws2rst ...

sage test FILES
sage test --all
  options: --long, --verbose, --optional, --sagenb, --help,
  [-p|--parallel], --randorder[=seed], --new, --initial, --debug,
  --failed, --warn-long [timeout]

sage preparse <FILE>
sage startuptime
sage coverage [-a, --all]
sage search? search_src? search_doc? grep? grep_doc?

sage sdist
sage valgrind [--cacherind] [--callgrind] [--massif] [--memcheck] [--omega]

sage docbuild
  options: (run sage --docbuild --help to see)

sage --nodotsage
sage --root
sage -q
sage --min
sage [-v, --version]
sage dumpversion?

sage fixdoctests ...

sage build
sage build --force
sage build test?  (currently sage -bt ...)
sage build run?  (currently sage -br)

Maybe some of these can be removed. Maybe some can be consolidated: do we need separate commands for gap, gp, maxima, ecl, R, etc., or can they be combined under a single command, like "sage run "? There is endless bikeshedding available.

[dimpase]

Comment by embray created at 2017-06-02 08:22:20

Yes, something quite like that.

And I was thinking of writing up some kind of declarative list(s) of subcommands. In particular I was thinking two separate lists:

1. One list of sage-specific sub-commands (such as sage package in your example above), which would automatically be translated to running individual scripts that implement them that would be named sage-<subcommand>. This is mostly how git works as well.

2. One list of programs installed in the Sage distribution (sage sh, sage gap, etc.) that can be launched from the interface. In principle one could make this automatic but I think it's better to have a hard-coded list. I don't think a sage run is really necessary. sage sh <whatever> is essentially the same as this, but I think it's still convenient to have shortcuts for common programs included in the Sage distribution.

Although somewhat redundant, because it's common I would also have sage --help as an alias for sage help and sage --help-advanced for sage help --advanced, though one could bikeshed about whether those should do the exact same thing or not.

I might also hide more of the development-specific commands (sage coverage, sage startuptime, etc.) behind a sub-command.

[dimpase]

Comment by embray created at 2017-06-02 08:23:08

(I should add, that's a very nice mock-up of what such an interface would look like, so thank you for that.)

[dimpase]

Comment by jdemeyer created at 2017-06-02 13:01:28

Replying to [comment:55 embray]:

E.g. replace sage -t with sage test.

What about sage -btp? I use that all the time. I would hate it if that would become
sage buildtest -p or worse, make build && sage test -p.

[dimpase]

Comment by jdemeyer created at 2017-06-02 13:12:10

We should also think to what extent the build system should be exposed under the sage command. For example, we now have

make FOO         # Build dependencies of FOO + FOO
sage -i FOO      # Build toolchain + dependencies of FOO + FOO
sage -f FOO      # Build toolchain + dependencies of FOO + *rebuild* FOO
sage -p FOO      # Build FOO *without* dependencies

and

make sagelib     # Build Sage library with dependencies
sage -i sagelib  # Build Sage library with toolchain and dependencies
sage -b          # Build Sage library *without* dependencies
sage -f sagelib  # Rebuild all of the Sage library with toolchain and dependencies
sage -ba         # Rebuild all of the Sage library *without* dependencies

This is all for historical and accidental reasons, but this ticket should clean that up too.

[dimpase]

Comment by jdemeyer created at 2017-06-02 13:13:22

Needless to say, many people don't even know the subtle differences between the above commands.

[dimpase]

Comment by embray created at 2017-06-06 09:42:41

Replying to [comment:59 jdemeyer]:

Replying to [comment:55 embray]:

E.g. replace sage -t with sage test.

What about sage -btp? I use that all the time. I would hate it if that would become
sage buildtest -p or worse, make build && sage test -p.

I was actually thinking of allowing subcommands to be chained, like in setup.py. So sage build test, where each can take optional flags if desired.

[dimpase]

Comment by mkoeppe created at 2020-08-11 19:29:26

Strong -1 "wontfix" for the idea of making an incompatible change toward using "subcommands" after 14 years of the existence of the sage script.

[dimpase]

Comment by embray created at 2020-08-31 15:42:10

I think the current interface of the sage script is pretty clumsy and dated by modern standards. It can and should be made more user-friendly. Just as one example of an advantage of subcommands is it makes the help documentation vastly more digestible. I wouldn't propose changing it without maintaining backwards compatibility though.

[dimpase]

Comment by @williamstein created at 2020-08-31 17:58:19

+1 from me to changing the sage script to support subcommands, especially if we can somehow do it in a way that preserves compatibility with the current parsing. I wonder to what extent the following is possible:

• 1. run the current shell script and if it "works" then done (with maybe some slight tightening)
• 2. parse using subcommands.

Or something else, e.g,. if you do "sage [explicit list of subcommands]" uses the subcommand approach; otherwise, fall back to the current parser.

I wrote at least the first version of the current sage command line parser, and frankly I didn't know what I was doing at the time, and just sort of stupidly copied random bits and pieces of design from programs I had used. Using python's subcommands support is a lot more systematic, and can also result in very nice modular code.

[dimpase]

Comment by embray created at 2020-09-01 13:32:32

Since all of the sage script's current "subcommands" (e.g. sage -b, sage -i) all start with hyphens, and new subcommands with be non-hyphenated, I think it would work to support both without too much ambiguity but I'd be interested in a counter-example. sage <filename.{py,sage}> would still work since no sub-command would be confused with a runnable script filename.

[dimpase]

Comment by embray created at 2020-09-01 13:37:14

Replying to [comment:68 was]:

Using python's subcommands support is a lot more systematic, and can also result in very nice modular code.

I might still just write it as a shell script. Reason being, based on my experience implementing CLIs in Python, it tends to be much much slower to run a single command. At the very least I would do this for the top-level sage script. Most subcommands would delegate to another program which might be another shell script, or could be written in Python (as is already the case). For subcommands written in Python it's not always so bad as long as most operations you would do with that command are long enough to make the Python interpreter startup time negligible. Those could also have further subcommands.

[dimpase]

Comment by @williamstein created at 2020-09-01 13:39:59

I might still just write it as a shell script.

+1

I should have just said "in my experience, structuring command line parsing code as subcommands (implemented in any language) can result in very nice modular code."

[dimpase]

Comment by mkoeppe created at 2020-09-03 02:52:21

Changing priority from critical to minor.

[dimpase]

Comment by nbruin created at 2020-09-04 16:17:04

Replying to [comment:68 was]:

+1 from me to changing the sage script to support subcommands,

One very frustrating part of subcommands is the failure of standard tab-completion to work with it:
for instance, for jupyter notebook sheet.ipynb. If there were just a command jupyter-notebook, it would be much easier and faster to type. Particularly with jupyter, which you nearly always use to start its notebook, it's rather frustrating. For git somehow it feels a little more natural, probably because there is naturally a larger variety of actions you want to take through it. It also helps that most preconfigured bash tabcompletions are aware of git subcommands. (although still, distinct commands git-push, git-branch, git-pull would be faster)

[dimpase]

Comment by jhpalmieri created at 2020-09-04 16:27:21

Replying to [comment:73 nbruin]:

Replying to [comment:68 was]:

+1 from me to changing the sage script to support subcommands,

One very frustrating part of subcommands is the failure of standard tab-completion to work with it:

The question so far has been (for example) sage --notebook vs. sage notebook, and tab-completion won't work with either. Are you suggesting adding sage-notebook and other scripts?

[dimpase]

Comment by nbruin created at 2020-09-04 16:48:07

Replying to [comment:74 jhpalmieri]:

The question so far has been (for example) sage --notebook vs. sage notebook, and tab-completion won't work with either. Are you suggesting adding sage-notebook and other scripts?

From a tab-completion point of view that would make sense, yes. (that, or learn how to extend the tab completion patterns). For this particular example, I'd use jupyter notebook anyway, with the sage kernel installed in the system jupyter server.

The traditional reason for having dashes in front of options/subcommands is to remove ambiguity from sage notebook (to run the file notebook) and sage notebook (to start the notebook). For that reason, I think we can only have subcommands for sage if sage <file> would have no meaning. I don't think we can discard this main function of sage. I think with jupyter, where there is one VERY common use, it's already a mistake to go with a subcommand design.

Extrapolating from that, I think that using subcommands for the sage script is also the wrong fit. It works well with git, but users really interact differently with git.

[dimpase]

Comment by mkoeppe created at 2020-09-04 17:48:39

Replying to [comment:75 nbruin]:

The traditional reason for having dashes in front of options/subcommands is to remove ambiguity from sage notebook (to run the file notebook) and sage notebook (to start the notebook). For that reason, I think we can only have subcommands for sage if sage <file> would have no meaning. I don't think we can discard this main function of sage. I think with jupyter, where there is one VERY common use, it's already a mistake to go with a subcommand design.

Extrapolating from that, I think that using subcommands for the sage script is also the wrong fit. It works well with git, but users really interact differently with git.

I fully agree.

[dimpase]

Comment by kini created at 2020-09-04 18:00:40

Replying to [comment:75 nbruin]:

(that, or learn how to extend the tab completion patterns).

FWIW, there are tools which can do this for you if your command line subcommands and options are all handled by argparse (rather than e.g. a top level bash script that calls out to python programs that use argparse for each subcommand, as embray suggested).

argcomplete will dynamically provide on-the-fly completion candidates by actually running the argument parsing logic from the sage program every time you hit tab in the shell. This is always accurate but could be slow if sage takes a long time to get to the line of code where the argument parser is run (e.g. if it has some heavy imports).

shtab also runs the argument parsing logic from the sage program, but it statically generates a bash completion script which you can then register with bash-completion by putting it in a relevant place (if you don't have root access, this can be ~/.local/share/bash-completion/completions/). Then when you press tab in the shell, completions should be pretty instantaneous, but the completion script needs to be kept up to date with the command line interface of sage.

If you use a shell other than bash, it may be harder. zsh, at least, is supported by shtab and to some extent argcomplete as well.

I suggest that shtab be run as part of the build process of Sage and that the resulting bash completion script be installed as part of the installation process. That should give out-of-the-box completion functionality to the majority of users, which would be nice. (Again, though, this would only work if the top-level sage does argument parsing in Python with argparse, and I understand that might not end up being the case for other reasons.)