several updates to config and environment settings

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

CHANGELOG.md

@@ -14,6 +14,10 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.scom/singularityhub/singularity-hpc/tree/master) (0.0.x)
+ - Allow environment variables in settings (0.0.29)
+   - User settings file creation and use with shpc config inituser
+   - registry is now a list to support multiple registry locations
+   - config supports add/remove to append/delete from list
  - Add test for docker and podman (0.0.28)
    - namespace as format string for command named renamed to repository
    - shpc test/uninstall should be run for all tests

docs/getting_started/user-guide.rst

@@ -99,15 +99,27 @@ Setup
 =====
 
 Setup includes, after installation, editing any configuration values to
-customize your install. The defaults are likely suitable for most.
-For any configuration value that you might set, the following variables
-are available to you:
+customize your install. The configuration file will default to ``shpc/settings.yml``
+in the installed module, however you can create your own user settings file to
+take preference over this one as follows:
+
+.. code-block:: console
+
+    $ shpc config userinit
+
+
+The defaults in either file are likely suitable for most. For any configuration value 
+that you might set, the following variables are available to you:
 
  - ``$install_dir``: the shpc folder
  - ``$root_dir``: the parent directory of shpc (where this README.md is located)
 
 
-A summary table of variables is included below, and then further discussed in detail.
+Additionally, the variables ``module_base``, ``container_base``, and ``registry``
+can be set with environment variables that will be expanded at runtime. You cannot
+use the protected set of substitution variables (``$install_dir`` and ``$install_root``)
+as environment variables, as they will be subbed in by shpc before environment
+variable replacement. A summary table of variables is included below, and then further discussed in detail.
 
 
 .. list-table:: Title
@@ -121,8 +133,8 @@ A summary table of variables is included below, and then further discussed in de
      - Set a default module system. Currently lmod and tcl are supported
      - [lmod, tcl]
    * - registry
-     - The full path to the registry folder (with subfolders with container.yaml recipes)
-     - $root_dir/registry
+     - A list of full paths to one or more registry folders (with subfolders with container.yaml recipes)
+     - [$root_dir/registry]
    * - module_base
      - The install directory for modules. Defaults to the install directory/modules
      - $root_dir/modules
@@ -271,8 +283,10 @@ directory.
 Registry
 --------
 
-The registry folder in the root of the repository, but you can change it to
-be a custom one with the config variable ``registry``
+The registry parameter is a list of one or more registry locations (filesystem
+directories) where shpc will search for ``container.yaml`` files. The default
+registry shipped with shpc is the folder in the root of the repository, but 
+you can add or remove entries via the config variable ``registry``
 
 
 .. code-block:: console
@@ -409,6 +423,8 @@ file directly, or you can use ``shpc config``, which will accept:
 
  - set to set a parameter and value
  - get to get a parameter by name
+ - add to add a value to a parameter that is a list (e.g., registry)
+ - remove to remove a value from a parameter that is a list
 
 The following example shows changing the default module_base path from the install directory modules folder.
 
@@ -428,6 +444,14 @@ And then to get values:
     $ shpc config get module_base
 
 
+And to add and remove a value to a list:
+
+.. code-block::console
+
+    $ shpc config add registry:/tmp/registry
+    $ shpc config remove registry:/tmp/registry
+
+
 You can also open the config in the editor defined in settings at ``config_editor``
 
 .. code-block:: console

docs/index.rst

@@ -20,11 +20,11 @@ And container technologies:
 
  - `Singularity <https://github.com/sylabs/singularity>`_
  - `Podman <https://podman.io>`_
+ - `Docker <https://www.docker.com/>`_
 
 And coming soon:
 
  - `Shifter <https://github.com/NERSC/shifter>`_
- - `Docker <https://www.docker.com/>`_
  - `Sarus <https://github.com/eth-cscs/sarus>`_
 
 

shpc/client/__init__.py

@@ -145,7 +145,13 @@ def get_parser():
     config.add_argument(
         "params",
         nargs="*",
-        help="Set or get a config value, or edit the config.\nshpc config set key:value\nshpc config get key\nshpc edit",
+        help="""Set or get a config value, edit the config, add or remove a list variable, or create a user-specific config.
+shpc config set key:value
+shpc config get key
+shpc edit
+shpc config inituser
+shpc config add registry:/tmp/registry
+shpc config remove registry:/tmp/registry""",
         type=str,
     )
     # Generate markdown docs for a container registry entry

shpc/client/config.py

@@ -24,9 +24,11 @@ def main(args, parser, extra, subparser):
     )
 
     # For each new setting, update and save!
+    if command == "inituser":
+        return cli.settings.inituser()
     if command == "edit":
         return cli.settings.edit()
-    elif command == "set":
+    elif command in ["set", "add", "remove"]:
         for param in args.params:
             if ":" not in param:
                 logger.warning(
@@ -35,8 +37,15 @@ def main(args, parser, extra, subparser):
                 )
                 continue
             key, value = param.split(":", 1)
-            cli.settings.set(key, value)
-            logger.info("Updated %s to be %s" % (key, value))
+            if command == "set":
+                cli.settings.set(key, value)
+                logger.info("Updated %s to be %s" % (key, value))
+            elif command == "add":
+                cli.settings.add(key, value)
+                logger.info("Added %s to %s" % (key, value))
+            elif command == "remove":
+                cli.settings.remove(key, value)
+                logger.info("Removed %s from %s" % (key, value))
 
         # Save settings
         cli.settings.save()

shpc/defaults.py

@@ -12,6 +12,14 @@
 # The default settings file in the install root
 default_settings_file = os.path.join(reps["$install_dir"], "settings.yml")
 
+# The user settings file can be created to over-ride default
+user_settings_file = os.path.join(
+    os.path.expanduser("~/.singularity-hpc"), "settings.yml"
+)
+
+# variables in settings that allow environment variable expansion
+allowed_envars = ["container_base", "module_base", "registry"]
+
 # The GitHub repository with recipes
 github_url = "https://github.com/singularityhub/singularity-hpc"
 

shpc/main/__init__.py

@@ -7,6 +7,7 @@
 
 import shpc.utils
 from shpc.logger import logger
+import shpc.defaults
 
 
 def get_client(quiet=False, **kwargs):

shpc/main/client.py

@@ -86,6 +86,18 @@ def add_namespace(self, name):
             name = "%s/%s" % (self.settings.namespace.strip("/"), name)
         return name
 
+    def load_registry_config(self, name):
+        """
+        Given an identifier, find the first match in the registry.
+        """
+        for registry, fullpath in self.container.iter_registry():
+            package_dir = os.path.join(registry, name)
+            package_file = os.path.join(package_dir, "container.yaml")
+            if package_file == fullpath:
+                return container.ContainerConfig(package_file)
+
+        logger.exit("%s is not a known recipe in any registry." % name)
+
     def _load_container(self, name, tag=None):
         """
         Given a name and an optional tag to default to, load a package
@@ -94,12 +106,8 @@ def _load_container(self, name, tag=None):
         if ":" in name:
             name, tag = name.split(":", 1)
 
-        # The recipe folder must exist in the registry
-        package_dir = os.path.join(self.settings.registry, name)
-        package_file = os.path.join(package_dir, "container.yaml")
-        config = container.ContainerConfig(package_file)
-
         # If the user provides a tag, set it
+        config = self.load_registry_config(name)
         config.set_tag(tag)
         return config
 
@@ -205,12 +213,10 @@ def show(self, name, names_only=False, out=None, filter_string=None):
             out = out or sys.stdout
 
             # List the known registry modules
-            for fullpath in utils.recursive_find(self.settings.registry):
+            for registry, fullpath in self.container.iter_registry():
                 if fullpath.endswith("container.yaml"):
                     module_name = (
-                        os.path.dirname(fullpath)
-                        .replace(self.settings.registry, "")
-                        .strip(os.sep)
+                        os.path.dirname(fullpath).replace(registry, "").strip(os.sep)
                     )
 
                     # If the user has provided a filter, honor it

shpc/main/container/base.py

@@ -90,6 +90,14 @@ def container_dir(self, name):
             return os.path.join(self.settings.module_base, name)
         return os.path.join(self.settings.container_base, name)
 
+    def iter_registry(self):
+        """
+        Iterate over known registries defined in settings.
+        """
+        for registry in self.settings.registry:
+            for filename in shpc.utils.recursive_find(registry):
+                yield registry, filename
+
     def guess_tag(self, module_name, allow_fail=False):
         """
         If a user asks for a name without a tag, try to figure it out.

shpc/main/container/singularity.py

@@ -71,21 +71,20 @@ def add(self, sif, module_name, modulefile, template, **kwargs):
         """
         Manually add a registry container.
         """
-        registry_dir = self.settings.registry
-
         # Ensure the container exists
         sif = os.path.abspath(sif)
         if not os.path.exists(sif):
             logger.exit("%s does not exist." % sif)
 
         # First ensure that we aren't using a known namespace
-        for subfolder in module_name.split("/"):
-            registry_dir = os.path.join(registry_dir, subfolder)
-            if os.path.exists(registry_dir):
-                logger.exit(
-                    "%s is a known registry namespace, choose another for a custom addition."
-                    % subfolder
-                )
+        for registry_dir, _ in self.iter_registry():
+            for subfolder in module_name.split("/"):
+                registry_dir = os.path.join(registry_dir, subfolder)
+                if os.path.exists(registry_dir):
+                    logger.exit(
+                        "%s is a known registry namespace, choose another for a custom addition."
+                        % subfolder
+                    )
 
         # The user can have a different container directory defined
         container_dir = self.container_dir(module_name)

shpc/main/schemas.py

@@ -107,9 +107,8 @@
 shells = ["/bin/bash", "/bin/sh", "/bin/csh"]
 
 # Currently all of these are required
-
 settingsProperties = {
-    "registry": {"type": "string"},
+    "registry": {"type": "array", "items": {"type": "string"}},
     "module_base": {"type": "string"},
     "container_base": {"type": ["string", "null"]},
     "namespace": {"type": ["string", "null"]},