diff --git a/docs/source/fundamentals/data_passing.rst b/docs/source/fundamentals/data_passing.rst index 4886b6ecc3..78540fdc45 100644 --- a/docs/source/fundamentals/data_passing.rst +++ b/docs/source/fundamentals/data_passing.rst @@ -147,3 +147,70 @@ Having the above order of connections, step 3's ``input_data`` becomes: } Top-to-bottom in the visual editor corresponds to left-to-right in ``unnamed``. + +.. _r: + +R example +--------- + +.. tip:: + 👉 Import the `example project + `_ showcasing **R** straight + in Orchest (:ref:`how to import a project `). + +The Orchest SDK in **R** works through the `reticulate `_ +package. To explain its usage, an example project is provided below. + +First, create an Orchest environment which uses the ``orchest/base-kernel-r`` base image (you can +find more details :ref:`here `). Next you want to install ``reticulate`` and configure +access to Python and the Orchest SDK. You can do so by having a script (let's say ``Install.r``) in +your project with the following content: + +.. code-block:: r + + install.packages("reticulate", repos = "http://cran.us.r-project.org") + library(reticulate) + + # Dynamically find system Python + python_path <- system("which python", intern=TRUE) + use_python(python_path) + + # Pre compile orchest deps + orchest <- import("orchest") + + print(orchest) + +and having the environment set-up script perform ``Rscript Install.r``. You will then be able to +access the Orchest SDK through R **in every step that makes use of this environment** . To do data +passing, for example, you would do the following: + +.. code-block:: r + + library(reticulate); + python_path <- system("which python", intern=TRUE); + use_python(python_path); + orchest <- import("orchest"); + orchest$transfer$output(2, name="Test"); + +In a child step you will be able to retrieve the output: + +.. code-block:: r + + library(reticulate); + python_path <- system("which python", intern=TRUE); + use_python(python_path); + orchest <- import("orchest") + step_inputs = orchest$transfer$get_inputs() + step_inputs$Test + +Julia example +------------- + +Refer to the `Julia example project `_ +showcasing **Julia** in Orchest (:ref:`how to import a project `). + +JavaScript example +------------------ + +Refer to the `JavaScript example project `_ +showcasing **JavaScript** in Orchest (:ref:`how to import a project `). diff --git a/docs/source/fundamentals/sdk/index.rst b/docs/source/fundamentals/sdk/index.rst deleted file mode 100644 index 0f10818c79..0000000000 --- a/docs/source/fundamentals/sdk/index.rst +++ /dev/null @@ -1,114 +0,0 @@ -.. _orchest sdk: - -Orchest SDK -=========== - -.. note:: - 💡 The Orchest SDK comes pre-installed when using Orchest. - -Python ------- - -Data passing -~~~~~~~~~~~~ -Please refer to the dedicated section on :ref:`data passing `. - -.. _sdk-quickstart-parameters: - -Parameters -~~~~~~~~~~ -Check out an example in the :ref:`parameters section in jobs `. - -.. _r: - -R -- -.. tip:: - 👉 Import the `example project - `_ showcasing **R** straight - in Orchest (:ref:`how to import a project `). - -The Orchest SDK in **R** works through the `reticulate `_ -package. To explain its usage, an example project is provided below. - -First, create an Orchest environment which uses the ``orchest/base-kernel-r`` base image (you can -find more details :ref:`here `). Next you want to install ``reticulate`` and configure -access to Python and the Orchest SDK. You can do so by having a script (let's say ``Install.r``) in -your project with the following content: - -.. code-block:: r - - install.packages("reticulate", repos = "http://cran.us.r-project.org") - library(reticulate) - - # Dynamically find system Python - python_path <- system("which python", intern=TRUE) - use_python(python_path) - - # Pre compile orchest deps - orchest <- import("orchest") - - print(orchest) - -and having the environment set-up script perform ``Rscript Install.r``. You will then be able to -access the Orchest SDK through R **in every step that makes use of this environment** . To do data -passing, for example, you would do the following: - -.. code-block:: r - - library(reticulate); - python_path <- system("which python", intern=TRUE); - use_python(python_path); - orchest <- import("orchest"); - orchest$transfer$output(2, name="Test"); - -In a child step you will be able to retrieve the output: - -.. code-block:: r - - library(reticulate); - python_path <- system("which python", intern=TRUE); - use_python(python_path); - orchest <- import("orchest") - step_inputs = orchest$transfer$get_inputs() - step_inputs$Test - -Julia ------ -Refer to the `Julia example project `_ -showcasing **Julia** in Orchest (:ref:`how to import a project `). - - -JavaScript ----------- -Refer to the `JavaScript example project `_ -showcasing **JavaScript** in Orchest (:ref:`how to import a project `). - - -API reference -------------- - -.. _api transfer: - -orchest.transfer -~~~~~~~~~~~~~~~~ - -.. automodule:: orchest.transfer - :members: - :exclude-members: output_to_memory - -.. _api parameters: - -orchest.parameters -~~~~~~~~~~~~~~~~~~ - -.. automodule:: orchest.parameters - :members: - -.. _api services: - -orchest.services -~~~~~~~~~~~~~~~~~~ - -.. automodule:: orchest.services - :members: diff --git a/docs/source/getting_started/how_to.md b/docs/source/getting_started/faq.md similarity index 61% rename from docs/source/getting_started/how_to.md rename to docs/source/getting_started/faq.md index 7016bd906d..ba4e3ad5d9 100644 --- a/docs/source/getting_started/how_to.md +++ b/docs/source/getting_started/faq.md @@ -1,10 +1,6 @@ -# How to... +# Frequently Asked Questions (FAQ) -## Pass data between pipeline steps - -Please refer to the dedicated section on {ref}`data passing `. - -## Install new packages +## How to install new packages? ```{tip} 👉 Would you rather watch a short video tutorial? Check it our here: [installing additional @@ -27,17 +23,9 @@ Do **not** install new packages by running bash commands inside the Notebooks. T packages to be installed every time you do a pipeline run, since the state of the kernel environment is ephemeral. -## Use `git` inside Orchest - -Please refer to the dedicated section on {ref}`using git inside Orchest `. - (how-to-import-a-project)= -## Import a project - -Check out our video: [importing a project](https://www.tella.tv/video/cknr7of9c000409jr5gx4efjy/view). - -## Share code between steps +## How to share code between steps? ```{note} 💡 This approach also works to share code between pipelines. @@ -57,7 +45,7 @@ import utils utils.transform(...) ``` -## Minimize Orchest's disk size +## How to minimize Orchest's disk size? To keep Orchest's disk footprint to a minimal you can use the following best practices: @@ -70,101 +58,13 @@ To keep Orchest's disk footprint to a minimal you can use the following best pra number of pipeline runs and automatically delete the older ones. Steps: (1) edit an existing job or create a new one, (2) go to _pipeline runs_, and (3) select _auto clean-up_. -## Use a GPU in Orchest +## How to use a GPU in Orchest? Currently GPU support is not yet available. Coming soon! -## Use the Orchest CLI - -Below you will find the most important `orchest-cli` commands that you need to know (you can also get all this -information by running `orchest -h`): - -```sh -orchest start - -# Stop Orchest (shuts down Orchest completely). -orchest stop - -# Install Orchest (check out the dedicated `Installation` guide in -# the `Getting started` section). -orchest install - -# Update Orchest to a newer version (NOTE: this can also be done -# through the settings in the UI). -orchest update - -# Get extensive version information. Useful to see whether the -# installation was successful. -orchest version -``` - -## Use Orchest shortcuts like a pro - -### Command palette - -```{eval-rst} -.. list-table:: - :widths: 25 25 - :header-rows: 1 - :align: left - - * - Key(s) - - Action - - * - :kbd:`Control`/:kbd:`Command` + :kbd:`K` - - Open command palette - - * - :kbd:`↑`/:kbd:`↓` - - Navigate command palette commands - - * - :kbd:`PageUp`/:kbd:`PageDown` - - Navigate command palette commands - - * - :kbd:`Escape` - - Dismiss command palette -``` - -### Pipeline editor - -```{eval-rst} -.. list-table:: - :widths: 25 25 - :header-rows: 1 - :align: left - - * - Key(s) - - Action - - * - :kbd:`Space` + click + drag - - Pan canvas* - - * - :kbd:`Ctrl` + click - - Select multiple steps - - * - :kbd:`Ctrl` + :kbd:`A` - - Select all steps* - - * - :kbd:`Ctrl` + :kbd:`Enter` - - Run selected steps* - - * - :kbd:`H` - - Center view and reset zoom - - * - :kbd:`Escape` - - Deselect steps - - * - :kbd:`Delete`/:kbd:`Backspace` - - Delete selected step(s) - - * - Double click a step - - Open file in JupyterLab -``` - -\* Requires mouse to hover the canvas - (skip-notebook-cells)= -## Skip notebook cells +## How to skip notebook cells? Notebooks facilitate an experimental workflow, meaning that there will be cells that should not be run when executing the notebook (from top to bottom). Since {term}`pipeline runs ` @@ -180,7 +80,7 @@ To skip a cell during pipeline runs: The cells with the _skip_ tag are still runnable through JupyterLab, but when executing these notebooks as part of pipelines in Orchest they will not be run. -## Migrate to Kubernetes +## How to migrate to Kubernetes? The moment we have moved to a Kubernetes backed Orchest version (and deprecated the Docker based version), we will update this section of the documentation to include steps on how to migrate your diff --git a/docs/source/index.md b/docs/source/index.md index 94f4b36190..45df8c9127 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -38,7 +38,6 @@ getting_started/installation getting_started/quickstart getting_started/when_to_use_orchest -getting_started/how_to ``` ```{toctree} @@ -54,9 +53,19 @@ fundamentals/notifications fundamentals/environments fundamentals/environment_variables fundamentals/services -fundamentals/sdk/index fundamentals/settings fundamentals/glossary +getting_started/faq +``` + +```{toctree} +:caption: Reference +:hidden: true +:maxdepth: 1 + +reference/shortcuts +reference/sdk/index +reference/cli/index ``` ```{toctree} diff --git a/docs/source/reference/cli/index.md b/docs/source/reference/cli/index.md new file mode 100644 index 0000000000..b80bf932a3 --- /dev/null +++ b/docs/source/reference/cli/index.md @@ -0,0 +1,23 @@ +# Orchest CLI reference + +Below you will find the most important `orchest-cli` commands that you need to know (you can also get all this +information by running `orchest -h`): + +```sh +orchest start + +# Stop Orchest (shuts down Orchest completely). +orchest stop + +# Install Orchest (check out the dedicated `Installation` guide in +# the `Getting started` section). +orchest install + +# Update Orchest to a newer version (NOTE: this can also be done +# through the settings in the UI). +orchest update + +# Get extensive version information. Useful to see whether the +# installation was successful. +orchest version +``` diff --git a/docs/source/reference/sdk/index.rst b/docs/source/reference/sdk/index.rst new file mode 100644 index 0000000000..72e21db70d --- /dev/null +++ b/docs/source/reference/sdk/index.rst @@ -0,0 +1,34 @@ +.. _orchest sdk: + +Orchest SDK reference +===================== + +.. note:: + 💡 The Orchest SDK comes pre-installed when using Orchest. + +To see how to use the SDK, please refer to the dedicated section on :ref:`data passing `. + +.. _api transfer: + +orchest.transfer +~~~~~~~~~~~~~~~~ + +.. automodule:: orchest.transfer + :members: + :exclude-members: output_to_memory + +.. _api parameters: + +orchest.parameters +~~~~~~~~~~~~~~~~~~ + +.. automodule:: orchest.parameters + :members: + +.. _api services: + +orchest.services +~~~~~~~~~~~~~~~~~~ + +.. automodule:: orchest.services + :members: diff --git a/docs/source/reference/shortcuts.md b/docs/source/reference/shortcuts.md new file mode 100644 index 0000000000..8682fad436 --- /dev/null +++ b/docs/source/reference/shortcuts.md @@ -0,0 +1,63 @@ +# Keyboard shortcuts + +## Command palette + +```{eval-rst} +.. list-table:: + :widths: 25 25 + :header-rows: 1 + :align: left + + * - Key(s) + - Action + + * - :kbd:`Control`/:kbd:`Command` + :kbd:`K` + - Open command palette + + * - :kbd:`↑`/:kbd:`↓` + - Navigate command palette commands + + * - :kbd:`PageUp`/:kbd:`PageDown` + - Navigate command palette commands + + * - :kbd:`Escape` + - Dismiss command palette +``` + +## Pipeline editor + +```{eval-rst} +.. list-table:: + :widths: 25 25 + :header-rows: 1 + :align: left + + * - Key(s) + - Action + + * - :kbd:`Space` + click + drag + - Pan canvas* + + * - :kbd:`Ctrl` + click + - Select multiple steps + + * - :kbd:`Ctrl` + :kbd:`A` + - Select all steps* + + * - :kbd:`Ctrl` + :kbd:`Enter` + - Run selected steps* + + * - :kbd:`H` + - Center view and reset zoom + + * - :kbd:`Escape` + - Deselect steps + + * - :kbd:`Delete`/:kbd:`Backspace` + - Delete selected step(s) + + * - Double click a step + - Open file in JupyterLab +``` + +\* Requires mouse to hover the canvas