Skip to content

Commit

Permalink
Kubernetes support for Metaflow (#644)
Browse files Browse the repository at this point in the history
* Refactor @resources decorator

@resources decorator is shared by all compute related decorators -
@Batch, @lambda, @K8s, @titus. This patch moves it out of
batch_decorator.py so that other decorators can cleanly reference
it.

* Update __init__.py

* Refactor @Batch decorator

* more change

* more changes

* more changes

* @kubernetes

* Kubernetes

* More changes

* More changes

* more changes

* some more changes

* more changes

* add disk space

* Add todos

* some fixes

* add k8s testing context

* more changes

* some more changes

* minor fixups

* better error handling for evicted pods (#711)

* fixes for pod/job metadata race conditions (#704)

* K8S: label value sanitizer (#719)

* rename name_space to namespace for k8s plugin (#750)

* fix k8s attribute handling bug (#753)

* tweak k8s test resources (to run on kind) (#754)

* add k8s api retries (#756)

* update done marker

* Use linux binaries in @conda when run in k8s (#758)

Conda environment should pack linux python binary when run on MacOS
to avoid an error

metaflow_PlayListFlow_osx-64_179c56284704ca8e53622f848a3df27cdd1f4327/bin/python: cannot execute binary file: Exec format error

* fix comment

* fix merge conflict

* update char

Co-authored-by: Oleg Avdeev <oleg.v.avdeev@gmail.com>
Co-authored-by: Roman Kindruk <36699371+sappier@users.noreply.github.com>
  • Loading branch information
3 people committed Oct 15, 2021
1 parent e2aa6a3 commit f7d54d6
Show file tree
Hide file tree
Showing 14 changed files with 1,954 additions and 218 deletions.
4 changes: 4 additions & 0 deletions metaflow/plugins/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,11 +88,13 @@ def get_plugin_cli():
# Add new CLI commands in this list
from . import package_cli
from .aws.batch import batch_cli
from .aws.eks import kubernetes_cli
from .aws.step_functions import step_functions_cli

return _ext_plugins.get_plugin_cli() + [
package_cli.cli,
batch_cli.cli,
kubernetes_cli.cli,
step_functions_cli.cli]


Expand All @@ -113,6 +115,7 @@ def _merge_lists(base, overrides, attr):
from .retry_decorator import RetryDecorator
from .resources_decorator import ResourcesDecorator
from .aws.batch.batch_decorator import BatchDecorator
from .aws.eks.kubernetes_decorator import KubernetesDecorator
from .aws.step_functions.step_functions_decorator \
import StepFunctionsInternalDecorator
from .test_unbounded_foreach_decorator\
Expand All @@ -125,6 +128,7 @@ def _merge_lists(base, overrides, attr):
ResourcesDecorator,
RetryDecorator,
BatchDecorator,
KubernetesDecorator,
StepFunctionsInternalDecorator,
CondaStepDecorator,
InternalTestUnboundedForeachDecorator],
Expand Down

0 comments on commit f7d54d6

Please sign in to comment.