New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Help needed] Pass config to custom executor operator #5054
Comments
what information are you trying to pass through component_config? |
I want to pass some strings which helps define the specifications of the distributed computing cluster I want to spawn. You think the type of information is relevant ? |
Also to be more precise, I would like to use my custom executor operator with the KubeflowDagRunner. I seem to have an additional issue which is to make aware the containerEntrypoint of my new executor operator. As the parameter a. All first-party ExecutorOperators developed by TFX team should already be in the DEFAULT_EXECUTOR_OPERATORS dictionary in the Launcher module. b. For custom ExecutorOperator not in above dict, please inject it into the launcher constructor via the custom_executor_operators argument. But I don't see how I am suppose to either update the DEFAULT_EXECUTOR_OPERATORS dictionary or the other solution. |
I think in the end my issue is with the container_entrypoint.py for kubeflow orchestrated pipelines as it does not expose the parameters The Is this the right place to discuss this ? Should I be opening an issue/feature request regarding kubeflow container entrypoint file ? Thank you in advance for your time. |
Hello ! I am currently trying to migrate from a custom launcher to a custom executor operator as shown here: Migrate from custom Launcher to custom ExecutorOperator but I have some trouble understanding how to replicate the behavior offered by the previous
component_config
argument of the Launcher class. This allowed us to pass some very useful configuration. I see in the example given in the readme, in the DockerExecutorOperator, that adocker_config
is initialized but no argument can be passed this far it seems to me.Any help would be appreciated as this subject is quite complex I might have missed something. To give a bit more context, the goal here of the custom executor operator will be to spawn a compute cluster to use for the beam component computations then to tear it down in order to better manage ressources. So the new CustomExecutorOperator is inheriting from the
BeamExecutorOperator
.Thank you in advance !
The text was updated successfully, but these errors were encountered: