You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current notebook runs without issue and works as a starting point. For me (and I presume others) the next step is naturally to organize the code in a more production-like pipeline which means adapting the notebook and fitting it into something like the templates described in this guide.
However, if one adapts the recommender tutorial to run outside of InteractiveContext, then the code fails to run. In particular, the Channel's that we pass to the Trainer component are empty when the pipeline is run using LocalDagRunner. When the MovielensModel calls movies_uri.get()[0] in its constructor, the program will throw a RuntimeError because we are indexing into an empty list. This is in spite of the fact that the artifacts do exist in the local file system and previous components have run correctly.
I created a fork and (arbitrarily) pushed my code here to illustrate exactly what I am running.
You can see the logs from a run here. In particular, look from this line onwards and you will see what custom_config evaluates to and the error.
From my brief attempt at tracing through the TFX code for the Trainer component, it seems that the executors track or resolve the artifacts for the non-custom_config arguments (like examples, transform_graph, and schema) differently than the custom_config arguments. That is why train_files in run_fn() is a valid path while the custom_config values are empty Channel's. But I am uncertain of why there is a difference depending on the orchestrator used and what the correct way to resolve this is.
This is not a new confusion, as you can see others have come across the same situation as myself. Unfortunately, that question was never answered and I was also unable to find any answers in any of the TensorFlow repos/docs or other stack overflow posts. I hope that this issue can clarify the correct way to approach this situation and help others avoid the same mistake in the future.
Why this should be changed:
I would like to request that the tutorial be updated because:
I feel like the tutorial should teach users how to use TFX in a manner that works independently of the type of orchestrator
it will help users avoid facing unexpected failures when they apply what they learned from the tutorial
there is currently a lack of clarity on why these artifacts are empty in some situations
The text was updated successfully, but these errors were encountered:
@rcrowe-google I see you published the original tutorial. Would you be willing to share your thoughts on this, whether its a worthwhile improvement, and possibly clarify why the code works when using the InteractiveContext but not LocalDagRunner?
Hi @TaylorZowtuk - Yes, it would be worthwhile to update the example to work in LocalDagRunner, and I should have written it that way in the first place. It's been on my list to update it for what seems like forever, and I just haven't had time yet. The code works in InteractiveContext because the artifacts are in memory, but they really should have been passed in Channels.
URL(s) with the issue:
Description of issue (what needs changing):
The current notebook runs without issue and works as a starting point. For me (and I presume others) the next step is naturally to organize the code in a more production-like pipeline which means adapting the notebook and fitting it into something like the templates described in this guide.
However, if one adapts the recommender tutorial to run outside of
InteractiveContext
, then the code fails to run. In particular, theChannel
's that we pass to theTrainer
component are empty when the pipeline is run usingLocalDagRunner
. When theMovielensModel
callsmovies_uri.get()[0]
in its constructor, the program will throw aRuntimeError
because we are indexing into an empty list. This is in spite of the fact that the artifacts do exist in the local file system and previous components have run correctly.I created a fork and (arbitrarily) pushed my code here to illustrate exactly what I am running.
You can see the logs from a run here. In particular, look from this line onwards and you will see what
custom_config
evaluates to and the error.From my brief attempt at tracing through the TFX code for the
Trainer
component, it seems that the executors track or resolve the artifacts for the non-custom_config
arguments (likeexamples
,transform_graph
, andschema
) differently than thecustom_config
arguments. That is whytrain_files
inrun_fn()
is a valid path while the custom_config values are emptyChannel
's. But I am uncertain of why there is a difference depending on the orchestrator used and what the correct way to resolve this is.This is not a new confusion, as you can see others have come across the same situation as myself. Unfortunately, that question was never answered and I was also unable to find any answers in any of the TensorFlow repos/docs or other stack overflow posts. I hope that this issue can clarify the correct way to approach this situation and help others avoid the same mistake in the future.
Why this should be changed:
I would like to request that the tutorial be updated because:
The text was updated successfully, but these errors were encountered: