Python API to map between md5 and a git tag

[mohitanchlia]

It would be good to have an API to map from MD5 to a git tag

It would also be nice to know the tag that was used when passing a null in "rev" in python API

[efiop]

Converted to discussion since this is not actionable right now.

We don't really have that mapping internally, so this is more of an active operation that every user can do himself by parsing his own git repo. I'm not sure we have a use for it ourselves that will justify us implementing it as a feature.

[mohitanchlia]

Use case for us is to be able to get the tag name and the revision/version using Python API and log this info to mlflow for a full view of the experiment and tracking. Is there a utility function that is available in the meantime?

[mohitanchlia]

What we were hoping was to have DVC give us all the details instead of relying on a programmer or data scientists to execute git wrapper commands or something else. Both md5 and git information are available to DVC, it would be extremely helpful for us to get a tight integration of DVC to our experiment info in MLFlow.

We are currently using mlflow and that's why we haven't looked at dvc exp. Not sure what support we would get to make that switch at this point.

[shcheklein]

It's a reasonable request. The challenge here is that Git provides tremendous amount of information. DVC provides additional information on top of that (md5). I personally don't think we could meaningfully replace Git Python libraries (that already do their job) well to extract all the possible Git information.

Do you have in mind in how that interface would look like?

instead of relying on a programmer or data scientists to execute git wrapper commands or something else.

absolutely. It should be packaged a single wrapper that is being used by DS / ML team. They definitely should not have to deal with Git or even DVC ger_url to my mind. Usually it can be done once, standardized, and even made as part of some callbacks (depends on what framework you use).

Not sure what support we would get to make that switch at this point.

not 100% sure I understand the support part here, but nw, it's absolutely up to you. I was asking only to get some feedback in case you tried it before.

[mohitanchlia]

Currently, we don't have a good way to get a handle of the location of the dataset that we want to run in an experiment other than calling something like get_url. get_url somehow is able to determine the correct version of the file for a given git tag or gives latest when version is null. I was wondering if we could use the same logic and return additional metadata eg: get_metadata(revision: optional) where revision is a git tag. When None is specified it returns the latest information. Metadata struct could be {git_info: {}, file_version:md5, ...}

By "support" I meant support within the organization. We are currently committed to using mlflow. I believe mlflow and dvc exp are competing technologies.

[mohitanchlia]

@shcheklein Following up to see if you need any more info

[shcheklein]

@mohitanchlia hey, no, I think the use case is clear and your need is clear. We can't prioritize building this API on our end atm. It's still not clear to me if we want to cover the Git portion of metadata via DVC API. I would first recommend still to try to use a combination of DVC + Git Python library to get this combination. If you do this please share (or ask questions) and we can publish this as an example in docs as a first step I think.