Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: migrate Memory.cache when a constant is promoted to a function parameter #1530

Open
JohannesBuchner opened this issue Dec 3, 2023 · 0 comments

Comments

@JohannesBuchner
Copy link

A common workflow for me with joblib.Memory.cache is that I have a function which I investigate:

@mem.cache
def myfunc(a, b):
   return sum([costly_function(b, 42, callback=otherfunction) for i in range(a)])

for b in range(10):
    print(myfunc(1000, 10))

Then some time later, I realise that I want to also test variations of the other hard-coded variables. I change to:

@mem.cache
def myfunc(a, b, c=42):
   return sum([costly_function(b, c, callback=otherfunction) for i in range(a)])

or

@mem.cache
def myfunc(a, b, callback=otherfunction):
   return sum([costly_function(b, 42, callback=callback) for i in range(a)])

for example.

Now joblib.Memory realises that the function is different, discards the cache.

I was wondering whether for simple situations, where a single value (constant, function, class) that is used directly once, is promoted to a parameter, could be identified as the same identical function.

This would require either
a) a hash(function, parameters) that gives the same value when the argument to the function is not given but hardcoded in the function, or,
b) upon encountering such a rewrite to see whether the old cache can be explained with a simple function generalisation (the new function where newly added parameters with their defaults is equivalent to the old function). Then the old cache can be migrated for the new function code with updated arguments.

b) seems easier.

This is probably wishful thinking and difficult to implement, so feel free to close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant