Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reap job inputs #39

Open
rgbkrk opened this issue Jan 5, 2015 · 2 comments
Open

Reap job inputs #39

rgbkrk opened this issue Jan 5, 2015 · 2 comments

Comments

@rgbkrk
Copy link
Member

rgbkrk commented Jan 5, 2015

Over time, we should toss the job input from the database. Reasons:

  • Size - People will send over large pickled python objects because they didn't know about all the globals they set. This may sound like premature optimization, but we'll want to be ready for it...
  • Privacy - Might as well tear down the input to the job since it shouldn't be needed any more. If someone wants to re-run they should re-submit.
@smashwilson
Copy link
Member

We'll need to leave the rest of the job metadata in place, so you can trawl through your history and figure out how you used 120 hours of compute three months ago. Looking through Job and SubmittedJob, I would say that we should age off:

  • Environment
  • Stdin
  • Result
  • Stderr
  • Stdout

I'm debating Cmd as well. Because we're really running arbitrary processes, it's possible that some user (or some SDK) could pass input by command-line parameters instead of stdin. Maybe we could truncate it?

@rgbkrk
Copy link
Member Author

rgbkrk commented Jan 8, 2015

That makes sense, since we're still keeping the metering data. Some shortening of the Command makes sense, but maybe we don't bother reaping that until we notice a problem.

Stdin should definitely be reaped, as that's our interface for sending pickled python over.

@smashwilson smashwilson added this to the v0.0.4: Caps and Meters milestone Feb 2, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants