Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ngcc): pause async processing if another process has the lockfile #35131

Conversation

petebacondarwin
Copy link
Member

PR Checklist

Please check if your PR fulfills the following requirements:

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Documentation content changes
  • angular.io application / infrastructure changes
  • Other... Please describe:

What is the current behavior?

Issue Number: N/A

What is the new behavior?

Does this PR introduce a breaking change?

  • Yes
  • No

Other information

@petebacondarwin petebacondarwin changed the title feat(ngcc): WIP feat(ngcc): pause async processing if another process has the lockfile Feb 3, 2020
@pullapprove pullapprove bot requested a review from alxhub February 3, 2020 22:06
@petebacondarwin petebacondarwin force-pushed the ngcc-async-lockfile-pause branch 2 times, most recently from 4e8cf5b to 9210143 Compare February 4, 2020 14:22
@petebacondarwin petebacondarwin added comp: ngcc state: WIP target: patch This PR is targeted for the next patch release feature Issue that requests a new feature labels Feb 4, 2020
@ngbot ngbot bot modified the milestone: needsTriage Feb 4, 2020
@petebacondarwin petebacondarwin added action: review The PR is still awaiting reviews from at least one requested reviewer and removed state: WIP labels Feb 5, 2020
@petebacondarwin petebacondarwin marked this pull request as ready for review February 5, 2020 16:40
@petebacondarwin petebacondarwin force-pushed the ngcc-async-lockfile-pause branch 2 times, most recently from fb27dc1 to 15e611e Compare February 6, 2020 07:47
Copy link
Member

@gkalpak gkalpak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, except I think we should reset the waiting period when the pid in the lockfile changes.
Really excited to incorporate this into the CLI workflow \o/

This is needed by ngcc when reading volatile files that may
be changed by an external process (e.g. the lockfile).
…ckfile

ngcc uses a lockfile to prevent two ngcc instances from executing at the
same time. Previously, if a lockfile was found the current process would
error and exit.

Now, when in async mode, the current process is able to wait for the previous
process to release the lockfile before continuing itself.
@petebacondarwin petebacondarwin added the action: merge The PR is ready for merge by the caretaker label Feb 10, 2020
@ngbot
Copy link

ngbot bot commented Feb 10, 2020

I see that you just added the PR action: merge label, but the following checks are still failing:
    failure status "pullapprove" is failing
    pending forbidden labels detected: PR action: review
    pending status "google3" is pending
    pending 3 pending code reviews

If you want your PR to be merged, it has to pass all the CI checks.

If you can't get the PR to a green state due to flakes or broken master, please try rebasing to master and/or restarting the CI job. If that fails and you believe that the issue is not due to your change, please contact the caretaker and ask for help.

@petebacondarwin petebacondarwin added action: presubmit The PR is in need of a google3 presubmit and removed action: review The PR is still awaiting reviews from at least one requested reviewer labels Feb 10, 2020
]]);
// We need to write to the rawFs to ensure that we don't update the cache at this point
rawFs.writeFile(lockFile.lockFilePath, '444');
await new Promise(resolve => setTimeout(resolve, 250));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a potential source of flakes 😁
Could we use fakeAsync() and tick() to make this test deterministic?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was a little bit worried about this. We cannot use the Angular fakeAsync stuff, since that is driven via Zone.js and we definitely not going to bring that in here just for testing...

I tried to make the timeouts as reliable as possible so that it would be extremely unlikely for it to flake:

  • having proven that the class has already checked the lockfile (and read its value) we are immediately writing the new one synchronously so there is no chance for the setTimeout inside LockFile to be triggered.
  • we then set a timeout for 250ms before the next check but the class has been configured to check the contents of the lockfile every 100ms. So there should be at least 2 opportunities for a setTimeout to be triggered before this testing one is triggered.
  • The class is setup not to fail until 10 attempts have occurred, which is a full second, so there is very little chance that the 250ms timeout is not triggered before we run out of attempts.

Do you have any better ideas?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough. I am fine leaving this as is and address it in the future if we start seeing flakes.

For future reference, one way to make this deterministic would be using Jasmine's Clock. There is one twist, because we need to also flush microtasks after each retryDelay ms, so it would be something like:

const clock = jasmine.clock();
const moveToNextIteration = () => {
  clock().tick(100);     // Or whatever `retryDelay` we have specified.
  return Promise.resolve();  // To wait for microtask queue to be flushed.
};

clock.withMock(async () => {
  const rawFs = getFileSystem();
  const fs = new CachedFileSystem(rawFs);
  const lockFile = new LockFileUnderTest(fs);

  fs.writeFile(lockFile.lockFilePath, '188');
  const promise = lockFile.lock(async () => lockFile.log.push('fn()'));
  expect(lockFile.log).toEqual(['create()']);
  expect(lockFile.getLogger().logs.info).toEqual([...]);

  rawFs.writeFile(lockFile.lockFilePath, '444');
  await moveToNextIteration();
  await moveToNextIteration();
  expect(lockFile.getLogger().logs.info).toEqual([...]]);

  ...
});

@gkalpak gkalpak added action: cleanup The PR is in need of cleanup, either due to needing a rebase or in response to comments from reviews and removed action: merge The PR is ready for merge by the caretaker labels Feb 10, 2020
@petebacondarwin petebacondarwin added action: merge The PR is ready for merge by the caretaker and removed action: cleanup The PR is in need of cleanup, either due to needing a rebase or in response to comments from reviews labels Feb 10, 2020
@kara
Copy link
Contributor

kara commented Feb 11, 2020

@petebacondarwin Are you still waiting on a review from @alxhub or is this ready to presubmit/merge?

@kara kara added the action: review The PR is still awaiting reviews from at least one requested reviewer label Feb 11, 2020
@petebacondarwin
Copy link
Member Author

Just a presubmit please @kara

@petebacondarwin petebacondarwin removed the request for review from alxhub February 11, 2020 09:04
@alxhub alxhub removed the action: review The PR is still awaiting reviews from at least one requested reviewer label Feb 19, 2020
@alxhub alxhub closed this in 7e8ce24 Feb 19, 2020
alxhub pushed a commit that referenced this pull request Feb 19, 2020
…ckfile (#35131)

ngcc uses a lockfile to prevent two ngcc instances from executing at the
same time. Previously, if a lockfile was found the current process would
error and exit.

Now, when in async mode, the current process is able to wait for the previous
process to release the lockfile before continuing itself.

PR Close #35131
alxhub pushed a commit that referenced this pull request Feb 19, 2020
…35131)

This is needed by ngcc when reading volatile files that may
be changed by an external process (e.g. the lockfile).

PR Close #35131
alxhub pushed a commit that referenced this pull request Feb 19, 2020
…ckfile (#35131)

ngcc uses a lockfile to prevent two ngcc instances from executing at the
same time. Previously, if a lockfile was found the current process would
error and exit.

Now, when in async mode, the current process is able to wait for the previous
process to release the lockfile before continuing itself.

PR Close #35131
alxhub added a commit that referenced this pull request Feb 19, 2020
…s the lockfile (#35131)"

This reverts commit b970028.

This is a feature commit and was improperly merged to the patch branch.
@petebacondarwin petebacondarwin deleted the ngcc-async-lockfile-pause branch February 23, 2020 19:06
@angular-automatic-lock-bot
Copy link

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

This action has been performed automatically by a bot.

@angular-automatic-lock-bot angular-automatic-lock-bot bot locked and limited conversation to collaborators Mar 25, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
action: merge The PR is ready for merge by the caretaker action: presubmit The PR is in need of a google3 presubmit cla: yes feature Issue that requests a new feature target: patch This PR is targeted for the next patch release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants