Skip to content

[tune] Cross-Node Recovery#3725

Merged
richardliaw merged 10 commits into
ray-project:masterfrom
richardliaw:tune_cross_node
Jan 15, 2019
Merged

[tune] Cross-Node Recovery#3725
richardliaw merged 10 commits into
ray-project:masterfrom
richardliaw:tune_cross_node

Conversation

@richardliaw

@richardliaw richardliaw commented Jan 9, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor

Augments trial restore to also check if the runner is at the same
location. If not, the checkpoint files are pushed onto the new location.

TODO:

  • Test this on EC2
  • Autoscaler test (maybe add into stress tests?)
    • Option 1: Have a helper fn that blocks until the ray cluster is the expected size, then submit tune job, down scale cluster, and then make sure the exit code of the submit job is 0.

We don't have any tests for this right now...
@AmplabJenkins

AmplabJenkins commented Jan 9, 2019

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10691/
Test FAILed.
@richardliaw richardliaw mentioned this pull request Jan 9, 2019
Closed
3 tasks
richardliaw
richardliaw commented Jan 9, 2019
View reviewed changes
Comment thread python/ray/tune/ray_trial_executor.py Outdated
@AmplabJenkins

AmplabJenkins commented Jan 9, 2019

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10703/
Test PASSed.
@AmplabJenkins

AmplabJenkins commented Jan 14, 2019

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10830/
Test FAILed.
@richardliaw richardliaw changed the title [wip][tune] Cross-Node Recovery Jan 14, 2019
ericl
ericl reviewed Jan 14, 2019
View reviewed changes
Comment thread python/ray/tune/logger.py

def update_location(self, worker_ip):
"""Sends the current log directory to the remote node."""
if worker_ip != self._log_syncer.worker_ip:

@ericl ericl Jan 14, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this crash on non-autoscaler clusters?

Also, log a warning if sync is happening?

@richardliaw richardliaw Jan 15, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done; and no, won't crash (1sync_to_worker_if_possible1 catches it)
ericl
ericl reviewed Jan 14, 2019
View reviewed changes
Comment thread python/ray/tune/logger.py Outdated
self._log_syncer.sync_now(force=True)
self._log_syncer.wait()

def update_location(self, worker_ip):

@ericl ericl Jan 14, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update_location() sounds more harmless than this is, maybe sync_results_to_new_location()?
@AmplabJenkins

AmplabJenkins commented Jan 14, 2019

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10855/
Test PASSed.
@AmplabJenkins

AmplabJenkins commented Jan 15, 2019

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10857/
Test PASSed.
@AmplabJenkins

AmplabJenkins commented Jan 15, 2019

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10866/
Test PASSed.
richardliaw
richardliaw commented Jan 15, 2019
View reviewed changes
Comment thread python/ray/tune/ray_trial_executor.py Outdated
richardliaw
richardliaw commented Jan 15, 2019
View reviewed changes
Comment thread python/ray/tune/ray_trial_executor.py Outdated
@richardliaw

richardliaw commented Jan 15, 2019

Copy link
Copy Markdown
Contributor Author

This is ready for another review.
@AmplabJenkins

AmplabJenkins commented Jan 15, 2019

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10885/
Test PASSed.
@AmplabJenkins

AmplabJenkins commented Jan 15, 2019

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10886/
Test PASSed.
@richardliaw richardliaw added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Jan 15, 2019
@richardliaw richardliaw merged commit 3918934 into ray-project:master Jan 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests-ok The tagger certifies test failures are unrelated and assumes personal liability.

3 participants

@richardliaw @AmplabJenkins @ericl