[tune] Fix Pausing and Error Propogation

[richardliaw]

What do these changes do?

Addresses #2814 and #2810.

• Manages error handling with recovery, even when error occurs at ray.get.
• Also introduces tests for pausing via trial executor.

Related issue number

Above.

[ericl]

Can you also change the smoke test iterations for the pbt test to something like 20? That catches the bug and also runs reasonably fast.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7958/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7961/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7968/
Test PASSed.

[richardliaw]

@joyyoj Can you take a look please?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7972/
Test FAILed.

[richardliaw]

jenkins retest this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7974/
Test FAILed.

[richardliaw]

Jenkins retest this please

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7975/
Test PASSed.

[joyyoj]

the changes make sense.

[ericl]

lgtm based on tests and @joyyoj 's approval (I did not look carefully at the unpause crash, it looks complicated)

One minor suggestion on the error handling that might give a cleaner interface.

[ericl]

So one other way of doing this is to have fetch_one_result() return an exception object for the result if there is an error. This avoids having a separate call just for this error handling.

[ericl]

Merging now since this is a fairly severe regression. Comments can be done in a follow-up...