Prepare socket file when start ray

[guoyuhong]

What do these changes do?

Sometimes, users want to specify their own socket file name. However, there are two cases that they could fail.

1. The directory specified does not exist. In this case, ray should make the directory.
2. The socket file exists before they start the cluster, which may due to that we are test failover functionality and send SIGKILL to raylet. In this case, ray should clean the uncleaned file.

For the first case, raylet will crash with:

libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::  error_info_injector<boost::system::system_error> >: bind: No such file or directory
*** Aborted at 1549001100 (unix time) try "date -d @1549001100" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x7fff6972ab86) received by PID 76085 (TID 0x10cef95c0) stack trace: ***
    @     0x7fff697d5b3d _sigtramp

For the second case, raylet will crash with:

libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >: bind: Address already in use
*** Aborted at 1549001712 (unix time) try "date -d @1549001712" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x7fff6972ab86) received by PID 77036 (TID 0x1161875c0) stack trace: ***
    @     0x7fff697d5b3d _sigtramp

Related issue number

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11390/
Test FAILed.

[jovany-wang]

Great!

[jovany-wang]

if not os.path.exist(socket_path) will be more explicit?

[raulchen]

typo, existent. or maybe shorter test_socket_dir_not_existing

[raulchen]

nit, let's put the socket under something like /tmp/ray/tests to avoid creating too many files in /tmp

[raulchen]

I'm not sure if it's a good idea to remove the existing socket file. If a raylet is still running and user starts ray again, this will break the previous raylet.
We can check if this socket file is still used by any processes before removing it. Or maybe simply raise an error if it already exists.

[guoyuhong]

Because this function is used in raylet socket and plasma socket, it not easy to connect to it to check whether it is in use. I raise exception here. It is better than nothing. If we don't do this and start cluster using ray start, there will be no error after this command. We need to check it in raylet.err. With this exception, ray start will stop immediately.

[raulchen]

Yep, raising the error earlier is better.
BTW, we can detect if a socket file is in use with the lsof command. But raising an error is okay as well.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11402/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11403/
Test PASSed.

[guoyuhong]

@robertnishihara The Linux Wheel test fails in all PRs. Shall we change the number of expecting wheels to 4 to unblock the test?