[rllib] Native support for Dict and Tuple spaces; fix Tuple action spaces; add prev a, r to LSTM by ericl · Pull Request #3051 · ray-project/ray

ericl · 2018-10-12T09:53:15Z

What do these changes do?

Add _build_layers_v2 to model which takes in an input dict of tensors. Deprecate _build_layers.
Pass in a_{t-1}, r_{t-1} through this input dict for use by LSTMs.
Unflatten Dict and Tuple observations automatically and expose their values in the input dict.
Fix tuple action space sample: the output should be a batch of tuples, not a tuple of batches.
Add verbose flag for PPO logging.

Related issue number

AmplabJenkins · 2018-10-12T10:44:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8599/
Test FAILed.

AmplabJenkins · 2018-10-12T10:57:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8600/
Test FAILed.

AmplabJenkins · 2018-10-12T11:05:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8601/
Test FAILed.

AmplabJenkins · 2018-10-12T11:10:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8602/
Test FAILed.

AmplabJenkins · 2018-10-12T23:57:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8605/
Test FAILed.

AmplabJenkins · 2018-10-13T05:22:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8608/
Test FAILed.

AmplabJenkins · 2018-10-13T08:00:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8610/
Test FAILed.

AmplabJenkins · 2018-10-13T09:16:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8611/
Test FAILed.

AmplabJenkins · 2018-10-14T03:14:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8632/
Test FAILed.

AmplabJenkins · 2018-10-14T03:17:46Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8633/
Test FAILed.

AmplabJenkins · 2018-10-14T03:22:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8634/
Test FAILed.

AmplabJenkins · 2018-10-14T04:33:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8635/
Test PASSed.

AmplabJenkins · 2018-10-14T07:41:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8641/
Test PASSed.

AmplabJenkins · 2018-10-15T04:47:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8648/
Test FAILed.

AmplabJenkins · 2018-10-15T05:22:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8649/
Test FAILed.

ericl · 2018-10-15T07:24:45Z

jenkins retest this please

AmplabJenkins · 2018-10-15T08:46:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8660/
Test FAILed.

AmplabJenkins · 2018-10-15T18:40:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8668/
Test FAILed.

AmplabJenkins · 2018-10-16T19:36:04Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8683/
Test FAILed.

AmplabJenkins · 2018-10-16T23:40:57Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8691/
Test FAILed.

AmplabJenkins · 2018-10-17T06:16:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8695/
Test PASSed.

ericl · 2018-10-19T02:33:50Z

Ping @richardliaw

richardliaw · 2018-10-19T06:51:18Z

    # Use the sync samples optimizer instead of the multi-gpu one
    "simple_optimizer": False,
+    # Verbose logging during SGD
+    "verbose": True,


aren't we trying to avoid this?

I don't have a better solution... perhaps: "log_level": "INFO"?

I removed this flag for now.

AmplabJenkins · 2018-10-20T01:24:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8763/
Test FAILed.

ericl added 9 commits October 12, 2018 00:13

wip

50aac0e

prev action reward

54725be

test

59ede85

fix tuple spaces

d6a6196

fix test

72e2d26

fix

97a1e68

lint

d3d439e

docs

bf11029

v2

abb1703

ericl assigned richardliaw Oct 12, 2018

ericl added 3 commits October 12, 2018 02:59

doc

edeebe2

remove batch check

59ff1f5

v2

cf3588a

ericl mentioned this pull request Oct 12, 2018

[RLlib] action passed into env.step has wrong shape with Tuple space #3048

Closed

fix

c97af9e

ericl force-pushed the layers2 branch from afdf62b to c97af9e Compare October 12, 2018 23:00

ericl mentioned this pull request Oct 12, 2018

[Rllib] Using Dict state space throws exception (not supported yet?) #3042

Closed

fix test

68b9c36

1.0

71bd2bd

ericl added 3 commits October 13, 2018 01:17

make keyword

6966db3

format

929b096

typo

1ceaef4

revert rnn refactor

706ecd5

fix catalog test and prep docs

5afdf1b

ericl added 2 commits October 14, 2018 20:57

fix PPO dict support

d44540b

revert test

d242687

ericl force-pushed the layers2 branch from b6aa318 to d242687 Compare October 15, 2018 04:00

ericl added 2 commits October 15, 2018 11:04

Merge remote-tracking branch 'upstream/master' into layers2

d271d8f

format

bbdebb4

fix obs

0b464f1

Merge remote-tracking branch 'upstream/master' into layers2

9a76303

fix test

49584c7

richardliaw approved these changes Oct 19, 2018

View reviewed changes

remove verbose

8f25479

add better error

59b5310

ericl merged commit 59901a8 into ray-project:master Oct 20, 2018

ericl mentioned this pull request Nov 18, 2018

no CUDA-capable device is detected #3265

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[rllib] Native support for Dict and Tuple spaces; fix Tuple action spaces; add prev a, r to LSTM#3051

[rllib] Native support for Dict and Tuple spaces; fix Tuple action spaces; add prev a, r to LSTM#3051
ericl merged 39 commits into
ray-project:masterfrom
ericl:layers2

ericl commented Oct 12, 2018

AmplabJenkins commented Oct 12, 2018

AmplabJenkins commented Oct 12, 2018

AmplabJenkins commented Oct 12, 2018

AmplabJenkins commented Oct 12, 2018

AmplabJenkins commented Oct 12, 2018

AmplabJenkins commented Oct 13, 2018

AmplabJenkins commented Oct 13, 2018

AmplabJenkins commented Oct 13, 2018

AmplabJenkins commented Oct 14, 2018

AmplabJenkins commented Oct 14, 2018

AmplabJenkins commented Oct 14, 2018

AmplabJenkins commented Oct 14, 2018

AmplabJenkins commented Oct 14, 2018

AmplabJenkins commented Oct 15, 2018

AmplabJenkins commented Oct 15, 2018

ericl commented Oct 15, 2018

AmplabJenkins commented Oct 15, 2018

AmplabJenkins commented Oct 15, 2018

AmplabJenkins commented Oct 16, 2018

AmplabJenkins commented Oct 16, 2018

AmplabJenkins commented Oct 17, 2018

ericl commented Oct 19, 2018

richardliaw Oct 19, 2018

ericl Oct 19, 2018 •

edited

Loading

ericl Oct 20, 2018

Uh oh!

Uh oh!

AmplabJenkins commented Oct 20, 2018

Labels

3 participants

Uh oh!

Conversation

ericl commented Oct 12, 2018

What do these changes do?

Related issue number

AmplabJenkins commented Oct 12, 2018

AmplabJenkins commented Oct 12, 2018

AmplabJenkins commented Oct 12, 2018

AmplabJenkins commented Oct 12, 2018

AmplabJenkins commented Oct 12, 2018

AmplabJenkins commented Oct 13, 2018

AmplabJenkins commented Oct 13, 2018

AmplabJenkins commented Oct 13, 2018

AmplabJenkins commented Oct 14, 2018

AmplabJenkins commented Oct 14, 2018

AmplabJenkins commented Oct 14, 2018

AmplabJenkins commented Oct 14, 2018

AmplabJenkins commented Oct 14, 2018

AmplabJenkins commented Oct 15, 2018

AmplabJenkins commented Oct 15, 2018

ericl commented Oct 15, 2018

AmplabJenkins commented Oct 15, 2018

AmplabJenkins commented Oct 15, 2018

AmplabJenkins commented Oct 16, 2018

AmplabJenkins commented Oct 16, 2018

AmplabJenkins commented Oct 17, 2018

ericl commented Oct 19, 2018

richardliaw Oct 19, 2018

Choose a reason for hiding this comment

ericl Oct 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

ericl Oct 20, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AmplabJenkins commented Oct 20, 2018

Labels

3 participants

ericl Oct 19, 2018 •

edited

Loading