Skip to content

Add functionality to automatically upload logs to Vertex Tensorboard#570

Merged
copybara-service[bot] merged 1 commit into
mainfrom
vertex_tb
Apr 10, 2024
Merged

Add functionality to automatically upload logs to Vertex Tensorboard#570
copybara-service[bot] merged 1 commit into
mainfrom
vertex_tb

Conversation

@SurbhiJainUSC

@SurbhiJainUSC SurbhiJainUSC commented Mar 29, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator
  • Add functionality to upload data in config.tensorboard_dir to Tensorboard in Vertex AI
  • XPK users won't have to create a Tensorboard in Vertex AI. XPK will automatically handle that.
  • For non-XPK cases, users can choose to manually create Tensorboard instance in Vertex AI cloud console or set the configuration to create instance in MaxText

Note: Uploader to upload logs to Vertex Tensorboard is only supported for Tensorflow < 2.15.0. Vertex AI team is working on fixing the issue to support the latest TF versions.
@SurbhiJainUSC SurbhiJainUSC requested a review from rwitten as a code owner March 29, 2024 22:28
rwitten
rwitten suggested changes Mar 30, 2024
View reviewed changes

@rwitten rwitten left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of questions! But looks amazing!
Comment thread MaxText/configs/base.yml Outdated
Comment thread MaxText/configs/base.yml
Comment thread MaxText/configs/base.yml Outdated
Comment thread MaxText/configs/base.yml Outdated
Comment thread MaxText/configs/base.yml Outdated
Comment thread MaxText/configs/base.yml Outdated
Comment thread MaxText/train.py Outdated
Comment thread MaxText/train.py Outdated
Comment thread README.md
Comment thread README.md Outdated
@rwitten rwitten removed their assignment Mar 30, 2024
@SurbhiJainUSC

SurbhiJainUSC commented Apr 2, 2024

Copy link
Copy Markdown
Collaborator Author

Lots of questions! But looks amazing!

Thank you Rafi for your suggestions!
I have simplified the process of creating Vertex AI Tensorboard for testing MaxText on GCE. I have also updated the README to explain different scenarios to upload logs to Vertex Tensorboard. Let me know if this looks good.
@SurbhiJainUSC SurbhiJainUSC requested a review from rwitten April 2, 2024 19:16
rwitten
rwitten suggested changes Apr 3, 2024
View reviewed changes

@rwitten rwitten left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The explicit teardown is still a bit weird but looks good otherwise!
Comment thread MaxText/configs/base.yml
Comment thread MaxText/train.py Outdated
@rwitten rwitten removed their assignment Apr 3, 2024
@SurbhiJainUSC SurbhiJainUSC assigned rwitten and unassigned rwitten Apr 8, 2024
@SurbhiJainUSC SurbhiJainUSC requested a review from rwitten April 9, 2024 00:21
rwitten
rwitten approved these changes Apr 9, 2024
View reviewed changes

@rwitten rwitten left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved (but there are some nits I want you to cleanup)
Comment thread MaxText/configs/base.yml
Comment thread MaxText/train.py Outdated
Comment thread README.md Outdated
@rwitten rwitten removed their assignment Apr 9, 2024
@copybara-service copybara-service Bot merged commit e366da6 into main Apr 10, 2024
@copybara-service copybara-service Bot deleted the vertex_tb branch April 10, 2024 00:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pull ready

2 participants

@SurbhiJainUSC @rwitten