[Data] Print data context in JSON (Dictionary) Format #61428

goutamvenkat-anyscale · 2026-03-03T01:30:58Z

will this truncate nesting inside the datacontext, or does it truncate the whole context? I believe we want to log the full context and not lose keys

This might truncate nesting inside the datacontext, e.g.

'execution_options': 'ExecutionOptions(resource_limits=ExecutionResources(cpu=inf, gpu=inf, object_store_memory=inf, memor...'

I tried to mitigate this issue by setting truncate_length to DATA_CONTEXT_LOG_TRUNCATE_LENGTH. My testing shows that none of the fields would be truncated in the debug output with the current configuration (log.txt)

+1 @rayhhome I have the same question

Based on the current truncate_length passed into (which is DATA_CONTEXT_LOG_TRUNCATE_LENGTH, i.e. 10000), the DataContext will not be truncated unless there's any string with more than 10000 characters or any list with more than 10000 elements, which I believe is unlikely.

Currently, calling json.dumps directly on DataContext raises a exception because there are a few fields in DataContext that are not JSON serializable (I've documented such fields in the PR description). I can use _json_default to get around this though, do we want to switch to using json.dumps instead of sanitize_for_struct?

-Original file line number
+Diff line change
@@ -1,6 +1,5 @@
     import logging
     import math
-    import pprint
     import threading
     import time
     import typing
@@ Expand Down Expand Up / @@ -45,7 +44,10 @@ @@
         register_dataset_logger,
         unregister_dataset_logger,
     )
-    from ray.data._internal.metadata_exporter import Topology as TopologyMetadata
+    from ray.data._internal.metadata_exporter import (
+        Topology as TopologyMetadata,
+        sanitize_for_struct,
+    )
     from ray.data._internal.operator_schema_exporter import (
         OperatorSchema,
         get_operator_schema_exporter,
@@ Expand All / @@ -65,6 +67,10 @@ @@
     # Interval for logging execution progress updates and operator metrics.
     DEBUG_LOG_INTERVAL_SECONDS = 5
+    # Maximum string/sequence length for DataContext logging. Set high to avoid truncation
+    # while still protecting against pathological cases.
+    DATA_CONTEXT_LOG_TRUNCATE_LENGTH = 10000
     # Visible for testing.
     _num_shutdown = 0
@@ Expand Down Expand Up / @@ -196,7 +202,10 @@ def execute( @@
                 ):
                     logger.debug(
                         f"Data Context for dataset {self._dataset_id}:\n%s",
-                        pprint.pformat(self._data_context),
+                        sanitize_for_struct(
+                            self._data_context,
+                            truncate_length=DATA_CONTEXT_LOG_TRUNCATE_LENGTH,
+                        ),
                     )
             # Setup the streaming DAG topology and start the runner thread.
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Data] Print data context in JSON (Dictionary) Format #61428

Uh oh!

Diff view

Diff view

There are no files selected for viewing

goutamvenkat-anyscale Mar 3, 2026

rayhhome Mar 3, 2026

bveeramani Mar 5, 2026

rayhhome Mar 5, 2026

Uh oh!

[Data] Print data context in JSON (Dictionary) Format #61428

Uh oh!

[Data] Print data context in JSON (Dictionary) Format #61428

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

goutamvenkat-anyscale Mar 3, 2026

Choose a reason for hiding this comment

rayhhome Mar 3, 2026

Choose a reason for hiding this comment

bveeramani Mar 5, 2026

Choose a reason for hiding this comment

rayhhome Mar 5, 2026

Choose a reason for hiding this comment