[2/n] [Serve] Refactor replica rank to prepare for node local ranks by abrarsheikh · Pull Request #58473 · ray-project/ray

abrarsheikh · 2025-11-08T06:01:16Z

Summary

This PR refactors the replica rank system to support multi-dimensional ranking (global, node-level, and local ranks) in preparation for node-local rank tracking. The ReplicaRank object now contains three fields instead of being a simple integer, enabling better coordination of replicas across nodes.

Motivation

Currently, Ray Serve only tracks a single global rank per replica. For advanced use cases like tensor parallelism, model sharding across nodes, and node-aware coordination, we need to track:

Global rank: Replica's rank across all nodes (0 to N-1)
Node rank: Which node the replica is on (0 to M-1)
Local rank: Replica's rank on its specific node (0 to K-1)

This PR lays the groundwork by introducing the expanded ReplicaRank schema while maintaining backward compatibility in feature.

Changes

Core Implementation

schema.py: Extended ReplicaRank to include node_rank and local_rank fields (currently set to -1 as placeholders)
replica.py: Updated replica actors to handle ReplicaRank objects
context.py: Changed ReplicaContext.rank type from Optional[int] to ReplicaRank

Current Behavior

node_rank and local_rank are set to -1 (placeholder values). Will change in future
Global rank assignment and management works as before
All existing functionality is preserved

Breaking Changes

Rank is changing from int to ReplicaRank

Next PR #58477

Signed-off-by: abrar <abrar@anyscale.com>

kouroshHakha

The changes make sense. Needs more detailed review from serve team

Signed-off-by: abrar <abrar@anyscale.com>

…ar-rank-p1

Signed-off-by: abrar <abrar@anyscale.com>

…58471) 2. **Extracted generic `RankManager` class** - Created reusable rank management logic separated from deployment-specific concerns 3. **Introduced `ReplicaRank` schema** - Type-safe rank representation replacing raw integers 4. **Simplified error handling** - not supporting self healing 5. **Updated tests** - Refactored unit tests to use new API and removed flag-dependent test cases **Impact:** - Cleaner separation of concerns in rank management - Foundation for future multi-level rank support Next PR #58473 --------- Signed-off-by: abrar <abrar@anyscale.com>

…ar-rank-p2

cursor

Bug: Replica Rank: Reconfiguration Parameter Type Error

In _stop_or_update_outdated_version_replicas, the code passes current_rank.rank (an integer) to replica.reconfigure(), but the method signature expects a ReplicaRank object. This causes a type mismatch since current_rank is already a ReplicaRank object returned from get_replica_rank(). The code should pass current_rank directly instead of extracting its .rank field.

python/ray/serve/_private/deployment_state.py#L2415-L2421

ray/python/ray/serve/_private/deployment_state.py

Lines 2415 to 2421 in 372819f

    
           # Get current rank for the replica 
        
           current_rank = self._rank_manager.get_replica_rank( 
        
               replica.replica_id.unique_id 
        
           ) 
        
           actor_updating = replica.reconfigure( 
        
               self._target_state.version, rank=current_rank.rank 
        
           )

…ar-rank-p2

…ay-project#58471) 2. **Extracted generic `RankManager` class** - Created reusable rank management logic separated from deployment-specific concerns 3. **Introduced `ReplicaRank` schema** - Type-safe rank representation replacing raw integers 4. **Simplified error handling** - not supporting self healing 5. **Updated tests** - Refactored unit tests to use new API and removed flag-dependent test cases **Impact:** - Cleaner separation of concerns in rank management - Foundation for future multi-level rank support Next PR ray-project#58473 --------- Signed-off-by: abrar <abrar@anyscale.com>

…ay-project#58473) ### Summary This PR refactors the replica rank system to support multi-dimensional ranking (global, node-level, and local ranks) in preparation for node-local rank tracking. The `ReplicaRank` object now contains three fields instead of being a simple integer, enabling better coordination of replicas across nodes. ### Motivation Currently, Ray Serve only tracks a single global rank per replica. For advanced use cases like tensor parallelism, model sharding across nodes, and node-aware coordination, we need to track: - **Global rank**: Replica's rank across all nodes (0 to N-1) - **Node rank**: Which node the replica is on (0 to M-1) - **Local rank**: Replica's rank on its specific node (0 to K-1) This PR lays the groundwork by introducing the expanded `ReplicaRank` schema while maintaining backward compatibility in feature. ### Changes #### Core Implementation - **`schema.py`**: Extended `ReplicaRank` to include `node_rank` and `local_rank` fields (currently set to -1 as placeholders) - **`replica.py`**: Updated replica actors to handle `ReplicaRank` objects - **`context.py`**: Changed `ReplicaContext.rank` type from `Optional[int]` to `ReplicaRank` ### Current Behavior - `node_rank` and `local_rank` are set to `-1` (placeholder values). Will change in future - Global rank assignment and management works as before - All existing functionality is preserved ### Breaking Changes Rank is changing from `int` to `ReplicaRank` Next PR ray-project#58477 --------- Signed-off-by: abrar <abrar@anyscale.com>

…ay-project#58471) 2. **Extracted generic `RankManager` class** - Created reusable rank management logic separated from deployment-specific concerns 3. **Introduced `ReplicaRank` schema** - Type-safe rank representation replacing raw integers 4. **Simplified error handling** - not supporting self healing 5. **Updated tests** - Refactored unit tests to use new API and removed flag-dependent test cases **Impact:** - Cleaner separation of concerns in rank management - Foundation for future multi-level rank support Next PR ray-project#58473 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

…ay-project#58473) ### Summary This PR refactors the replica rank system to support multi-dimensional ranking (global, node-level, and local ranks) in preparation for node-local rank tracking. The `ReplicaRank` object now contains three fields instead of being a simple integer, enabling better coordination of replicas across nodes. ### Motivation Currently, Ray Serve only tracks a single global rank per replica. For advanced use cases like tensor parallelism, model sharding across nodes, and node-aware coordination, we need to track: - **Global rank**: Replica's rank across all nodes (0 to N-1) - **Node rank**: Which node the replica is on (0 to M-1) - **Local rank**: Replica's rank on its specific node (0 to K-1) This PR lays the groundwork by introducing the expanded `ReplicaRank` schema while maintaining backward compatibility in feature. ### Changes #### Core Implementation - **`schema.py`**: Extended `ReplicaRank` to include `node_rank` and `local_rank` fields (currently set to -1 as placeholders) - **`replica.py`**: Updated replica actors to handle `ReplicaRank` objects - **`context.py`**: Changed `ReplicaContext.rank` type from `Optional[int]` to `ReplicaRank` ### Current Behavior - `node_rank` and `local_rank` are set to `-1` (placeholder values). Will change in future - Global rank assignment and management works as before - All existing functionality is preserved ### Breaking Changes Rank is changing from `int` to `ReplicaRank` Next PR ray-project#58477 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

abrarsheikh added 3 commits November 8, 2025 02:18

Refactor replica rank to prepare for node local ranks

8e8f393

Signed-off-by: abrar <abrar@anyscale.com>

referance schema

5f118c2

Signed-off-by: abrar <abrar@anyscale.com>

[Serve] Refactor replica rank to prepare for node local ranks

0f7ae3c

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh added the go add ONLY when ready to merge, run all tests label Nov 8, 2025

abrarsheikh mentioned this pull request Nov 8, 2025

[1/n] [Serve] Refactor replica rank to prepare for node local ranks #58471

Merged

abrarsheikh changed the title ~~[Serve] Refactor replica rank to prepare for node local ranks~~ Nov 8, 2025

kouroshHakha approved these changes Nov 11, 2025

View reviewed changes

abrarsheikh added 5 commits November 12, 2025 23:16

add fail on rank

ad660b3

Signed-off-by: abrar <abrar@anyscale.com>

Merge branch 'master' of github.com:ray-project/ray into LLM-2497-abr…

41a95a1

…ar-rank-p1

add fail on rank

dd960d3

Signed-off-by: abrar <abrar@anyscale.com>

remove unused code

6d40e49

Signed-off-by: abrar <abrar@anyscale.com>

Merge branch 'LLM-2497-abrar-rank-p1' into LLM-2497-abrar-rank-p2

7fa563e

Base automatically changed from LLM-2497-abrar-rank-p1 to master November 13, 2025 19:41

Merge branch 'master' of github.com:ray-project/ray into LLM-2497-abr…

372819f

…ar-rank-p2

abrarsheikh marked this pull request as ready for review November 13, 2025 20:05

abrarsheikh requested review from a team as code owners November 13, 2025 20:05

abrarsheikh requested a review from zcin November 13, 2025 20:06

cursor Bot reviewed Nov 13, 2025

View reviewed changes

zcin reviewed Nov 13, 2025

View reviewed changes

Comment thread python/ray/serve/_private/deployment_state.py

zcin approved these changes Nov 13, 2025

View reviewed changes

ray-gardener Bot added the serve Ray Serve Related Issue label Nov 14, 2025

Merge branch 'master' of github.com:ray-project/ray into LLM-2497-abr…

5b3de81

…ar-rank-p2

abrarsheikh merged commit 80eb240 into master Nov 14, 2025
6 checks passed

abrarsheikh deleted the LLM-2497-abrar-rank-p2 branch November 14, 2025 06:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[2/n] [Serve] Refactor replica rank to prepare for node local ranks#58473

[2/n] [Serve] Refactor replica rank to prepare for node local ranks#58473
abrarsheikh merged 10 commits into
masterfrom
LLM-2497-abrar-rank-p2

abrarsheikh commented Nov 8, 2025 •

edited

Loading

kouroshHakha left a comment

cursor Bot left a comment

Uh oh!

Uh oh!

Labels

3 participants

	# Get current rank for the replica
	current_rank = self._rank_manager.get_replica_rank(
	replica.replica_id.unique_id
	)
	actor_updating = replica.reconfigure(
	self._target_state.version, rank=current_rank.rank
	)

Uh oh!

Conversation

abrarsheikh commented Nov 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Changes

Core Implementation

Current Behavior

Breaking Changes

kouroshHakha left a comment

Choose a reason for hiding this comment

cursor Bot left a comment

Choose a reason for hiding this comment

Bug: Replica Rank: Reconfiguration Parameter Type Error

Uh oh!

Uh oh!

Labels

3 participants

abrarsheikh commented Nov 8, 2025 •

edited

Loading