Repository navigation
Don't walk route_ht while holding peertab->lock - #147
Open
randomizedcoder wants to merge 1 commit into
Open
randomizedcoder wants to merge 1 commit into
randomizedcoder wants to merge 1 commit into
Conversation
The rhashtable walk functions acquire the table's internal lock with spin_lock, and the rhashtable rehash worker holds that lock with softirqs enabled. peertab->lock is acquired in softirq context (homa_route_get, homa_route_free), so holding it while walking the table can deadlock: lockdep reports "SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected" from homa_peer_free_net at netns exit, and homa_route_gc has the same problem. Add peertab->walk_mutex to serialize the two walks (and protect route_ht_iter), and acquire peertab->lock only around the removals. homa_route_gc now holds the RCU read lock from choosing victims until it has removed them, since another thread may remove a victim in between. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Heads-up: this conflicts with #144 at the end of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rhashtable_walk_enter,_startand_stoptake the table'sht->lockwith plainspin_lock, and the rehash worker holds it with softirqs enabled.peertab->lockis taken in softirq (homa_route_getfromhoma_rpc_alloc_server,homa_route_freefrom an RCU callback). Two places holdpeertab->lockwhile walkingroute_ht:homa_peer_free_net(netns exit);homa_route_gc→homa_route_pick_victims.That can hard-hang: CPU0's rehash worker holds
ht->lockand a Homa softirq on CPU0 spins onpeertab->lock, while CPU1 holdspeertab->lockand spins onht->lock. It's rare (a rehash, a netns exit or GC, and traffic at the same moment), but lockdep reports it on every run that exits a netns:Fix:
peertab->walk_mutexserializes the two walks and protectsroute_ht_iter(it's acquired beforepeertab->lock; both callers run in process context).homa_peer_free_nettakespeertab->lockonly around each removal and counter update.homa_route_gcpicks victims under the mutex andrcu_read_lock(), then takespeertab->lockfor the removals.rhashtable_remove_fast's existing check skips a victim someone else removed in between, and RCU keeps it valid until then.READ_ONCEon the locklessnum_routesreads.If you'd rather not add a lock: a fresh local iterator per GC pass would also work, but it loses the persistent cursor, so GC would always restart from the beginning of the table.
Checked:
homa_peer_free_net__*,homa_route_gc__*,homa_route_pick_victims__*).__STRIP__=1.No test: the unit harness has no lockdep, and the lock order isn't otherwise observable.
🤖 Generated with Claude Code