From: "Paul E. McKenney" Adds variant of RCU that permits sleeping in read-side critical sections. SRCU is as follows: o Each use of SRCU creates its own srcu_struct, and each srcu_struct has its own set of grace periods. This is critical, as it prevents one subsystem with a blocking reader from holding up SRCU grace periods for other subsystems. o The SRCU primitives (srcu_read_lock(), srcu_read_unlock(), and synchronize_srcu()) all take a pointer to a srcu_struct. o The SRCU primitives must be called from process context. o srcu_read_lock() returns an int that must be passed to the matching srcu_read_unlock(). Realtime RCU avoids the need for this by storing the state in the task struct, but SRCU needs to allow a given code path to pass through multiple SRCU domains -- storing state in the task struct would therefore require either arbitrary space in the task struct or arbitrary limits on SRCU nesting. So I kicked the state-storage problem up to the caller. o There is no call_srcu(). It would not be hard to implement one, but it seems like too easy a way to OOM the system. (Hey, we have enough trouble with call_rcu(), which does -not- permit readers to sleep!!!) So, if you want it, please tell me why... Signed-off-by: Paul E. McKenney Signed-off-by: Andrew Morton --- Documentation/RCU/checklist.txt | 32 +++++ Documentation/RCU/rcu.txt | 3 Documentation/RCU/whatisRCU.txt | 3 include/linux/srcu.h | 42 +++++++ kernel/Makefile | 2 kernel/srcu.c | 173 ++++++++++++++++++++++++++++++ 6 files changed, 253 insertions(+), 2 deletions(-) diff -puN Documentation/RCU/checklist.txt~srcu-rcu-variant-permitting-read-side-blocking Documentation/RCU/checklist.txt --- a/Documentation/RCU/checklist.txt~srcu-rcu-variant-permitting-read-side-blocking +++ a/Documentation/RCU/checklist.txt @@ -221,3 +221,35 @@ over a rather long period of time, but i disable irq on a given acquisition of that lock will result in deadlock as soon as the RCU callback happens to interrupt that acquisition's critical section. + +13. SRCU (srcu_read_lock(), srcu_read_unlock(), and synchronize_srcu()) + may only be invoked from process context. Unlike other forms of + RCU, it -is- permissible to block in an SRCU read-side critical + section (demarked by srcu_read_lock() and srcu_read_unlock()), + hence the "SRCU": "sleepable RCU". + + Also unlike other forms of RCU, explicit initialization + and cleanup is required via init_srcu_struct() and + cleanup_srcu_struct(). These are passed a "struct srcu_struct" + that defines the scope of a given SRCU domain. Once initialized, + the srcu_struct is passed to srcu_read_lock(), srcu_read_unlock() + and synchronize_srcu(). A given synchronize_srcu() waits only + for SRCU read-side critical sections governed by srcu_read_lock() + and srcu_read_unlock() calls that have been passd the same + srcu_struct. This property is what makes sleeping read-side + critical sections tolerable -- a given subsystem delays only + its own updates, not those of other subsystems using SRCU. + + This benefit does not come for free. First, corresponding + srcu_read_lock() and srcu_read_unlock() calls must be passed + the same srcu_struct. Second, grace-period-detection overhead + is amortized only over those updates sharing a given srcu_struct, + rather than being globally amortized as they are for other forms + of RCU. This means that SRCU should be used only in extremely + read-intensive situations if performance is the goal (of course, + if the goal is instead read-side deadlock avoidance or read-side + realtime latency, then SRCU might be applicable to less extreme + degrees of read intensity). + + Note that, rcu_assign_pointer() and rcu_dereference() relate to + SRCU just as they do to other forms of RCU. diff -puN Documentation/RCU/rcu.txt~srcu-rcu-variant-permitting-read-side-blocking Documentation/RCU/rcu.txt --- a/Documentation/RCU/rcu.txt~srcu-rcu-variant-permitting-read-side-blocking +++ a/Documentation/RCU/rcu.txt @@ -45,7 +45,8 @@ o How can I see where RCU is currently u Search for "rcu_read_lock", "rcu_read_unlock", "call_rcu", "rcu_read_lock_bh", "rcu_read_unlock_bh", "call_rcu_bh", - "synchronize_rcu", and "synchronize_net". + "srcu_read_lock", "srcu_read_unlock", "synchronize_rcu", + "synchronize_net", and "synchronize_srcu". o What guidelines should I follow when writing code that uses RCU? diff -puN Documentation/RCU/whatisRCU.txt~srcu-rcu-variant-permitting-read-side-blocking Documentation/RCU/whatisRCU.txt --- a/Documentation/RCU/whatisRCU.txt~srcu-rcu-variant-permitting-read-side-blocking +++ a/Documentation/RCU/whatisRCU.txt @@ -777,6 +777,8 @@ Markers for RCU read-side critical secti rcu_read_unlock rcu_read_lock_bh rcu_read_unlock_bh + srcu_read_lock + srcu_read_unlock RCU pointer/list traversal: @@ -803,6 +805,7 @@ RCU grace period: synchronize_net synchronize_sched synchronize_rcu + synchronize_srcu call_rcu call_rcu_bh diff -puN /dev/null include/linux/srcu.h --- /dev/null +++ a/include/linux/srcu.h @@ -0,0 +1,42 @@ +/* + * Sleepable Read-Copy Update mechanism for mutual exclusion + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * Copyright (C) IBM Corporation, 2006 + * + * Author: Paul McKenney + * + * For detailed explanation of Read-Copy Update mechanism see - + * Documentation/RCU/ *.txt + * + */ + +struct srcu_struct_array { + int c[2]; +} ____cacheline_internode_aligned_in_smp; + +struct srcu_struct { + int completed; + struct srcu_struct_array *per_cpu_ref; + struct mutex mutex; +}; + +void init_srcu_struct(struct srcu_struct *sp); +void cleanup_srcu_struct(struct srcu_struct *sp); +int srcu_read_lock(struct srcu_struct *sp); +void srcu_read_unlock(struct srcu_struct *sp, int idx); +void synchronize_srcu(struct srcu_struct *sp); +long srcu_batches_completed(struct srcu_struct *sp); diff -puN kernel/Makefile~srcu-rcu-variant-permitting-read-side-blocking kernel/Makefile --- a/kernel/Makefile~srcu-rcu-variant-permitting-read-side-blocking +++ a/kernel/Makefile @@ -8,7 +8,7 @@ obj-y = sched.o fork.o exec_domain.o signal.o sys.o kmod.o workqueue.o pid.o \ rcupdate.o extable.o params.o posix-timers.o \ kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \ - hrtimer.o rwsem.o nsproxy.o + hrtimer.o rwsem.o nsproxy.o srcu.o obj-$(CONFIG_STACKTRACE) += stacktrace.o obj-y += time/ diff -puN /dev/null kernel/srcu.c --- /dev/null +++ a/kernel/srcu.c @@ -0,0 +1,173 @@ +/* + * Sleepable Read-Copy Update mechanism for mutual exclusion. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * Copyright (C) IBM Corporation, 2006 + * + * Author: Paul McKenney + * + * For detailed explanation of Read-Copy Update mechanism see - + * Documentation/RCU/ *.txt + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +/** + * init_srcu_struct - initialize a sleep-RCU structure + * @sp: structure to initialize. + * + * Must invoke this on a given srcu_struct before passing that srcu_struct + * to any other function. Each srcu_struct represents a separate domain + * of SRCU protection. + */ +void init_srcu_struct(struct srcu_struct *sp) +{ + int cpu; + + sp->completed = 0; + sp->per_cpu_ref = (struct srcu_struct_array *) + kmalloc(NR_CPUS * sizeof(*sp->per_cpu_ref), + GFP_KERNEL); + for_each_cpu(cpu) { + sp->per_cpu_ref[cpu].c[0] = 0; + sp->per_cpu_ref[cpu].c[1] = 0; + } + mutex_init(&sp->mutex); +} + +/** + * cleanup_srcu_struct - deconstruct a sleep-RCU structure + * @sp: structure to clean up. + * + * Must invoke this after you are finished using a given srcu_struct. + * Failure to do so will result in a memory leak. + */ +void cleanup_srcu_struct(struct srcu_struct *sp) +{ + kfree(sp->per_cpu_ref); + sp->per_cpu_ref = NULL; +} + +/** + * srcu_read_lock - register a new reader for an SRCU-protected structure. + * @sp: srcu_struct in which to register the new reader. + * + * Counts the new reader in the appropriate per-CPU element of the + * srcu_struct. Must be called from process context. + * Returns an index that must be passed to the matching srcu_read_unlock(). + */ +int srcu_read_lock(struct srcu_struct *sp) +{ + int idx; + + preempt_disable(); + idx = sp->completed & 0x1; + barrier(); + sp->per_cpu_ref[smp_processor_id()].c[idx]++; + preempt_enable(); + return idx; +} + +/** + * srcu_read_unlock - unregister a old reader from an SRCU-protected structure. + * @sp: srcu_struct in which to unregister the old reader. + * @idx: return value from corresponding srcu_read_lock(). + * + * Removes the count for the old reader from the appropriate per-CPU + * element of the srcu_struct. Note that this may well be a different + * CPU than that which was incremented by the corresponding srcu_read_lock(). + * Must be called from process context. + */ +void srcu_read_unlock(struct srcu_struct *sp, int idx) +{ + preempt_disable(); + sp->per_cpu_ref[smp_processor_id()].c[idx]--; + preempt_enable(); +} + +/** + * synchronize_srcu - wait for prior SRCU read-side critical-section completion + * @sp: srcu_struct with which to synchronize. + * + * Flip the completed counter, and wait for the old count to drain to zero. + * As with classic RCU, the updater must use some separate means of + * synchronizing concurrent updates. Can block; must be called from + * process context. + */ +void synchronize_srcu(struct srcu_struct *sp) +{ + int cpu; + int idx; + int sum; + + might_sleep(); + + mutex_lock(&sp->mutex); + + smp_mb(); /* Prevent operations from leaking in. */ + + idx = sp->completed & 0x1; + sp->completed++; + + synchronize_sched(); /* forces memory barriers all around. */ + + /* + * At this point, all srcu_read_lock() calls using the + * old counters have completed. + */ + + for (;;) { + sum = 0; + for_each_cpu(cpu) { + sum += sp->per_cpu_ref[cpu].c[idx]; + } + if (sum == 0) + break; + schedule_timeout_interruptible(1); + } + + synchronize_sched(); /* forces memory barriers all around. */ + + mutex_unlock(&sp->mutex); +} + +/** + * srcu_batches_completed - return batches completed. + * @sp: srcu_struct on which to report batch completion. + * + * Report the number of batches, correlated with, but not necessarily + * precisely the same as, the number of grace periods that have elapsed. + */ + +long srcu_batches_completed(struct srcu_struct *sp) +{ + return (sp->completed); +} + +EXPORT_SYMBOL_GPL(init_srcu_struct); +EXPORT_SYMBOL_GPL(cleanup_srcu_struct); +EXPORT_SYMBOL_GPL(srcu_read_lock); +EXPORT_SYMBOL_GPL(srcu_read_unlock); +EXPORT_SYMBOL_GPL(synchronize_srcu); +EXPORT_SYMBOL_GPL(srcu_batches_completed); _