Subject: Introduce this_cpu_ptr() and generic this_cpu_* operations this_cpu_ptr ------------ this_cpu_ptr(xx) = per_cpu_ptr(xx, smp_processor_id). The problem with per_cpu_ptr(x, smp_processor_id) is that it requires an array lookup to find the offset for the cpu. Processors typically have the offset for the current cpu area in some kind of (arch dependent) efficiently accessible register or memory location. We can use that instead of doing the array lookup to speed up the determination of the addressof the percpu variable. This is particularly significant because these lookups occur in performance critical paths of the core kernel. this_cpu_ptr comes in two flavors. The preemption context matters since we are referring the the currently executing processor. In many cases we must insure that the processor does not change while a code segment is executed. __this_cpu_ptr -> Do not check for preemption context this_cpu_ptr -> Check preemption context The parameter to these operations is a per cpu pointer. Typically the pointer would point to a structure with per cpu data. This can be the address of a statically defined per cpu variable (&per_cpu_var(xxx)) or the address of a per cpu variable allocated with the per cpu allocator. Generic this_cpu_* operations ----------------------------- this_cpu_* operations operate on scalars that are members of structures allocated with the new per cpu allocator. They can also operate on static per_cpu variables if the address of those is determined with per_cpu_var(). Generic fallback functions are used if an arch does not define optimized this_cpu operations. The functions come also come in the two flavors used for this_cpu_ptr(). The firstparameter is a scalar that is a member of a structure allocated through allocpercpu or a per cpu variable (use per_cpu_var(xxx)). The operations are similar to what percpu_add() and friends. The operations are guaranteed to be atomic vs preemption if they modify the scalar (unless they are prefixed by __ in which case they do not need to be). The calculation of the per cpu offset is also guaranteed to be atomic in the same way. A this_cpu_* operation can be safely in a context where interrupts are enabled and preemption is allowed. this_cpu_read(scalar) this_cpu_write(scalar, value) this_cpu_add(scale, value) this_cpu_sub(scalar, value) this_cpu_inc(scalar) this_cpu_dec(scalar) this_cpu_and(scalar, value) this_cpu_or(scalar, value) this_cpu_xor(scalar, value) Arch code can override the generic functions and provide optimized atomic per cpu operations. These atomic operations must provide both the relocation (x86 does it through a segment override) and the operation on the data in a single instruction. Otherwise preempt needs to be disabled and there is no gain from providing arch implementations. A third variant is provided prefixed by irqsafe_. These variants are safe against hardware interrupts on the *same* processor (all per cpu atomic primitives are *always* *only* providing safety for code running on the *same* processor!). The increment needs to be implemented by the hardware in such a way that it is a single RMW instruction that is either processed before or after an interrupt. cc: David Howells cc: Tejun Heo cc: Ingo Molnar cc: Rusty Russell cc: Eric Dumazet Signed-off-by: Christoph Lameter --- include/asm-generic/percpu.h | 5 include/linux/percpu.h | 434 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 439 insertions(+) Index: linux-2.6/include/linux/percpu.h =================================================================== --- linux-2.6.orig/include/linux/percpu.h 2009-09-29 11:57:01.000000000 -0500 +++ linux-2.6/include/linux/percpu.h 2009-09-29 12:18:06.000000000 -0500 @@ -243,4 +243,438 @@ do { \ # define percpu_xor(var, val) __percpu_generic_to_op(var, (val), ^=) #endif +/* + * Branching function to split up a function into a set of functions that + * are called for different scalar sizes of the objects handled. + */ + +extern void __bad_size_call_parameter(void); + +#define __size_call_return(stem, variable) \ +({ \ + switch(sizeof(variable)) { \ + case 1: return stem##1(variable); \ + case 2: return stem##2(variable); \ + case 4: return stem##4(variable); \ + case 8: return stem##8(variable); \ + default: \ + __bad_size_call_parameter(); \ + } \ +}) + +#define __size_call(stem, variable, ...) \ +do { \ + switch(sizeof(variable)) { \ + case 1: stem##1(variable, __VA_ARGS__);break; \ + case 2: stem##2(variable, __VA_ARGS__);break; \ + case 4: stem##4(variable, __VA_ARGS__);break; \ + case 8: stem##8(variable, __VA_ARGS__);break; \ + default: \ + __bad_size_call_parameter();break; \ + } \ +} while (0) + +/* + * Optimized manipulation for memory allocated through the per cpu + * allocator or for addresses of per cpu variables (can be determined + * using per_cpu_var(xx). + * + * These operation guarantee exclusivity of access for other operations + * on the *same* processor. The assumption is that per cpu data is only + * accessed by a single processor instance (the current one). + * + * The first group is used for accesses that must be done in a + * preemption safe way since we know that the context is not preempt + * safe. Interrupts may occur. If the interrupt modifies the variable + * too then RMW actions will not be reliable. + * + * The arch code can provide optimized functions in two ways: + * + * 1. Override the function completely. F.e. define this_cpu_add(). + * The arch must then ensure that the various scalar format passed + * are handled correctly. + * + * 2. Provide functions for certain scalar sizes. F.e. provide + * this_cpu_add_2() to provide per cpu atomic operations for 2 byte + * sized RMW actions. If arch code does not provide operations for + * a scalar size then the fallback in the generic code will be + * used. + */ + + +/* + * Read operations are generally atomic up to the word size of the + * processor. + */ +#ifndef this_cpu_read +# define this_cpu_read(pcp) \ + ({ \ + *this_cpu_ptr(&(pcp)); \ + }) +#else +# ifndef this_cpu_read_1 +# define this_cpu_read_1(pcp) (*this_cpu_ptr(&(pcp)) +# endif +# ifndef this_cpu_read_2 +# define this_cpu_read_2(pcp) (*this_cpu_ptr(&(pcp)) +# endif +# ifndef this_cpu_read_4 +# define this_cpu_read_4(pcp) (*this_cpu_ptr(&(pcp)) +# endif +# ifndef this_cpu_read_8 +# ifdef CONFIG_64Bit +# define this_cpu_read_8(pcp) (*this_cpu_ptr(&(pcp)) +# else +# define this_cpu_read_8(pcp) (preempt_disable();*this_cpu_ptr(&(pcp);preempt_enable()) +# endif +# endif +# define this_cpu_read(pcp) __size_call_return(this_cpu_read_, (pcp) +#endif + +#define _this_cpu_generic_to_op(pcp, val, op) \ +do { \ + preempt_disable(); \ + *__this_cpu_ptr(&pcp) op val; \ + preempt_enable_no_resched(); \ +} while (0) + +#ifndef this_cpu_write +# define this_cpu_write(pcp, val) _this_cpu_generic_to_op((pcp), (val), =) +#else +# ifndef this_cpu_write_1 +# define this_cpu_write_1(pcp, val) _this_cpu_generic_to_op((pcp), (val), =) +# endif +# ifndef this_cpu_write_2 +# define this_cpu_write_2(pcp, val) _this_cpu_generic_to_op((pcp), (val), =) +# endif +# ifndef this_cpu_write_4 +# define this_cpu_write_4(pcp, val) _this_cpu_generic_to_op((pcp), (val), =) +# endif +# ifndef this_cpu_write_8 +# define this_cpu_write_8(pcp, val) _this_cpu_generic_to_op((pcp), (val), =) +# endif +# define this_cpu_write(pcp, val) __size_call(this_cpu_write_, (pcp), (val)) +#endif + +#ifndef this_cpu_add +# define this_cpu_add(pcp, val) _this_cpu_generic_to_op((pcp), (val), +=) +#else +# ifndef this_cpu_add_1 +# define this_cpu_add_1(pcp, val) _this_cpu_generic_to_op((pcp), (val), +=) +# endif +# ifndef this_cpu_add_2 +# define this_cpu_add_2(pcp, val) _this_cpu_generic_to_op((pcp), (val), +=) +# endif +# ifndef this_cpu_add_4 +# define this_cpu_add_4(pcp, val) _this_cpu_generic_to_op((pcp), (val), +=) +# endif +# ifndef this_cpu_add_8 +# define this_cpu_add_8(pcp, val) _this_cpu_generic_to_op((pcp), (val), +=) +# endif +# define this_cpu_add(pcp, val) __size_call(this_cpu_add_, (pcp), (val)) +#endif + +#ifndef this_cpu_sub +# define this_cpu_sub(pcp, val) this_cpu_add((pcp), -(val)) +#endif + +#ifndef this_cpu_inc +# define this_cpu_inc(pcp) this_cpu_add((pcp), 1) +#endif + +#ifndef this_cpu_dec +# define this_cpu_dec(pcp) this_cpu_sub((pcp), 1) +#endif + +#ifndef this_cpu_and +# define this_cpu_and(pcp, val) _this_cpu_generic_to_op((pcp), (val), &=) +#else +# ifndef this_cpu_and_1 +# define this_cpu_and_1(pcp, val) _this_cpu_generic_to_op((pcp), (val), &=) +# endif +# ifndef this_cpu_and_2 +# define this_cpu_and_2(pcp, val) _this_cpu_generic_to_op((pcp), (val), &=) +# endif +# ifndef this_cpu_and_4 +# define this_cpu_and_4(pcp, val) _this_cpu_generic_to_op((pcp), (val), &=) +# endif +# ifndef this_cpu_and_8 +# define this_cpu_and_8(pcp, val) _this_cpu_generic_to_op((pcp), (val), &=) +# endif +# define this_cpu_and(pcp, val) __size_call(this_cpu_and_, (pcp), (val)) +#endif + +#ifndef this_cpu_or +# define this_cpu_or(pcp, val) _this_cpu_generic_to_op((pcp), (val), |=) +#else +# ifndef this_cpu_or_1 +# define this_cpu_or_1(pcp, val) _this_cpu_generic_to_op((pcp), (val), |=) +# endif +# ifndef this_cpu_or_2 +# define this_cpu_or_2(pcp, val) _this_cpu_generic_to_op((pcp), (val), |=) +# endif +# ifndef this_cpu_or_4 +# define this_cpu_or_4(pcp, val) _this_cpu_generic_to_op((pcp), (val), |=) +# endif +# ifndef this_cpu_or_8 +# define this_cpu_or_8(pcp, val) _this_cpu_generic_to_op((pcp), (val), |=) +# endif +# define this_cpu_or(pcp, val) __size_call(this_cpu_or_, (pcp), (val)) +#endif + +#ifndef this_cpu_xor +# define this_cpu_xor(pcp, val) _this_cpu_generic_to_op((pcp), (val), ^=) +#else +# ifndef this_cpu_xor_1 +# define this_cpu_xor_1(pcp, val) _this_cpu_generic_to_op((pcp), (val), ^=) +# endif +# ifndef this_cpu_xor_2 +# define this_cpu_xor_2(pcp, val) _this_cpu_generic_to_op((pcp), (val), ^=) +# endif +# ifndef this_cpu_xor_4 +# define this_cpu_xor_4(pcp, val) _this_cpu_generic_to_op((pcp), (val), ^=) +# endif +# ifndef this_cpu_xor_8 +# define this_cpu_xor_8(pcp, val) _this_cpu_generic_to_op((pcp), (val), ^=) +# endif +# define this_cpu_xor(pcp, val) __size_call(this_cpu_or_, (pcp), (val)) +#endif + +/* + * Generic percpu operations that do not require preemption handling. + * Either we do not care about races or the caller has the + * responsibility of handling preemptions issues. + * + * If there is no other protection through preempt disable and/or + * disabling interupts then one of these RMW operations can show unexpected + * behavior because the execution thread was rescheduled on another processor + * or an interrupt occurred and the same percpu variable was modified from + * the interrupt context. + */ +#ifndef __this_cpu_read +# define __this_cpu_read(pcp) \ + ({ \ + *__this_cpu_ptr(&(pcp)); \ + }) +#else +# ifndef __this_cpu_read_1 +# define __this_cpu_read_1(pcp) (*this_cpu_ptr(&(pcp)) +# endif +# ifndef __this_cpu_read_2 +# define __this_cpu_read_2(pcp) (*this_cpu_ptr(&(pcp)) +# endif +# ifndef __this_cpu_read_4 +# define __this_cpu_read_4(pcp) (*this_cpu_ptr(&(pcp)) +# endif +# ifndef __this_cpu_read_8 +# define __this_cpu_read_8(pcp) (*this_cpu_ptr(&(pcp)) +# endif +# define __this_cpu_read(pcp) __size_call_return(__this_cpu_read_, (pcp) +#endif + +#define __this_cpu_generic_to_op(pcp, val, op) \ +do { \ + *__this_cpu_ptr(&(pcp)) op val; \ +} while (0) + +#ifndef __this_cpu_write +# define __this_cpu_write(pcp, val) __this_cpu_generic_to_op((pcp), (val), =) +#else +# ifndef __this_cpu_write_1 +# define __this_cpu_write_1(pcp,, val) __this_cpu_generic_to_op((pcp), (val), =) +# endif +# ifndef __this_cpu_write_2 +# define __this_cpu_write_2(pcp,, val) __this_cpu_generic_to_op((pcp), (val), =) +# endif +# ifndef __this_cpu_write_4 +# define __this_cpu_write_4(pcp,, val) __this_cpu_generic_to_op((pcp), (val), =) +# endif +# ifndef __this_cpu_write_8 +# define __this_cpu_write_8(pcp,, val) __this_cpu_generic_to_op((pcp), (val), =) +# endif +# define __this_cpu_write(pcp, val) __size_call(__this_cpu_write_, pcp, val) +#endif + +#ifndef __this_cpu_add +# define __this_cpu_add(pcp, val) __this_cpu_generic_to_op((pcp), (val), +=) +#else +# ifndef __this_cpu_add_1 +# define __this_cpu_add_1(pcp, val) __this_cpu_generic_to_op((pcp), (val), +=) +# endif +# ifndef __this_cpu_add_2 +# define __this_cpu_add_2(pcp, val) __this_cpu_generic_to_op((pcp), (val), +=) +# endif +# ifndef __this_cpu_add_4 +# define __this_cpu_add_4(pcp, val) __this_cpu_generic_to_op((pcp), (val), +=) +# endif +# ifndef __this_cpu_add_8 +# define __this_cpu_add_8(pcp, val) __this_cpu_generic_to_op((pcp), (val), +=) +# endif +# define __this_cpu_add(pcp, val) __size_call(__this_cpu_add_, pcp, val) +#endif + +#ifndef __this_cpu_sub +# define __this_cpu_sub(pcp, val) __this_cpu_add((pcp), -(var)) +#endif + +#ifndef __this_cpu_inc +# define __this_cpu_inc(pcp) __this_cpu_add((pcp), 1) +#endif + +#ifndef __this_cpu_dec +# define __this_cpu_dec(pcp) __this_cpu_sub((pcp), 1) +#endif + +#ifndef __this_cpu_and +# define __this_cpu_and(pcp, val) __this_cpu_generic_to_op((pcp), (val), &=) +#else +# ifndef __this_cpu_and_1 +# define __this_cpu_and_1(pcp, val) __this_cpu_generic_to_op((pcp), (val), &=) +# endif +# ifndef __this_cpu_and_2 +# define __this_cpu_and_2(pcp, val) __this_cpu_generic_to_op((pcp), (val), &=) +# endif +# ifndef __this_cpu_and_4 +# define __this_cpu_and_4(pcp, val) __this_cpu_generic_to_op((pcp), (val), &=) +# endif +# ifndef __this_cpu_and_8 +# define __this_cpu_and_8(pcp, val) __this_cpu_generic_to_op((pcp), (val), &=) +# endif +# define __this_cpu_and(pcp, val) __size_call(__this_cpu_and_, pcp, val) +#endif + +#ifndef __this_cpu_or +# define __this_cpu_or(pcp, val) __this_cpu_generic_to_op((pcp), (val), |=) +#else +# ifndef __this_cpu_or_1 +# define __this_cpu_or_1(pcp, val) __this_cpu_generic_to_op((pcp), (val), |=) +# endif +# ifndef __this_cpu_or_2 +# define __this_cpu_or_2(pcp, val) __this_cpu_generic_to_op((pcp), (val), |=) +# endif +# ifndef __this_cpu_or_4 +# define __this_cpu_or_4(pcp, val) __this_cpu_generic_to_op((pcp), (val), |=) +# endif +# ifndef __this_cpu_or_8 +# define __this_cpu_or_8(pcp, val) __this_cpu_generic_to_op((pcp), (val), |=) +# endif +# define __this_cpu_or(pcp, val) __size_call(__this_cpu_or_, pcp, val) +#endif + +#ifndef __this_cpu_xor +# define __this_cpu_xor(pcp, val) __this_cpu_generic_to_op((pcp), (val), ^=) +#else +# ifndef __this_cpu_xor_1 +# define __this_cpu_xor_1(pcp,, val) __this_cpu_generic_to_op((pcp), (val), ^=) +# endif +# ifndef __this_cpu_xor_2 +# define __this_cpu_xor_2(pcp,, val) __this_cpu_generic_to_op((pcp), (val), ^=) +# endif +# ifndef __this_cpu_xor_4 +# define __this_cpu_xor_4(pcp,, val) __this_cpu_generic_to_op((pcp), (val), ^=) +# endif +# ifndef __this_cpu_xor_8 +# define __this_cpu_xor_8(pcp,, val) __this_cpu_generic_to_op((pcp), (val), ^=) +# endif +# define __this_cpu_xor(pcp, val) __size_call(__this_cpu_xor_, pcp, val) +#endif + +/* + * IRQ safe versions of the per cpu RMW operations. Note that these operations + * are *not* safe against modification of the same variable from another + * processors. They are guaranteed to be atomic vs. local interrupts and + * preemption. + */ +#define irqsafe_cpu_generic_to_op(pcp, val, op) \ +do { \ + unsigned long flags; \ + local_irqsave(flags); \ + *__this_cpu_ptr(&(pcp)) op val; \ + local_irqrestore(flags); \ +} while (0) + +#ifndef irqsafe_this_cpu_add +# define irqsafe_this_cpu_add(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), +=) +#else +# ifndef irqsafe_this_cpu_add_1 +# define(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), +=) +# endif +# ifndef irqsafe_this_cpu_add_2 +# define(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), +=) +# endif +# ifndef irqsafe_this_cpu_add_4 +# define(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), +=) +# endif +# ifndef irqsafe_this_cpu_add_8 +# define(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), +=) +# endif +# define irqsafe_this_cpu_add(pcp, val) __size_call(irqsafe_this_cpu_add_, (val)) +#endif + +#ifndef irqsafe_this_cpu_sub +# define irqsafe_this_cpu_sub(pcp, val) irqsafe_cpu_add((pcp), -(var)) +#endif + +#ifndef irqsafe_this_cpu_inc +# define irqsafe_this_cpu_inc(pcp) irqsafe_cpu_add((pcp), 1) +#endif + +#ifndef irqsafe_this_cpu_dec +# define irqsafe_this_cpu_dec(pcp) irqsafe_cpu_sub((pcp), 1) +#endif + +#ifndef irqsafe_this_cpu_and +# define irqsafe_this_cpu_and(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), &=) +#else +# ifndef irqsafe_this_cpu_and_1 +# define irqsafe_this_cpu_and_1(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), &=) +# endif +# ifndef irqsafe_this_cpu_and_2 +# define irqsafe_this_cpu_and_2(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), &=) +# endif +# ifndef irqsafe_this_cpu_and_4 +# define irqsafe_this_cpu_and_4(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), &=) +# endif +# ifndef irqsafe_this_cpu_and_8 +# define irqsafe_this_cpu_and_8(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), &=) +# endif +# define irqsafe_this_cpu_and(pcp, val) __size_call(irqsafe_this_cpu_and_, (val)) +#endif + +#ifndef irqsafe_this_cpu_or +# define irqsafe_this_cpu_or(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), |=) +#else +# ifndef irqsafe_this_cpu_or_1 +# define irqsafe_this_cpu_or_1(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), |=) +# endif +# ifndef irqsafe_this_cpu_or_2 +# define irqsafe_this_cpu_or_2(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), |=) +# endif +# ifndef irqsafe_this_cpu_or_4 +# define irqsafe_this_cpu_or_4(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), |=) +# endif +# ifndef irqsafe_this_cpu_or_8 +# define irqsafe_this_cpu_or(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), |=) +# endif +# define irqsafe_this_cpu_or(pcp, val) __size_call(irqsafe_this_cpu_or_, (val)) +#endif + +#ifndef irqsafe_this_cpu_xor +# define irqsafe_this_cpu_xor(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), ^=) +#else +# ifndef irqsafe_this_cpu_xor_1 +# define irqsafe_this_cpu_xor_1(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), ^=) +# endif +# ifndef irqsafe_this_cpu_xor_2 +# define irqsafe_this_cpu_xor_2(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), ^=) +# endif +# ifndef irqsafe_this_cpu_xor_4 +# define irqsafe_this_cpu_xor_4(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), ^=) +# endif +# ifndef irqsafe_this_cpu_xor_8 +# define irqsafe_this_cpu_xor_8(pcp, val) irqsafe_cpu_generic_to_op((pcp), (val), ^=) +# endif +# define irqsafe_this_cpu_add(pcp, val) __size_call(irqsafe_this_cpu_add_, (val)) +#endif + #endif /* __LINUX_PERCPU_H */ Index: linux-2.6/include/asm-generic/percpu.h =================================================================== --- linux-2.6.orig/include/asm-generic/percpu.h 2009-09-29 11:57:01.000000000 -0500 +++ linux-2.6/include/asm-generic/percpu.h 2009-09-29 12:08:04.000000000 -0500 @@ -56,6 +56,9 @@ extern unsigned long __per_cpu_offset[NR #define __raw_get_cpu_var(var) \ (*SHIFT_PERCPU_PTR(&per_cpu_var(var), __my_cpu_offset)) +#define this_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, my_cpu_offset) +#define __this_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, __my_cpu_offset) + #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA extern void setup_per_cpu_areas(void); @@ -66,6 +69,8 @@ extern void setup_per_cpu_areas(void); #define per_cpu(var, cpu) (*((void)(cpu), &per_cpu_var(var))) #define __get_cpu_var(var) per_cpu_var(var) #define __raw_get_cpu_var(var) per_cpu_var(var) +#define this_cpu_ptr(ptr) per_cpu_ptr(ptr, 0) +#define __this_cpu_ptr(ptr) this_cpu_ptr(ptr) #endif /* SMP */