Extending High res Timers Adding New Platforms to

Extending High res Timers Adding New Platforms to the POSIX high resolution timers code

Presentation Objective n Basic high-res-timers patch adds POSIX timers to all platforms. This does not include high-res code for platforms other than the x 86. n This presentation will discuss how to add the high-res part to new platforms. Powering the Embedded Revolution

Outline n Header file linkage n Time keeping vs. Timer interrupts n Functions and macros Powering the Embedded Revolution

Header File Linkage n Only linux/hrtime. h is included from core system files. n Linux/hrtime. h: #ifdef CONFIG_HIGH_RES_TIMERS #include <asm/hrtime. h> n CONFIG_HIGH_RES_TIMERS is defined in the arch config. in file. Powering the Embedded Revolution

linux/hrtime. h #ifndef _HRTIME_H #define DEBUGx #if defined( DEBUG) && defined(CONFIG_KGDB) #define IF_DEBUG(a) a #include <asm/kgdb. h> #else #define IF_DEBUG(a) #endif #ifndef kgdb_ts #define kgdb_ts(data 0, data 1) #endif Powering the Embedded Revolution

linux/hrtime. h (continued) /* * This file is the glue to bring in the platform stuff. * We make it all depend on the CONFIG option so all archs * will work as long as the CONFIG is not set. Once an * arch defines the CONFIG, it had better have the * asm/hrtime. h file in place. */ #include <linux/config. h> #ifdef CONFIG_HIGH_RES_TIMERS #include <asm/hrtime. h> Powering the Embedded Revolution

Time vs. tick interrupts n In the normal linux, tick interrupts are used to drive the clock. n We want to separate these two so that we can use the interrupts to drive timer expiration events. n We also want to be able to determine time to at least 1 microsecond resolution at any time. Powering the Embedded Revolution

Time vs. tick interrupts (continued) n To the extent that a platforms gettimeofday() function has resolution greater than a jiffie this is already present in most platforms. n However, it may be pegged in some way to the last tick interrupt. n We want to remove dependence on the tick interrupt from time calculations. Powering the Embedded Revolution

Good sources for system time n What are good sources for time? n It should be a cpu register that is clocked uniformly at a rate that gives at least microsecond resolution. n It need not have any interrupt capability. n It should roll over at some binary value greater than several jiffies. Powering the Embedded Revolution

Why a cpu register? n It should be a cpu register so that it can be read quickly. Access to a source external to the cpu is an I/O access and is not only slow, but is a sync point in most archs. (Sync points are points where the processor must not do OOE (Out of Order Execution) and usually must flush the pipe line. ) Powering the Embedded Revolution

Why no need to interrupt? n We will be looking at this source every tick and in the best case this will be every jiffie. We can, therefore, detect any issues that an interrupt might want to signal. n Systems that use this source to derive tick interrupts are ok as long as this usage does not affect the count accuracy. (E. g. PPC, HPPA) Powering the Embedded Revolution

Why several jiffies? n If the counter rolls over in less than several jiffies, we need to interrupt more often so as not to miss a roll over. “Several jiffies” allows us to be a “little” late and still not loose system time. Powering the Embedded Revolution

Why binary? n It should be binary so we can do simple roll over math. * These are specific to the ACPI pm counter * The spec says the counter can be either 32 or 24 bits wide. * We treat them both as 24 bits. Its faster than doing the test. */ #define SIZE_MASK 0 xffffff extern int acpi_pm_tmr_address; extern inline unsigned long get_cpuctr(void) { static long old; old = last_update; last_update = inl(acpi_pm_tmr_address); return (last_update - old) & SIZE_MASK; } Powering the Embedded Revolution

A word about sub_jiffie units n You may choose the units to use for the sub_jiffie part of time, however… n It is probably best to use the natural units of your high resolution clock source. This will keep the conversion overhead to a minimum. n For the three x 86 clock options I used three different units (PIT clock, ACPI pm timer clock, TSC frequency) Powering the Embedded Revolution

Externally visible macros n Macros visible to core code are: n sub_jiffie() macro n update_jiffies() n schedule_next_int(a, b) n high_res_test() macro Powering the Embedded Revolution

Sub_jiffie macro (linux/hrtime. h) /* * The sub_jiffie() macro should return the current time offset from the latest * jiffie. This will be in "arch" defined units and is used to determine if * a timer has expired. Since no sub_expire value will be used if "arch" * has not defined the high-res package, 0 will work well here. * * In addition, to save time if there is no high-res package (or it is not * configured), we define the sub expression for the run_timer_list. */ #ifndef sub_jiffie #undef CONFIG_HIGH_RES_TIMERS #define sub_jiffie() 0 #endif // sub_jiffie And from i 386_asm/hrtime. h extern int _sub_jiffie; #define sub_jiffie() _sub_jiffie Powering the Embedded Revolution

update_jiffies macro (linux/hrtime. h) /* * These should have been defined in the platform hrtimer. h * If not (or HIGH_RES_TIMERS not configured) define the default. */ #ifndef update_jiffies extern u 64 jiffies_64; #define update_jiffies() (*(u 64 *)&jiffies_64)++ #endif #ifndef new_jiffie #define new_jiffie() 0 #endif Powering the Embedded Revolution

update_jiffies macro (include/i 386_asm/hrtime. h) : : #define update_jiffies() update_jiffies_sub() : : /* * This routine is always called under the write_lockirq(xtime_lock) */ extern inline void update_jiffies_sub(void) { unsigned long cycles_update; cycles_update = get_cpuctr(); arch_update_jiffies(cycles_update); } Powering the Embedded Revolution

arch_update_jiffies (include/i 386_asm/hrtime. h) extern inline void arch_update_jiffies (unsigned long update) { int _new_jiffie = 0; /* * update is the delta in sub_jiffies */ kgdb_ts(update, _sub_jiffie); _sub_jiffie += update; while (_sub_jiffie > cycles_per_jiffies){ _sub_jiffie -= cycles_per_jiffies; _new_jiffie++; } if (_new_jiffie) { jiffies_intr += _new_jiffie; jiffies_64 += _new_jiffie; } } Powering the Embedded Revolution

schedule_next_int macro (linux/hrtime. h) /* * * * */ The schedule_next_int function is to be defined by the "arch" code when an "arch" is implementing the high-res part of POSIX timers. The actual function will be called with the offset in "arch" (parm 2) defined sub_jiffie units from the reference jiffie boundry (parm 1)to the next required sub_jiffie timer interrupt. This value will be -1 if the next timer interrupt should be the next jiffie value. The "arch" code must determine how far out the interrupt is, based on current jiffie, sub_jiffie time and set up the hardware to interrupt at that time. It is possible that the time will already have passed, in which case the function should return true (no interrupt is needed), otherwise the return should be 0. If the requested interrupt is "close" to a jiffie interrupt, the two can be rolled into one by calling do_timer(), however, this will advance "jiffie" so "close" should be in the order of the interrupt overhead time. Powering the Embedded Revolution

schedule_next_int macro (linux/hrtime. h) #ifndef schedule_next_int #define schedule_next_int(s, d) 0 #undef CONFIG_HIGH_RES_TIMERS #endif // schedule_next_int Powering the Embedded Revolution

_schedule_next_int function (arch/i 386/kernel/time. c) static int last_was_long = 0; int _schedule_next_int(unsigned long jiffie_f, long sub_jiffie_in) { long sub_jiff_offset; IF_ALL_PERIODIC( if ((sub_jiffie_in == -1) && last_was_long) return 0); /* * First figure where we are in time. * A note on locking. We are under the timerlist_lock here. This * means that interrupts are off already, so don't use irq versions. */ if_SMP( read_lock(&xtime_lock)); sub_jiff_offset = quick_update_jiffies_sub(jiffie_f); if_SMP( read_unlock(&xtime_lock)); Powering the Embedded Revolution

_schedule_next_int function (arch/i 386/kernel/time. c) if ((IF_ALL_PERIODIC( last_was_long =) (sub_jiffie_in == -1 ))) { sub_jiff_offset = cycles_per_jiffies - sub_jiff_offset; }else{ sub_jiff_offset = sub_jiffie_in - sub_jiff_offset; } /* * If time is already passed, just return saying so. */ if (sub_jiff_offset < high_res_test_val){ IF_ALL_PERIODIC( last_was_long = 0); return 1; } reload_timer_chip(sub_jiff_offset); return 0; } Powering the Embedded Revolution

high_res_test macro (linux/hrtime. h) /* * * * The high_res_test() macro should set up a test mode that will do a worst case timer interrupt. I. e. it may be that a call to schedule_next_int() could return -1 indicating that the time has already expired. This macro says to set it so that schedule_next_int() will always set up a timer interrupt. This is used during init to calculate the worst case loop time from timer set up to int to the signal code. * high_res_end_test() cancels the above state and allows the no * interrupt return from schedule_next_int() */ #ifndef high_res_test #define high_res_test() #define high_res_end_test() #endif Powering the Embedded Revolution

high_res_test macro (include/i 386_asm/hrtime. h) extern int high_res_test_val; #define update_jiffies() update_jiffies_sub() new_jiffie() jiffies_intr high_res_test() high_res_test_val = - cycles_per_jiffies; high_res_end_test() high_res_test_val = 0; Powering the Embedded Revolution

Externally visible functions n Cycles_per_jiffies (really an int) n CONFIG_HIGH_RESOLUTION n Update_jiffies_sub() n Quick_update_jiffies_sub() n Conversion inlines. Powering the Embedded Revolution

Cycles_per_jiffies (an int) n This is really just a define that is the number of units of sub_jiffies that comprise a jiffie. n Do note that the whole of the timers patch is independent of the value of HZ and thus jiffie (which is 1/HZ sec. ) Powering the Embedded Revolution

CONFIG_HIGH_RESOLUTION n CONFIG_HIGH_RESOLUTION is the resolution in nanoseconds of the high resolution clocks: CLOCK_REALTIME_HR and CLOCK_MONOTONIC_HR n It should be defined in the platform part and made available by including asm/hrtime. h n It may or may not be a config defined value. Powering the Embedded Revolution

quick_update_jiffies_sub (include/i 386_asm/hrtime. h) /* * If smp, this must be called with the read_lockirq(&xtime_lock) held. * No lock is needed if not SMP. */ extern inline long quick_update_jiffies_sub(unsigned long ref_jiff) { unsigned long update; unsigned long rtn; unsigned long jiffies_f; long _sub_jiffie_f; get_rat_jiffies( &jiffies_f, &_sub_jiffie_f, &update); rtn = _sub_jiffie_f + (unsigned long) update; rtn += (jiffies_f - ref_jiff) * cycles_per_jiffies; return rtn; } Powering the Embedded Revolution

get_rat_jiffies (smp) (include/i 386_asm/hrtime. h) /* * quick_update_jiffies_sub returns the sub_jiffie offset of * current time from the "ref_jiff" jiffie value. We do this * with out updating any memory values and thus do not need to * take any locks, if we are careful. * * I don't know how to eliminate the lock in the SMP case, so. . * Oh, and also the PIT case requires a lock anyway, so. . */ #if defined (CONFIG_SMP) || defined(CONFIG_HIGH_RES_TIMER_PIT) static inline void get_rat_jiffies(unsigned long *jiffies_f, long * _sub_jiffie_f, unsigned long *update) { unsigned long flags; read_lock_irqsave(&xtime_lock, flags); *jiffies_f = jiffies; *_sub_jiffie_f = _sub_jiffie; *update = quick_get_cpuctr(); read_unlock_irqrestore(&xtime_lock, flags); } #else Powering the Embedded Revolution

get_rat_jiffies (non smp) (include/i 386_asm/hrtime. h) static inline void get_rat_jiffies(unsigned long *jiffies_f, long *_sub_jiffie_f, unsigned long *update) { unsigned long last_update_f; do { *jiffies_f = jiffies; last_update_f = last_update; barrier(); *_sub_jiffie_f = _sub_jiffie; *update = quick_get_cpuctr(); barrier(); }while (*jiffies_f != jiffies || last_update_f != last_update); } #endif /* CONFIG_SMP */ : : extern inline unsigned long quick_get_cpuctr(void) { return (inl(acpi_pm_tmr_address) - last_update) & SIZE_MASK; } Powering the Embedded Revolution

Conversion code n The problem is converting units that are on the same order of magnitude as the desired unit. For example the x 86 TSC on an 800 MHZ box is 10/8 away from the nanosecond. n We need to preserve precision as well as speed when we do these conversions. n The answer is scaled math and NO divides. Powering the Embedded Revolution

Scaled math n The basic idea is to scale the fraction (e. g. 10/8 above) by a power of 2 (I. e shift left), do the math (as a mpy) and then unscale (I. e. shift right). n The hassle we run into is that while most (all) platforms have long*long=long and long/long=long+long rem The C standard does not. Powering the Embedded Revolution

32 -bit scale (include/i 386_asm/sc_math. h) /* * Pre scaling defines */ #define SC_32(x) ((long)x<<32) #define SC_n(n, x) (((long)x)<<n) /* * This routine performs the following calculation: * * X = (a*b)>>32 * we could, (but don't) also get the part shifted out. */ extern inline long mpy_ex 32(long a, long b) { long edx; __asm__("imull %2" : "=a" (a), "=d" (edx) : "rm" (b), "0" (a)); return edx; } Powering the Embedded Revolution

32 -bit de-scale (include/i 386_asm/sc_math. h) /* * X = (a/b)<<32 or more precisely x = (a<<32)/b */ extern inline long div_ex 32(long a, long b) { long dum; __asm__("divl %2" : "=a" (b), "=d" (dum) : "r" (b), "0" (0), "1" (a)); return b; } Powering the Embedded Revolution

N-bit scale (include/i 386_asm/sc_math. h) /* * These routines allow you to do x = (a/b) << N and * x=(a*b)>>N for values of N from 1 to 32. */ #define mpy_sc_n(N, aa, bb) ({long edx, a=aa, b=bb; __asm__("imull %2nt" "shldl $(32 -"MATH_STR(N)"), %0, %1" : "=a" (a), "=d" (edx) : "rm" (b), "0" (a)); edx; }) #define div_sc_n(N, aa, bb) ({long dum=aa, dum 2, b=bb; __asm__("shrdl $(32 -"MATH_STR(N)"), %4, %3nt" "sarl $(32 -"MATH_STR(N)"), %4nt" "divl %2" : "=a" (dum 2), "=d" (dum) : "rm" (b), "0" (0), "1" (dum)); dum 2; }) Powering the Embedded Revolution

Long long/long with rem (include/i 386_asm/sc_math. h) /* * (long)X = ((long)divs) / (long)div * (long)rem = ((long)divs) % (long)div * * Warning, this will do an exception if X overflows. */ #define div_long_rem(a, b, c) div_ll_X_l_rem(a, b, c) extern inline long div_ll_X_l_rem(long divs, long div, long * rem) { long dum 2; __asm__( "divl %2" : "=a" (dum 2), "=d" (*rem) : "rm" (div), "A" (divs)); return dum 2; } Powering the Embedded Revolution

Long long/long with rem (include/linux/hrtime. h) /* * If we included a high-res file, we may have gotten a more efficient * u 64/u 32, u 64%u 32 routine. The one in div 64. h actually handles a * u 64 result, something we don't need, and, since it is more expensive * arch porters are encouraged to implement the div_long_rem(). * * int div_long_rem(u 64 dividend, int divisor, int* remainder) * which returns dividend/divisor and remainder. * * Here we provide default code for those who, for what ever reason, * have not provided the above. */ #ifndef div_long_rem #include <asm/div 64. h> #define div_long_rem(dividend, divisor, remainder) ({ u 64 result = dividend; *remainder = do_div(result, divisor); result; }) #endif /* ifndef div_long_rem */ Powering the Embedded Revolution

Scaling in x 86 example-1 (include/i 386_asm/hrtime_Macpi. h) /* * We use various scaling. The ex 32 scales by 2**32, sc_n by the first parm. * When working with constants, choose a scale such that x/n>>(32 -scale)< 1/2. * So for 1/3 <1/2 so scale of 32, where as 3/1 must be shifted 3 times (3/8) * to be less than 1/2 so scale should be 29 * */ #define HR_SCALE_ARCH_NSEC 22 #define HR_SCALE_ARCH_USEC 32 #define HR_SCALE_NSEC_ARCH 32 #define HR_SCALE_USEC_ARCH 29 #ifndef PM_TIMER_FREQUENCY #define PM_TIMER_FREQUENCY 3579545 /* counts per second */ #endif extern inline int arch_cycles_to_usec(unsigned long update) { return (mpy_ex 32(update , (SC_32(1000000)/(long)PM_TIMER_FREQUENCY))); } #define arch_to_nsec (SC_n(HR_SCALE_ARCH_NSEC, 100000)/ (long)PM_TIMER_FREQUENCY) Powering the Embedded Revolution

Scaling in x 86 example-2 (include/i 386_asm/hrtime_Macpi. h) extern inline int arch_cycles_to_nsec(long update) { return mpy_sc_n(HR_SCALE_ARCH_NSEC, update, arch_to_nsec); } /* * And the other way. . . */ #define usec_to_arch (SC_n( HR_SCALE_USEC_ARCH, PM_TIMER_FREQUENCY)/ (long)1000000) extern inline int usec_to_arch_cycles(unsigned long usec) { return mpy_sc_n(HR_SCALE_USEC_ARCH, usec_to_arch); } #define nsec_to_arch (SC_n( HR_SCALE_NSEC_ARCH, PM_TIMER_FREQUENCY)/ (long)100000) extern inline int nsec_to_arch_cycles(unsigned long nsec) { return mpy_ex 32(nsec, nsec_to_arch); } Powering the Embedded Revolution

What is kgdb_ts(x, y)? n Kgdb_ts() is a macro defined in the Monta. Vista x 86 kgdb patch that captures “time stamps” and stores them for later viewing. n This allows you to examine the order of events and, hopefully to understand what you did wrong. n It is purely a debug tool and is not intended to stay in production code. Powering the Embedded Revolution

Kgdb. h… #ifdef CONFIG_KGDB_TS void kgdb_tstamp(int line, char * source, int data 0, int data 1); /* * This is the time stamp function. The macro adds the source info and * does a cast on the data to allow most any 32 -bit value. */ #define kgdb_ts(data 0, data 1) kgdb_tstamp(__LINE__, __FILE__, (int)data 0, (int)data 1) #else #define kgdb_ts(data 0, data 1) #endif Powering the Embedded Revolution

Kgdb_stub code (arch/i 386/kernel/kgdb_stub. c) void kgdb_tstamp(int line, char * source, int data 0, int data 1) { static spinlock_t ts_spin = SPIN_LOCK_UNLOCKED; int flags; local_irq_save(flags); spin_lock(&ts_spin); rdtscll(kgdb_and_then->at_time); #ifdef CONFIG_SMP kgdb_and_then->on_cpu = smp_processor_id(); #endif kgdb_and_then->from_ln = line; kgdb_and_then->in_src = source; kgdb_and_then->from = __builtin_return_address(0); kgdb_and_then->with_if = (flags & 0 x 200) >> 9; kgdb_and_then->data 0 = data 0; kgdb_and_then->data 1 = data 1; kgdb_and_then = &kgdb_data[++kgdb_and_then_count & INDEX_MASK]; spin_unlock(&ts_spin); local_irq_restore(flags); return; } Powering the Embedded Revolution

And the gdb macros-1 define set_andthen set var $thp=0 set var $thp=(struct kgdb_and_then_struct *)&kgdb_data[0] set var $at_size = (sizeof kgdb_data)/(sizeof *$thp) set var $at_oc=kgdb_and_then_count set var $at_cc=$at_oc end define beforethat andthen_set_edge if ($at_cc <= $at_low) printf "Outside window. Window size is %dn", ($at_oc-$at_low) else printf "%d: ", $at_cc-1 output *($thp+(--$at_cc % $at_size )) printf "n" end Powering the Embedded Revolution

And the gdb macros-2 define andthen_set_edge set var $at_oc=kgdb_and_then_count set var $at_low = $at_oc - $at_size if ($at_low < 0 ) set var $at_low = 0 end if (( $at_cc > $at_oc) || ($at_cc < $at_low)) printf "Count outside of window, setting count to " if ($at_cc >= $at_oc) set var $at_cc = $at_oc else set var $at_cc = $at_low end printf "%dn", $at_cc end Powering the Embedded Revolution