Real-time execution - kernel scheduler information

[Erik Eliasson]

Regarding the audio quality and media processing, here are some thoughts
I have.

It would be interesting to know exactly how the threads execute on the
processor(s). We would have more knowledge about what situations the 
buffering system must handle. I expect I would be surprised (as usual)
when digging into the details. I don't really know what kind of jitter
the scheduling introduces. How does moving the mouse pointer affects us?
Other I/O? Does the RTP threads stop being "responsive" when the device
is under high load and do we start processing packets in bursts? When a
signaling packet arrives, how does that affect audio? Should we set
thread priorities/how much would it improve minisip?

To measure on the application layer is not good enought, and I did not
find any application that satisfied my needs to do what I wanted
(although I think it _should_ exist).

It would be nice to have output in a graphical form that could look
something like this (excluding timestamp information that could be in
milli/micro/nano seconds or clock cycles):

q = in run queue
r = executing on processor
| = sleeping 

OtherApp SIGN  RTP1  RTP2  PLAY App. timestamps
  |        |    |     |     |
  |        |    r     |     |
  |        |    r     q     |
  |        |    r     q     |
  |        |    |     r     |
  |        |    |     r     |
  |        |    |     r     |
  |        |    |     |     |
  |        |    |     |     |
  |        |    |     |     r
  |        |    |     |     r<--resample
  |        |    |     |     r
  |        |    q     |     r<--spatial mixing
  |        |    q     |     r
  |        |    r     |     |
  |        |    r     |     |
  |        |    r     |     |
  |        |    |     |     |

The vertical axes is time, and they contain information about what the
processor(s) is doing (different application threads, interrupt handling
etc). 

Quite some time ago I did some simple testing to see how hard it is, and
what I did was
- Modify the linux kernel scheduler to store a list of events
   (such as switching in or out a task) and a timestamp when that 
   happens (i.e. the processor TSC which is the number of clock cycles 
   since boot)
- Write a kernel module that makes this data available via the /dev/ 
  file system (creating a new system call would also work). 

It was not as hard as you might expect (about an afternoons work, and
not many lines of code at all), but to go the whole way you need to
spend some time to:
- Combine the data with timestamp data from the application (so 
   that the data makes more sence - you see what your process is doing)
- Display the information in a human readable format. 

My (very limited) experience after this is that some applications are
schedules onto the processor surprisingly often (such as the printer
daemon on my system).

If anyone would like to do this, you can contact me to get my source
code (nothing spectacular) or just a more detailed description of it -
I'd be very interested in seeing the result. I did not write a
blocking /dev device even if I should have (i.e. the application
collecting the data was polling the device).

--Erik

-- 
Erik Eliasson <eliasson at it.kth.se>