Linux下的万能观测工具

Click to edit Master title style,Click to edit Master text styles,Second level,Third level,Fourth level,Fifth level,*,*,*,SystemTap,Linux下的万能观测工具,褚霸,核心系统数据库组,http:/yufeng.info,2010/11/18,Agenda,介绍,SystemTap,安装和系统要求,实践例子,参考和杂项,结论,SystemTap是什么？,According to,http:/sourceware.org/systemtap/,SystemTap provides free software(GPL)infrastructure to,simplify the gathering of information about the running,Linux system.This assists diagnosis of a performance or,functional problem.SystemTap eliminates the need for,the developer to go through the tedious and disruptive,instrument,recompile,install,and reboot sequence that,may be otherwise required to collect data.,观察活体系统最佳工具，前提是你,懂得,如何观察！,SystemTap是如何工作的,1.,write or choose a script describing what you want to observe,2.,stap translates it into a kernel module,3.,stap loads the module and communicates with it,4.,just wait for your data,五步走,#stap-uv test.stp,Pass 1:,parsed user script and 74 library script(s)using 86868virt/20488res/1792shr kb,in 190usr/20sys/209real ms.,Pass 2:,analyzed script:1 probe(s),0 function(s),0 embed(s),0 global(s)using 87264virt/21148res/1976shr kb,in 10usr/0sys/7real ms.,Pass 3:,translated to C into/tmp/stapz2iv97/stap_aef621603e006af62084b361e0a0c981_553.c using 87264virt/21332res/2144shr kb,in 0usr/0sys/0real ms.,Pass 4:,compiled C into stap_aef621603e006af62084b361e0a0c981_553.ko in 1230usr/160sys/1384real ms.,Pass 5:,starting run.,Pass 5:,run completed in 10usr/20sys/12331real ms.,SystemTap 探测点例子,SystemTap is all about executing certain actions when hitting,certain probe points.,syscall.read,when entering read()system call,syscall.close.return,when returning from the close()system call,module(floppy).function(*),when entering any function from the floppy module,kernel.function(,*net/socket.c).return,when returning from any function in le net/socket.c,kernel.statement(,*kernel/sched.c:2917),when hitting line 2917 of le kernel/sched.c,更多探测点例子,timer.ms(200),every 200 milliseconds,process(/bin/ls).function(*),when entering any function in/bin/ls(not its libraries orsyscalls),process(/lib/libc.so.6).function(*malloc*),when entering any glibc function which has malloc in its name,kernel.function(*exit*).return,when returningfrom any kernel function which has exit in its name,RTFM for more(man stapprobes).,SystemTap编程语言,mostly C-style syntax with a feeling of awk,builtin associative arrays,builtin aggregates of statistical data,very easy to collect data and do statistics on it(average,min,max,count,.),many helper functions(builtin and in tapsets),RTFM:SystemTap Language Reference shipped with SystemTap(langref.pdf),Performances and safety,language-level safety features,no pointers,no unbounded loops,type inference,you can also write probe handlers in C(with-g)but dont complain if you break stuff,runtime safety features,stap enforces maximum run time for each probe handler,various concurrency constraints are enforced,overload processing(dont allow stap to take up all the CPU time),many things can be overriden manually if you really want,see SAFETY AND SECURITY section of stap(1),The overhead depends a lot of what you are trying to do but in,general stap will try to stop you from doing something stupid(but,then you can still force it to do it).,Some helper functions youll see a lot,pid(),which process is this?,uid(),which user is running this?,execname(),what is the name of this process?,tid(),which thread is this?,gettimeofday_s(),epoch time in seconds,probefunc(),what function are we in?,print_backtrace(),figure out how we ended up here,There are many many more.RTFM(man stapfuncs)and explore/usr/share/systemtap/tapset/.,Some cool stap options,-x,trace only speci ed PID(only for userland probing),-c,run given command and only trace it and its children(will still trace all threads for kernel probes),-L,list probe points matching given pattern along withavailable variables,-d,load given module debuginfo to help with symbol resolution in backtraces,-g,embed C code in stap script unsafe,dangerous and fun,Agenda,介绍,SystemTap,安装和系统要求,实践例子,参考,结论,Requirements,SystemTap探测用户空间程序需要utrace的支持，但是这个特性还没有被Linux上游吸收。Redhat的发行版本目前支持这个特性。,源码级别跟踪需要安装符号信息,包层面需要安装package-debuginfo on RPM distros,用户自己的程序需要gcc-g-gdwarf-2-g3编译,stap脚本是编译成内核模块运行的，需要root权限,安装SystemTap,RHEL5U4,需要安装内核符号信息:,rpm-i kernel-debuginfo-common-2.6.18-164.el5.x86_64.rpmrpm-i kernel-debuginfo-2.6.18-164.el5.x86_64.rpm,由于5U4带的SystemTap是0.97版本，需要升级到1.3:,./configure prefix=/usr&make&make install,如何验证是否成功：,#stap topsys.stp,SYSCALL COUNT,read 48,fcntl 42,.fstat 1-,Agenda,介绍,SystemTap,安装和系统要求,实践例子,参考和杂项,结论,Example:谁在执行我们的程序,Listing:exec.stp,probe syscall.exec*printf(exec%s%sn,execname(),argstr),$stap-L syscall.exec*syscall.execve name:string filename:string args:string argstr:string$filename:char*$argv:char*$envp:char*$regs:struct pt_regs*,#stap exec.stpexec sshd/usr/sbin/sshd-Rexec sshd/bin/bash,例子:谁杀了我的程序,Listing:sigkill.stp,probe signal.sendif(sig_name=SIGKILL)printf(%s was sent to%s(pid:%d)by%s uid:%dn,s