ãæå¶ãããã»ã¹ã®èšºæ
é »ç¹ã«çºçããåé¡ãã©ãããããã§åçŸããè¯ãäŸã次ã«ç€ºããŸãããŠãŒã¶ãŒã¯ã
find
çµæãè¿ããã«ãã¯ããã«é ãããšäžå¹³ãèšã£ãŠã
find
ã åé¡ãäœã§ããããç¥ã£ãŠãåé¡ã解決ããŸããã ããããç§ã¯ãã®ãããªåé¡ã解決ããäœç³»çãªã¢ãããŒããèšå®ããããã«é ŒãŸããŸããã
幞ããªããšã«ãã·ã¹ãã ã¯OEL6ãå®è¡ããŠããŸãã ããªãæ°ããã«ãŒãã«ïŒã€ãŸã-2.6.39 UEK2ïŒ
ããã§ã¯ã蚺æãå§ããŸãããã
æåã«ãæ€çŽ¢ããã»ã¹ããŸã åäœããŠãããã©ããã確èªããŸãã
[root @ oel6ã]ïŒps -ef | grep find
ã«ãŒã27288 27245 4 11:57ãã€ã³ã/ 0 00:00:01 æ€çŽ¢ -ã¿ã€ãf
ã«ãŒã27334 27315 0 11:57ãã€ã³ã/ 1 00:00:00 grepæ€çŽ¢
ã¯ãããããŸã-PID 27288ïŒããã«èšºæäŸã§ã¯ãã®pidã䜿çšããŸãïŒ
åºæ¬ããå§ããŠããã®ããã»ã¹ã®ããã«ããã¯ãèŠãŠã¿ãŸãããã äœã«ããããã¯ãããŠããªãå ŽåïŒããšãã°ãå¿ èŠãªãã®ããã¹ãŠãã£ãã·ã¥ããèªã¿åãïŒãCPUã100ïŒ äœ¿çšããå¿ èŠããããŸãã IOãŸãã¯ãããã¯ãŒã¯ã®åé¡ãåå ã§ãããã¯ãããŠããå Žåãããã»ããµã®è² è·ã¯äœããªããããŸã£ãããªããªãã¯ãã§ãã
[root @ oel6ã]ïŒtop -cbp 27288
top-11:58:15æ倧7æ¥éã3ïŒ38ã2ãŠãŒã¶ãŒãè² è·å¹³åïŒ1.21ã0.65ã0.47
ã¿ã¹ã¯ïŒåèš1ãå®è¡0ãã¹ãªãŒã1ãåæ¢0ããŸã³ã0
CPUïŒ0.1ïŒ usã0.1ïŒ syã0.0ïŒ niã99.8ïŒ idã0.0ïŒ waã0.0ïŒ hiã0.0ïŒ siã0.0ïŒ st
ã¡ã¢ãªïŒåèš2026460kã1935780k䜿çšã90680kç¡æã64416kãããã¡
ã¹ã¯ããïŒåèš4128764kã䜿çšæžã¿251004kã3877760kç¡æã662280kãã£ãã·ã¥
PIDãŠãŒã¶ãŒPR NI VIRT RES SHR SïŒ CPUïŒ MEMæé+ã³ãã³ã
27288ã«ãŒã20 0 109m 1160 844 D 0.0 0.1 0ïŒ01.11æ€çŽ¢ -ã¿ã€ãf
ãtopãã³ãã³ãã®åºåã¯ããã®ããã»ã¹ãCPUã«ãŸã£ããè² è·ããããªãããšã瀺ããŠããŸãïŒãŸãã¯è² è·ãéåžžã«å°ããããããŒããšèŠãªãããšãã§ããŸãïŒã ãã ããå®å šã«ããªãŒãºããããã»ããµã¯ã©ã³ã¿ã ãååŸããæ©äŒããŸã£ãããªãããã»ã¹ãšãã¹ã¿ã³ãã€ç¶æ ããåžžã«èµ·åããŠããã«åã³ã¹ãªãŒãç¶æ ã«ãªãããã»ã¹ïŒããšãã°ãåžžã«ã¿ã€ã ã¢ãŠãã§çµäºããããŒãªã³ã°æäœããããŠãããã»ã¹ã¯åã³ãããåŒã³åºããŠã¹ãªãŒãç¶æ ã«ãªããŸãïŒã æ®å¿µãªããããããã®2ã€ã®ç¶æ ãèªèããã«ã¯ãããããã³ãã³ãã§ã¯äžååã§ãã ããããå°ãªããšããã®ããã»ã¹ã¯ããã»ããµæéã浪費ããªãããšããã§ã«ããã£ãŠããŸãã
ä»ã®ããŒã«ãè©ŠããŠã¿ãŸãããã éåžžãããã»ã¹ããã®ããã«ãã³ã°ããããã«èŠããå ŽåïŒCPUã®0ïŒ ã¯ãéåžžãããã»ã¹ãäœããã®ã·ã¹ãã ã³ãŒã«ããããã¯ããŠããããšãæå³ããŸã-ããã«ãããã«ãŒãã«ãããã»ã¹ãã¹ãªãŒããããŸãïŒããã®ããã»ã¹ã§
strace
ãå®è¡ããŠãã©ã®ã·ã¹ãã ã³ãŒã«ã远跡ããŸã圌ã¯çŸåšç«ã¡åŸçããŠããŸãã ããã»ã¹ãå®éã«å®å šã«ãã³ã°ãããå®æçã«ã·ã¹ãã ã³ãŒã«ããæ»ã£ãŠèµ·åããå Žåãããã¯straceã®åºåã«ã衚瀺ãããŸãïŒã·ã¹ãã ã³ãŒã«ã¯å®æçã«çµäºããå床åŒã³åºãããŸãïŒã
[root @ oel6ã]ïŒstrace -cp 27288
ããã»ã¹27288ãæ·»ä»ãããŸãã-äžæããŠäžæããŸã
^ C
^ Z
[1] + strace -cp 27288ãåæ¢ããŸãã
[root @ oel6ã]ïŒkill -9 %%
[1] + strace -cp 27288ãåæ¢ããŸãã
[root @ oel6ã]ïŒ
[1] +殺ãããçè·¡-cp 27288
ãã£ãš... straceããŒã èªäœããã³ã°ããŠããããã§ãïŒ åœŒå¥³ã¯é·ãéäœã衚瀺ãããCTRL + CãæŒããŠãå¿çããªãã£ããããæåã«CTRL + ZãæŒããŠåœŒå¥³ãããã¯ã°ã©ãŠã³ãã«éãããã®åŸå®å šã«æ®ºãå¿ èŠããããŸããã ç°¡åãªèšºæã§ã¯ãããŸããïŒ
pstackãè©ŠããŠã¿ãŸãããïŒLinuxã§ã¯ãpstackã¯GDBãããã¬ãŒã®åãªãã©ãããŒã¹ã¯ãªããã§ãïŒã Pstackã¯ã«ãŒãã«ã®å éšãããã³ã«ã€ããŠã¯äœãæããŸããããå°ãªããšããããã©ã®ãããªã·ã¹ãã ã³ãŒã«ã§ãããã瀺ããŸãïŒéåžžã¯ãŠãŒã¶ãŒã¹ã¿ãã¯äžã®libcã©ã€ãã©ãªé¢æ°åŒã³åºãã®ããã«èŠããŸãïŒã
[root @ oel6ã]ïŒpstack 27288
^ C
^ Z
[1] + pstack 27288ãåæ¢ããŸãã
[root @ oel6ã]ïŒ%%ã殺ã
[1] + pstack 27288ãåæ¢ããŸãã
[root @ oel6ã]ïŒ
[1] +çµäºããpstack 27288
Pstackã説æãªãã§ãã³ã°ããŸããïŒ
ãã®ãããããã»ã¹ã100ïŒ ç¡æïŒalasïŒãªã®ãã99.99ïŒ ïŒç®èŠããŠããã«ç ãã«ã€ãïŒãªã®ãã¯ãŸã ããããŸããããŸãããããã©ãã§èµ·ãã£ãã®ããããããŸããã
ä»ã«ã©ããèŠããŸããïŒ ããã¢ã¯ã»ã¹ã§ããå¥ã®å ŽæããããŸããã¹ããŒã¿ã¹ãšWCHANãã£ãŒã«ãã¯ãå€ãè¯ãpsã³ãã³ãã䜿çšããŠå 容ã調ã¹ãããšãã§ããŸãïŒãã¶ããããã«å®è¡ããŠãŸã³ãã«å¯ŸåŠããŠããªãããšã確èªããŠãã ããïŒã
[root @ oel6ã]ïŒps -flp 27288
FS UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 Dã«ãŒã27288 27245 0 80 0-28070 rpc_wa 11:57ãã€ã³ã/ 0 00:00:01æ€çŽ¢ -ã¿ã€ãf
ããã»ã¹ãåŒãç¶ãåãç¶æ ã«ããããšã確èªããã«ã¯ãpsãé£ç¶ããŠæ°åå®è¡ããå¿ èŠããããŸãïŒééã£ããšãã«è¡ãããå¯äžã®è©Šè¡ã«åºã¥ããŠèª€ã£ãçµè«ã«éããããªãã§ããïŒïŒã ãã ããããã§ã¯åæ°ãæã£ãŠ1åã ãã®èµ·åã瀺ããŠããŸãã
ããã»ã¹ã¯ç¶æ DïŒããµãŠã³ãã¹ãªãŒããïŒã«ãããéåžžã¯ãã£ã¹ã¯I / Oã«é¢é£ä»ããããŠããŸãïŒpsã®ããã¥ã¢ã«ããŒãžã«ãèšèŒãããŠããŸãïŒã ããã«ãWCHANãã£ãŒã«ãïŒããã»ã¹ãäŒæ¢ç¶æ ãŸãã¯åŸ æ©ç¶æ ã«ããé¢æ°ã®ååïŒã¯å°ãããªãã³ã°ãããŠããŸãã psåŒã³åºãã«ãªãã·ã§ã³ãè¿œå ããŠãã®ãã£ãŒã«ãã®åºåãå°ãåºãããããšãã§ããŸããããã®å 容ã¯ãããã®å Žåã/ procã·ã¹ãã ããååŸãããããããœãŒã¹ãçŽæ¥èŠãŠã¿ãŸãããïŒç¹°ãè¿ããŸãããåçµã確èªããããã«ãããæ°åè¡ããšããã§ãããïŒããã»ã¹ãå®äºããŠãããã©ããããŸãã¯éåžžã«é »ç¹ã«ããããå¯ãŠãããã©ããïŒïŒ
[root @ oel6ã]ïŒcat / proc / 27288 / wchan
rpc_wait_bit_killable
ããŒã...ããã»ã¹ã¯äœããã®RPCã³ãŒã«ãæåŸ ããŠããããã§ãã ããã¯éåžžãããã»ã¹ãä»ã®ããã»ã¹ãšéä¿¡ããŠããããšãæå³ããŸãïŒããŒã«ã«ãã·ã³äžããŸãã¯ãªã¢ãŒããµãŒããŒäžã§ãïŒã ãããããŸã çç±ã¯ããããŸããã
äœãåãããããŸããããŸãã¯ããã»ã¹ãå®å šã«è¡ãè©°ã£ãŠããŸããïŒ
èšäºã®ãèãã«é²ãåã«ãããã»ã¹ãå®å šã«ã¹ã¿ãã¯ããŠãããã©ãããå€æããŸãããã ææ°ã®ã«ãŒãã«ã§ã¯ã/ proc / PID / statusã調ã¹ãããšã§ããã«ã€ããŠç¥ãããšãã§ããŸãã ããããããããããã«ãèå³æ·±ãå€ã匷調ããŠããŸãã
[root @ oel6ã]ïŒcat / proc / 27288 / status
ååïŒæ€çŽ¢
ç¶æ ïŒDïŒãã£ã¹ã¯ã¹ãªãŒãïŒ
TgidïŒ27288
PidïŒ27288
PPidïŒ27245
TracerPidïŒ0
UidïŒ0 0 0 0
GidïŒ0 0 0 0
FdsizeïŒ256
ã°ã«ãŒãïŒ0 1 2 3 4 6 10
VmPeakïŒ112628 kB
VmSizeïŒ112280 kB
VmLckïŒ0 kB
VmHWMïŒ1508 kB
VmRSSïŒ1160 kB
VmDataïŒ260 kB
VmStkïŒ136 kB
VmExeïŒ224 kB
VmLibïŒ2468 kB
VmPTEïŒ88 kB
VmSwapïŒ0 kB
ã¹ã¬ããïŒ1
SigQïŒ4/15831
SigPndïŒ0000000000040000
ShdPndïŒ0000000000000000
SigBlkïŒ0000000000000000
SigIgnïŒ0000000000000000
SigCgtïŒ0000000180000000
CapInhïŒ0000000000000000
CapPrmïŒffffffffffffffff
CapEffïŒffffffffffffffff
CapbndïŒffffffffffffffff
Cpus_allowedïŒffffffffãffffffff
Cpus_allowed_listïŒ0-63
Mems_allowedïŒ00000000,00000000,00000000,00,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000ã 00000000.00000000.00000000.00000000.00000000.00000000.00000000.00000001
Mems_allowed_listïŒ0
voluntary_ctxt_switchesïŒ9950
nonvoluntary_ctxt_switchesïŒ17104
ããã»ã¹ã¯ç¶æ D-ãã£ã¹ã¯ã¹ãªãŒãïŒããµãŠã³ããã¹ãªãŒãïŒã§ãã ãŸãã
voluntary_ctxt_switches
ãš
nonvoluntary_ctxt_switches
å€ã«ã泚æããŠãã ãã-ããã»ããµãŒãCPUã®
nonvoluntary_ctxt_switches
åãåã£ãïŒãŸãã¯è¿ããïŒåæ°ãããããŸãã 次ã«ãæ°ç§åŸã«ã³ãã³ããå床å®è¡ããå€ãå¢å ãããã©ããã確èªããŸãã ç§ã®å Žåãæ°ã¯å¢ããŸããã§ããããããã£ãŠãããã»ã¹ããã€ããã³ã°ããããšããå§ãããŸãïŒãŸããå°ãªããšãããŒã éã§æ°ç§éèµ·ããªãã§ãã ããïŒã ãã®ãããããã»ã¹ãå®å šã«ããªãŒãºããŠããããšã確信ã§ããããã«ãªããŸããïŒã¬ãŒããŒã®äžãé£ãã§ããã ãã§ãªããããã»ããµã®æéã®0.04ïŒ ãåžžã«æ¶è²»ããŠããŸãïŒã
ã¡ãªã¿ã«ãã³ã³ããã¹ãã¹ã€ããã®æ°ã確èªã§ããå Žæãããã«2ã€ãããŸãïŒããã«ã2ã€ç®ã¯å€ä»£ã®ã«ãŒãã«ã§äœ¿çšå¯èœã§ãïŒã
[root @ oel6ã]ïŒcat / proc / 27288 / sched
æ€çŽ¢ïŒ27288ãïŒthreadsïŒ1ïŒ
-se.exec_startïŒ617547410.689282
se.vruntimeïŒ2471987.542895
se.sum_exec_runtimeïŒ1119.480311
se.statistics.wait_startïŒ0.000000
se.statistics.sleep_startïŒ0.000000
se.statistics.block_startïŒ617547410.689282
se.statistics.sleep_maxïŒ0.089192
se.statistics.block_maxïŒ60082.951331
se.statistics.exec_maxïŒ1.110465
se.statistics.slice_maxïŒ0.334211
se.statistics.wait_maxïŒ0.812834
se.statistics.wait_sumïŒ724.745506
se.statistics.wait_countïŒ27211
se.statistics.iowait_sumïŒ0.000000
se.statistics.iowait_countïŒ0
se.nr_migrationsïŒ312
se.statistics.nr_migrations_coldïŒ0
se.statistics.nr_failed_migrations_affineïŒ0
se.statistics.nr_failed_migrations_runningïŒ96
se.statistics.nr_failed_migrations_hotïŒ1794
se.statistics.nr_forced_migrationsïŒ150
se.statistics.nr_wakeupsïŒ18507
se.statistics.nr_wakeups_syncïŒ1
se.statistics.nr_wakeups_migrateïŒ155
se.statistics.nr_wakeups_localïŒ18504
se.statistics.nr_wakeups_remoteïŒ3
se.statistics.nr_wakeups_affineïŒ155
se.statistics.nr_wakeups_affine_attemptsïŒ158
se.statistics.nr_wakeups_passiveïŒ0
se.statistics.nr_wakeups_idleïŒ0
avg_atomïŒ0.041379
avg_per_cpuïŒ3.588077
nr_switchesïŒ27054
nr_voluntary_switchesïŒ9950
nr_involuntary_switchesïŒ17104
se.load.weightïŒ1024
ããªã·ãŒïŒ0
ããªãªïŒ120
ã¯ããã¯ãã«ã¿ïŒ72
ããã§ã¯ã
nr_switches
ïŒ
nr_voluntary_switches
+
nr_involuntary_switches
çããïŒã®æ°ã調ã¹ãå¿ èŠããããŸãã
äžèšã®éšåã®nr_switchesã®ç·æ°ã¯27054ã§ãããåºå/ proc / PID / schedstatã®3çªç®ã®ãã£ãŒã«ãã«ããããŸãã
[root @ oel6ã]ïŒcat / proc / 27288 / schedstat
1119480311 724745506 27054
ãããŠãããã¯å¢å ããŸãã...
/ procãã¡ã€ã«ã·ã¹ãã ã䜿çšããã³ã¢ã«ãŒãã«ãããã³ã®æ¢çŽ¢
ç§ãã¡ã®ããã»ã¹ã¯éåžžã«åºãåçµãããŠããããã§ã:)ã Straceãšpstackã¯åœ¹ã«ç«ã¡ãŸããã 圌ãã¯ptraceïŒïŒã·ã¹ãã ã³ãŒã«ã䜿çšããŠããã»ã¹ã«æ¥ç¶ããã¡ã¢ãªã«æ¿å ¥ããŸãããããã»ã¹ã絶æçã«ââãã³ã°ãããããããããäœããã®ã·ã¹ãã ã³ãŒã«ã§ãã³ã°ããã®ã§ãptraceïŒïŒã³ãŒã«ãããèªäœã§ãã³ã°ãããšä»®å®ããŸãïŒãšããã§ãç§ã¯ã©ãããããããã¿ãŒã²ããããã»ã¹ã«æ¥ç¶ããç¬éã«straceèªäœã§straceãå®è¡ããããšããŸããããããã«ããããã»ã¹ãã¯ã©ãã·ã¥ããŸããã
ã©ã®ã·ã¹ãã ã³ãŒã«ã§straceãŸãã¯pstackãªãã§é»è©±ãåã£ãã®ããç¥ãã«ã¯ã©ãããã°ããã§ããïŒ å¹žããªããšã«ãç§ãã¡ã¯ææ°ã®ã³ã¢ã«åãçµãã§ããŸãïŒ ãããã/ proc / PID / syscallïŒ
[root @ oel6ã]ïŒcat / proc / 27288 / syscall
262 0xffffffffffffffff9c 0x20cf6c8 0x7fff97c52710 0x100 0x100 0x676e776f645f616d 0x7fff97c52658 0x390e2da8ea
ã€ãŸããïŒ ãããŠãç§ã¯ããã§äœããã¹ãã§ããïŒ
ãŸããéåžžããããã®æ°åã¯äœãã®ããã«å¿ èŠã§ãã â 0xAVeryBigNumberâã®ãããªãã®ã衚瀺ãããå Žåãããã¯éåžžã¡ã¢ãªå ã®ã¢ãã¬ã¹ã§ããïŒpmapãªã©ã®ãŠãŒãã£ãªãã£ã§äœ¿çšã§ããŸãïŒãæ°å€ãå°ããå Žåã¯ãããããããçš®ã®é åã®ã€ã³ããã¯ã¹ã§ãã ããšãã°ãéããŠãããã¡ã€ã«èšè¿°åã®é åïŒ/ proc / PID / fdã§ç¢ºèªã§ããŸãïŒããŸãã¯ãã®å Žåãã·ã¹ãã ã³ãŒã«ãåŠçããŠãããããããã¯ããã»ã¹ãååšããã·ã¹ãã ã³ãŒã«ã®æ°ã§ãïŒ ãã®ãããããã»ã¹ãã·ã¹ãã ã³ãŒã«ïŒ262ã§ã¹ã¿ãã¯ããŠããããšãããããŸããã
ã·ã¹ãã ã³ãŒã«çªå·ã¯ããªãã¬ãŒãã£ã³ã°ã·ã¹ãã ãOSããŒãžã§ã³ããã©ãããã©ãŒã ã«ãã£ãŠç°ãªãå ŽåããããããOSããæ£ããããããŒãã¡ã€ã«ãå¿ èŠã«ãªãããšã«æ³šæããŠãã ããã / usr / includeãã©ã«ããŒã§ãsyscall *ããæ¢ãããšããå§ããã®ãããã§ãããã ç§ã®ããŒãžã§ã³ããã³Linuxãã©ãããã©ãŒã ïŒ64ãããïŒã§ã¯ãã·ã¹ãã ã³ãŒã«ã¯ãã¡ã€ã«
/usr/include/asm/unistd_64.h
å®çŸ©ãããŠããŸãã
[root @ oel6ã]ïŒgrep 262 /usr/include/asm/unistd_64.h
#define __NR_ newfstatat 262
ããããã§ãïŒ 262ã·ã¹ãã ã³ãŒã«ã¯
newfstatat
ãšåŒã°ãããã®
newfstatat
ã 人ãåããããããäœã§ããããç¥ãããšã¯æ®ã£ãŠããŸãã ã·ã¹ãã ã³ãŒã«ã®ååã«ã€ããŠç°¡åãªãã³ãã瀺ããŸããããã¥ã¢ã«ããŒãžã§å¿ èŠãªååãèŠã€ãããªãå Žåã¯ãããããçš®é¡ã®ãµãã£ãã¯ã¹ãšãã¬ãã£ãã¯ã¹ãªãã§æ€çŽ¢ããŠã¿ãŠãã ããïŒããšãã°ããman pread64ãã§ã¯ãªããman preadãïŒ-ãã®å Žåããnewãprefix-
man fstatat
ãªãã§manãå®è¡ããŸã
man fstatat
ã ãŸãããŸãã¯åã«ãããã°ãŒã°ã«ã
å Žåã«ãã£ãŠã¯ããã®ãnew-fstat-atãã·ã¹ãã ã³ãŒã«ã«ãããéåžžã®statã·ã¹ãã ã³ãŒã«ãšéåžžã«ãã䌌ããã¡ã€ã«ããããã£ãèªã¿åãããšãã§ããŸãã ãããŠãç§ãã¡ã¯ãã¡ã€ã«ã®ã¡ã¿ããŒã¿ãèªã¿åãæäœã«å°å¿µããŸããã ããã§ãç§ãã¡ã¯ããã«äžæ©åé²ããŸããã ãã ããåçµãçºçããçç±ã¯ãŸã ããããŸããã
ããŠã ç§ã®å°ããªå人 / proc / PID / stackã«æšæ¶ããæãæ¥ãŸãããããã«ãããprocãã¡ã€ã«ã®å 容ãå°å·ããã ãã§ãããã»ã¹ã®æ žã¹ã¿ãã¯ã®ãã§ãŒã³ãèŠãããšãã§ããŸã!!!
[root @ oel6ã]ïŒcat / proc / 27288 / ã¹ã¿ãã¯
[] rpc_wait_bit_killable + 0x24 / 0x40 [sunrpc]
[] __rpc_execute + 0xf5 / 0x1d0 [sunrpc]
[] rpc_execute + 0x43 / 0x50 [sunrpc]
[] rpc_run_task + 0x75 / 0x90 [sunrpc]
[] rpc_call_sync + 0x42 / 0x70 [sunrpc]
[] nfs3_rpc_wrapper.clone.0 + 0x35 / 0x80 [nfs]
[] nfs3_proc_getattr + 0x47 / 0x90 [nfs]
[] __nfs_revalidate_inode + 0xcc / 0x1f0 [nfs]
[] nfs_revalidate_inode + 0x36 / 0x60 [nfs]
[] nfs _getattr + 0x5f / 0x110 [nfs]
[] vfs_getattr + 0x4e / 0x80
[] vfs_fstatat + 0x70 / 0x90
[] sys_ newfstatat + 0x24 / 0x50
[] system_call_fastpath + 0x16 / 0x1b
[] 0xffffffffffffffff
äžçªäžã®æ©èœã¯ãã«ãŒãã«ã³ãŒãå ã®å Žæã§ãã ããã¯ãŸãã«WCHANãã£ãŒã«ããæ¢ã«èšã£ãããšã§ãïŒãã ããå®éã«ã¯ãå¿ èŠã«å¿ããŠããã»ã¹ãã¹ãªãŒããŸãã¯ãŠã§ã€ã¯ã¢ããããã«ãŒãã«
schedule()
é¢æ°ãªã©ãããã€ãã®æ©èœãããããšã«æ³šæããŠãã ããããã ãããããã®æ©èœã¯ããã«ã¯ç€ºããŸããïŒããããããããã¯ãã§ã«æåŸ ã®ç¶æ ã®çµæã§ããããã®åå ã§ã¯ãªãããã§ãïŒã
ãã®ã¿ã¹ã¯ã®ããã®å®å šãªã«ãŒãã«ã¹ã¿ãã¯ãæå ã«çšæããŠãããã ã¢ããã§èª¿æ»ãã
rpc_wait_bit_killable
é¢æ°
rpc_wait_bit_killable
ãã¹ã±ãžã¥ãŒã©ãŒã®åŒã³åºãã«ã©ã®ããã«
rpc_wait_bit_killable
ã
rpc_wait_bit_killable
ç¶æ ã«ãªã£ãããææ¡ããããšã
rpc_wait_bit_killable
ãŸãã
system_call_fastpath
ã³ãŒã«ã¯ãåŠçããŠããnewfstatatïŒ
sys_newfstatat
ïŒã·ã¹ãã ã³ãŒã«ãå®è£ ããã³ãŒãã«ã€ãªãã£ãæšæºã«ãŒãã«ã·ã¹ãã ã³ãŒã«ãã³ãã©ã§ãã ããã«ãåãé¢æ°ã«ç§»åãããšãNFSã«é¢é£ããããã€ãã®ããšãããããŸãã ããã¯ãã§ã«ãNFSåŠçã³ãŒãé åã®ã©ããã«ãããšãã100ïŒ åè«ã§ããªã蚌æ ã§ãïŒ äžèšã§èŠãéãããããã®NFSé¢æ°ã®æåŸã¯ãRPCé¢æ°ïŒrpc_call_syncïŒãåŒã³åºããŠå¥ã®ããã»ã¹ã«ã¢ã¯ã»ã¹ããŸãã ãã®å Žåããããã
[kworker/N:N]
ã
[nfsiod]
ã
[lockd]
ãŸãã¯
[rpciod]
IOã«ãŒãã«ã¹ã¬ããã§ãã ãããŠããããã®ã¹ã¬ããã®ããã€ãã¯äœããã®çç±ã§å¿çããŸããïŒéåžžããããã¯ãŒã¯æ¥ç¶ã®åæããã±ããæ倱ããŸãã¯äœããã®ãããã¯ãŒã¯ã®åé¡ãçãå¿ èŠããããŸãïŒã
ãããã®è£å©ã¹ã¬ããã®ããããããããã¯ãŒã¯é¢é£ã®ã³ãŒãã§ãã³ã°ããŠãããã©ãããå€æããã«ã¯ãããšãã°ãkworkersãRPCãNFSã«æ¥ç¶ããã ãã§ãªããã¯ããã«å€ãã®æ©èœãå®è¡ããå Žåã§ããã¹ã¿ãã¯ãèæ ®ããå¿ èŠããããŸã å¥ã®å®éšïŒNFSãä»ããŠå€§ããªãã¡ã€ã«ãã³ããŒããã ãïŒã§ãã³ãŒãã§ãããã¯ãŒã¯ãšéä¿¡ããããã«åŸ æ©ããŠããkworkersã®1人ãèŠã€ããŸããã
[root @ oel6 proc]ïŒ `pgrep worker`ã®iã®å Žå; ps -fp $ i; cat / proc / $ i /ã¹ã¿ãã¯; ãã£ã
UID PID PPID C STIME TTY TIME CMD
ã«ãŒã53 2 0 2æ14æ¥ïŒ 00:04:34 [ kworker / 1ïŒ1]
[] __cond_resched + 0x2a / 0x40
[] lock_sock_nested + 0x35 / 0x70
[] tcp_sendmsg + 0x29 / 0xbe0
[] inet_sendmsg + 0x48 / 0xb0
[] sock_sendmsg + 0xef / 0x120
[] kernel_sendmsg + 0x41 / 0x60
[] xs_send_kvec + 0x8e / 0xa0 [sunrpc]
[] xs_sendpages + 0x173 / 0x220 [sunrpc]
[] xs_tcp_send_request + 0x5d / 0x160 [sunrpc]
[] xprt_transmit + 0x83 / 0x2e0 [sunrpc]
[] call_transmit + 0xa8 / 0x130 [sunrpc]
[] __rpc_execute + 0x66 / 0x1d0 [sunrpc]
[] rpc_async_schedule + 0x15 / 0x20 [sunrpc]
[] process_one_work + 0x13e / 0x460
[] worker_thread + 0x17c / 0x3b0
[] kthread + 0x96 / 0xa0
[] kernel_thread_helper + 0x4 / 0x10
ãããããã«ãŒãã«ãã¬ãŒã¹ãæå¹ã«ããŠãã©ã®ã«ãŒãã«ã¹ã¬ãããçžäºã«éä¿¡ããŠããããæ£ç¢ºã«èŠã€ããããšã¯é£ãããããŸãããããã®èšäºã§ã¯ãããè¡ããããããŸããã å®çšçãªïŒãããŠç°¡åãªïŒèšºææŒç¿ã«ããŠãã ããïŒ
ç解ããŠãä¿®ââçããã
ãããã®å Žåã§ããææ°ã®Linuxã§ã«ãŒãã«ã¹ã¿ãã¯ã®å°å·ãéåžžã«ç°¡åã«ååŸã§ããããïŒã©ã®ããŒãžã§ã³ã®ã«ãŒãã«ãç»å Žãããæ£ç¢ºã«ã¯èšããŸããïŒãfindã³ãã³ãããã³ã°ããå Žæãã€ãŸãã«ãŒãã«ã®NFSã³ãŒãã§äžè²«ããŠçµè«ãââåºãããšãã§ããŸããLinux ãŸããNFSã«é¢é£ãããã³ã°ã¢ãããåŠçããŠããå Žåã¯ããããããããã¯ãŒã¯ã®åé¡ãçãå¿ èŠããããŸãã 瀺ãããåé¡ã®åçŸæ¹æ³ã«èå³ãããå Žåã¯ããã¹ãŠãéåžžã«ç°¡åã§ããä»®æ³ãã·ã³ããNFSããªã¥ãŒã ãããŠã³ãããfindã³ãã³ããå®è¡ããŠããããã·ã³ãåæ¢ããŸããã ããã«ãããTCPãšã³ãããŒãã«éç¥ããã«æ¥ç¶ããµã€ã¬ã³ãã«çµäºããããŸãã¯äœããã®çç±ã§ãã±ãããåã«ééããªããããã¯ãŒã¯ïŒæ§æããã¡ã€ã¢ãŠã©ãŒã«ïŒã®åé¡ãããå Žåãšåãçç¶ãçããŸããã
ã¹ã¿ãã¯ã®äžçªäžã«ã¯ãkillableãé¢æ°ã®1ã€ïŒå®å šã«äžæã§ãã
rpc_wait_bit_killable
ïŒãããããã
kill -9
ã³ãã³ãã§ããã
rpc_wait_bit_killable
ã§ããŸãã
[root @ oel6ã]ïŒps -fp 27288
UID PID PPID C STIME TTY TIME CMD
ã«ãŒã27288 27245 0 11:57ãã€ã³ã/ 0 00:00:01 æ€çŽ¢ -ã¿ã€ãf
[root @ oel6ã]ïŒkill -9 27288
[root @ oel6ã]ïŒls -l / proc / 27288 / stack
lsïŒ/ proc / 27288 / stackã«ã¢ã¯ã»ã¹ã§ããŸããïŒãã®ãããªãã¡ã€ã«ãŸãã¯ãã£ã¬ã¯ããªã¯ãããŸãã
[root @ oel6ã]ïŒps -fp 27288
UID PID PPID C STIME TTY TIME CMD
[root @ oel6ã]ïŒ
ããã»ã¹ã¯å®äºã§ãã
ãã²ãã®äžã®ãã«ãŒãã«ã¹ã¬ããã®ãããã¡ã€ãªã³ã°
ãã¡ã€ã«/ proc / PID / stackã¯éåžžã®ããã¹ããã¡ã€ã«ã®ããã«èŠããããšã«æ³šæããŠãã ããã ãããã£ãŠãã«ãŒãã«ã¹ã¬ãããããã¡ã€ã«ãç°¡åã«ååŸã§ããŸãïŒ ä»¥äžã¯ããã²ããŸãããŠãçŸåšã®ã·ã¹ãã ã³ãŒã«ãšã«ãŒãã«ã¹ã¿ãã¯ã®ææ ŒïŒã·ã¹ãã ã³ãŒã«ã®å ŽåïŒã確èªããããããã¹ãŠãåéå±€ãããã¡ã€ã«ã«çµåããæ¹æ³ã§ãã
[root @ oel6ã]ïŒexport LC_ALL = C ; for iã®{1..100}; do cat / proc / 29797 / syscall | awk '{print $ 1}'; cat / proc / 29797 /ã¹ã¿ãã¯| / home / oracle / os_explain -k; usleep 100000; å®äº| sort -r | uniq -c
69ã©ã³ãã³ã° 1 ffffff81534c83 2 ffffff81534820 6,247 25,180
100 0xffffffffffffffff 1 thread_group_cputime 27 sysenter_dispatch 3 ia32_sysret 1 task_sched_runtime 27 sys32_pread 1 compat_sys_io_submit 2 compat_sys_io_getevents 27 sys_pread64 2 sys_io_getevents 1 do_io_submit 27 vfs_read 2 read_events 1 io_submit_one 27 do_sync_read 1 aio_run_iocb 27 generic_file_aio_read 1 aio_rw_vect_retry 27 generic_file_read_iter 1 generic_file_aio_read 27 mapping_direct_IO 1 generic_file_read_iter 27 blkdev_direct_IO 27 __blockdev_direct_IO 27 do_blockdev_direct_IO 27 dio_post_submission 27 dio_await_completion 6 blk_flush_plug_list
ããã«ãããããã»ã¹ãã«ãŒãã«å ã®ã©ãã§æéãè²»ãããŠããã®ããéåžžã«å€§ãŸãã«è©äŸ¡ã§ããŸãã ã·ã¹ãã ã³ãŒã«çªå·ã¯ãäžéšã§åå¥ã«åŒ·èª¿è¡šç€ºãããŸãã ãå®è¡äžããšã¯ã蚺æäžã«ããã»ã¹ããŠãŒã¶ãŒç©ºéã§æ©èœããããšãæå³ããŸãïŒã·ã¹ãã ã³ãŒã«ã§ã¯æ©èœããŸããïŒã ãã®ãããããã»ã¹ããŠãŒã¶ãŒã³ãŒãã§ãã£ãæéã®69ïŒ ã§ãã ã·ã¹ãã ã³ãŒã«ïŒ180ïŒç§ã®ã·ã¹ãã ã§ã¯nfsservctlïŒã§25ïŒ ãã·ã¹ãã ã³ãŒã«ïŒ247ïŒwaitidïŒã§6ïŒ ã
äžèšã§ã¯ãäœæè ã¯ã¹ã¯ãªãã/ home / oracle / os_explainã®åŒã³åºãã䜿çšããŠããŸãã åç §ã§ååŸã§ããŸã-çŽã perevã
åºåã«ã¯ãããã«2ã€ã®ãé¢æ°ãã衚瀺ãããŸãããäœããã®çç±ã§ååã§è¡šç€ºãããŸããã§ããã ãã ãããããã®ã¢ãã¬ã¹ã¯æ¢ç¥ã§ãããããæåã§ç¢ºèªã§ããŸãã
[root @ oel6ã]ïŒcat / proc / kallsyms | grep -i ffffff81534c83
ffffffff81534c83 t ia32_sysret
ããã¯32ããããµãã·ã¹ãã ã®ã·ã¹ãã ã³ãŒã«ããã®æ»ãå€ã®ããã§ããããã®é¢æ°èªäœã¯ã·ã¹ãã ã³ãŒã«ã§ã¯ãªãããïŒåãªãå éšè£å©é¢æ°ã§ãïŒãæããã«/ proc /ã¹ã¿ãã¯ãã³ãã©ã¯ãã®ååã衚瀺ããŸããã§ããã / procã®ç¶æ ã¯ãªãŒããŒãšäžèŽããªãããããããã®ã¢ãã¬ã¹ã衚瀺ãããå¯èœæ§ããããŸããäžæ¹ã1ã€ã®ã¹ã¬ããããããã®ã¡ã¢ãªæ§é ãšã¹ã¬ãããèªã¿åãèŠçŽ ãå€æŽãããšãå€ãããŒã¿ã衚瀺ãããå ŽåããããŸãã
2çªç®ã®ã¢ãã¬ã¹ãåæã«ç¢ºèªããŸãã
[root @ oel6ã]ïŒcat / proc / kallsyms | grep -i ffffff81534820
[root @ oel6ã]ïŒ
äœãèŠã€ãããŸããã§ãããïŒ ãã ãããããã°ã¯ãããŸã§ã§ã¯ãããŸããïŒ ãã®ã¢ãã¬ã¹ã®åšèŸºã§äœãé¢çœããã®ãæ¢ããŸãããã ããšãã°ãã¢ãã¬ã¹ã®æ«å°Ÿã«ããæåŸã®æ°æ¡ãåé€ããŸãã
[root @ oel6ã]ïŒcat / proc / kallsyms | grep -i ffffff815348
ffffffff8153480d t sysenter_do_call
ffffffff81534819 t sysenter_dispatch
ffffffff81534847 t sysexit_from_sys_call
ffffffff8153487a t sysenter_auditsys
ffffffff815348b9 t sysexit_audit
sysenter_dispatché¢æ°ã¯ã/ proc / PID / stackã«ãããã³ã°ãããå ã®ã¢ãã¬ã¹ã®1ãã€ãåã«æåéãéå§ããããã§ãã ãã®ãããã»ãšãã©ã®å Žå1ãã€ããã»ãŒå®äºããŸããïŒãããããåçãããã°çšã«ããã«ããNOPã³ãã³ãïŒã ãã ããã¹ã¿ãã¯ã®ãããã®éšåã¯
sysenter_dispatch
é¢æ°ã«ããããã§ã
sysenter_dispatch
é¢æ°ã¯ãããèªäœã¯ã·ã¹ãã ã³ãŒã«ã§ã¯ãªãããã«ããŒé¢æ°ã§ãã ïŒ å°ãæãäžããå Žå-éãã¯ãŸã 1ãã€ãã§ã¯ãªã7ãã€ãã§ããæ°åã¯16é²æ°ã§ãïŒ-ãããTranslã ïŒ
ã¹ã¿ãã¯ãããã¡ã€ãªã³ã°ã®è©³çŽ°
Linuxã®PerfãSolarisã®DTraceã®Oprofileããã³
profile
ãŠãŒãã£ãªãã£ãªã©ãããŸããŸãªã¹ã¿ãã¯ããã¢ãŒã·ã§ã³ãŠãŒãã£ãªãã£ã¯ãåœä»€ãã€ã³ã¿ãŒã¬ãžã¹ã¿ïŒ32ãããIntelã®EIPããŸãã¯x64ã®RIPïŒãšãã¹ã¿ãã¯ãã€ã³ã¿ãŒïŒ32ãããã®ESPããã³RSPã®ESP x64ïŒããã»ããµäžã®çŸåšã®ã¹ã¬ããã«å¯ŸããŠãã¹ã¿ãã¯ãã€ã³ã¿ã«æ²¿ã£ãŠæ»ããŸãã ãããã£ãŠããããã®ãŠãŒãã£ãªãã£ã¯ããã¹ãæã«CPUã§å®è¡ãããã¹ã¬ããã®ã¿ã衚瀺ã§ããŸãïŒããã¯ããã¡ãããããã»ããµã®è² è·ãé«ãåé¡ãæ¢ããŠãããšãã¯çŽ æŽãããããšã§ããããããããã¯ãŸãã¯é·æéåŸ æ©/äŒæ¢ç¶æ ã®ããã»ã¹ã®èšºæã«ã¯ãŸã£ãã圹ã«ç«ã¡ãŸããã
LinuxãSolarisãHP-UXãprocstackïŒAIXäžïŒãORADEBUG SHORT_STACKäžã®pstackãããã³/ proc / PID / stackç䌌ãã¡ã€ã«ã®èªã¿åããªã©ã®ãŠãŒãã£ãªãã£ã¯ãCPUãããã¡ã€ãªã³ã°ãŠãŒãã£ãªãã£ãžã®é©åãªè¿œå ïŒçœ®æã§ã¯ãããŸããïŒ-ããã»ã¹ã¡ã¢ãªã«ç¬ç«ããŠã¢ã¯ã»ã¹ã§ããéãã¹ã±ãžã¥ãŒã©ãŒã®ç¶æ ããã¹ã¿ãã¯ãèªã¿åããŸããããã»ã¹ãã¹ãªãŒãããŠCPUã«è§Šããªãå Žåãããã»ã¹ã¹ã¿ãã¯ã®æäžéšã¯ä¿åãããã³ã³ããã¹ãããèªã¿åãããšãã§ããŸããä¿åãããã³ã³ããã¹ãã¯ãã³ã³ããã¹ãã®åãæ¿ãäžã«OSã¹ã±ãžã¥ãŒã©ãŒã«ãã£ãŠã«ãŒãã«ã¡ã¢ãªã«ä¿åãããŸãã
ãã¡ãããããã»ããµã€ãã³ããããã¡ã€ãªã³ã°ãŠãŒãã£ãªãã£ã¯ãå€ãã®å ŽåãpstackãOProfileãPerfãããã«ã¯DTraceïŒSolaris11ïŒã®CPCãããã€ããŒä»¥äžã®ãã®ãå®è¡ã§ããŸããããšãã°ãå éšããã»ããµã«ãŠã³ã¿ãèšå®ããã³èªã¿åããã¡ã¢ãªã¢ã¯ã»ã¹ã®åŸ æ©äžã«ãã¹ããCPUãµã€ã¯ã«æ°ãªã©ãè©äŸ¡ããŸããL1 / L2ãã£ãã·ã¥ã®ãã¹æ°ãªã© ããããKevin Clossonãããã«ã€ããŠæžããŠããããšãããèªãã§ãã ããïŒïŒPerfãOprofileïŒ
幞éãç¥ããŸã:-)
é¢é£ãããã¯
- Oracleã€ã³ã¹ã¿ã³ã¹ã䜿çšããŠããOracleããŒã ã®æ€çŽ¢...
- ãããã¬ãŒã®å±éº-ããŒã2
- 以åã®ã»ã°ã¡ã³ãã¬ãã«ãã§ãã¯ãã€ã³ãã®ç®çã¯äœã§ããïŒ
- æªäºããªã©ã¯ã«ã§èµ·ãã£ãŠããŸã
- Oracle Exadata Performanceã·ãªãŒãº-ããŒã1ïŒç§ã¯...