Messages in this thread | | | Subject | | From | Mike Sullivan <> | Date | Mon, 8 Sep 2003 13:58:36 -0500 |
| |
In 2.6.0-test3 I am seeing a significant slowdown (~40%) during heavy database workloads when using the default anticipatory I/O scheduler. When the same kernel is booted with the elevator=deadline argument the slowdown does not occur. I've attached excerpts from the database test for vmstat and iostat output as well as readprofiles for each of the schedulers. Is anyone else seeing similiar behaviors?
Hardware: IBM x440 8way with hypertheading enabled, 16GB RAM, 4 QLA2310 FC adapters attached to 2 FastT900 controllers (112 disks total).
1a. vmstat for anticipatory scheduler procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 14 41 0 3489576 3840 168280 0 0 25 2 70 30 1 2 96 0 4 103 0 3329760 3860 172160 0 0 233549 97 6606 21284 27 6 0 67 7 91 0 3327496 3864 172156 0 0 137106 14 7364 32957 20 5 0 75 2 84 0 3325416 3868 172412 0 0 138979 73 7096 30445 19 5 0 76 6 78 0 3324528 3868 173192 0 0 143213 444 7328 24901 13 4 0 83 0 105 0 3316408 3876 173964 0 0 142413 524 7531 26854 12 5 0 83 7 94 0 3313664 3880 174220 0 0 134939 821 7336 29037 14 5 0 82 1 110 0 3313560 3884 174216 0 0 114651 923 6990 27063 11 4 0 84 5 109 0 3313008 3884 174216 0 0 110115 862 6937 26381 11 4 0 85 3 111 0 3325400 3920 174180 0 0 108664 1030 6892 26721 11 4 0 85
1b. vmstat for deadline scheduler procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 19 0 0 3518364 3104 119096 0 0 43 1 78 62 1 2 97 1 70 23 0 3361380 3776 170684 0 0 351035 538 3521 19435 68 17 9 5 96 25 0 3321060 3816 171684 0 0 588377 349 4973 25000 91 9 0 0 79 41 0 3316492 3824 171676 0 0 621683 831 5037 26853 91 9 0 0 86 48 0 3318684 3836 172964 0 0 609092 3073 4882 27878 91 9 0 0 71 44 0 3310164 3848 173212 0 0 551741 3746 4291 28492 92 8 0 0 91 35 0 3297708 3852 174248 0 0 508096 1425 4410 29533 91 8 0 1
2a. iostat for anticipatory scheduler
Linux 2.6.0-test3lp Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/sdj 1.92 0.00 363.48 0.88 18908.11 29.12 9454.06 14.56 51.98 4.80 13.14 1.22 44.57 /dev/sdi 1.97 0.00 364.65 0.61 18541.72 20.66 9270.86 10.33 50.82 4.86 13.27 1.23 44.75 /dev/sdh 1.95 0.00 361.35 0.85 18901.05 28.30 9450.53 14.15 52.26 4.79 13.20 1.23 44.72 /dev/sdg 1.66 0.00 357.14 0.84 18155.59 28.06 9077.80 14.03 50.79 4.74 13.23 1.25 44.61 /dev/sdf 3.20 0.00 328.57 0.87 17293.65 28.98 8646.82 14.49 52.58 4.73 14.34 1.36 44.74 /dev/sde 3.44 0.00 330.51 0.75 17180.30 18.55 8590.15 9.28 51.92 4.76 14.35 1.36 45.08 /dev/sdd 3.36 0.00 323.76 0.42 16911.58 14.56 8455.79 7.28 52.21 4.80 14.78 1.38 44.75 /dev/sdc 3.04 0.00 325.47 0.93 16665.03 31.01 8332.51 15.50 51.15 4.94 15.13 1.37 44.74
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/sdj 4.80 0.00 863.10 2.00 39945.60 64.00 19972.80 32.00 46.25 11.25 13.08 1.16 100.02 /dev/sdi 4.50 0.00 787.50 14.90 34438.40 483.20 17219.20 241.60 43.52 14.13 17.58 1.25 100.02 /dev/sdh 3.40 0.00 851.00 2.10 33180.80 70.40 16590.40 35.20 38.98 11.65 13.60 1.17 100.02 /dev/sdg 5.10 0.00 840.60 9.10 37772.80 288.00 18886.40 144.00 44.79 13.01 15.27 1.18 100.02 /dev/sdf 8.90 0.00 740.70 3.90 33075.20 128.00 16537.60 64.00 44.59 11.66 15.67 1.34 100.02 /dev/sde 8.30 0.00 760.40 2.80 36169.60 96.00 18084.80 48.00 47.52 11.70 15.23 1.31 100.02 /dev/sdd 12.40 0.00 769.70 0.00 30147.20 0.00 15073.60 0.00 39.17 11.05 14.29 1.30 100.02 /dev/sdc 9.20 0.00 746.00 3.00 31568.00 92.80 15784.00 46.40 42.27 11.65 15.59 1.34 100.03
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/sdj 5.01 0.00 895.80 0.00 38073.27 0.00 19036.64 0.00 42.50 11.06 12.29 1.12 100.12 /dev/sdi 5.91 0.00 818.02 12.61 32515.72 406.81 16257.86 203.40 39.64 13.84 16.71 1.21 100.12 /dev/sdh 7.41 0.00 862.16 1.80 34652.25 54.45 17326.13 27.23 40.17 11.46 13.31 1.16 100.12 /dev/sdg 4.40 0.00 866.67 2.20 33271.67 70.47 16635.84 35.24 38.37 11.67 13.43 1.15 100.13 /dev/sdf 8.01 0.00 784.28 2.80 33242.84 86.49 16621.42 43.24 42.35 11.52 14.59 1.27 100.13 /dev/sde 6.11 0.20 728.53 17.62 30488.09 576.58 15244.04 288.29 41.63 15.00 20.15 1.34 100.13 /dev/sdd 6.01 0.40 705.51 12.11 29690.49 413.21 14845.25 206.61 41.95 13.73 19.17 1.40 100.13 /dev/sdc 7.41 0.80 738.04 7.61 33338.94 281.88 16669.47 140.94 45.09 12.13 16.17 1.34 100.12
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/sdj 4.29 0.00 864.37 0.00 33411.58 0.00 16705.79 0.00 38.65 11.17 12.97 1.15 99.83 /dev/sdi 6.29 0.00 826.35 6.49 35002.00 198.00 17501.00 99.00 42.27 12.34 14.75 1.20 99.83 /dev/sdh 6.69 0.00 844.91 0.90 36563.67 28.74 18281.84 14.37 43.26 10.92 13.03 1.18 99.83 /dev/sdg 3.39 0.00 860.78 0.70 33705.39 22.36 16852.69 11.18 39.15 11.27 13.09 1.16 99.82 /dev/sdf 8.28 0.00 749.40 1.20 32661.08 38.32 16330.54 19.16 43.56 11.24 14.97 1.33 99.82 /dev/sde 5.79 0.00 731.44 13.47 34328.14 427.94 17164.07 213.97 46.66 13.86 18.67 1.34 99.82 /dev/sdd 4.59 0.00 713.77 17.37 31853.09 552.50 15926.55 276.25 44.32 14.61 20.09 1.37 99.82 /dev/sdc 11.88 0.00 699.00 16.07 30914.17 514.17 15457.09 257.09 43.95 14.46 20.26 1.40 99.82
2b. io for deadline Linux 2.6.0-test3lp
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/sdj 0.04 0.00 103.39 1.02 30230.24 36.05 15115.12 18.02 289.87 0.68 6.51 1.60 16.72 /dev/sdi 0.01 0.00 64.93 0.82 18642.75 29.55 9321.38 14.77 283.99 0.50 7.57 1.90 12.49 /dev/sdh 0.02 0.00 109.26 1.02 30215.96 36.05 15107.98 18.02 274.30 0.70 6.37 1.55 17.14 /dev/sdg 0.03 0.00 62.97 0.21 18346.02 10.04 9173.01 5.02 290.50 0.48 7.53 1.92 12.11 /dev/sdf 0.02 0.00 98.78 0.92 27471.17 32.80 13735.58 16.40 275.87 0.75 7.47 1.64 16.37 /dev/sde 1.64 0.00 105.42 1.75 27301.07 41.84 13650.54 20.92 255.15 0.77 7.21 1.63 17.46 /dev/sdd 0.02 0.00 103.89 1.02 28228.90 36.05 14114.45 18.02 269.40 0.77 7.36 1.58 16.54 /dev/sdc 0.05 0.00 108.91 1.63 31749.13 55.56 15874.56 27.78 287.71 0.77 6.94 1.67 18.49
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/sdj 0.10 0.00 581.30 0.00 152032.00 0.00 76016.00 0.00 261.54 4.16 7.16 1.62 94.39 /dev/sdi 0.20 0.00 497.70 0.00 152601.60 0.00 76300.80 0.00 306.61 3.75 7.52 1.92 95.35 /dev/sdh 0.20 0.00 552.40 0.00 150166.40 0.00 75083.20 0.00 271.84 3.97 7.18 1.71 94.71 /dev/sdg 0.20 0.00 545.70 0.00 148086.40 0.00 74043.20 0.00 271.37 3.93 7.19 1.74 95.15 /dev/sdf 0.20 0.00 661.90 0.00 160950.40 0.00 80475.20 0.00 243.16 4.97 7.50 1.47 97.56 /dev/sde 0.80 0.00 725.50 0.00 148051.20 0.00 74025.60 0.00 204.07 5.39 7.42 1.31 95.00 /dev/sdd 0.90 0.00 688.60 0.00 158950.40 0.00 79475.20 0.00 230.83 5.12 7.43 1.40 96.40 /dev/sdc 0.70 0.00 644.30 13.80 147622.40 441.60 73811.20 220.80 224.99 4.99 7.58 1.41 92.52
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/sdj 0.00 0.00 448.10 0.00 132969.60 0.00 66484.80 0.00 296.74 3.03 6.77 1.88 84.40 /dev/sdi 0.00 0.00 514.20 0.00 141683.20 0.00 70841.60 0.00 275.54 3.53 6.88 1.80 92.70 /dev/sdh 0.20 0.00 537.90 0.00 145718.40 0.00 72859.20 0.00 270.90 3.93 7.31 1.71 91.92 /dev/sdg 0.60 0.00 764.00 0.00 149760.00 0.00 74880.00 0.00 196.02 5.25 6.87 1.30 99.41 /dev/sdf 0.70 0.00 803.10 0.00 138851.20 0.00 69425.60 0.00 172.89 5.94 7.40 1.20 96.11 /dev/sde 1.00 0.00 993.30 0.00 164857.60 0.00 82428.80 0.00 165.97 6.97 7.02 1.01 99.96 /dev/sdd 1.80 0.00 899.70 0.00 166409.60 0.00 83204.80 0.00 184.96 5.93 6.59 1.11 99.49 /dev/sdc 0.20 0.00 445.30 0.00 131558.40 0.00 65779.20 0.00 295.44 3.37 7.57 1.93 85.86
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/sdj 0.10 0.00 402.60 0.00 154393.60 0.00 77196.80 0.00 383.49 2.77 6.89 2.01 80.84 /dev/sdi 0.00 0.00 506.60 9.20 132547.20 294.40 66273.60 147.20 257.54 3.47 6.73 1.49 76.64 /dev/sdh 0.20 0.00 323.90 0.00 103331.20 0.00 51665.60 0.00 319.02 2.28 7.04 2.04 66.11 /dev/sdg 0.10 0.00 553.00 0.00 174540.80 0.00 87270.40 0.00 315.63 3.56 6.44 1.71 94.53 /dev/sdf 0.00 0.00 447.70 0.00 154179.20 0.00 77089.60 0.00 344.38 3.33 7.44 1.99 88.89 /dev/sde 0.20 0.00 426.20 4.60 155136.00 147.20 77568.00 73.60 360.45 3.21 7.47 2.06 88.80 /dev/sdd 0.10 0.00 448.90 0.00 159312.00 0.00 79656.00 0.00 354.89 3.28 7.30 1.98 88.70 /dev/sdc 0.00 0.00 410.50 0.00 146473.60 0.00 73236.80 0.00 356.82 3.04 7.41 2.16 88.65
3a. readprofile for anticipatory scheduler 1699600 total 0.8206 1609934 poll_idle 27757.4828 12233 schedule 10.1772 11380 do_softirq 54.9758 9795 __make_request 8.0485 5708 scsi_request_fn 8.3450 5282 scsi_end_request 28.2460 3482 qla2x00_start_scsi 3.4960 3466 dio_bio_end_io 28.8833 2279 dio_await_one 14.6090 2065 cpu_idle 32.2656 1706 qla2x00_queuecommand 1.1685 1647 add_timer 11.6809 1572 put_io_context 18.2791 1503 unlock_page 17.0795 1433 follow_hugetlb_page 3.0042 1292 sys_msgsnd 1.7969 1260 current_kernel_time 18.2609 1081 scsi_dispatch_cmd 2.8078 918 qla2x00_get_new_sp 8.2703 758 try_to_wake_up 1.3512 702 system_call 15.9545 691 del_timer_sync 5.4841 687 scsi_io_completion 0.6314 628 do_no_page 0.5004 589 blk_recount_segments 0.8951 577 blk_run_queues 3.6752 560 __end_that_request_first 1.0219 533 get_io_context 4.1318 506 scsi_device_unbusy 4.0480 501 set_page_dirty_lock 5.5055 493 get_request 0.7515 451 direct_io_worker 0.4275 450 do_direct_IO 0.4306 441 dio_bio_complete 1.6036 409 worker_thread 0.6059 406 submit_page_section 0.9442 386 kmem_cache_alloc 5.0130 358 bio_alloc 0.8689 344 add_disk_randomness 6.6154 339 kmem_cache_free 3.8966 319 dio_bio_add_page 3.0673 317 __generic_file_aio_read 0.5671 316 ipc_lock 4.3288 296 kfree 2.2256 281 add_timer_randomness 1.2165 268 as_work_handler 3.3924 236 fsync_buffers_list 0.6396 230 get_offset_cyclone 4.1818 223 huge_pte_offset 5.4390 213 qla2x00_next 0.2359 211 io_schedule_timeout 4.1373 208 scsi_put_command 1.4545 197 mempool_alloc 0.5988 186 blockdev_direct_IO 0.5360 180 finished_one_bio 1.5517 173 testmsg 2.4714 173 batch_entropy_store 1.1234 171 sys_msgrcv 0.1913 168 blk_run_queue 2.3014 166 mempool_free_slab 7.9048 159 dio_refill_pages 0.5179 157 generic_make_request 0.3601 149 bio_add_page 0.4357 140 dnotify_parent 1.3592 136 fget_light 1.0543 135 vfs_read 0.4576 122 do_page_fault 0.1083 118 update_atime 0.5315 116 get_jiffies_64 2.0000 113 sd_rw_intr 0.2770 111 scsi_run_queue 0.5663 105 .text.lock.util 4.3750 97 blkdev_get_blocks 0.8220 92 __copy_from_user_ll 0.7541 90 generic_file_direct_IO 0.5172 88 add_wait_queue 1.2571 85 submit_bio 0.5556 84 sys_semtimedop 0.0635 84 __copy_to_user_ll 0.7000 82 remove_wait_queue 1.1081 82 dio_bio_submit 0.8119 81 io_schedule 1.6875 78 do_gettimeofday 0.5417 78 dio_send_cur_page 0.4561 77 find_vma 0.9506 75 page_add_rmap 0.1974 75 memcpy 1.3158 75 bio_put 1.3889 74 page_waitqueue 1.5745 72 scsi_init_cmd_errh 0.3564 72 generic_file_read 0.4186 71 max_block 0.6174 70 blkdev_direct_IO 0.6140 66 sys_ipc 0.0965 65 proc_pid_stat 0.0543 64 kmap_atomic 0.3951 63 scsi_softirq 0.2864 63 del_timer 0.6000 62 default_idle 1.3778 60 mempool_free 0.4878 60 bio_destructor 0.6383 56 get_user_pages 0.0556 56 find_extend_vma 0.4444 55 task_mem 0.2148 55 __d_lookup 0.2000 52 number 0.0692 51 syscall_exit 4.6364 51 dio_await_completion 0.6220 50 bio_get_nr_vecs 0.9434 49 as_set_request 0.4375 48 sys_pread64 0.3780 48 dio_get_page 0.5517 45 find_get_page 0.5769 41 do_anonymous_page 0.0429 40 do_getitimer 0.1455 38 dio_new_bio 0.3016 38 __set_page_dirty_buffers 0.1279 36 mempool_alloc_slab 1.7143 35 sys_setitimer 0.1268 35 do_clock_nanosleep 0.0448 35 __kmalloc 0.2869 34 __copy_user_intel 0.1977 33 adjust_abs_time 0.1158 31 page_remove_rmap 0.0571 31 blkdev_writepage 0.8857 30 inode_times_differ 0.4412 30 get_more_blocks 0.1911 29 s_show 0.0458 28 scsi_decide_disposition 0.0875 28 link_path_walk 0.0133 27 sys_nanosleep 0.1034 26 show_regs 0.0741 25 wait_on_page_bit 0.1196 25 do_setitimer 0.0529 25 bio_endio 0.2049 23 vsnprintf 0.0201 23 queue_delayed_work 0.1655 23 ipcperms 0.1411 22 do_readv_writev 0.0316 21 zap_pte_range 0.0297 21 scsi_finish_command 0.1167 20 scsi_add_timer 0.1980 20 generic_file_aio_write_nolock 0.0068 19 update_queue 0.1652 19 dio_zero_block 0.1131 19 blk_queue_bounce 0.2468 18 syscall_call 1.6364 18 store_msg 0.0783 17 try_atomic_semop 0.0570 17 do_wp_page 0.0130 17 buffered_rmqueue 0.0489 16 sys_gettimeofday 0.0812 16 handle_mm_fault 0.0329 15 scsi_free_sgtable 0.3333 15 qla2x00_calc_iocbs_64 0.2542 15 free_hot_cold_page 0.0600 15 flush_tlb_mm 0.1136 15 elv_set_request 0.2586 14 kmap_atomic_to_page 0.0921 14 .text.lock.direct_io 0.1386 13 wake_up_process 0.3095 13 dio_bio_alloc 0.1429 13 default_llseek 0.0491 12 scsi_next_command 0.2927 12 pid_revalidate 0.1121 11 show_cpuinfo 0.0162 11 render_sigset_t 0.0598 11 dio_bio_reap 0.0965 11 __lookup 0.0539 10 proc_pid_status 0.0090 10 path_lookup 0.0299 10 get_pid_list 0.0855 9 free_msg 0.1957 9 clear_page_tables 0.0201 9 atomic_dec_and_lock 0.1385 9 __pagevec_lru_add_active 0.0354 8 local_bh_enable 0.0597 8 load_msg 0.0193 8 __wake_up 0.1231 7 vfs_write 0.0237 7 release_pages 0.0216 7 pte_alloc_one 0.0547 7 pte_alloc_map 0.0215 7 kernel_to_ipc64_perm 0.1296 7 ipc_unlock 0.8750 7 fput 0.3333 7 copy_process 0.0027 6 vfs_readv 0.0600 6 tstojiffie 0.0594 6 sock_alloc_send_pskb 0.0117 6 generic_file_readv 0.0385 6 end_that_request_chunk 1.2000 6 dio_complete 0.0800 5 write_profile 0.0276 5 sock_wfree 0.0658 5 radix_tree_lookup 0.0676 5 pgd_ctor 0.1316 5 page_address 0.0289 5 lru_cache_add_active 0.0602 5 get_empty_filp 0.0209 5 generic_file_write_nolock 0.0321 5 filemap_nopage 0.0054 5 bad_range 0.0442 4 zap_pmd_range 0.0342 4 vma_link 0.0244 4 unix_stream_recvmsg 0.0029 4 unix_stream_data_wait 0.0134 4 unix_release_sock 0.0060 4 unix_create1 0.0113 4 tg3_poll 0.0137 4 sync_mapping_buffers 0.0889 4 split_vma 0.0137 4 skb_recv_datagram 0.0169 4 qla2x00_blink_led 0.0214 4 pte_chain_alloc 0.0625 4 get_wchan 0.0328 4 generic_osync_inode 0.0156 4 follow_page 0.0095 4 follow_mount 0.0286 4 flush_scheduled_work 0.4000 4 elv_put_request 0.1212 4 copy_page_range 0.0030 4 __pte_chain_free 0.0455 4 __get_user_4 0.1739 4 __constant_c_and_count_memset 0.0312 4 .text.lock.time 0.1212 3 unix_stream_sendmsg 0.0030 3 tcp_v4_rcv 0.0013 3 tcp_sendmsg 0.0006 3 sys_llseek 0.0118 3 sys_fstat64 0.0526 3 sys_close 0.0265 3 strncpy_from_user 0.0303 3 sock_fasync 0.0052 3 scsi_init_io 0.0095 3 poll_freewait 0.0469 3 may_open 0.0068 3 kmap_high 0.0065 3 ipc_buildid 0.2500 3 generic_fillattr 0.0176 3 fget 0.0476 3 exit_notify 0.0016 3 eth_type_trans 0.0179 3 dput 0.0074 3 do_poll 0.0152 3 do_mmap_pgoff 0.0017 3 dio_cleanup 0.0195 3 bdev_read_only 0.0517 3 alloc_ldt 0.0071 3 access_process_vm 0.0065 3 __mark_inode_dirty 0.0123 3 __filemap_copy_from_user_iovec 0.0171 3 __alloc_pages 0.0038 2 wake_up_process_kick 0.0476 2 vsprintf 0.0465 2 vfs_writev 0.0200 2 unmap_region 0.0058 2 unix_write_space 0.0142 2 unix_poll 0.0129 2 unix_find_socket_byinode 0.0233 2 sys_stat64 0.0351 2 sys_readv 0.0202 2 sys_open 0.0144 2 sys_clock_nanosleep 0.0067 2 ss_wakeup 0.0426 2 sock_poll 0.0408 2 sock_map_fd 0.0061 2 set_page_address 0.0060 2 remove_suid 0.0182 2 remove_shared_vm_struct 0.0177 2 release_task 0.0049 2 raw_file_aio_write 0.0345 2 prepare_to_wait 0.0222 2 pid_alive 0.0571 2 normal_poll 0.0056 2 mmput 0.0120 2 iput 0.0161 2 ip_rcv 0.0017 2 get_write_access 0.0351 2 fn_hash_lookup 0.0085 2 eventpoll_init_file 0.0741 2 end_that_request_first 0.2000 2 do_sync_read 0.0109 2 do_aic7xxx_isr 0.0129 2 d_alloc 0.0044 2 cp_new_stat64 0.0075 2 copy_files 0.0030 2 change_protection 0.0035 2 cache_grow 0.0025 2 __block_prepare_write 0.0019 2 __blk_queue_bounce 0.0034 1 work_resched 0.0455 1 wait_for_packet 0.0031 1 vm_acct_memory 0.0185 1 vfs_permission 0.0031 1 vfs_follow_link 0.0022 1 unix_stream_connect 0.0010 1 unix_peer_get 0.0238 1 unix_accept 0.0033 1 tg3_rx 0.0011 1 tcp_transmit_skb 0.0007 1 tcp_poll 0.0026 1 sys_timer_settime 0.0012 1 sys_select 0.0007 1 sys_pwrite64 0.0079 1 sys_munmap 0.0098 1 sys_mprotect 0.0017 1 sys_getuid 0.0625 1 sys_getdents64 0.0048 1 strnlen_user 0.0147 1 sock_wmalloc 0.0106 1 sock_destroy_inode 0.0323 1 sock_def_readable 0.0079 1 sock_create 0.0027 1 skip_atoi 0.0154 1 skb_clone 0.0030 1 show_interrupts 0.0014 1 shm_get_stat 0.0064 1 set_huge_pte 0.0057 1 scsi_setup_cmd_retry 0.0093 1 schedule_timeout 0.0053 1 schedule_delayed_work 0.0714 1 restore_fpu 0.0323 1 recalc_bh_state 0.0076 1 rb_insert_color 0.0044 1 qla2x00_build_scsi_iocbs_32 0.0018 1 pty_write 0.0024 1 pty_chars_in_buffer 0.0137 1 profile_exit_task 0.0132 1 proc_lookup 0.0037 1 proc_calc_metrics 0.0123 1 prepare_to_wait_exclusive 0.0112 1 prep_new_page 0.0122 1 pid_fd_revalidate 0.0051 1 path_release 0.0159 1 parse_table 0.0034 1 page_cache_readahead 0.0026 1 old_mmap 0.0032 1 nr_running 0.0244 1 notify_change 0.0026 1 next_thread 0.0127 1 net_rx_action 0.0039 1 monotonic_clock_cyclone 0.0061 1 mm_init 0.0040 1 math_state_restore 0.0152 1 lru_add_drain 0.0119 1 locks_remove_posix 0.0036 1 load_elf_binary 0.0003 1 kobj_lookup 0.0034 1 kill_fasync 0.0169 1 kill_bdev 0.0147 1 ipc_checkid 0.0303 1 invalidate_inode_buffers 0.0111 1 inode_init_once 0.0027 1 inet_sendmsg 0.0112 1 in_group_p 0.0222 1 i_waitq_head 0.0455 1 handle_signal 0.0032 1 get_unused_fd 0.0025 1 get_signal_to_deliver 0.0012 1 generic_file_open 0.0111 1 flush_tlb_page 0.0072 1 find_lock_page 0.0036 1 filp_close 0.0083 1 filldir64 0.0032 1 fd_install 0.0179 1 ext2_update_inode 0.0013 1 ext2_get_inode 0.0033 1 ext2_alloc_block 0.0050 1 dup_task_struct 0.0050 1 do_sys_settimeofday 0.0051 1 do_select 0.0014 1 do_page_cache_readahead 0.0027 1 do_flush_tlb_all 0.0108 1 disk_round_stats 0.0047 1 device_not_available 0.0244 1 dev_seq_printf_stats 0.0044 1 detach_vmas_to_be_unmapped 0.0101 1 destroy_context 0.0058 1 dentry_open 0.0027 1 dcache_readdir 0.0021 1 d_rehash 0.0111 1 d_lookup 0.0125 1 d_instantiate 0.0111 1 csum_partial 0.0035 1 create_elf_tables 0.0011 1 count_semncnt 0.0091 1 copy_strings 0.0016 1 copy_mm 0.0011 1 check_unthrottle 0.0164 1 can_vma_merge_before 0.0119 1 cache_init_objs 0.0085 1 c_start 0.0278 1 bd_claim 0.0172 1 batch_entropy_process 0.0047 1 alloc_skb 0.0045 1 alloc_inode 0.0030 1 add_softcursor 0.0042 1 __vma_link 0.0063 1 __unix_remove_socket 0.0084 1 __tcp_select_window 0.0040 1 __posix_lock_file 0.0007 1 __kfree_skb 0.0046 1 __fput 0.0050 1 __copy_user_zeroing_intel 0.0058 1 __block_commit_write 0.0068 1 __alloc_percpu 0.0047 1 SHATransform 0.0033 1 .text.lock.tty_io 0.0023 1 .text.lock.read_write 0.0192 1 .text.lock.open 0.0058 1 .text.lock.dnotify 0.0233
3b. readprofile for deadline scheduler 171653 total 0.0829 19968 do_softirq 96.4638 18055 schedule 15.0208 14925 scsi_request_fn 21.8202 11718 qla2x00_start_scsi 11.7651 9323 follow_hugetlb_page 19.5451 5626 __make_request 4.6228 4567 sys_msgsnd 6.3519 4108 poll_idle 70.8276 4070 dio_bio_end_io 33.9167 4044 unlock_page 45.9545 3882 do_direct_IO 3.7148 3690 scsi_end_request 19.7326 3577 submit_page_section 8.3186 3146 set_page_dirty_lock 34.5714 2502 dio_bio_complete 9.0982 2313 qla2x00_queuecommand 1.5842 2298 dio_bio_add_page 22.0962 2230 system_call 50.6818 2163 del_timer_sync 17.1667 1905 try_to_wake_up 3.3957 1782 get_offset_cyclone 32.4000 1721 fsync_buffers_list 4.6640 1677 get_request 2.5564 1658 bio_add_page 4.8480 1394 add_timer 9.8865 1181 dio_await_one 7.5705 1159 dio_refill_pages 3.7752 1121 scsi_dispatch_cmd 2.9117 1056 current_kernel_time 15.3043 939 bio_alloc 2.2791 884 qla2x00_get_new_sp 7.9640 863 direct_io_worker 0.8180 858 ipc_lock 11.7534 812 do_no_page 0.6470 802 put_io_context 9.3256 798 blkdev_get_blocks 6.7627 765 .text.lock.util 31.8750 751 max_block 6.5304 717 __generic_file_aio_read 1.2826 691 scsi_io_completion 0.6351 683 kmem_cache_alloc 8.8701 677 scsi_device_unbusy 5.4160 658 kfree 4.9474 644 sys_msgrcv 0.7204 624 kmem_cache_free 7.1724 610 get_jiffies_64 10.5172 582 blk_run_queue 7.9726 581 __end_that_request_first 1.0602 563 page_add_rmap 1.4816 546 find_vma 6.7407 541 dio_send_cur_page 3.1637 517 scsi_put_command 3.6154 502 generic_unplug_device 4.7810 483 blk_run_queues 3.0764 471 testmsg 6.7286 471 page_waitqueue 10.0213 461 blk_recount_segments 0.7006 455 __copy_from_user_ll 3.7295 433 default_llseek 1.6340 405 get_user_pages 0.4018 379 huge_pte_offset 9.2439 360 fget_light 2.7907 348 copy_page_range 0.2632 337 add_disk_randomness 6.4808 301 get_io_context 2.3333 301 do_readv_writev 0.4325 301 blockdev_direct_IO 0.8674 289 do_page_fault 0.2564 280 __copy_to_user_ll 2.3333 273 sd_rw_intr 0.6691 266 add_timer_randomness 1.1515 258 qla2x00_next 0.2857 254 mempool_alloc 0.7720 239 generic_make_request 0.5482 236 update_atime 1.0631 235 batch_entropy_store 1.5260 231 __kmalloc 1.8934 227 do_gettimeofday 1.5764 223 sys_semtimedop 0.1687 219 do_wp_page 0.1681 211 get_more_blocks 1.3439 208 sys_ipc 0.3041 198 io_schedule 4.1250 186 __set_page_dirty_buffers 0.6263 183 finished_one_bio 1.5776 183 del_timer 1.7429 182 kmap_atomic 1.1235 181 find_extend_vma 1.4365 180 wait_on_page_bit 0.8612 180 generic_file_direct_IO 1.0345 180 dio_get_page 2.0690 179 dnotify_parent 1.7379 178 io_schedule_timeout 3.4902 172 generic_file_readv 1.1026 158 blkdev_direct_IO 1.3860 152 submit_bio 0.9935 146 sys_llseek 0.5748 144 do_anonymous_page 0.1506 137 scsi_run_queue 0.6990 137 ipcperms 0.8405 134 bio_destructor 1.4255 126 do_getitimer 0.4582 120 find_get_page 1.5385 118 mempool_free 0.9593 116 syscall_exit 10.5455 113 do_clock_nanosleep 0.1445 112 do_setitimer 0.2368 108 scsi_softirq 0.4909 108 deadline_set_request 1.1020 102 store_msg 0.4435 102 scsi_init_cmd_errh 0.5050 101 sys_setitimer 0.3659 97 proc_pid_stat 0.0810 97 dio_bio_submit 0.9604 95 page_remove_rmap 0.1750 92 bio_put 1.7037 92 __d_lookup 0.3345 88 dio_await_completion 1.0732 85 memcpy 1.4912 80 sys_gettimeofday 0.4061 79 wake_up_process 1.8810 76 blk_queue_bounce 0.9870 73 vfs_read 0.2475 72 pte_alloc_one 0.5625 69 load_msg 0.1663 68 task_mem 0.2656 68 scsi_decide_disposition 0.2125 65 bio_get_nr_vecs 1.2264 64 number 0.0852 63 mempool_alloc_slab 3.0000 61 zap_pte_range 0.0863 60 handle_mm_fault 0.1235 60 dio_new_bio 0.4762 58 adjust_abs_time 0.2035 57 .text.lock.direct_io 0.5644 56 sys_readv 0.5657 54 scsi_finish_command 0.3000 53 blkdev_writepage 1.5143 52 huge_page_release 1.2381 50 vsnprintf 0.0436 49 scsi_next_command 1.1951 48 try_atomic_semop 0.1611 46 syscall_call 4.1818 46 scsi_add_timer 0.4554 44 vfs_readv 0.4400 44 inode_times_differ 0.6471 43 sys_select 0.0318 42 sys_nanosleep 0.1609 42 elv_set_request 0.7241 41 find_lock_page 0.1475 40 generic_file_read 0.2326 39 __copy_user_intel 0.2267 37 buffered_rmqueue 0.1063 36 update_queue 0.3130 36 sys_pread64 0.2835 36 radix_tree_lookup 0.4865 36 link_path_walk 0.0171 36 do_select 0.0495 35 work_resched 1.5909 35 qla2x00_calc_iocbs_64 0.5932 35 device_not_available 0.8537 33 s_show 0.0521 32 dio_bio_alloc 0.3516 30 pte_chain_alloc 0.4688 28 free_msg 0.6087 27 kmap_atomic_to_page 0.1776 27 bio_endio 0.2213 26 free_hot_cold_page 0.1040 26 __lookup 0.1275 24 release_pages 0.0741 24 pte_alloc_map 0.0738 24 flush_tlb_mm 0.1818 24 __pte_chain_free 0.2727 23 mempool_free_slab 1.0952 23 generic_file_aio_write_nolock 0.0078 23 filemap_nopage 0.0249 23 dio_zero_block 0.1369 22 ipc_unlock 2.7500 22 dio_complete 0.2933 21 tstojiffie 0.2079 20 fget 0.3175 20 clear_page_tables 0.0446 20 __alloc_pages 0.0255 19 set_huge_pte 0.1092 18 dio_bio_reap 0.1579 18 bad_range 0.1593 18 __wake_up 0.2769 17 shmem_getpage 0.0062 17 fput 0.8095 17 find_vma_prev 0.2099 16 local_bh_enable 0.1194 16 kstat_read_proc 0.0175 16 end_that_request_chunk 3.2000 16 copy_files 0.0237 15 queue_delayed_work 0.1079 15 pid_revalidate 0.1402 15 may_open 0.0339 14 tcp_poll 0.0365 14 scsi_free_sgtable 0.3111 14 render_sigset_t 0.0761 14 cache_grow 0.0175 14 __pagevec_lru_add_active 0.0551 13 shmem_nopage 0.1204 13 math_state_restore 0.1970 13 copy_mm 0.0140 13 .text.lock.time 0.3939 12 pid_alive 0.3429 12 hugetlb_prefault 0.0327 12 atomic_dec_and_lock 0.1846 11 get_unused_fd 0.0279 11 get_pid_list 0.0940 11 generic_file_write_nolock 0.0705 11 cpu_idle 0.1719 11 __get_user_4 0.4783 10 show_interrupts 0.0136 10 remove_shared_vm_struct 0.0885 10 ipc_checkid 0.3030 10 end_that_request_first 1.0000 9 vfs_write 0.0305 9 sock_poll 0.1837 9 restore_fpu 0.2903 9 qla2x00_build_scsi_iocbs_32 0.0161 9 generic_osync_inode 0.0350 9 d_alloc 0.0199 9 __posix_lock_file 0.0060 9 __mark_inode_dirty 0.0370 8 sys_close 0.0708 8 strncpy_from_user 0.0808 8 qla2x00_blink_led 0.0428 8 proc_pid_status 0.0072 8 kernel_to_ipc64_perm 0.1481 8 ipc_parse_version 0.2759 8 flush_scheduled_work 0.8000 7 unmap_hugepage_range 0.0281 7 schedule_delayed_work 0.5000 7 poll_freewait 0.1094 7 nr_blockdev_pages 0.0700 7 free_pages 0.0909 7 do_sys_settimeofday 0.0354 7 copy_process 0.0027 7 bdev_read_only 0.1207 7 __alloc_percpu 0.0327 6 write_profile 0.0331 6 vfs_writev 0.0600 6 unix_write_space 0.0426 6 sys_shmat 0.0083 6 ss_wakeup 0.1277 6 pgd_ctor 0.1579 6 mark_page_accessed 0.1154 6 lru_cache_add_active 0.0723 6 ipc_buildid 0.5000 6 flush_tlb_page 0.0432 6 do_sigaction 0.0095 6 do_mmap_pgoff 0.0034 6 dio_cleanup 0.0390 6 cp_new_stat64 0.0226 5 unmap_vmas 0.0069 5 unix_stream_recvmsg 0.0036 5 sys_open 0.0360 5 schedule_timeout 0.0267 5 path_lookup 0.0150 5 locks_remove_posix 0.0182 5 do_aic7xxx_isr 0.0323 5 chrdev_open 0.0121 5 access_process_vm 0.0108 5 __pollwait 0.0254 5 __free_pages 0.0694 4 unix_stream_data_wait 0.0134 4 unix_release_sock 0.0060 4 tg3_rx 0.0043 4 show_cpuinfo 0.0059 4 setup_sigcontext 0.0135 4 setup_rt_frame 0.0059 4 remove_wait_queue 0.0541 4 proc_lookup 0.0148 4 prep_new_page 0.0488 4 page_cache_readahead 0.0105 4 notify_change 0.0102 4 new_inode 0.0345 4 max_select_fd 0.0177 4 kill_bdev 0.0588 4 ip_route_input_slow 0.0018 4 get_empty_filp 0.0167 4 eventpoll_init_file 0.1481 4 dput 0.0099 4 do_generic_mapping_read 0.0031 4 alloc_skb 0.0179 4 __constant_c_and_count_memset 0.0312 3 wake_up_process_kick 0.0714 3 unix_stream_sendmsg 0.0030 3 unix_poll 0.0194 3 tg3_poll 0.0103 3 sys_pwrite64 0.0236 3 split_vma 0.0103 3 sock_map_fd 0.0091 3 sock_alloc_send_pskb 0.0059 3 sigprocmask 0.0142 3 scsi_init_io 0.0095 3 page_address 0.0173 3 open_namei 0.0029 3 lru_add_drain 0.0357 3 kmap_high 0.0065 3 hugetlbfs_statfs 0.0370 3 generic_fillattr 0.0176 3 follow_mount 0.0214 3 find_vma_prepare 0.0297 3 file_move 0.0526 3 file_kill 0.0682 3 fd_install 0.0536 3 exit_notify 0.0016 3 dup_task_struct 0.0149 3 do_settimeofday 0.0099 3 do_munmap 0.0071 3 disk_stats_read 0.0041 3 count_semncnt 0.0273 3 calc_crc_errors 0.0155 3 alloc_inode 0.0090 3 __get_free_pages 0.0303 3 __fput 0.0151 3 __copy_user_zeroing_intel 0.0174 3 __blk_queue_bounce 0.0052 3 .text.lock.read_write 0.0577 2 wait_for_packet 0.0062 2 vma_link 0.0122 2 vm_acct_memory 0.0370 2 unmap_region 0.0058 2 unix_accept 0.0066 2 task_prio 0.1429 2 sys_fstat64 0.0351 2 sys_connect 0.0113 2 sys_brk 0.0073 2 sys_alarm 0.0241 2 sys_accept 0.0063 2 sync_mapping_buffers 0.0444 2 sock_def_readable 0.0157 2 skb_recv_datagram 0.0084 2 shmem_swp_alloc 0.0032 2 set_page_address 0.0060 2 seq_read 0.0027 2 schedule_tail 0.0130 2 proc_pident_lookup 0.0037 2 old_mmap 0.0063 2 nr_free_pages 0.0290 2 memcpy_fromiovec 0.0118 2 lookup_mnt 0.0192 2 locks_remove_flock 0.0103 2 load_elf_binary 0.0007 2 kunmap_atomic 0.1667 2 kobj_lookup 0.0069 2 kfree_skbmem 0.0455 2 invalidate_vcache 0.0139 2 init_new_context 0.0085 2 get_signal_to_deliver 0.0024 2 free_pages_and_swap_cache 0.0153 2 fn_hash_lookup 0.0085 2 filp_close 0.0165 2 file_read_actor 0.0085 2 ext2_get_inode 0.0066 2 eligible_child 0.0114 2 elf_map 0.0107 2 do_sync_read 0.0109 2 do_lookup 0.0124 2 do_irq_balance 0.0019 2 dnotify_flush 0.0128 2 dentry_open 0.0053 2 create_elf_tables 0.0022 2 copy_strings 0.0031 2 convert_fxsr_to_user 0.0057 2 change_protection 0.0035 2 bounce_copy_vec 0.0175 2 alloc_ldt 0.0047 2 add_wait_queue 0.0286 2 __user_walk 0.0213 2 __bio_map_user 0.0030 1 yield 0.0455 1 wake_up_inode 0.0156 1 vsscanf 0.0008 1 vgacon_cursor 0.0024 1 vfs_stat 0.0110 1 vfs_permission 0.0031 1 vfs_fstat 0.0125 1 unix_stream_connect 0.0010 1 unix_sock_destructor 0.0036 1 unix_peer_get 0.0238 1 uart_wait_until_sent 0.0045 1 sys_write 0.0101 1 sys_wait4 0.0016 1 sys_socketcall 0.0015 1 sys_shmdt 0.0030 1 sys_rt_sigreturn 0.0032 1 sys_read 0.0101 1 sys_mmap2 0.0059 1 sys_ioctl 0.0016 1 sys_dup2 0.0044 1 sys_clock_nanosleep 0.0034 1 sys_access 0.0030 1 sync_sb_inodes 0.0017 1 supplemental_group_member 0.0204 1 submit_bh 0.0029 1 sockfd_lookup 0.0084 1 sock_wfree 0.0132 1 sock_init_data 0.0030 1 sock_destroy_inode 0.0323 1 sock_create 0.0027 1 sock_alloc_send_skb 0.0196 1 sock_alloc 0.0106 1 skip_atoi 0.0154 1 skb_release_data 0.0045 1 skb_free_datagram 0.0256 1 shm_open 0.0111 1 shm_get_stat 0.0064 1 shm_close 0.0063 1 serial_in 0.0156 1 seq_printf 0.0116 1 sched_migrate_task 0.0127 1 save_i387_fxsave 0.0050 1 save_i387 0.0055 1 s_start 0.0024 1 run_timer_softirq 0.0026 1 restore_i387 0.0079 1 reserve_blocks 0.0055 1 release_task 0.0024 1 rb_insert_color 0.0044 1 rb_erase 0.0044 1 raw_release 0.0069 1 raw_open 0.0033 1 proc_readdir 0.0022 1 proc_info_read 0.0029 1 proc_destroy_inode 0.0323 1 prepare_binprm 0.0044 1 prep_compound_page 0.0185 1 posix_locks_conflict 0.0058 1 pipe_read 0.0017 1 path_release 0.0159 1 netif_receive_skb 0.0022 1 mpage_writepages 0.0014 1 move_addr_to_kernel 0.0069 1 monotonic_clock_cyclone 0.0061 1 mmput 0.0060 1 mm_init 0.0040 1 meminfo_read_proc 0.0020 1 locks_insert_lock 0.0143 1 locks_conflict 0.0323 1 load_elf_interp 0.0017 1 lease_alloc 0.0044 1 kmap 0.0455 1 ipc_addid 0.0058 1 inode_update_time 0.0059 1 huge_pte_alloc 0.0244 1 hash_vcache 0.0400 1 handle_signal 0.0032 1 handle_IRQ_event 0.0100 1 get_zeroed_page 0.0082 1 get_wchan 0.0082 1 get_unmapped_area 0.0029 1 generic_forget_inode 0.0033 1 follow_page 0.0024 1 flush_signal_handlers 0.0149 1 flush_old_exec 0.0005 1 flush_all_zero_pkmaps 0.0056 1 finish_wait 0.0111 1 find_task_by_pid 0.0130 1 find_group_orlov 0.0012 1 find_busiest_node 0.0015 1 filldir64 0.0032 1 file_ra_state_init 0.0312 1 fcntl_setlk 0.0014 1 fasync_helper 0.0043 1 ext2_new_block 0.0007 1 ext2_free_blocks 0.0014 1 ext2_find_entry 0.0015 1 ext2_block_to_path 0.0031 1 ext2_alloc_branch 0.0019 1 ext2_add_link 0.0009 1 exit_thread 0.0185 1 error_code 0.0179 1 end_buffer_async_write 0.0051 1 elv_put_request 0.0303 1 do_sync_write 0.0055 1 do_signal 0.0038 1 do_sigaltstack 0.0028 1 do_rw_proc 0.0062 1 do_pollfd 0.0069 1 do_poll 0.0051 1 destroy_inode 0.0125 1 destroy_context 0.0058 1 deny_write_access 0.0159 1 default_idle 0.0222 1 dcache_readdir 0.0021 1 d_rehash 0.0111 1 copy_siginfo_to_user 0.0025 1 con_write_room 0.0370 1 complete 0.0123 1 cap_vm_enough_memory 0.0053 1 cap_task_post_setuid 0.0034 1 can_vma_merge_after 0.0109 1 can_share_swap_page 0.0097 1 call_rcu 0.0192 1 batch_entropy_process 0.0047 1 assign_type 0.0417 1 alloc_buffer_head 0.0135 1 __vma_link 0.0063 1 __unix_insert_socket 0.0109 1 __rb_erase_color 0.0024 1 __pagevec_lru_add 0.0048 1 __kfree_skb 0.0046 1 __find_get_block_slow 0.0029 1 __filemap_copy_from_user_iovec 0.0057 1 SHATransform 0.0033 1 .text.lock.swap 0.0103 1 .text.lock.file_table 0.0092
Mike Sullivan
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |