lkml.org 
[lkml]   [2011]   [Oct]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())
This fix also went into 3.0.5 via
http://git.kernel.org/?p=linux/kernel/git/stable/stable-queue.git;a=blob;f=releases/3.0.5/block-free-queue-resources-at-blk_release_queue.patch
(originated at http://marc.info/?l=linux-scsi&m=131669751909474&w=2 and
http://marc.info/?l=linux-scsi&m=131669414205696&w=2)

However, it seems we still have a use-after-free bug, now causing the
following oops with kernel 3.0.6 when removing paths to storage while
generating load on device-mapper multipath disks:

> Unable to handle kernel pointer dereference at virtual kernel address 6b6b6b6b6b6b6000
> Oops: 0038 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> Modules linked in: iptable_filter ip_tables x_tables dm_round_robin sunrpc qeth_l3 binfmt_misc dm_multipath scsi_dh dm_mod ipv6 qeth ccwgroup [last unloaded: scsi_wait_scan]
> CPU: 1 Not tainted 3.0.6-50.x.20111006-s390xdefault #1
> Process blast.LzS_64_SL (pid: 26613, task: 0000000063e2a398, ksp: 0000000064de3560)
> Krnl PSW : 0704100180000000 000000000048a038 (scsi_print_command+0x44/0xf8)
> R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:0 CC:1 PM:0 EA:3
> Krnl GPRS: 000000000000006b 6b6b6b6b6b6b6b6b 000000006717f800 000000000094f2e0
> 000000000061242e 000000000062bd88 0000000066fb90d8 0000000065391ad7
> 000000006717f800 000000006717f800 000000006716a090 000000006717f800
> 0000000000000004 0000000000672f88 0000000064de3838 0000000064de3808
> Krnl Code: 000000000048a026: f0b80004ebbf srp 4(12,%r0),3007(%r14),8
> 000000000048a02c: f0a0000407f4 srp 4(11,%r0),2036,0
> 000000000048a032: e31020800004 lg %r1,128(%r2)
> >000000000048a038: e31010b00004 lg %r1,176(%r1)
> 000000000048a03e: b9020011 ltgr %r1,%r1
> 000000000048a042: a7840032 brc 8,48a0a6
> 000000000048a046: e33020000004 lg %r3,0(%r2)
> 000000000048a04c: c04000182f4c larl %r4,78fee4
> Call Trace:
> ([<000000006717f800>] 0x6717f800)
> [<0000000000487f28>] scsi_log_send+0xf0/0x130
> [<000000000048824c>] scsi_dispatch_cmd+0xc8/0x4bc
> [<0000000000490694>] scsi_request_fn+0x3b8/0x480
> [<00000000003fe4c8>] __elv_add_request+0x174/0x35c
> [<0000000000402fd6>] blk_insert_cloned_request+0x72/0xcc
> [<000003e001e134f2>] dm_dispatch_request+0x5a/0x94 [dm_mod]
> [<000003e001e15f36>] dm_request_fn+0x216/0x398 [dm_mod]
> [<0000000000405590>] queue_unplugged+0x60/0x1a8
> [<0000000000407470>] blk_flush_plug_list+0x218/0x29c
> [<00000000006136ec>] io_schedule+0x94/0x100
> [<000000000020e6d6>] sleep_on_page_killable+0x22/0x60
> [<00000000006140a8>] __wait_on_bit_lock+0xc0/0x124
> [<000000000020e5f8>] __lock_page_killable+0x78/0x84
> [<0000000000210690>] generic_file_aio_read+0x56c/0x800
> [<000000000027a0a0>] do_sync_read+0xcc/0x11c
> [<000000000027a974>] vfs_read+0xb0/0x194
> [<000000000027aab0>] SyS_read+0x58/0xb4
> [<00000000006185f6>] sysc_noemu+0x16/0x1c
> [<000003fffd473bd4>] 0x3fffd473bd4
> INFO: lockdep is turned off.
> Last Breaking-Event-Address:
> [<000000000048a020>] scsi_print_command+0x2c/0xf8

During a different run, we also got a similar oops which was not logged
but only its panic_on_oops:

> Kernel panic - not syncing: Fatal exception: panic_on_oops
> CPU: 1 Tainted: G D 3.0.6-50.x.20111006-s390xdefault #1
> Process blast.LzS_64_SL (pid: 6551, task: 00000000a2942398, ksp: 00000000a11834e0)
> 00000000a1183630 00000000a11835b0 0000000000000002 0000000000000000
> 00000000a1183650 00000000a11835c8 00000000a11835c8 000000000061242e
> 0000000000000000 0000000000000000 00000000a3176ed7 0000000000100ed2
> 000000000000000d 000000000000000c 00000000a1183618 0000000000000000
> 0000000000000000 0000000000100a18 00000000a11835b0 00000000a11835f0
> Call Trace:
> ([<0000000000100926>] show_trace+0xee/0x144)
> [<0000000000612258>] panic+0xb0/0x234
> [<0000000000100ed2>] die+0x166/0x168
> [<000000000011eede>] do_no_context+0xba/0xf8
> [<000000000061ad28>] do_asce_exception+0x130/0x18c
> [<0000000000618918>] pgm_exit+0x0/0x4
> [<000000000048a0d4>] scsi_print_command+0xe0/0xf8
> ([<000000000048a0e6>] scsi_print_command+0xf2/0xf8)
> [<0000000000487f28>] scsi_log_send+0xf0/0x130
> [<000000000048824c>] scsi_dispatch_cmd+0xc8/0x4bc
> [<0000000000490694>] scsi_request_fn+0x3b8/0x480
> [<00000000003fe4c8>] __elv_add_request+0x174/0x35c
> [<0000000000402fd6>] blk_insert_cloned_request+0x72/0xcc
> [<000003e001e134f2>] dm_dispatch_request+0x5a/0x94 [dm_mod]
> [<000003e001e15f36>] dm_request_fn+0x216/0x398 [dm_mod]
> [<0000000000405590>] queue_unplugged+0x60/0x1a8
> [<0000000000407470>] blk_flush_plug_list+0x218/0x29c
> [<000000000040751e>] blk_finish_plug+0x2a/0x58
> [<000000000021034e>] generic_file_aio_read+0x22a/0x800
> [<000000000027a0a0>] do_sync_read+0xcc/0x11c
> [<000000000027a974>] vfs_read+0xb0/0x194
> [<000000000027aab0>] SyS_read+0x58/0xb4
> [<00000000006185f6>] sysc_noemu+0x16/0x1c
> [<000003fffd395bd4>] 0x3fffd395bd4
> INFO: lockdep is turned off.

The other CPU was doing scsi_print_command successfully to the console
(sclp_console_write) when this panic occurred.

Earlier, we tried Alan's patch
http://marc.info/?l=linux-scsi&m=130963676807728&w=2
on 3.0.4, but in contrast to Cascardo's experience
http://marc.info/?l=linux-scsi&m=131482024119345&w=2
it did not work for us.

Initially, we encountered use-after-free bugs in
scsi_print_command / scsi_dispatch_cmd
http://marc.info/?l=linux-scsi&m=130824013229933&w=2
__elv_add_request
http://marc.info/?l=linux-scsi&m=130858383702949&w=2

Also related is the following thread:
http://marc.info/?l=linux-scsi&m=130977887419062&w=2

Unfortunately, I don't understand what exactly is going wrong here.
Can you shed some light on this?

On 09/28/2011 04:14 PM, Jens Axboe wrote:
> Alright, lets call this fully tested and fixed then. Linus, I committed
> it, please pull:
>
> git://git.kernel.dk/linux-block.git for-linus
>
> Hannes Reinecke (1):
> block: Free queue resources at blk_release_queue()
>
> block/blk-core.c | 13 ++++++-------
> block/blk-sysfs.c | 5 +++++
> 2 files changed, 11 insertions(+), 7 deletions(-)
>
> diff --git a/block/blk-core.c b/block/blk-core.c
> index b2ed78a..d34433a 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -348,9 +348,10 @@ void blk_put_queue(struct request_queue *q)
> EXPORT_SYMBOL(blk_put_queue);
>
> /*
> - * Note: If a driver supplied the queue lock, it should not zap that lock
> - * unexpectedly as some queue cleanup components like elevator_exit() and
> - * blk_throtl_exit() need queue lock.
> + * Note: If a driver supplied the queue lock, it is disconnected
> + * by this function. The actual state of the lock doesn't matter
> + * here as the request_queue isn't accessible after this point
> + * (QUEUE_FLAG_DEAD is set) and no other requests will be queued.
> */
> void blk_cleanup_queue(struct request_queue *q)
> {
> @@ -367,10 +368,8 @@ void blk_cleanup_queue(struct request_queue *q)
> queue_flag_set_unlocked(QUEUE_FLAG_DEAD, q);
> mutex_unlock(&q->sysfs_lock);
>
> - if (q->elevator)
> - elevator_exit(q->elevator);
> -
> - blk_throtl_exit(q);
> + if (q->queue_lock !=&q->__queue_lock)
> + q->queue_lock =&q->__queue_lock;
>
> blk_put_queue(q);
> }
> diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
> index e681805..60fda88 100644
> --- a/block/blk-sysfs.c
> +++ b/block/blk-sysfs.c
> @@ -479,6 +479,11 @@ static void blk_release_queue(struct kobject *kobj)
>
> blk_sync_queue(q);
>
> + if (q->elevator)
> + elevator_exit(q->elevator);
> +
> + blk_throtl_exit(q);
> +
> if (rl->rq_pool)
> mempool_destroy(rl->rq_pool);
>
>
>

--
Steffen Maier

Linux on System z Development

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2011-10-13 15:11    [W:0.130 / U:0.060 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site