lkml.org 
[lkml]   [2021]   [Jul]   [1]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patch in this message
/
From
SubjectRe: [PATCH] sched/fair: Ensure _sum and _avg values stay consistent
Date


> On 24-Jun-2021, at 4:48 PM, Odin Ugedal <odin@uged.al> wrote:
>
> The _sum and _avg values are in general sync together with the PELT
> divider. They are however not always completely in perfect sync,
> resulting in situations where _sum gets to zero while _avg stays
> positive. Such situations are undesirable.
>
> This comes from the fact that PELT will increase period_contrib, also
> increasing the PELT divider, without updating _sum and _avg values to
> stay in perfect sync where (_sum == _avg * divider). However, such PELT
> change will never lower _sum, making it impossible to end up in a
> situation where _sum is zero and _avg is not.
>
> Therefore, we need to ensure that when subtracting load outside PELT,
> that when _sum is zero, _avg is also set to zero. This occurs when
> (_sum < _avg * divider), and the subtracted (_avg * divider) is bigger
> or equal to the current _sum, while the subtracted _avg is smaller than
> the current _avg.
>
> Reported-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
> Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
> Signed-off-by: Odin Ugedal <odin@uged.al>

Hello Odin, Vincent,

The issue of kernel warning(during boot) seen on Power sever, reported few days back is not completely fixed.
I am able to recreate this issue with latest 5.13 kernel(which has this fix)

# git log --oneline kernel/sched/fair.c
a6eaf3850cb1 Merge tag 'sched-urgent-2021-06-30' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
54a728dc5e4f Merge tag 'sched-core-2021-06-28' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
1c35b07e6d39 sched/fair: Ensure _sum and _avg values stay consistent <<==


[ 65.407631] ------------[ cut here ]------------
[ 65.407656] cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
[ 65.407666] WARNING: CPU: 18 PID: 6642 at kernel/sched/fair.c:3308 update_blocked_averages+0x748/0x7a0
[ 65.407693] Modules linked in: dm_mod bonding nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink sunrpc xfs libcrc32c pseries_rng xts uio_pdrv_genirq vmx_crypto uio sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse
[ 65.407776] CPU: 18 PID: 6642 Comm: gcc Not tainted 5.13.0-05357-gdbe69e433722 #1
[ 65.407789] NIP: c0000000001b2e48 LR: c0000000001b2e44 CTR: c00000000072cac0
[ 65.407798] REGS: c00000003f2cf660 TRAP: 0700 Not tainted (5.13.0-05357-gdbe69e433722)
[ 65.407809] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48088224 XER: 00000005
[ 65.407837] CFAR: c00000000014d520 IRQMASK: 1
GPR00: c0000000001b2e44 c00000003f2cf900 c0000000029bc700 0000000000000048
GPR04: 00000000ffff7fff c00000003f2cf5c0 0000000000000027 c0000008baa07e18
GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000002876888
GPR12: 0000000000008000 c00000001ec35a80 c0000008baa92580 0000000f3a993c3c
GPR16: 0000000000000000 c0000008baa92580 c00000001a53b400 0000000000000001
GPR20: 0000000000000000 c0000000029fdfe0 0000000000000000 0000000000000012
GPR24: 0000000000000000 c0000008baa92f90 0000000000000001 c0000008baa92600
GPR28: 0000000000000012 0000000f3a993c3c c00000001a53b5c0 0000000000000001
[ 65.407984] NIP [c0000000001b2e48] update_blocked_averages+0x748/0x7a0
[ 65.407995] LR [c0000000001b2e44] update_blocked_averages+0x744/0x7a0
[ 65.408005] Call Trace:
[ 65.408010] [c00000003f2cf900] [c0000000001b2e44] update_blocked_averages+0x744/0x7a0 (unreliable)
[ 65.408025] [c00000003f2cfa20] [c0000000001bdd78] newidle_balance+0x258/0x5c0
[ 65.408038] [c00000003f2cfab0] [c0000000001be1bc] pick_next_task_fair+0x7c/0x4c0
[ 65.408051] [c00000003f2cfb10] [c000000000cf223c] __schedule+0x15c/0x17b0
[ 65.408063] [c00000003f2cfc50] [c0000000001a6044] do_task_dead+0x64/0x70
[ 65.408076] [c00000003f2cfc80] [c0000000001567d8] do_exit+0x868/0xce0
[ 65.408089] [c00000003f2cfd50] [c000000000156d24] do_group_exit+0x64/0xe0
[ 65.408101] [c00000003f2cfd90] [c000000000156dc4] sys_exit_group+0x24/0x30
[ 65.408115] [c00000003f2cfdb0] [c0000000000307b0] system_call_exception+0x150/0x2d0
[ 65.408128] [c00000003f2cfe10] [c00000000000cc5c] system_call_common+0xec/0x278
[ 65.408141] --- interrupt: c00 at 0x7fffa163b80c
[ 65.408150] NIP: 00007fffa163b80c LR: 00007fffa1597004 CTR: 0000000000000000
[ 65.408159] REGS: c00000003f2cfe80 TRAP: 0c00 Not tainted (5.13.0-05357-gdbe69e433722)
[ 65.408168] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28000222 XER: 00000000
[ 65.408199] IRQMASK: 0
GPR00: 00000000000000ea 00007fffdf5cd220 00007fffa1757300 0000000000000000
GPR04: 0000000000000000 00007fffdf5cd078 0000000000000000 0000000000000000
GPR08: 0000000000000001 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffa192a930 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 00007fffa1754ee0 0000000000000004 0000000000000001
GPR24: 00007fffa17508a0 0000000000000000 0000000000000000 0000000000000001
GPR28: 0000000000000000 0000000000000000 00007fffa1923950 0000000000000000
[ 65.408308] NIP [00007fffa163b80c] 0x7fffa163b80c
[ 65.408317] LR [00007fffa1597004] 0x7fffa1597004
[ 65.408324] --- interrupt: c00
[ 65.408331] Instruction dump:
[ 65.408339] 2f890000 409efa3c 4bffa119 4bfffa34 60000000 60000000 e9210070 e8610088
[ 65.408361] 39400001 99490003 4bf9a679 60000000 <0fe00000> e95201ba 2faa0000 4bfffc50
[ 65.408383] ---[ end trace 14708e73bf91cf1c ]—


Let me know if any additional information is required.

I have attached the dmesg log captured after the failure.

Thanks
-Sachin

> ---
>
> Reports and discussion can be found here:
>
> https://lore.kernel.org/lkml/2ED1BDF5-BC0C-47CD-8F33-9A46C738F8CF@linux.vnet.ibm.com/
> https://lore.kernel.org/lkml/CA+G9fYsMXELmjGUzw4SY1bghTYz_PeR2diM6dRp2J37bBZzMSA@mail.gmail.com/
>
> kernel/sched/fair.c | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index bfaa6e1f6067..def48bc2e90b 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -3688,15 +3688,15 @@ update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq)
>
> r = removed_load;
> sub_positive(&sa->load_avg, r);
> - sub_positive(&sa->load_sum, r * divider);
> + sa->load_sum = sa->load_avg * divider;
>
> r = removed_util;
> sub_positive(&sa->util_avg, r);
> - sub_positive(&sa->util_sum, r * divider);
> + sa->util_sum = sa->util_avg * divider;
>
> r = removed_runnable;
> sub_positive(&sa->runnable_avg, r);
> - sub_positive(&sa->runnable_sum, r * divider);
> + sa->runnable_sum = sa->runnable_avg * divider;
>
> /*
> * removed_runnable is the unweighted version of removed_load so we
> --
> 2.32.0
>


[ 0.000000] crashkernel: memory value expected
[ 0.000000] hash-mmu: Page sizes from device-tree:
[ 0.000000] hash-mmu: base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
[ 0.000000] hash-mmu: base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7
[ 0.000000] hash-mmu: base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56
[ 0.000000] hash-mmu: base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1
[ 0.000000] hash-mmu: base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=8
[ 0.000000] hash-mmu: base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0
[ 0.000000] hash-mmu: base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, tlbiel=0, penc=3
[ 0.000000] Enabling pkeys with max key count 31
[ 0.000000] Activating Kernel Userspace Execution Prevention
[ 0.000000] Activating Kernel Userspace Access Prevention
[ 0.000000] Page orders: linear mapping = 24, virtual = 16, io = 16, vmemmap = 24
[ 0.000000] Using 1TB segments
[ 0.000000] hash-mmu: Initializing hash mmu with SLB
[ 0.000000] Linux version 5.13.0-05357-gdbe69e433722 (root@ltc-zzci-4.aus.stglabs.ibm.com) (gcc (GCC) 8.4.1 20200928 (Red Hat 8.4.1-1), GNU ld version 2.30-93.el8) #1 SMP Thu Jul 1 04:24:02 EDT 2021
[ 0.000000] Found initrd at 0xc00000000d500000:0xc00000001365b29d
[ 0.000000] Using pSeries machine description
[ 0.000000] printk: bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 40 cpus.
[ 0.000000] CPU maps initialized for 8 threads per core
[ 0.000000] (thread shift is 3)
[ 0.000000] Allocated 4800 bytes for 40 pacas
[ 0.000000] -----------------------------------------------------
[ 0.000000] phys_mem_size = 0x8c0000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x0001c07b8f5f9187
[ 0.000000] possible = 0x000ffbfbcf5fb187
[ 0.000000] always = 0x0000000380008181
[ 0.000000] cpu_user_features = 0xdc0065c2 0xeff00000
[ 0.000000] mmu_features = 0x7c006e01
[ 0.000000] firmware_features = 0x0000009fc45bfc57
[ 0.000000] vmalloc start = 0xc008000000000000
[ 0.000000] IO start = 0xc00a000000000000
[ 0.000000] vmemmap start = 0xc00c000000000000
[ 0.000000] hash-mmu: ppc64_pft_size = 0x1c
[ 0.000000] hash-mmu: htab_hash_mask = 0x1fffff
[ 0.000000] -----------------------------------------------------
[ 0.000000] numa: NODE_DATA [mem 0x8bfe9af80-0x8bfe9ffff]
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] rfi-flush: patched 13 locations (mttrig type flush)
[ 0.000000] count-cache-flush: flush disabled.
[ 0.000000] link-stack-flush: software flush enabled.
[ 0.000000] entry-flush: patched 61 locations (mttrig type flush)
[ 0.000000] uaccess-flush: patched 1 locations (mttrig type flush)
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] stf-barrier: patched 61 entry locations (eieio barrier)
[ 0.000000] stf-barrier: patched 13 exit locations (eieio barrier)
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:0 psize:0 block size:8
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:0 psize:2 block size:8
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:0 psize:10 block size:8
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:2 psize:2 block size:8
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:2 psize:10 block size:8
[ 0.000000] PPC64 nvram contains 15360 bytes
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] barrier-nospec: patched 385 locations
[ 0.000000] Top of RAM: 0x8c0000000, Total RAM: 0x8c0000000
[ 0.000000] Memory hole size: 0MB
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000000000000-0x00000008bfffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x00000008bfffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000008bfffffff]
[ 0.000000] percpu: Embedded 10 pages/cpu s605976 r0 d49384 u1048576
[ 0.000000] pcpu-alloc: s605976 r0 d49384 u1048576 alloc=1*1048576
[ 0.000000] pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0] 07
[ 0.000000] pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0] 15
[ 0.000000] pcpu-alloc: [0] 16 [0] 17 [0] 18 [0] 19 [0] 20 [0] 21 [0] 22 [0] 23
[ 0.000000] pcpu-alloc: [0] 24 [0] 25 [0] 26 [0] 27 [0] 28 [0] 29 [0] 30 [0] 31
[ 0.000000] pcpu-alloc: [0] 32 [0] 33 [0] 34 [0] 35 [0] 36 [0] 37 [0] 38 [0] 39
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 572880
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-5.13.0-05357-gdbe69e433722 root=UUID=a95e51fa-8962-4cb2-8189-8c8cd8d653c9 ro crashkernel=auto biosdevname=0
[ 0.000000] Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 36290240K/36700160K available (13376K kernel code, 5440K rwdata, 4096K rodata, 4992K init, 2863K bss, 409920K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=40, Nodes=2
[ 0.000000] ftrace: allocating 33136 entries in 13 pages
[ 0.000000] ftrace: allocated 13 pages with 3 groups
[ 0.000000] trace event string verifier disabled
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=40.
[ 0.000000] Rude variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=40
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000000] pic: no ISA interrupt controller
[ 0.000000] rcu: Offload RCU callbacks from CPUs: (none).
[ 0.000000] random: get_random_u64 called from start_kernel+0x834/0xa90 with crng_init=0
[ 0.000000] time_init: decrementer frequency = 512.000000 MHz
[ 0.000000] time_init: processor frequency = 2900.000000 MHz
[ 0.000002] time_init: 56 bit decrementer (max: 7fffffffffffff)
[ 0.000040] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[ 0.000106] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.000149] clockevent: decrementer mult[83126f] shift[24] cpu[0]
[ 0.000205] Console: colour dummy device 80x25
[ 0.000237] printk: console [hvc0] enabled
[ 0.000266] printk: bootconsole [udbg0] disabled
[ 0.000324] pid_max: default: 40960 minimum: 320
[ 0.000372] LSM: Security Framework initializing
[ 0.000387] Yama: becoming mindful.
[ 0.000399] SELinux: Initializing.
[ 0.000521] Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
[ 0.000599] Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
[ 0.001300] POWER9 performance monitor hardware support registered
[ 0.001335] rcu: Hierarchical SRCU implementation.
[ 0.002406] smp: Bringing up secondary CPUs ...
[ 0.017195] smp: Brought up 1 node, 40 CPUs
[ 0.017206] numa: Node 0 CPUs: 0-39
[ 0.017214] Big cores detected but using small core scheduling
[ 0.018638] devtmpfs: initialized
[ 0.022718] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.022727] futex hash table entries: 16384 (order: 5, 2097152 bytes, linear)
[ 0.023011] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[ 0.023112] audit: initializing netlink subsys (disabled)
[ 0.023179] audit: type=2000 audit(1625128776.020:1): state=initialized audit_enabled=0 res=1
[ 0.023234] thermal_sys: Registered thermal governor 'fair_share'
[ 0.023236] thermal_sys: Registered thermal governor 'step_wise'
[ 0.023339] cpuidle: using governor menu
[ 0.023407] RTAS daemon started
[ 0.023543] pstore: Registered nvram as persistent store backend
[ 0.024150] EEH: pSeries platform initialized
[ 0.029601] PCI: Probing PCI hardware
[ 0.029605] EEH: No capable adapters found: recovery disabled.
[ 0.029609] PCI: Probing PCI hardware done
[ 0.029704] pseries-rng: Registering arch random hook.
[ 0.030975] Kprobes globally optimized
[ 0.031162] HugeTLB registered 16.0 MiB page size, pre-allocated 0 pages
[ 0.031167] HugeTLB registered 16.0 GiB page size, pre-allocated 0 pages
[ 0.065972] wait_for_initramfs() called before rootfs_initcalls
[ 0.069388] iommu: Default domain type: Translated
[ 0.069440] vgaarb: loaded
[ 0.069515] SCSI subsystem initialized
[ 0.069549] usbcore: registered new interface driver usbfs
[ 0.069559] usbcore: registered new interface driver hub
[ 0.069575] usbcore: registered new device driver usb
[ 0.069657] EDAC MC: Ver: 3.0.0
[ 0.069874] NetLabel: Initializing
[ 0.069877] NetLabel: domain hash size = 128
[ 0.069880] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
[ 0.069894] NetLabel: unlabeled traffic allowed by default
[ 0.070078] clocksource: Switched to clocksource timebase
[ 0.080481] VFS: Disk quotas dquot_6.6.0
[ 0.080504] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[ 0.082731] NET: Registered PF_INET protocol family
[ 0.082877] IP idents hash table entries: 262144 (order: 5, 2097152 bytes, linear)
[ 0.085639] tcp_listen_portaddr_hash hash table entries: 32768 (order: 3, 524288 bytes, linear)
[ 0.085703] TCP established hash table entries: 524288 (order: 6, 4194304 bytes, linear)
[ 0.086557] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes, linear)
[ 0.086650] TCP: Hash tables configured (established 524288 bind 65536)
[ 0.086700] UDP hash table entries: 32768 (order: 4, 1048576 bytes, linear)
[ 0.086799] UDP-Lite hash table entries: 32768 (order: 4, 1048576 bytes, linear)
[ 0.086963] NET: Registered PF_UNIX/PF_LOCAL protocol family
[ 0.086972] PCI: CLS 0 bytes, default 128
[ 0.087015] Trying to unpack rootfs image as initramfs...
[ 0.087662] IOMMU table initialized, virtual merging enabled
[ 0.101220] hv-24x7: read 1530 catalog entries, created 509 event attrs (0 failures), 275 descs
[ 0.101890] random: fast init done
[ 0.104385] Initialise system trusted keyrings
[ 0.104442] workingset: timestamp_bits=38 max_order=20 bucket_order=0
[ 0.105898] zbud: loaded
[ 0.127277] NET: Registered PF_ALG protocol family
[ 0.127282] Key type asymmetric registered
[ 0.127285] Asymmetric key parser 'x509' registered
[ 0.127294] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
[ 0.127350] io scheduler mq-deadline registered
[ 0.127356] io scheduler kyber registered
[ 0.128111] atomic64_test: passed
[ 0.128154] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 0.128499] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.128752] Non-volatile memory driver v1.3
[ 0.128784] Linux agpgart interface v0.103
[ 0.128946] tpm_ibmvtpm 30000003: CRQ initialization completed
[ 1.200907] rdac: device handler registered
[ 1.200975] hp_sw: device handler registered
[ 1.200978] emc: device handler registered
[ 1.201039] alua: device handler registered
[ 1.201138] libphy: Fixed MDIO Bus: probed
[ 1.201183] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 1.201195] ehci-pci: EHCI PCI platform driver
[ 1.201205] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 1.201215] ohci-pci: OHCI PCI platform driver
[ 1.201226] uhci_hcd: USB Universal Host Controller Interface driver
[ 1.201263] usbcore: registered new interface driver usbserial_generic
[ 1.201271] usbserial: USB Serial support registered for generic
[ 1.201333] mousedev: PS/2 mouse device common for all mice
[ 1.201439] rtc-generic rtc-generic: registered as rtc0
[ 1.201550] rtc-generic rtc-generic: setting system clock to 2021-07-01T08:39:37 UTC (1625128777)
[ 1.201639] xcede: xcede_record_size = 10
[ 1.201642] xcede: Record 0 : hint = 1, latency = 0x400 tb ticks, Wake-on-irq = 1
[ 1.201647] xcede: Record 1 : hint = 2, latency = 0x3e8000 tb ticks, Wake-on-irq = 0
[ 1.201652] cpuidle: Skipping the 2 Extended CEDE idle states
[ 1.201656] cpuidle: Fixed up CEDE exit latency to 1 us
[ 1.202109] pseries_idle_driver registered
[ 1.202203] nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sync_size new:65536 old:0
[ 1.202209] nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sync_sg new:510 old:0
[ 1.202215] nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sg_len new:4080 old:0
[ 1.202451] alg: No test for 842 (842-nx)
[ 1.202499] hid: raw HID events driver (C) Jiri Kosina
[ 1.202603] usbcore: registered new interface driver usbhid
[ 1.202606] usbhid: USB HID core driver
[ 1.202627] drop_monitor: Initializing network drop monitor service
[ 1.202688] Initializing XFRM netlink socket
[ 1.202774] NET: Registered PF_INET6 protocol family
[ 1.203059] Segment Routing with IPv6
[ 1.203076] NET: Registered PF_PACKET protocol family
[ 1.203082] mpls_gso: MPLS GSO support
[ 1.204026] Running MSI bitmap self-tests ...
[ 1.205583] registered taskstats version 1
[ 1.205597] Loading compiled-in X.509 certificates
[ 1.540712] Freeing initrd memory: 99648K
[ 1.543608] alg: No test for pkcs1pad(rsa,sha1) (pkcs1pad(rsa-generic,sha1))
[ 1.544279] Loaded X.509 cert 'Build time autogenerated kernel key: 2bba0209abfc79c0306841f3505f786c172fa1b0'
[ 1.544400] zswap: loaded using pool lzo/zbud
[ 1.544520] pstore: Using crash dump compression: deflate
[ 1.546192] Freeing unused kernel image (initmem) memory: 4992K
[ 1.590239] Run /init as init process
[ 1.590244] with arguments:
[ 1.590245] /init
[ 1.590247] with environment:
[ 1.590248] HOME=/
[ 1.590249] TERM=linux
[ 1.590250] BOOT_IMAGE=/vmlinuz-5.13.0-05357-gdbe69e433722
[ 1.590251] crashkernel=auto
[ 1.590252] biosdevname=0
[ 1.598752] systemd[1]: systemd 239 (239-45.el8) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
[ 1.598975] systemd[1]: Detected architecture ppc64-le.
[ 1.598981] systemd[1]: Running in initial RAM disk.
[ 1.650496] systemd[1]: Set hostname to <ltc-zzci-4.aus.stglabs.ibm.com>.
[ 1.710918] random: systemd: uninitialized urandom read (16 bytes read)
[ 1.710999] systemd[1]: Listening on udev Kernel Socket.
[ 1.711220] random: systemd: uninitialized urandom read (16 bytes read)
[ 1.711306] systemd[1]: Listening on udev Control Socket.
[ 1.711427] random: systemd: uninitialized urandom read (16 bytes read)
[ 1.711440] systemd[1]: Reached target Slices.
[ 1.711637] systemd[1]: Listening on Journal Socket (/dev/log).
[ 1.711754] systemd[1]: Reached target Timers.
[ 1.711867] systemd[1]: Reached target Local File Systems.
[ 1.769510] fuse: init (API version 7.33)
[ 1.953986] synth uevent: /devices/vio: failed to send uevent
[ 1.953998] vio vio: uevent: failed to send synthetic uevent
[ 1.954079] synth uevent: /devices/vio/4000: failed to send uevent
[ 1.954083] vio 4000: uevent: failed to send synthetic uevent
[ 1.954099] synth uevent: /devices/vio/4001: failed to send uevent
[ 1.954103] vio 4001: uevent: failed to send synthetic uevent
[ 1.954118] synth uevent: /devices/vio/4002: failed to send uevent
[ 1.954122] vio 4002: uevent: failed to send synthetic uevent
[ 1.954137] synth uevent: /devices/vio/4004: failed to send uevent
[ 1.954141] vio 4004: uevent: failed to send synthetic uevent
[ 1.986172] ibmveth: IBM Power Virtual Ethernet Driver 1.06
[ 1.987465] ibmveth 30000002 net0: renamed from eth0
[ 1.988322] ibmvscsi 30000068: SRP_VERSION: 16.a
[ 1.988426] ibmvscsi 30000068: Maximum ID: 64 Maximum LUN: 32 Maximum Channel: 3
[ 1.988433] scsi host0: IBM POWER Virtual SCSI Adapter 1.5.9
[ 1.988582] ibmvscsi 30000068: partner initialization complete
[ 1.988618] ibmvscsi 30000068: host srp version: 16.a, host partition ltc-zzci-vios1 (100), OS 3, max io 262144
[ 1.988672] ibmvscsi 30000068: Client reserve enabled
[ 1.988679] ibmvscsi 30000068: sent SRP login
[ 1.988710] ibmvscsi 30000068: SRP_LOGIN succeeded
[ 2.021106] scsi 0:0:1:0: Direct-Access AIX VDASD 0001 PQ: 0 ANSI: 3
[ 2.057643] scsi 0:0:1:0: Attached scsi generic sg0 type 0
[ 2.072743] sd 0:0:1:0: [sda] 139466752 4096-byte logical blocks: (571 GB/532 GiB)
[ 2.072783] sd 0:0:1:0: [sda] Write Protect is off
[ 2.072788] sd 0:0:1:0: [sda] Mode Sense: 17 00 00 08
[ 2.072822] sd 0:0:1:0: [sda] Cache data unavailable
[ 2.072827] sd 0:0:1:0: [sda] Assuming drive cache: write through
[ 2.161746] sda: sda1 sda2 sda3 sda4 < sda5 >
[ 2.162697] sd 0:0:1:0: [sda] Attached SCSI disk
[ 2.619863] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 3.164023] printk: systemd: 15 output lines suppressed due to ratelimiting
[ 3.507303] SELinux: Runtime disable is deprecated, use selinux=0 on the kernel cmdline.
[ 3.507312] SELinux: Disabled at runtime.
[ 3.660100] audit: type=1404 audit(1625128779.950:2): enforcing=0 old_enforcing=0 auid=4294967295 ses=4294967295 enabled=0 old-enabled=1 lsm=selinux res=1
[ 4.203691] systemd[1]: systemd 239 (239-45.el8) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
[ 4.203904] systemd[1]: Detected architecture ppc64-le.
[ 4.209939] systemd[1]: Set hostname to <ltc-zzci-4.aus.stglabs.ibm.com>.
[ 4.240883] random: crng init done
[ 4.240889] random: 7 urandom warning(s) missed due to ratelimiting
[ 5.445640] systemd[1]: systemd-journald.service: Succeeded.
[ 5.446405] systemd[1]: initrd-switch-root.service: Succeeded.
[ 5.446757] systemd[1]: Stopped Switch Root.
[ 5.447376] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart.
[ 5.447489] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[ 5.447582] systemd[1]: Stopped Journal Service.
[ 5.448412] systemd[1]: Starting Journal Service...
[ 5.602303] synth uevent: /devices/vio: failed to send uevent
[ 5.602314] vio vio: uevent: failed to send synthetic uevent
[ 5.602560] synth uevent: /devices/vio/4000: failed to send uevent
[ 5.602565] vio 4000: uevent: failed to send synthetic uevent
[ 5.602580] synth uevent: /devices/vio/4001: failed to send uevent
[ 5.602584] vio 4001: uevent: failed to send synthetic uevent
[ 5.602597] synth uevent: /devices/vio/4002: failed to send uevent
[ 5.602601] vio 4002: uevent: failed to send synthetic uevent
[ 5.602615] synth uevent: /devices/vio/4004: failed to send uevent
[ 5.602619] vio 4004: uevent: failed to send synthetic uevent
[ 5.621109] Adding 41942976k swap on /dev/sda3. Priority:-2 extents:1 across:41942976k FS
[ 5.624915] EXT4-fs (sda5): re-mounted. Opts: (null). Quota mode: none.
[ 6.331259] pseries_rng: Registering IBM pSeries RNG driver
[ 7.532889] SGI XFS with ACLs, security attributes, quota, no debug enabled
[ 7.534521] XFS (sda2): Mounting V5 Filesystem
[ 7.598310] XFS (sda2): Ending clean mount
[ 7.611036] xfs filesystem being mounted at /boot supports timestamps until 2038 (0x7fffffff)
[ 8.126320] RPC: Registered named UNIX socket transport module.
[ 8.126329] RPC: Registered udp transport module.
[ 8.126333] RPC: Registered tcp transport module.
[ 8.126336] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 16.172919] device-mapper: uevent: version 1.0.3
[ 16.173009] device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
[ 65.407631] ------------[ cut here ]------------
[ 65.407656] cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
[ 65.407666] WARNING: CPU: 18 PID: 6642 at kernel/sched/fair.c:3308 update_blocked_averages+0x748/0x7a0
[ 65.407693] Modules linked in: dm_mod bonding nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink sunrpc xfs libcrc32c pseries_rng xts uio_pdrv_genirq vmx_crypto uio sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse
[ 65.407776] CPU: 18 PID: 6642 Comm: gcc Not tainted 5.13.0-05357-gdbe69e433722 #1
[ 65.407789] NIP: c0000000001b2e48 LR: c0000000001b2e44 CTR: c00000000072cac0
[ 65.407798] REGS: c00000003f2cf660 TRAP: 0700 Not tainted (5.13.0-05357-gdbe69e433722)
[ 65.407809] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48088224 XER: 00000005
[ 65.407837] CFAR: c00000000014d520 IRQMASK: 1
GPR00: c0000000001b2e44 c00000003f2cf900 c0000000029bc700 0000000000000048
GPR04: 00000000ffff7fff c00000003f2cf5c0 0000000000000027 c0000008baa07e18
GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000002876888
GPR12: 0000000000008000 c00000001ec35a80 c0000008baa92580 0000000f3a993c3c
GPR16: 0000000000000000 c0000008baa92580 c00000001a53b400 0000000000000001
GPR20: 0000000000000000 c0000000029fdfe0 0000000000000000 0000000000000012
GPR24: 0000000000000000 c0000008baa92f90 0000000000000001 c0000008baa92600
GPR28: 0000000000000012 0000000f3a993c3c c00000001a53b5c0 0000000000000001
[ 65.407984] NIP [c0000000001b2e48] update_blocked_averages+0x748/0x7a0
[ 65.407995] LR [c0000000001b2e44] update_blocked_averages+0x744/0x7a0
[ 65.408005] Call Trace:
[ 65.408010] [c00000003f2cf900] [c0000000001b2e44] update_blocked_averages+0x744/0x7a0 (unreliable)
[ 65.408025] [c00000003f2cfa20] [c0000000001bdd78] newidle_balance+0x258/0x5c0
[ 65.408038] [c00000003f2cfab0] [c0000000001be1bc] pick_next_task_fair+0x7c/0x4c0
[ 65.408051] [c00000003f2cfb10] [c000000000cf223c] __schedule+0x15c/0x17b0
[ 65.408063] [c00000003f2cfc50] [c0000000001a6044] do_task_dead+0x64/0x70
[ 65.408076] [c00000003f2cfc80] [c0000000001567d8] do_exit+0x868/0xce0
[ 65.408089] [c00000003f2cfd50] [c000000000156d24] do_group_exit+0x64/0xe0
[ 65.408101] [c00000003f2cfd90] [c000000000156dc4] sys_exit_group+0x24/0x30
[ 65.408115] [c00000003f2cfdb0] [c0000000000307b0] system_call_exception+0x150/0x2d0
[ 65.408128] [c00000003f2cfe10] [c00000000000cc5c] system_call_common+0xec/0x278
[ 65.408141] --- interrupt: c00 at 0x7fffa163b80c
[ 65.408150] NIP: 00007fffa163b80c LR: 00007fffa1597004 CTR: 0000000000000000
[ 65.408159] REGS: c00000003f2cfe80 TRAP: 0c00 Not tainted (5.13.0-05357-gdbe69e433722)
[ 65.408168] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28000222 XER: 00000000
[ 65.408199] IRQMASK: 0
GPR00: 00000000000000ea 00007fffdf5cd220 00007fffa1757300 0000000000000000
GPR04: 0000000000000000 00007fffdf5cd078 0000000000000000 0000000000000000
GPR08: 0000000000000001 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffa192a930 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 00007fffa1754ee0 0000000000000004 0000000000000001
GPR24: 00007fffa17508a0 0000000000000000 0000000000000000 0000000000000001
GPR28: 0000000000000000 0000000000000000 00007fffa1923950 0000000000000000
[ 65.408308] NIP [00007fffa163b80c] 0x7fffa163b80c
[ 65.408317] LR [00007fffa1597004] 0x7fffa1597004
[ 65.408324] --- interrupt: c00
[ 65.408331] Instruction dump:
[ 65.408339] 2f890000 409efa3c 4bffa119 4bfffa34 60000000 60000000 e9210070 e8610088
[ 65.408361] 39400001 99490003 4bf9a679 60000000 <0fe00000> e95201ba 2faa0000 4bfffc50
[ 65.408383] ---[ end trace 14708e73bf91cf1c ]---
\
 
 \ /
  Last update: 2021-07-01 11:50    [W:2.035 / U:0.244 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site