pfSense filesystem corruption after power outage
pfSense has a serious flaw that has affected some of our customers. The UFS filesystem used by pfSense is sensitive to sudden power loss. When pfSense system losses power at a wrong moment, the UFS filesystem may get corrupted, which may result in unbootable router. This problem is not related to hardware, it affects all routers with pfSense.
Most customers that reported this problem to us said that it appeared right aftter a sudden electricity outage. One customer reported that it happened during a pfSense upgrade.
We have tried to replicate this by pulling the plug while the router was running, 15 times in a row, and managed to reproduce it once.
There are many threads on pfSense forum that decriebe exactly the same problem: unbootable or boot-looping operating system after electricity loss.
pfSense team is aware of this issue and recently added ability to choose ZFS instead of UFS when installing the OS. In our experience ZFS is immune to the issue, so we strongly recommend using ZFS instead of UFS.
The quickest way to fix this problem to reinstall pfSense, as described in pfSense installation instructions
If you don't want to reinstall, you may try to restore the system by running fsck as descived in the docs. In my experience this method not always works.
Here's one of the tracebacks that you could see:
SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! Timecounter "TSC" frequency 1000026765 Hz quality 1000 Trying to mount root from ufs:/dev/ufsid/1833a427785e9dd3 [rw]… Configuring crash dumps… Using /dev/label/swap0 for dump device. /dev/ufsid/1833a427785e9dd3: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufsid/1833a427785e9dd3: clean, 12988930 free (2712 frags, 1418277 blocks, 0.0% fragmentation) Filesystems are clean, continuing... Mounting filesystems... Disabling APM on /dev/ad8 pid 52 (mdconfig), uid 0: exited on signal 11 (core dumped)
This makes the system die after the last line.Here's a different traceback, this time kernel panics - this is the one we managed to reproduce.
\KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p1 #83 r313908+d77c47fe50c(RELENG_2_4_0): Tue Oct 10 06:48:42 CDT 2017
root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
VT(vga): resolution 640x480
CPU: AMD GX-412TC SOC (998.15-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x730f01 Family=0x16 Model=0x30 Stepping=1
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C>
AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
AMD Features2=0x1d4037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PTSC,PL2I>
Structured Extended Features=0x8<BMI1>
XSAVE Features=0x1<XSAVEOPT>
SVM: NP,NRIP,AFlush,DAssist,NAsids=8
TSC: P-state invariant, performance statistics
real memory = 4815060992 (4592 MB)
avail memory = 4073357312 (3884 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <CORE COREBOOT>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic1: Changing APIC ID to 5
ioapic0 <Version 2.1> irqs 0-23 on motherboard
ioapic1 <Version 2.1> irqs 24-55 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC" frequency 998147093 Hz quality 1000
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80683490, 0) error 1
random: entropy device external interface
wlan: mac acl policy registered
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8065c570, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8065c620, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8065c6d0, 0) error 1
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80683330, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff806833e0, 0) error 1
kbd0 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff81158310, 0) error 19
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
padlock0: No ACE support.
acpi0: <CORE COREBOOT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x818-0x81b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.2 on pci0
pcib1: failed to allocate initial I/O port window: 0x1000-0x1fff
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xfe500000-0xfe51ffff,0xfe520000-0xfe523fff at device 0.0 on pci1
igb0: Using MSIX interrupts with 3 vectors
igb0: Ethernet address: 00:0d:b9:4a:42:48
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
pcib2: <ACPI PCI-PCI bridge> at device 2.3 on pci0
pci2: <ACPI PCI bus> on pcib2
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x2000-0x201f mem 0xfe600000-0xfe61ffff,0xfe620000-0xfe623fff at device 0.0 on pci2
igb1: Using MSIX interrupts with 3 vectors
igb1: Ethernet address: 00:0d:b9:4a:42:49
igb1: Bound queue 0 to cpu 2
igb1: Bound queue 1 to cpu 3
igb1: netmap queues/slots: TX 2/1024, RX 2/1024
pcib3: <ACPI PCI-PCI bridge> at device 2.4 on pci0
pci3: <ACPI PCI bus> on pcib3
igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x3000-0x301f mem 0xfe700000-0xfe71ffff,0xfe720000-0xfe723fff at device 0.0 on pci3
igb2: Using MSIX interrupts with 3 vectors
igb2: Ethernet address: 00:0d:b9:4a:42:4a
igb2: Bound queue 0 to cpu 0
igb2: Bound queue 1 to cpu 1
igb2: netmap queues/slots: TX 2/1024, RX 2/1024
pcib4: <ACPI PCI-PCI bridge> at device 2.5 on pci0
pci4: <ACPI PCI bus> on pcib4
ath0: <Atheros 9280> mem 0xfe800000-0xfe80ffff at device 0.0 on pci4
[ath] enabling AN_TOP2_FIXUP
ath0: [HT] enabling HT modes
ath0: [HT] 1 stream STBC receive enabled
ath0: [HT] 1 stream STBC transmit enabled
ath0: [HT] 2 RX streams; 2 TX streams
ath0: AR9280 mac 128.2 RF5133 phy 13.0
ath0: 2GHz radio: 0x0000; 5GHz radio: 0x00c0
pci0: <encrypt/decrypt> at device 8.0 (no driver attached)
xhci0: <AMD FCH USB 3.0 controller> mem 0xfeb22000-0xfeb23fff at device 16.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Unable to map MSI-X table
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
ahci0: <AMD Hudson-2 AHCI SATA controller> port 0x4010-0x4017,0x4020-0x4023,0x4018-0x401f,0x4024-0x4027,0x4000-0x400f mem 0xfeb25000-0xfeb253ff at device 17.0 on pci0
ahci0: AHCI v1.30 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ehci0: <AMD FCH USB 2.0 controller> mem 0xfeb25400-0xfeb254ff at device 18.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
usbus1: 480Mbps High Speed USB v2.0
ehci1: <AMD FCH USB 2.0 controller> mem 0xfeb25500-0xfeb255ff at device 19.0 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci1
usbus2: 480Mbps High Speed USB v2.0
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
sdhci_pci0: <Generic SD HCI> mem 0xfeb25600-0xfeb256ff at device 20.7 on pci0
sdhci_pci0: 1 slot(s) allocated
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
orm0: <ISA Option ROM> at iomem 0xef000-0xeffff on isa0
ppc0: cannot reserve I/O port range
uart1: <16550 or compatible> at port 0x2f8 irq 3 on isa0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
nvme cam probe device init
ugen0.1: <0x1022 XHCI root HUB> at usbus0
ugen1.1: <AMD EHCI root HUB> at usbus1
ugen2.1: <AMD EHCI root HUB> at usbus2
uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub2: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
uhub0: 4 ports with 4 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
ugen2.2: <vendor 0x0438 product 0x7900> at usbus2
uhub3 on uhub2
uhub3: <vendor 0x0438 product 0x7900, class 9/0, rev 2.00/0.18, addr 2> on usbus2
ugen1.2: <vendor 0x0438 product 0x7900> at usbus1
uhub4 on uhub1
uhub4: <vendor 0x0438 product 0x7900, class 9/0, rev 2.00/0.18, addr 2> on usbus1
uhub4: 4 ports with 4 removable, self powered
uhub3: 4 ports with 4 removable, self powered
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <SATA SSD SBFM01.1> ACS-4 ATA SATA 3.x device
ada0: Serial Number FF1A077B02B601912944
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 15272MB (31277232 512 byte sectors)
Trying to mount root from ufs:/dev/gptid/c7e1e0b7-fee3-11e6-ba90-000db94a4248 [rw]...
WARNING: / was not properly dismounted
Configuring crash dumps...
Using /dev/gptid/c7e4f4f5-fee3-11e6-ba90-000db94a4248 for dump device.
** SU+J Recovering /dev/gptid/c7e1e0b7-fee3-11e6-ba90-000db94a4248
** Reading 33554432 byte journal from inode 4.
** Building recovery table.
** Resolving unreferenced inode list.
** Processing journal entries.
** 48 journal records in 3072 bytes for 50.00% utilization
** Freed 0 inodes (0 dirs) 4 blocks, and 3 frags.
***** FILE SYSTEM MARKED CLEAN *****
Filesystems are clean, continuing...
Mounting filesystems...
__
_ __ / _|___ ___ _ __ ___ ___
| '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \
| |_) | _\__ \ __/ | | \__ \ __/
| .__/|_| |___/\___|_| |_|___/\___|
|_|
Welcome to pfSense 2.4.0-RELEASE...
savecore: reboot after panic: ffs_write: type 0xfffff80007bdb3b0 0 (0,8192)
savecore: writing core to /var/crash/textdump.tar.0
...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.24/mach/CORE
32-bit compatibility ldconfig path:
done.
panic: ffs_write: type 0xfffff800078af938 0 (0,8192)
cpuid = 3
KDB: enter: panic
[ thread pid 308 tid 100106 ]
Stopped at kdb_enter+0x3b: movq $0,kdb_why
db:0:kdb.enter.default> textdump set
textdump set
db:0:kdb.enter.default> capture on
db:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks> show alllocks
No such command
db:1:alllocks> show lockedvnods
Locked vnodes
vnode 0xfffff800078af938: tag ufs, type VNON
usecount 1, writecount 1, refcount 1 mountedhere 0
flags (VI_ACTIVE)
lock type ufs: EXCL by thread 0xfffff8000786a560 (pid 308, php-fpm, tid 100106)
ino 1694953, on dev gptid/c7e1e0b7-fee3-11e6-ba90-000db94a4248
db:0:kdb.enter.default> show pcpu
cpuid = 3
dynamic pcpu = 0xfffffe0172b09f80
curthread = 0xfffff8000786a560: pid 308 "php-fpm"
curpcb = 0xfffffe01205ffb80
fpcurthread = 0xfffff8000786a560: pid 308 "php-fpm"
idlethread = 0xfffff80003975560: tid 100006 "idle: cpu3"
curpmap = 0xfffff8000788f138
tssp = 0xffffffff82a73c48
commontssp = 0xffffffff82a73c48
rsp0 = 0xfffffe01205ffb80
gs32p = 0xffffffff82a7a4a0
ldt = 0xffffffff82a7a4e0
tss = 0xffffffff82a7a4d0
db:0:kdb.enter.default> bt
Tracing pid 308 tid 100106 td 0xfffff8000786a560
kdb_enter() at kdb_enter+0x3b/frame 0xfffffe01205ff4b0
vpanic() at vpanic+0x1a3/frame 0xfffffe01205ff530
panic() at panic+0x43/frame 0xfffffe01205ff590
ffs_write() at ffs_write+0x57c/frame 0xfffffe01205ff640
VOP_WRITE_APV() at VOP_WRITE_APV+0x111/frame 0xfffffe01205ff750
vn_write() at vn_write+0x230/frame 0xfffffe01205ff7d0
vn_io_fault() at vn_io_fault+0x112/frame 0xfffffe01205ff850
dofilewrite() at dofilewrite+0xc8/frame 0xfffffe01205ff8a0
sys_write() at sys_write+0xdb/frame 0xfffffe01205ff920
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe01205ffab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01205ffab0
--- syscall (4, FreeBSD ELF64, sys_write), rip = 0x801df729a, rsp = 0x7fffffffa278, rbp = 0x7fffffffa2b0 ---
db:0:kdb.enter.default> ps
pid ppid pgrp uid state wmesg wchan cmd
309 25 25 0 S+ sbwait 0xfffff80007a664a4 fcgicli
308 307 307 0 R CPU 3 php-fpm
307 1 307 0 Ss kqread 0xfffff800079fdb00 php-fpm
56 0 0 0 DL mdwait 0xfffff800076da000 [md0]
25 1 25 0 Ss+ pause 0xfffff80003973630 sh
24 0 0 0 DL syncer 0xffffffff829aed80 [syncer]
23 0 0 0 DL vlruwt 0xfffff80007017588 [vnlru]
22 0 0 0 DL (threaded) [bufdaemon]
100095 D psleep 0xffffffff829ad584 [bufdaemon]
100104 D sdflush 0xfffff800078668e8 [/ worker]
21 0 0 0 DL - 0xffffffff829ae23c [bufspacedaemon]
20 0 0 0 DL pgzero 0xffffffff829c29e4 [pagezero]
19 0 0 0 DL psleep 0xffffffff829bee94 [vmdaemon]
18 0 0 0 DL (threaded) [pagedaemon]
100091 D psleep 0xffffffff82a72f05 [pagedaemon]
100098 D launds 0xffffffff829bee44 [laundry: dom0]
100099 D umarcl 0xffffffff829be7b8 [uma]
17 0 0 0 DL - 0xffffffff82789680 [rand_harvestq]
16 0 0 0 DL pftm 0xffffffff80e93180 [pf purge]
9 0 0 0 DL waiting_ 0xffffffff82a61cf0 [sctp_iterator]
8 0 0 0 DL - 0xffffffff829acd94 [soaiod4]
7 0 0 0 DL - 0xffffffff829acd94 [soaiod3]
6 0 0 0 DL - 0xffffffff829acd94 [soaiod2]
5 0 0 0 DL - 0xffffffff829acd94 [soaiod1]
15 0 0 0 DL (threaded) [usb]
100057 D - 0xfffffe0001235460 [usbus0]
100058 D - 0xfffffe00012354b8 [usbus0]
100059 D - 0xfffffe0001235510 [usbus0]
100060 D - 0xfffffe0001235568 [usbus0]
100061 D - 0xfffffe00012355c0 [usbus0]
100064 D - 0xfffffe0001033d10 [usbus1]
100065 D - 0xfffffe0001033d68 [usbus1]
100066 D - 0xfffffe0001033dc0 [usbus1]
100067 D - 0xfffffe0001033e18 [usbus1]
100068 D - 0xfffffe0001033e70 [usbus1]
100069 D - 0xfffffe0001241d10 [usbus2]
100070 D - 0xfffffe0001241d68 [usbus2]
100071 D - 0xfffffe0001241dc0 [usbus2]
100072 D - 0xfffffe0001241e18 [usbus2]
100073 D - 0xfffffe0001241e70 [usbus2]
4 0 0 0 DL (threaded) [cam]
100037 D - 0xffffffff8265c400 [doneq0]
100090 D - 0xffffffff8265c248 [scanner]
3 0 0 0 DL crypto_r 0xffffffff829bd370 [crypto returns]
2 0 0 0 DL crypto_w 0xffffffff829bd218 [crypto]
14 0 0 0 DL (threaded) [geom]
100031 D - 0xffffffff82a39da0 [g_event]
100032 D - 0xffffffff82a39da8 [g_up]
100033 D - 0xffffffff82a39db0 [g_down]
13 0 0 0 DL (threaded) [ng_queue]
100027 D sleep 0xffffffff82615c50 [ng_queue0]
100028 D sleep 0xffffffff82615c50 [ng_queue1]
100029 D sleep 0xffffffff82615c50 [ng_queue2]
100030 D sleep 0xffffffff82615c50 [ng_queue3]
12 0 0 0 WL (threaded) [intr]
100007 I [swi1: netisr 0]
100008 I [swi1: netisr 1]
100009 I [swi1: netisr 2]
100010 I [swi1: netisr 3]
100011 I [swi3: vm]
100012 I [swi4: clock (0)]
100013 I [swi4: clock (1)]
100014 I [swi4: clock (2)]
100015 I [swi4: clock (3)]
100022 I [swi6: task queue]
100023 I [swi6: Giant taskq]
100026 I [swi5: fast taskq]
100038 I [irq256: igb0:que 0]
100040 I [irq257: igb0:que 1]
100042 I [irq258: igb0:link]
100043 I [irq259: igb1:que 0]
100045 I [irq260: igb1:que 1]
100047 I [irq261: igb1:link]
100048 I [irq262: igb2:que 0]
100050 I [irq263: igb2:que 1]
100052 I [irq264: igb2:link]
100053 I [irq40: ath0]
100056 I [irq265: xhci0]
100062 I [irq19: ahci0]
100063 I [irq18: ehci0 ehci1]
100074 I [irq16: sdhci_pci0]
100075 I [swi0: uart uart]
100083 I [swi1: pf send]
100084 I [swi1: pfsync]
11 0 0 0 RL (threaded) [idle]
100003 Run CPU 0 [idle: cpu0]
100004 Run CPU 1 [idle: cpu1]
100005 Run CPU 2 [idle: cpu2]
100006 CanRun [idle: cpu3]
1 0 1 0 SLs wait 0xfffff80003972588 [init]
10 0 0 0 DL audit_wo 0xffffffff82a68ec0 [audit]
0 0 0 0 DLs (threaded) [kernel]
100000 D swapin 0xffffffff82a39de8 [swapper]
100016 D - 0xfffff8000398a900 [if_config_tqg_0]
100017 D - 0xfffff8000398a700 [if_io_tqg_0]
100018 D - 0xfffff8000398a500 [if_io_tqg_1]
100019 D - 0xfffff8000398a300 [if_io_tqg_2]
100020 D - 0xfffff8000398a100 [if_io_tqg_3]
100021 D - 0xfffff80003989e00 [kqueue_ctx taskq]
100024 D - 0xfffff80003989600 [aiod_kick taskq]
100025 D - 0xfffff80003989400 [thread taskq]
100034 D - 0xfffff80003964100 [firmware taskq]
100039 D - 0xfffff80003c57500 [igb0 que (qid 0)]
100041 D - 0xfffff80003be9b00 [igb0 que (qid 1)]
100044 D - 0xfffff80003be8500 [igb1 que (qid 2)]
100046 D - 0xfffff80003be8200 [igb1 que (qid 3)]
100049 D - 0xfffff80003c08800 [igb2 que (qid 0)]
100051 D - 0xfffff800039dcd00 [igb2 que (qid 1)]
100054 D - 0xfffff800039de000 [ath0 taskq]
100055 D - 0xfffff80003964200 [ath0 net80211 taskq]
100076 D - 0xfffff80007079400 [mca taskq]
100085 D - 0xfffff800076d5600 [acpi_task_0]
100086 D - 0xfffff800076d5600 [acpi_task_1]
100087 D - 0xfffff800076d5600 [acpi_task_2]
100089 D - 0xfffff800039dcb00 [CAM taskq]
db:0:kdb.enter.default> alltrace
Tracing command fcgicli pid 309 tid 100102 td 0xfffff8000786b560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205eb5c0
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205eb5f0
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe01205eb660
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe01205eb690
_sleep() at _sleep+0x248/frame 0xfffffe01205eb710
sbwait() at sbwait+0x54/frame 0xfffffe01205eb730
soreceive_generic() at soreceive_generic+0x3bb/frame 0xfffffe01205eb7f0
kern_recvit() at kern_recvit+0x1dc/frame 0xfffffe01205eb8b0
sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe01205eb920
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe01205ebab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01205ebab0
--- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b1966a, rsp = 0x7fffffffe588, rbp = 0x7fffffffe5d0 ---
Tracing command php-fpm pid 308 tid 100106 td 0xfffff8000786a560
kdb_enter() at kdb_enter+0x3b/frame 0xfffffe01205ff4b0
vpanic() at vpanic+0x1a3/frame 0xfffffe01205ff530
panic() at panic+0x43/frame 0xfffffe01205ff590
ffs_write() at ffs_write+0x57c/frame 0xfffffe01205ff640
VOP_WRITE_APV() at VOP_WRITE_APV+0x111/frame 0xfffffe01205ff750
vn_write() at vn_write+0x230/frame 0xfffffe01205ff7d0
vn_io_fault() at vn_io_fault+0x112/frame 0xfffffe01205ff850
dofilewrite() at dofilewrite+0xc8/frame 0xfffffe01205ff8a0
sys_write() at sys_write+0xdb/frame 0xfffffe01205ff920
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe01205ffab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01205ffab0
--- syscall (4, FreeBSD ELF64, sys_write), rip = 0x801df729a, rsp = 0x7fffffffa278, rbp = 0x7fffffffa2b0 ---
Tracing command php-fpm pid 307 tid 100120 td 0xfffff80007862000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01206454c0
mi_switch() at mi_switch+0xe5/frame 0xfffffe01206454f0
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0120645560
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe01206455a0
_sleep() at _sleep+0x223/frame 0xfffffe0120645620
kqueue_kevent() at kqueue_kevent+0x294/frame 0xfffffe0120645800
kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0120645870
sys_kevent() at sys_kevent+0x138/frame 0xfffffe0120645920
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0120645ab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0120645ab0
--- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801d8ffba, rsp = 0x7fffffffc878, rbp = 0x7fffffffc8c0 ---
Tracing command md0 pid 56 tid 100108 td 0xfffff80007863000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120609940
mi_switch() at mi_switch+0xe5/frame 0xfffffe0120609970
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01206099a0
_sleep() at _sleep+0x255/frame 0xfffffe0120609a20
md_kthread() at md_kthread+0xd5/frame 0xfffffe0120609a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120609ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120609ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command sh pid 25 tid 100100 td 0xfffff80007753560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205e1720
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205e1750
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe01205e17c0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe01205e17f0
_sleep() at _sleep+0x248/frame 0xfffffe01205e1870
kern_sigsuspend() at kern_sigsuspend+0x1b4/frame 0xfffffe01205e18f0
sys_sigsuspend() at sys_sigsuspend+0x31/frame 0xfffffe01205e1920
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe01205e1ab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01205e1ab0
--- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800b62e5a, rsp = 0x7fffffffe5f8, rbp = 0x7fffffffe6b0 ---
Tracing command syncer pid 24 tid 100097 td 0xfffff80007754000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205d2910
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205d2940
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe01205d2980
_cv_timedwait_sbt() at _cv_timedwait_sbt+0x1a3/frame 0xfffffe01205d29e0
sched_sync() at sched_sync+0x1a9/frame 0xfffffe01205d2a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01205d2ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01205d2ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command vnlru pid 23 tid 100096 td 0xfffff80007754560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205cd8e0
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205cd910
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe01205cd950
_sleep() at _sleep+0x235/frame 0xfffffe01205cd9d0
vnlru_proc() at vnlru_proc+0x214/frame 0xfffffe01205cda70
fork_exit() at fork_exit+0x85/frame 0xfffffe01205cdab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01205cdab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command bufdaemon pid 22 tid 100095 td 0xfffff80007755000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205c8940
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205c8970
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe01205c89b0
_sleep() at _sleep+0x235/frame 0xfffffe01205c8a30
buf_daemon() at buf_daemon+0x88/frame 0xfffffe01205c8a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01205c8ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01205c8ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command bufdaemon pid 22 tid 100104 td 0xfffff800076d0560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205f5940
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205f5970
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe01205f59b0
_sleep() at _sleep+0x235/frame 0xfffffe01205f5a30
softdep_flush() at softdep_flush+0x1be/frame 0xfffffe01205f5a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01205f5ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01205f5ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command bufspacedaemon pid 21 tid 100094 td 0xfffff80007755560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205c3950
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205c3980
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe01205c39c0
_sleep() at _sleep+0x235/frame 0xfffffe01205c3a40
bufspace_daemon() at bufspace_daemon+0x65/frame 0xfffffe01205c3a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01205c3ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01205c3ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command pagezero pid 20 tid 100093 td 0xfffff80007756000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205be950
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205be980
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe01205be9c0
_sleep() at _sleep+0x235/frame 0xfffffe01205bea40
vm_pagezero() at vm_pagezero+0xb5/frame 0xfffffe01205bea70
fork_exit() at fork_exit+0x85/frame 0xfffffe01205beab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01205beab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command vmdaemon pid 19 tid 100092 td 0xfffff80007756560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205b9910
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205b9940
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01205b9970
_sleep() at _sleep+0x255/frame 0xfffffe01205b99f0
vm_daemon() at vm_daemon+0x9c/frame 0xfffffe01205b9a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01205b9ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01205b9ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command pagedaemon pid 18 tid 100091 td 0xfffff80007022560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205b4910
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205b4940
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe01205b4980
_sleep() at _sleep+0x235/frame 0xfffffe01205b4a00
vm_pageout() at vm_pageout+0x36a/frame 0xfffffe01205b4a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01205b4ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01205b4ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command pagedaemon pid 18 tid 100098 td 0xfffff800076cc560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205d78b0
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205d78e0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01205d7910
_sleep() at _sleep+0x255/frame 0xfffffe01205d7990
vm_pageout_laundry_worker() at vm_pageout_laundry_worker+0x1ac/frame 0xfffffe01205d7a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01205d7ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01205d7ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command pagedaemon pid 18 tid 100099 td 0xfffff800076cc000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205dc950
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205dc980
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01205dc9b0
_sleep() at _sleep+0x255/frame 0xfffffe01205dca30
uma_reclaim_worker() at uma_reclaim_worker+0x136/frame 0xfffffe01205dca70
fork_exit() at fork_exit+0x85/frame 0xfffffe01205dcab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01205dcab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command rand_harvestq pid 17 tid 100088 td 0xfffff800076ce000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012056d920
mi_switch() at mi_switch+0xe5/frame 0xfffffe012056d950
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe012056d990
_sleep() at _sleep+0x235/frame 0xfffffe012056da10
random_kthread() at random_kthread+0x264/frame 0xfffffe012056da70
fork_exit() at fork_exit+0x85/frame 0xfffffe012056dab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012056dab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command pf purge pid 16 tid 100082 td 0xfffff80003c5c560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012054f950
mi_switch() at mi_switch+0xe5/frame 0xfffffe012054f980
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe012054f9c0
_sleep() at _sleep+0x235/frame 0xfffffe012054fa40
pf_purge_thread() at pf_purge_thread+0xdd/frame 0xfffffe012054fa70
fork_exit() at fork_exit+0x85/frame 0xfffffe012054fab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012054fab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command sctp_iterator pid 9 tid 100081 td 0xfffff80003c68000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012054a970
mi_switch() at mi_switch+0xe5/frame 0xfffffe012054a9a0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe012054a9d0
_sleep() at _sleep+0x255/frame 0xfffffe012054aa50
sctp_iterator_thread() at sctp_iterator_thread+0x69/frame 0xfffffe012054aa70
fork_exit() at fork_exit+0x85/frame 0xfffffe012054aab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012054aab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command soaiod4 pid 8 tid 100080 td 0xfffff80003c68560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120544930
mi_switch() at mi_switch+0xe5/frame 0xfffffe0120544960
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe01205449a0
_sleep() at _sleep+0x235/frame 0xfffffe0120544a20
soaio_kproc_loop() at soaio_kproc_loop+0x1f3/frame 0xfffffe0120544a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120544ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120544ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command soaiod3 pid 7 tid 100079 td 0xfffff80003c69000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012053f930
mi_switch() at mi_switch+0xe5/frame 0xfffffe012053f960
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe012053f9a0
_sleep() at _sleep+0x235/frame 0xfffffe012053fa20
soaio_kproc_loop() at soaio_kproc_loop+0x1f3/frame 0xfffffe012053fa70
fork_exit() at fork_exit+0x85/frame 0xfffffe012053fab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012053fab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command soaiod2 pid 6 tid 100078 td 0xfffff80003c69560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012053a930
mi_switch() at mi_switch+0xe5/frame 0xfffffe012053a960
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe012053a9a0
_sleep() at _sleep+0x235/frame 0xfffffe012053aa20
soaio_kproc_loop() at soaio_kproc_loop+0x1f3/frame 0xfffffe012053aa70
fork_exit() at fork_exit+0x85/frame 0xfffffe012053aab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012053aab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command soaiod1 pid 5 tid 100077 td 0xfffff80003c6a000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120535930
mi_switch() at mi_switch+0xe5/frame 0xfffffe0120535960
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xfffffe01205359a0
_sleep() at _sleep+0x235/frame 0xfffffe0120535a20
soaio_kproc_loop() at soaio_kproc_loop+0x1f3/frame 0xfffffe0120535a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120535ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120535ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100057 td 0xfffff80003c99560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012019e990
mi_switch() at mi_switch+0xe5/frame 0xfffffe012019e9c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe012019e9f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe012019ea30
usb_process() at usb_process+0x137/frame 0xfffffe012019ea70
fork_exit() at fork_exit+0x85/frame 0xfffffe012019eab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012019eab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100058 td 0xfffff80003c99000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01201a3990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01201a39c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01201a39f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe01201a3a30
usb_process() at usb_process+0x137/frame 0xfffffe01201a3a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01201a3ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01201a3ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100059 td 0xfffff80003c6b560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01201a8990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01201a89c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01201a89f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe01201a8a30
usb_process() at usb_process+0x137/frame 0xfffffe01201a8a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01201a8ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01201a8ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100060 td 0xfffff80007022000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01201ad990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01201ad9c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01201ad9f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe01201ada30
usb_process() at usb_process+0x137/frame 0xfffffe01201ada70
fork_exit() at fork_exit+0x85/frame 0xfffffe01201adab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01201adab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100061 td 0xfffff80007021560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01201b2990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01201b29c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01201b29f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe01201b2a30
usb_process() at usb_process+0x137/frame 0xfffffe01201b2a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01201b2ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01201b2ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100064 td 0xfffff80007020000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012036c990
mi_switch() at mi_switch+0xe5/frame 0xfffffe012036c9c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe012036c9f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe012036ca30
usb_process() at usb_process+0x137/frame 0xfffffe012036ca70
fork_exit() at fork_exit+0x85/frame 0xfffffe012036cab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012036cab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100065 td 0xfffff8000701f560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120371990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01203719c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01203719f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe0120371a30
usb_process() at usb_process+0x137/frame 0xfffffe0120371a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120371ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120371ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100066 td 0xfffff8000701f000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120376990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01203769c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01203769f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe0120376a30
usb_process() at usb_process+0x137/frame 0xfffffe0120376a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120376ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120376ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100067 td 0xfffff8000701e560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012037b990
mi_switch() at mi_switch+0xe5/frame 0xfffffe012037b9c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe012037b9f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe012037ba30
usb_process() at usb_process+0x137/frame 0xfffffe012037ba70
fork_exit() at fork_exit+0x85/frame 0xfffffe012037bab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012037bab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100068 td 0xfffff8000701e000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120380990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01203809c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01203809f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe0120380a30
usb_process() at usb_process+0x137/frame 0xfffffe0120380a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120380ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120380ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100069 td 0xfffff80007057560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120508990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205089c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01205089f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe0120508a30
usb_process() at usb_process+0x137/frame 0xfffffe0120508a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120508ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120508ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100070 td 0xfffff80007057000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012050d990
mi_switch() at mi_switch+0xe5/frame 0xfffffe012050d9c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe012050d9f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe012050da30
usb_process() at usb_process+0x137/frame 0xfffffe012050da70
fork_exit() at fork_exit+0x85/frame 0xfffffe012050dab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012050dab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100071 td 0xfffff80007056560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120512990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205129c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01205129f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe0120512a30
usb_process() at usb_process+0x137/frame 0xfffffe0120512a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120512ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120512ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100072 td 0xfffff80007056000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120517990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205179c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01205179f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe0120517a30
usb_process() at usb_process+0x137/frame 0xfffffe0120517a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120517ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120517ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command usb pid 15 tid 100073 td 0xfffff80007055560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012051c990
mi_switch() at mi_switch+0xe5/frame 0xfffffe012051c9c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe012051c9f0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe012051ca30
usb_process() at usb_process+0x137/frame 0xfffffe012051ca70
fork_exit() at fork_exit+0x85/frame 0xfffffe012051cab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012051cab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command cam pid 4 tid 100037 td 0xfffff80003b9c000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f606b940
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f606b970
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f606b9a0
_sleep() at _sleep+0x255/frame 0xfffffe00f606ba20
xpt_done_td() at xpt_done_td+0xfe/frame 0xfffffe00f606ba70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f606bab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f606bab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command cam pid 4 tid 100090 td 0xfffff800076cd000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120577930
mi_switch() at mi_switch+0xe5/frame 0xfffffe0120577960
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0120577990
_sleep() at _sleep+0x255/frame 0xfffffe0120577a10
xpt_scanner_thread() at xpt_scanner_thread+0x237/frame 0xfffffe0120577a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120577ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120577ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command crypto returns pid 3 tid 100036 td 0xfffff800039aa560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f6066940
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f6066970
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f60669a0
_sleep() at _sleep+0x255/frame 0xfffffe00f6066a20
crypto_ret_proc() at crypto_ret_proc+0x1c8/frame 0xfffffe00f6066a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f6066ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f6066ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command crypto pid 2 tid 100035 td 0xfffff800039c2000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f6061940
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f6061970
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f60619a0
_sleep() at _sleep+0x255/frame 0xfffffe00f6061a20
crypto_proc() at crypto_proc+0x9d/frame 0xfffffe00f6061a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f6061ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f6061ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command geom pid 14 tid 100031 td 0xfffff800039c8000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f604d940
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f604d970
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f604d9a0
_sleep() at _sleep+0x255/frame 0xfffffe00f604da20
g_run_events() at g_run_events+0x1a0/frame 0xfffffe00f604da70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f604dab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f604dab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command geom pid 14 tid 100032 td 0xfffff800039c7560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f6052940
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f6052970
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f60529a0
_sleep() at _sleep+0x255/frame 0xfffffe00f6052a20
g_io_schedule_up() at g_io_schedule_up+0x1be/frame 0xfffffe00f6052a60
g_up_procbody() at g_up_procbody+0x7d/frame 0xfffffe00f6052a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f6052ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f6052ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command geom pid 14 tid 100033 td 0xfffff800039c7000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f6057940
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f6057970
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f60579a0
_sleep() at _sleep+0x255/frame 0xfffffe00f6057a20
g_io_schedule_down() at g_io_schedule_down+0x19c/frame 0xfffffe00f6057a60
g_down_procbody() at g_down_procbody+0x7d/frame 0xfffffe00f6057a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f6057ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f6057ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command ng_queue pid 13 tid 100027 td 0xfffff800039a6000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51ed950
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51ed980
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51ed9b0
_sleep() at _sleep+0x255/frame 0xfffffe00f51eda30
ngthread() at ngthread+0x8c/frame 0xfffffe00f51eda70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51edab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51edab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command ng_queue pid 13 tid 100028 td 0xfffff800039c9560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51f2950
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51f2980
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51f29b0
_sleep() at _sleep+0x255/frame 0xfffffe00f51f2a30
ngthread() at ngthread+0x8c/frame 0xfffffe00f51f2a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51f2ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51f2ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command ng_queue pid 13 tid 100029 td 0xfffff800039c9000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51f7950
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51f7980
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51f79b0
_sleep() at _sleep+0x255/frame 0xfffffe00f51f7a30
ngthread() at ngthread+0x8c/frame 0xfffffe00f51f7a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51f7ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51f7ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command ng_queue pid 13 tid 100030 td 0xfffff800039c8560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51fc950
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51fc980
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51fc9b0
_sleep() at _sleep+0x255/frame 0xfffffe00f51fca30
ngthread() at ngthread+0x8c/frame 0xfffffe00f51fca70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51fcab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51fcab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command intr pid 12 tid 100007 td 0xfffff80003975000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100008 td 0xfffff80003974560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100009 td 0xfffff80003974000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100010 td 0xfffff80003990560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100011 td 0xfffff80003990000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100012 td 0xfffff8000398f560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51a29f0
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51a2a20
ithread_loop() at ithread_loop+0x1d2/frame 0xfffffe00f51a2a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51a2ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51a2ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command intr pid 12 tid 100013 td 0xfffff8000398f000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100014 td 0xfffff8000398e560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100015 td 0xfffff8000398e000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100022 td 0xfffff800039a8560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100023 td 0xfffff800039a8000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51d99f0
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51d9a20
ithread_loop() at ithread_loop+0x1d2/frame 0xfffffe00f51d9a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51d9ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51d9ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command intr pid 12 tid 100026 td 0xfffff800039a6560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100038 td 0xfffff80003c49560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100040 td 0xfffff80003c6a560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100042 td 0xfffff80003b9b000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100043 td 0xfffff80003b96560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100045 td 0xfffff80003b95560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100047 td 0xfffff80003b94560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100048 td 0xfffff80003b94000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100050 td 0xfffff80003c48560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100052 td 0xfffff80003ca3000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100053 td 0xfffff80003ca2560
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100056 td 0xfffff80003ca1000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100062 td 0xfffff80007021000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01201b79f0
mi_switch() at mi_switch+0xe5/frame 0xfffffe01201b7a20
ithread_loop() at ithread_loop+0x1d2/frame 0xfffffe01201b7a70
fork_exit() at fork_exit+0x85/frame 0xfffffe01201b7ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01201b7ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command intr pid 12 tid 100063 td 0xfffff80007020560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01203679f0
mi_switch() at mi_switch+0xe5/frame 0xfffffe0120367a20
ithread_loop() at ithread_loop+0x1d2/frame 0xfffffe0120367a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120367ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120367ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command intr pid 12 tid 100074 td 0xfffff80007055000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100075 td 0xfffff80007054560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe01205279f0
mi_switch() at mi_switch+0xe5/frame 0xfffffe0120527a20
ithread_loop() at ithread_loop+0x1d2/frame 0xfffffe0120527a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120527ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120527ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command intr pid 12 tid 100083 td 0xfffff80003c5c000
fork_trampoline() at fork_trampoline
Tracing command intr pid 12 tid 100084 td 0xfffff800076d0000
fork_trampoline() at fork_trampoline
Tracing command idle pid 11 tid 100003 td 0xfffff80003977000
cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff829c6c60
ipi_nmi_handler() at ipi_nmi_handler+0x45/frame 0xffffffff829c6c80
trap() at trap+0x3a/frame 0xffffffff829c6e40
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff829c6e40
--- trap 0x13, rip = 0xffffffff81233f66, rsp = 0xfffffe00f5175910, rbp = 0xfffffe00f5175910 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00f5175910
acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe00f5175960
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe00f5175980
cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00f51759a0
sched_idletd() at sched_idletd+0xca/frame 0xfffffe00f5175a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f5175ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f5175ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100004 td 0xfffff80003976560
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe0119ffdd40
ipi_nmi_handler() at ipi_nmi_handler+0x45/frame 0xfffffe0119ffdd60
trap() at trap+0x3a/frame 0xfffffe0119ffdf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe0119ffdf20
--- trap 0x13, rip = 0xffffffff81233f66, rsp = 0xfffffe00f517a910, rbp = 0xfffffe00f517a910 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00f517a910
acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe00f517a960
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe00f517a980
cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00f517a9a0
sched_idletd() at sched_idletd+0xca/frame 0xfffffe00f517aa70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f517aab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f517aab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100005 td 0xfffff80003976000
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00f515bd40
ipi_nmi_handler() at ipi_nmi_handler+0x45/frame 0xfffffe00f515bd60
trap() at trap+0x3a/frame 0xfffffe00f515bf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00f515bf20
--- trap 0x13, rip = 0xffffffff81233f66, rsp = 0xfffffe00f517f910, rbp = 0xfffffe00f517f910 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00f517f910
acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe00f517f960
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe00f517f980
cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00f517f9a0
sched_idletd() at sched_idletd+0xca/frame 0xfffffe00f517fa70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f517fab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f517fab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command idle pid 11 tid 100006 td 0xfffff80003975560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f5184950
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f5184980
critical_exit() at critical_exit+0x7a/frame 0xfffffe00f51849a0
sched_idletd() at sched_idletd+0xca/frame 0xfffffe00f5184a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f5184ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f5184ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command init pid 1 tid 100002 td 0xfffff80003977560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f5170540
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f5170570
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00f51705e0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00f5170610
_sleep() at _sleep+0x248/frame 0xfffffe00f5170690
kern_wait6() at kern_wait6+0x267/frame 0xfffffe00f5170730
sys_wait4() at sys_wait4+0x78/frame 0xfffffe00f5170920
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00f5170ab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00f5170ab0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x421c2a, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe870 ---
Tracing command audit pid 10 tid 100001 td 0xfffff80003978000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f516b960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f516b990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f516b9c0
_cv_wait() at _cv_wait+0x169/frame 0xfffffe00f516ba00
audit_worker() at audit_worker+0x83/frame 0xfffffe00f516ba70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f516bab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f516bab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100000 td 0xffffffff82a3a380
sched_switch() at sched_switch+0x4aa/frame 0xffffffff82e45a30
mi_switch() at mi_switch+0xe5/frame 0xffffffff82e45a60
sleepq_timedwait() at sleepq_timedwait+0x42/frame 0xffffffff82e45aa0
_sleep() at _sleep+0x235/frame 0xffffffff82e45b20
swapper() at swapper+0xdf/frame 0xffffffff82e45b70
btext() at btext+0x2c
Tracing command kernel pid 0 tid 100016 td 0xfffff8000398d560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51b6990
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51b69c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51b69f0
msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe00f51b6a40
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf3/frame 0xfffffe00f51b6a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51b6ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51b6ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100017 td 0xfffff8000398d000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51bb990
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51bb9c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51bb9f0
msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe00f51bba40
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf3/frame 0xfffffe00f51bba70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51bbab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51bbab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100018 td 0xfffff80003978560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51c0990
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51c09c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51c09f0
msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe00f51c0a40
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf3/frame 0xfffffe00f51c0a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51c0ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51c0ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100019 td 0xfffff800039aa000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51c5990
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51c59c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51c59f0
msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe00f51c5a40
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf3/frame 0xfffffe00f51c5a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51c5ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51c5ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100020 td 0xfffff800039a9560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51ca990
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51ca9c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51ca9f0
msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe00f51caa40
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf3/frame 0xfffffe00f51caa70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51caab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51caab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100021 td 0xfffff800039a9000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51cf960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51cf990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51cf9c0
_sleep() at _sleep+0x255/frame 0xfffffe00f51cfa40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe00f51cfa70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51cfab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51cfab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100024 td 0xfffff800039a7560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51de960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51de990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51de9c0
_sleep() at _sleep+0x255/frame 0xfffffe00f51dea40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe00f51dea70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51deab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51deab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100025 td 0xfffff800039a7000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f51e3960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f51e3990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f51e39c0
_sleep() at _sleep+0x255/frame 0xfffffe00f51e3a40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe00f51e3a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f51e3ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f51e3ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100034 td 0xfffff800039c2560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f605c960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f605c990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f605c9c0
_sleep() at _sleep+0x255/frame 0xfffffe00f605ca40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe00f605ca70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f605cab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f605cab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100039 td 0xfffff80003c6b000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f6085960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f6085990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f60859c0
_sleep() at _sleep+0x255/frame 0xfffffe00f6085a40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe00f6085a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f6085ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f6085ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100041 td 0xfffff80003b9b560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f608f960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f608f990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f608f9c0
_sleep() at _sleep+0x255/frame 0xfffffe00f608fa40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe00f608fa70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f608fab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f608fab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100044 td 0xfffff80003b96000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f60ae960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f60ae990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f60ae9c0
_sleep() at _sleep+0x255/frame 0xfffffe00f60aea40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe00f60aea70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f60aeab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f60aeab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100046 td 0xfffff80003b95000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f60b8960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f60b8990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f60b89c0
_sleep() at _sleep+0x255/frame 0xfffffe00f60b8a40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe00f60b8a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f60b8ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f60b8ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100049 td 0xfffff80003c49000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f60d7960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f60d7990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f60d79c0
_sleep() at _sleep+0x255/frame 0xfffffe00f60d7a40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe00f60d7a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f60d7ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f60d7ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100051 td 0xfffff80003ca3560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00f60e1960
mi_switch() at mi_switch+0xe5/frame 0xfffffe00f60e1990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe00f60e19c0
_sleep() at _sleep+0x255/frame 0xfffffe00f60e1a40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe00f60e1a70
fork_exit() at fork_exit+0x85/frame 0xfffffe00f60e1ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f60e1ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100054 td 0xfffff80003ca2000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012018d960
mi_switch() at mi_switch+0xe5/frame 0xfffffe012018d990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe012018d9c0
_sleep() at _sleep+0x255/frame 0xfffffe012018da40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe012018da70
fork_exit() at fork_exit+0x85/frame 0xfffffe012018dab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012018dab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100055 td 0xfffff80003ca1560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120192960
mi_switch() at mi_switch+0xe5/frame 0xfffffe0120192990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01201929c0
_sleep() at _sleep+0x255/frame 0xfffffe0120192a40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe0120192a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120192ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120192ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100076 td 0xfffff80007054000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120530990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205309c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01205309f0
msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe0120530a40
taskqueue_thread_loop() at taskqueue_thread_loop+0xf3/frame 0xfffffe0120530a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120530ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120530ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100085 td 0xfffff800076cf560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe012055e990
mi_switch() at mi_switch+0xe5/frame 0xfffffe012055e9c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe012055e9f0
msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe012055ea40
taskqueue_thread_loop() at taskqueue_thread_loop+0xf3/frame 0xfffffe012055ea70
fork_exit() at fork_exit+0x85/frame 0xfffffe012055eab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012055eab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100086 td 0xfffff800076cf000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120563990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205639c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01205639f0
msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe0120563a40
taskqueue_thread_loop() at taskqueue_thread_loop+0xf3/frame 0xfffffe0120563a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120563ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120563ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100087 td 0xfffff800076ce560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120568990
mi_switch() at mi_switch+0xe5/frame 0xfffffe01205689c0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01205689f0
msleep_spin_sbt() at msleep_spin_sbt+0x18a/frame 0xfffffe0120568a40
taskqueue_thread_loop() at taskqueue_thread_loop+0xf3/frame 0xfffffe0120568a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120568ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120568ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Tracing command kernel pid 0 tid 100089 td 0xfffff800076cd560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0120572960
mi_switch() at mi_switch+0xe5/frame 0xfffffe0120572990
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe01205729c0
_sleep() at _sleep+0x255/frame 0xfffffe0120572a40
taskqueue_thread_loop() at taskqueue_thread_loop+0x121/frame 0xfffffe0120572a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0120572ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0120572ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:0:kdb.enter.default> capture off
db:0:kdb.enter.default> textdump dump
Textdump complete.
db:0:kdb.enter.default> reset
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 3
PCEngines apu3
coreboot build 20170302
4080 MB ECC DRAM
SeaBIOS (version rel-1.10.0.1)
Press F10 key now for boot menu
Here's the recording from the above boot-loop:
Below are few frequently asked questions:
Question: Should I upgrade BIOS on the router?
Answer: BIOS upgrade won't solve this problem, and won't make it less likely to happen.
Question: what can I do to prevent this from happening again?
Answer: the only option so far is to reinstall pfSense with ZFS instead of UFS, as described here pfSense installation instructions
Question: can you reinstall pfSense for me?
Answer: yes, we can reinstall if for you for free, but you will have to cover shipping both ways, or deliver the router to us in Stockholm or Täby.
Are you experiencing this or similar problem? Let us know