USB disconnects randomly

rosenrot
Posts: 33
Joined: Wed Apr 10, 2019 2:37 am
languages_spoken: english
ODROIDs: n2
Has thanked: 0
Been thanked: 1 time
Contact:

USB disconnects randomly

Unread post by rosenrot » Sun May 05, 2019 3:13 am

Dear all,

I do have a Sundtek dvb-s2 USB that seems to disconnect randomly. It happens while running Ubuntu and Coreelec. I was wondering if any USB issues like this are known.

I posted my issues at the Sundtek forum already. However, I got a hint in the coreelec forum that you guys might be aware of USB issues.

UPDATE: I just found this post:

viewtopic.php?f=181&t=34683

And my errors look quite the same.

Code: Select all

[16445.902985] usb 1-1.1: usbfs: USBDEVFS_CONTROL failed cmd mediasrv rqt 64 rq 187 len 1 ret -110
[16451.022508] usb 1-1.1: usbfs: USBDEVFS_CONTROL failed cmd mediasrv rqt 192 rq 187 len 1 ret -110
[16456.144299] usb 1-1.1: usbfs: USBDEVFS_CONTROL failed cmd mediasrv rqt 64 rq 187 len 1 ret -110
[16461.263251] usb 1-1.1: usbfs: USBDEVFS_CONTROL failed cmd mediasrv rqt 192 rq 187 len 1 ret -110
[16466.382001] usb 1-1.1: usbfs: USBDEVFS_CONTROL failed cmd mediasrv rqt 64 rq 187 len 2 ret -110
[16471.502248] usb 1-1.1: usbfs: USBDEVFS_CONTROL failed cmd mediasrv rqt 64 rq 187 len 2 ret -110
What does the return code -110 mean? It is the same I found in the other odroid topic. It is about power draw right?

Did someone measure the voltage of the USB supply circuit to see if it breaks down at some point?

I didn't look at the schematics yet but how comes that only a single device is disconnecting and not the entire USB port?

UPDATE2: A quick look in the schematics reveals that each USB port is powered and guarded individually by the GL3523-QFN76 chip.

Schematics: https://dn.odroid.com/S922X/ODROID-N2/S ... 190307.pdf

GL3523 datasheet: https://datasheet.lcsc.com/szlcsc/Genes ... 157360.pdf

The Micro-USB OTG doesn't have a protection circuit similar to the four other USB ports. Maybe it doesn't show this issue?

UPDATE3: It seems my USB HDD is also suffering from the USB issue.

Code: Select all

[35788.290316] sd 0:0:0:0: [sda] tag#11 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
[35788.290347] sd 0:0:0:0: [sda] tag#11 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
[35794.166491] sd 0:0:0:0: [sda] tag#16 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT
[35794.166520] sd 0:0:0:0: [sda] tag#16 CDB: Write(16) 8a 00 00 00 00 01 27 4a 2a 10 00 00 00 10 00 00
[35794.166608] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[35794.170724] xhci-hcd xhci-hcd.0.auto: @00000000cf612100 00000000 00000000 1b000000 05048001
[35794.180968] scsi host0: uas_eh_bus_reset_handler start
[35794.181131] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[35794.188949] xhci-hcd xhci-hcd.0.auto: @00000000cf612150 00000000 00000000 1b000000 05078001
[35794.198469] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[35794.207149] xhci-hcd xhci-hcd.0.auto: @00000000cf6121a0 00000000 00000000 1b000000 05078001
[35794.313795] usb 2-1.3: reset SuperSpeed USB device number 3 using xhci-hcd
[35794.336753] scsi host0: uas_eh_bus_reset_handler success
[35801.286168] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[35801.290270] xhci-hcd xhci-hcd.0.auto: @00000000cf612aa0 00000000 00000000 1b000000 05048001
[35815.185340] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[35815.189465] xhci-hcd xhci-hcd.0.auto: @00000000cf6122f0 00000000 00000000 1b000000 05048000
[35829.084322] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[35829.088488] xhci-hcd xhci-hcd.0.auto: @00000000cf612d40 00000000 00000000 1b000000 05048000
[35842.983365] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[35842.987508] xhci-hcd xhci-hcd.0.auto: @00000000cf6125a0 00000000 00000000 1b000000 05048001
[35856.882368] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[35856.886500] xhci-hcd xhci-hcd.0.auto: @00000000cf612e00 00000000 00000000 1b000000 05048001
[35870.781358] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[35870.785495] xhci-hcd xhci-hcd.0.auto: @00000000cf612840 00000000 00000000 1b000000 05048000
[35884.680424] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[35884.684556] xhci-hcd xhci-hcd.0.auto: @00000000cf6120a0 00000000 00000000 1b000000 05048001
[36253.004500] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36253.008600] xhci-hcd xhci-hcd.0.auto: @00000000cf612df0 00000000 00000000 1b000000 05048001
[36259.954046] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36259.958157] xhci-hcd xhci-hcd.0.auto: @00000000cf612140 00000000 00000000 1b000000 05048000
[36266.903604] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36266.907736] xhci-hcd xhci-hcd.0.auto: @00000000cf612680 00000000 00000000 1b000000 05048000
[36273.853043] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36273.857146] xhci-hcd xhci-hcd.0.auto: @00000000cf612bb0 00000000 00000000 1b000000 05048000
[36280.802652] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36280.806784] xhci-hcd xhci-hcd.0.auto: @00000000cf612f00 00000000 00000000 1b000000 05048000
[36287.752165] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36287.756299] xhci-hcd xhci-hcd.0.auto: @00000000cf612430 00000000 00000000 1b000000 05048001
[36294.701588] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36294.705697] xhci-hcd xhci-hcd.0.auto: @00000000cf612960 00000000 00000000 1b000000 05048001
[36301.651093] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36301.655192] xhci-hcd xhci-hcd.0.auto: @00000000cf612cb0 00000000 00000000 1b000000 05048001
[36308.600603] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36308.604717] xhci-hcd xhci-hcd.0.auto: @00000000cf6121f0 00000000 00000000 1b000000 05048000
[36308.613282] sd 0:0:0:0: [sda] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[36308.613291] sd 0:0:0:0: [sda] tag#17 Sense Key : Not Ready [current]
[36308.613298] sd 0:0:0:0: [sda] tag#17 Add. Sense: Logical unit is in process of becoming ready
[36308.613314] sd 0:0:0:0: [sda] tag#17 CDB: Write(16) 8a 00 00 00 00 00 7d 04 08 00 00 00 00 08 00 00
[36308.613321] blk_update_request: I/O error, dev sda, sector 2097416192
[36308.619787] blk_update_request: I/O error, dev sda, sector 2097416192
[36308.626369] Buffer I/O error on dev sda3, logical block 0, lost sync page write
[36308.634122] EXT4-fs error (device sda3): ext4_journal_check_start:56: Detected aborted journal
[36308.642984] EXT4-fs (sda3): Remounting filesystem read-only
[36308.650921] EXT4-fs (sda3): previous I/O error to superblock detected
[36315.600170] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36315.604286] xhci-hcd xhci-hcd.0.auto: @00000000cf6127c0 00000000 00000000 1b000000 05048000
[36322.549724] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36322.553856] xhci-hcd xhci-hcd.0.auto: @00000000cf612b10 00000000 00000000 1b000000 05048000
[36329.499240] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36329.503375] xhci-hcd xhci-hcd.0.auto: @00000000cf612040 00000000 00000000 1b000000 05048001
[36336.448664] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36336.452769] xhci-hcd xhci-hcd.0.auto: @00000000cf612580 00000000 00000000 1b000000 05048001
[36343.398310] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36343.402444] xhci-hcd xhci-hcd.0.auto: @00000000cf6128d0 00000000 00000000 1b000000 05048001
[36350.347818] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36350.351961] xhci-hcd xhci-hcd.0.auto: @00000000cf612e00 00000000 00000000 1b000000 05048001
[36357.297199] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36357.301303] xhci-hcd xhci-hcd.0.auto: @00000000cf612330 00000000 00000000 1b000000 05048000
[36364.246899] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[36364.251038] xhci-hcd xhci-hcd.0.auto: @00000000cf612680 00000000 00000000 1b000000 05048000
[36371.196293] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
UPDATE4: Don't the error messages suggest that for some reason an endpoint event drops and thus the endpoint is basically lost? I experienced that my dvb-s2 USB device continues streaming while receiving error messages. Thus, the streaming endpoint might continue to work while the controlling endpoint is lost. Does anyone have ideas here?

I found another thread about USB issues.

It would be nice if odroid would give us some feedback about it.

viewtopic.php?f=177&t=34790

User avatar
odroid
Site Admin
Posts: 32743
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 223 times
Been thanked: 369 times
Contact:

Re: USB disconnects randomly

Unread post by odroid » Tue May 07, 2019 11:49 am

Which USB HDD do you use? Is it UASP capable?
We couldn't reproduce the issue with our SATA bridge board and WD 1TB HDD.
https://www.hardkernel.com/shop/usb3-0- ... oard-plus/

rosenrot
Posts: 33
Joined: Wed Apr 10, 2019 2:37 am
languages_spoken: english
ODROIDs: n2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: USB disconnects randomly

Unread post by rosenrot » Tue May 07, 2019 2:28 pm

It is a Seagate Backup Plus 4TB 3.0 which is capable of UASP.

I'm not sure where the USB issue comes from - if it is hardware related or a software bug. However, seeing my (and the ones of other people) USB devices fail one after the other when connected to the USB3.0 boards of the N2 puzzles me.

For me the issue arises once every 48h on the USB HDD that sees random read write access every 30 minutes. Could you try to reproduce it under this circumstances? It doesn't fail immediately.

Also, please keep in mind that the USB HDD issue is arising from the overall USB issues seen on the N2. Thus, me and other users are affected by this issue which is not limited to USB HDDs.

I ordered dual power USB cables and a external powered USB hub to investigate if it is a power issue or not.

If you have any chance to track the issue down to be on the hardware or software side, I would highly appreciate it.

User avatar
odroid
Site Admin
Posts: 32743
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 223 times
Been thanked: 369 times
Contact:

Re: USB disconnects randomly

Unread post by odroid » Tue May 07, 2019 2:49 pm

I see.
We have to find a way to test the USB connection stability.
Is it a good idea that copy a bunch of files from/to two USB 3.0 HDD/SSD for a few days?

rosenrot
Posts: 33
Joined: Wed Apr 10, 2019 2:37 am
languages_spoken: english
ODROIDs: n2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: USB disconnects randomly

Unread post by rosenrot » Tue May 07, 2019 2:57 pm

I would say this is a good idea. The USB issue is happening randomly I would say but less often if there is no data transfer between the host and the slave. I wrote a script logging my errors here.

I can tell that until now no error happened when using the micro-usb. So maybe it is possible to say it is either a USB3.0 software bug or a hardware bug of the USB3.0 connectors.

Let's see what we will find.

elatllat
Posts: 1592
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 25 times
Been thanked: 75 times
Contact:

Re: USB disconnects randomly

Unread post by elatllat » Tue May 07, 2019 6:03 pm

rosenrot wrote:
Tue May 07, 2019 2:28 pm
...external powered USB hub...
Back in the rpi days I would use paper and tape to isolate the hub +5v from the SBC.

User avatar
odroid
Site Admin
Posts: 32743
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 223 times
Been thanked: 369 times
Contact:

Re: USB disconnects randomly

Unread post by odroid » Wed May 08, 2019 9:02 am

USB 3.0 SSD and HDD heavy copying test will start and we'll share the result by end of this week.

rosenrot
Posts: 33
Joined: Wed Apr 10, 2019 2:37 am
languages_spoken: english
ODROIDs: n2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: USB disconnects randomly

Unread post by rosenrot » Thu May 09, 2019 4:06 am

Thanks! I received the following error message recently...


[ 7.525920] sd 0:0:0:0: [sda] Write Protect is off
[ 7.525955] sd 0:0:0:0: [sda] Mode Sense: 4f 00 00 00
[ 7.526229] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 7.534613] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring
[ 7.544145] xhci-hcd xhci-hcd.0.auto: @00000000cf612c70 00000000 00000000 1b000000 05038001


Same as here: viewtopic.php?f=177&t=34790&p=255256#p255256

Soleil
Posts: 38
Joined: Tue Apr 30, 2019 9:20 am
languages_spoken: english
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: USB disconnects randomly

Unread post by Soleil » Thu May 09, 2019 9:13 am

Hi All,

I've run into USB issues too... All USBs die, though system works fine - it is complete shutdown of USBs. Since box has SSH I was able to connect via on-board Ethernet adapter.

Environment:
- 1xUSB3.0 GigabitEthernet adapter (AmazonBasics) - no real bandwidth used - ssh session, literally - nothing more
- 1xUSB3.0 HDD - Toshiba External HDD (not sdd) - somewhat busy - doing quite a bit of random reads, a lot of head movement I suppose = power.
- Filesystem on that HDD is Luks (twofish) and ext4 as btrfs is another set of issues.
- Power supply - currently 72W, 12V, 6A (on 230V) - (JOYLIT EU Plug AC110 V-240V ¨¤ 12V 6A 72W)
- no WiFi transmitter plugged to the system, though within 30cm distance access point is placed.

There shouldn't be shortage of power delivered to the box. Earlier had 2A power supply and to rule out this issue it has been replaced with this big one.
Compilation of Kodi on all cores for an hour went fine. So it should be good from stress test perspective.

The issue happened number of times already and renders board not really reliable.

What has been tried:
- connecting via powered USB3 HUB - this itself gives errors - disks are not discovered. Same hub on Dell laptop and other works fine.
- connecting both devices (net adapter and disk) to other ports. Originally these were one under another using USBs closer to onboard Ethernet adapter. During last issue were plugged to bottom ports.

Around that time (1am) I've ongoing backup processes running using Borgbackup which reads some nice set of data but it is not about bandwidth as more about number of blocks at different disk locations.

Hope it helps in investigation.

Code: Select all

May  9 01:00:35 hostname kernel: [22691.000935@3] xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.
May  9 01:00:35 hostname kernel: [22691.000940@3] xhci-hcd xhci-hcd.0.auto: Assuming host is dying, halting host.
May  9 01:00:35 hostname kernel: [22691.020814@3] xhci-hcd xhci-hcd.0.auto: Host not halted after 16000 microseconds.
May  9 01:00:35 hostname kernel: [22691.020818@3] xhci-hcd xhci-hcd.0.auto: Non-responsive xHCI host is not halting.
May  9 01:00:35 hostname kernel: [22691.020821@3] xhci-hcd xhci-hcd.0.auto: Completing active URBs anyway.
May  9 01:00:35 hostname kernel: [22691.021013@1] hub 2-1:1.0: hub_ext_port_status failed (err = -108)
May  9 01:00:35 hostname kernel: [22691.021186@3] xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
May  9 01:00:35 hostname kernel: [22691.022212@3] usb 1-1: USB disconnect, device number 2
May  9 01:00:35 hostname kernel: [22691.022591@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0000: -110
May  9 01:00:35 hostname kernel: [22691.022598@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0001: -19
May  9 01:00:35 hostname kernel: [22691.022602@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0009: -19
May  9 01:00:35 hostname kernel: [22691.022605@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x000a: -19
May  9 01:00:35 hostname kernel: [22691.022608@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0004: -19
May  9 01:00:35 hostname kernel: [22691.022610@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0005: -19
May  9 01:00:35 hostname kernel: [22691.022773@3] usb 2-1: USB disconnect, device number 2
May  9 01:00:35 hostname kernel: [22691.022778@3] usb 2-1.2: USB disconnect, device number 4
May  9 01:00:35 hostname kernel: [22691.022890@3] ax88179_178a 2-1.2:1.0 lan: unregister 'ax88179_178a' usb-xhci-hcd.0.auto-1.2, ASIX AX88179 USB 3.0 Gigabit Ethernet
May  9 01:00:35 hostname kernel: [22691.022942@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0002: -19
May  9 01:00:35 hostname kernel: [22691.022947@3] ax88179_178a 2-1.2:1.0 lan: Failed to write reg index 0x0002: -19
May  9 01:00:35 hostname avahi-daemon[2444]: Interface lan.IPv4 no longer relevant for mDNS.
May  9 01:00:35 hostname avahi-daemon[2444]: Leaving mDNS multicast group on interface lan.IPv4 with address 192.168.0.1.
May  9 01:00:35 hostname avahi-daemon[2444]: Withdrawing address record for 192.168.0.1 on lan.
May  9 01:00:35 hostname systemd-networkd[2048]: lan: Lost carrier
May  9 01:00:35 hostname systemd-timesyncd[2239]: Network configuration changed, trying to establish connection.
May  9 01:00:35 hostname kernel: [22691.057059@3] ax88179_178a 2-1.2:1.0 lan (unregistered): Failed to write reg index 0x0002: -19
May  9 01:00:35 hostname kernel: [22691.057065@3] ax88179_178a 2-1.2:1.0 lan (unregistered): Failed to write reg index 0x0001: -19
May  9 01:00:35 hostname kernel: [22691.057069@3] ax88179_178a 2-1.2:1.0 lan (unregistered): Failed to write reg index 0x0002: -19
May  9 01:00:35 hostname kernel: [22691.057556@3] usb 2-1.3: USB disconnect, device number 6
May  9 01:00:35 hostname kernel: [22691.058259@0] blk_update_request: 27 callbacks suppressed
May  9 01:00:35 hostname kernel: [22691.058264@0] blk_update_request: I/O error, dev sda, sector 1258739336
May  9 01:00:35 hostname kernel: [22691.059451@3] blk_update_request: I/O error, dev sda, sector 1950936992
May  9 01:00:35 hostname ModemManager[2468]: <info>  (net/lan): released by modem /sys/devices/platform/ff500000.dwc3/xhci-hcd.0.auto/usb2/2-1/2-1.2
May  9 01:00:35 hostname kernel: [22691.065999@1] blk_update_request: I/O error, dev sda, sector 1258739336
May  9 01:00:35 hostname kernel: [22691.068999@3] sd 0:0:0:0: [sda] Synchronizing SCSI cache
May  9 01:00:35 hostname kernel: [22691.072725@1] blk_update_request: I/O error, dev sda, sector 1258739336
May  9 01:00:35 hostname kernel: [22691.072804@0] blk_update_request: I/O error, dev sda, sector 5496
May  9 01:00:35 hostname kernel: [22691.072818@0] buffer_io_error: 14 callbacks suppressed
May  9 01:00:35 hostname kernel: [22691.072824@0] Buffer I/O error on dev dm-0, logical block 175, lost async page write
May  9 01:00:35 hostname systemd[1]: Stopping usbmount@dev-sda.service...
May  9 01:00:35 hostname systemd[1]: Stopped usbmount@dev-sda.service.
May  9 01:00:35 hostname kernel: [22691.093039@4] blk_update_request: I/O error, dev sda, sector 1258739440
May  9 01:00:35 hostname kernel: [22691.093042@3] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
May  9 01:00:35 hostname kernel: [22691.099463@2] Buffer I/O error on dev dm-0, logical block 62918740, lost async page write
May  9 01:00:35 hostname systemd-timesyncd[2239]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
May  9 01:00:35 hostname kernel: [22691.107673@2] Buffer I/O error on dev dm-0, logical block 62918804, lost async page write
May  9 01:00:35 hostname kernel: [22691.116447@3] Buffer I/O error on dev dm-0, logical block 62918904, lost async page write
May  9 01:00:35 hostname kernel: [22691.124225@3] Buffer I/O error on dev dm-0, logical block 62918905, lost async page write
May  9 01:00:35 hostname kernel: [22691.132099@3] Buffer I/O error on dev dm-0, logical block 365954448, lost async page write
May  9 01:00:35 hostname kernel: [22691.132107@3] Buffer I/O error on dev dm-0, logical block 366477317, lost async page write
May  9 01:00:35 hostname kernel: [22691.132112@0] Aborting journal on device dm-0-8.
May  9 01:00:35 hostname kernel: [22691.132118@3] EXT4-fs warning (device dm-0): ext4_end_bio:313: I/O error -5 writing to inode 91368006 (offset 51449856 size 4096 starting block 378020713)
May  9 01:00:35 hostname kernel: [22691.132122@3] Buffer I/O error on device dm-0, logical block 378020713
May  9 01:00:35 hostname kernel: [22691.132126@3] Buffer I/O error on device dm-0, logical block 378020714
May  9 01:00:35 hostname kernel: [22691.132140@3] scsi 0:0:0:0: rejecting I/O to dead device
May  9 01:00:35 hostname kernel: [22691.132143@3] blk_update_request: I/O error, dev sda, sector 503354008
May  9 01:00:35 hostname kernel: [22691.132151@3] Buffer I/O error on dev dm-0, logical block 62918739, lost async page write
May  9 01:00:35 hostname kernel: [22691.133267@2] Buffer I/O error on dev dm-0, logical block 243826688, lost sync page write
May  9 01:00:35 hostname kernel: [22691.134480@0] JBD2: Error -5 detected when updating journal superblock for dm-0-8.
May  9 01:00:35 hostname kernel: [22691.134515@0] JBD2: Detected IO errors while flushing file data on dm-0-8
May  9 01:00:36 hostname kernel: [22691.712674@3] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:963: inode #91881867: lblock 0: comm squid: error -5 reading directory block
May  9 01:00:39 hostname kernel: [22694.876623@2] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
May  9 01:00:39 hostname kernel: [22694.879230@0] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal
May  9 01:00:39 hostname kernel: [22694.895218@0] EXT4-fs (dm-0): Remounting filesystem read-only
May  9 01:00:39 hostname kernel: [22694.895351@0] EXT4-fs (dm-0): previous I/O error to superblock detected
# left it for some time and tried to recover situation by unplugging/plugging USB devices (Gigagbit Ethernet adapter & disk)
May  9 01:07:39 hostname kernel: [23115.086326@3] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:963: inode #91357705: lblock 0: comm perl: error -5 reading directory block

joshua.yang
Posts: 304
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 4 times
Been thanked: 50 times
Contact:

Re: USB disconnects randomly

Unread post by joshua.yang » Thu May 09, 2019 5:05 pm

Hi.

I've been testing 'copying bunch of files' about 5 hours until now and it looks fine for now.

Wrote a script file that creates 1000, 100 MB dummy files and copies from SSD to HDD infinitely. It removes and re-creates dummy files for every loop.

I will watch this until late next week and let you know if any issue occurs.

If you have any better ideas about this test, please let me know. :)
Attachments
스크린샷 2019-05-09 오후 4.56.16.png
스크린샷 2019-05-09 오후 4.56.16.png (811.91 KiB) Viewed 5616 times
20190509_165231.jpg
20190509_165231.jpg (537.99 KiB) Viewed 5616 times
These users thanked the author joshua.yang for the post:
thyandrecardoso (Fri May 10, 2019 4:58 am)

Soleil
Posts: 38
Joined: Tue Apr 30, 2019 9:20 am
languages_spoken: english
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: USB disconnects randomly

Unread post by Soleil » Fri May 10, 2019 4:20 am

Thanks for active work on this.

There might be some additional ingredient to the crash.
I've just experienced another one during the time when no real activity on HDD is performed. The only process was squid for one user with no web activity, so literally access logs with no active browsing.

To collect more details, dstat has been run, I'll share output.

Given it is possible to resurrect system by simple software reboot, it is hopefully not in hardware. Maybe a mem leal, good old C days?

Other idea, how about encrypting these drives for next round copy? And maybe a large number of small files? Maybe it has something with memory map issue and pointer getting data from wrong mem area?

Soleil
Posts: 38
Joined: Tue Apr 30, 2019 9:20 am
languages_spoken: english
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: USB disconnects randomly

Unread post by Soleil » Fri May 10, 2019 8:48 am

Got another crash.

This time during backup process, but hdd wasn't loaded.

Below is dstat dump run since last reboot, crash with new type of messages + slab and interrupts dump - hope that helps.

Let me know what else and in which direction to move data collection for your investigation.

Code: Select all

# cat ~/tmp/dstat_crash.1.log 
----system---- --total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system-- -------------interrupts------------ ---load-avg--- ------memory-usage----- -net/total- ---procs--- --io/total- ----swap--- ---system-- async --filesystem- ---file-locks-- -----virtual-memory---- ---advanced-virtual-memory--- ------zones-memory-----
     time     |usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw |  11    19    20    21    22    53 | 1m   5m  15m | used  free  buff  cach| recv  send|run blk new| read  writ| used  free| int   csw | #aio|files  inodes|pos lck rea wri|majpf minpf alloc  free|steal scanK scanD pgoru astll| d32F  d32H normF normH
09-05 21:16:29|  5   1  93   0   0|  71k   53k|   0     0 |   0     0 |1552  2214 | 973     0     0    39    87    60 |1.63 1.71 1.69| 700M 1981M  243M  714M|   0     0 |  0   0 0.4|3.64  5.14 |   0     0 |1552  2214 |   0 | 3264  47462 |3.0  12 1.0  15|   0     0     0     0 |   0     0   475     0     0 |   0     0     0     0 
09-05 21:17:29|  7   3  91   0   0|1365B   65k| 375k  379k|   0     0 |2219  2553 |1043     0     0   245   356    60 |1.81 1.75 1.70| 707M 1974M  243M  715M| 375k  379k|  0   0 0.6|0.20  9.13 |   0     0 |2219  2554 |   0 | 3296  47509 |3.0  16 2.0  18|   0     0     0     0 |   0     0  1413     0     0 |   0     0     0     0 
09-05 21:18:29|  7   2  90   0   0| 341B   85k| 474k  479k|   0     0 |2350  2496 |1026     0     0   308   462    60 |1.72 1.73 1.70| 707M 1973M  243M  715M| 474k  479k|  0   0 0.1|0.08  6.10 |   0     0 |2350  2496 |   0 | 3360  47624 |3.0  16 1.0  19|   0     0     0     0 |   0     0  1718     0     0 |   0     0     0     0 
09-05 21:19:29|  6   2  92   0   0| 273B   10k| 363k  366k|   0     0 |2056  2358 | 992     0     0   234   337    60 |1.41 1.64 1.67| 695M 1985M  243M  715M| 363k  366k|  0   0 0.0|0.07  1.83 |   0     0 |2056  2358 |   0 | 3200  47538 |3.0  13 1.0  16|   0     0     0     0 |   0     0  1318     0     0 |   0     0     0     0 
09-05 21:20:29|  6   2  91   0   0| 205B   43k| 443k  429k|   0     0 |2193  2427 |1005     0     0   277   395    60 |1.62 1.65 1.67| 694M 1984M  244M  716M| 443k  429k|2.0   0 0.1|0.05  2.87 |   0     0 |2194  2427 |   0 | 3136  47461 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1484     0     0 |   0     0     0     0 
09-05 21:21:29|  7   3  90   0   0| 273B   21k| 602k  511k|   0     0 |2348  2543 |1005     0     0   321   467    60 |1.84 1.71 1.69| 694M 1985M  244M  716M| 602k  511k|2.0   0 0.1|0.07  3.72 |   0     0 |2348  2543 |   0 | 3136  47452 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1664     0     0 |   0     0     0     0 
09-05 21:22:29|  7   2  91   0   0|1775B   44k| 406k  410k|   0     0 |2207  2469 |1018     0     0   264   379    60 |1.92 1.76 1.71| 694M 1984M  244M  717M| 406k  410k|  0   0 1.0|0.17  7.07 |   0     0 |2207  2469 |   0 | 3072  47460 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1530     0     0 |   0     0     0     0 
09-05 21:23:29|  6   2  91   0   0|4847B   28k| 399k  403k|   0     0 |2140  2419 | 999     0     0   258   378    60 |1.92 1.79 1.72| 694M 1984M  244M  717M| 399k  403k|  0   0 0.0|0.93  4.42 |   0     0 |2140  2419 |   0 | 3072  47513 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1373     0     0 |   0     0     0     0 
09-05 21:24:29|  7   3  91   0   0| 205B   21k| 540k  437k|   0     0 |2213  2495 |1000     0     0   278   402    60 |2.37 1.93 1.78| 694M 1984M  244M  717M| 540k  437k|  0   0 0.0|0.05  3.82 |   0     0 |2213  2495 |   0 | 3072  47516 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1446     0     0 |   0     0     0     0 
09-05 21:25:29|  6   2  91   0   0| 273B   13k| 410k  413k|   0     0 |2149  2425 |1017     0     0   259   371    60 |2.40 1.99 1.81| 694M 1983M  244M  717M| 410k  413k|1.0   0 0.0|0.07  2.43 |   0     0 |2149  2425 |   0 | 3072  47502 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1356     0     0 |   0     0     0     0 
09-05 21:26:29|  6   2  91   0   0| 273B   15k| 388k  390k|   0     0 |2085  2336 | 995     0     0   245   349    60 |1.72 1.87 1.78| 694M 1983M  244M  717M| 388k  390k|  0   0 0.1|0.07  2.42 |   0     0 |2085  2336 |   0 | 3072  47497 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1307     0     0 |   0     0     0     0 
09-05 21:27:29|  6   2  92   0   0| 273B   25k| 377k  380k|   0     0 |2102  2411 | 997     0     0   241   350    60 |1.44 1.77 1.75| 694M 1983M  244M  717M| 377k  380k|1.0   0 0.6|0.07  4.45 |   0     0 |2102  2411 |   0 | 3040  47494 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1312     0     0 |   0     0     0     0 
09-05 21:28:29|  6   2  92   0   0| 273B   30k| 354k  357k|   0     0 |2059  2372 |1006     0     0   229   333    60 |1.83 1.81 1.76| 694M 1982M  244M  718M| 354k  357k|2.0   0 0.3|0.07  3.55 |   0     0 |2058  2372 |   0 | 3040  47512 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1226     0     0 |   0     0     0     0 
09-05 21:29:29|  8   3  89   0   0| 273B   11k| 747k  595k|   0     0 |2483  2596 | 994     0     0   381   537    60 |1.70 1.81 1.77| 694M 1982M  244M  718M| 747k  595k|1.0   0 0.0|0.07  2.10 |   0     0 |2483  2597 |   0 | 3040  47511 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1909     0     0 |   0     0     0     0 
09-05 21:30:29|  6   2  92   0   0| 273B   19k| 376k  379k|   0     0 |2056  2365 | 976     0     0   238   344    60 |1.58 1.77 1.75| 694M 1982M  244M  718M| 376k  379k|1.0   0 0.0|0.07  3.83 |   0     0 |2057  2365 |   0 | 3040  47510 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1267     0     0 |   0     0     0     0 
09-05 21:31:29|  6   2  92   0   0| 273B   17k| 385k  388k|   0     0 |2094  2393 | 981     0     0   249   355    60 |1.39 1.67 1.72| 694M 1982M  244M  718M| 385k  388k|  0   0 0.1|0.07  3.48 |   0     0 |2094  2393 |   0 | 3040  47502 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1306     0     0 |   0     0     0     0 
09-05 21:32:29|  6   2  92   0   0| 205B   20k| 261k  264k|   0     0 |1895  2298 | 989     0     0   171   257    60 |1.57 1.66 1.71| 694M 1982M  244M  718M| 261k  264k|  0   0 0.2|0.05  3.68 |   0     0 |1895  2298 |   0 | 3040  47512 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1001     0     0 |   0     0     0     0 
09-05 21:33:29|  5   2  93   0   0| 273B   26k| 207k  209k|   0     0 |1759  2249 | 982     0     0   134   198    60 |1.64 1.66 1.71| 694M 1982M  244M  718M| 207k  209k|  0   0 0.3|0.07  4.28 |   0     0 |1758  2249 |   0 | 3072  47512 |3.0  12 1.0  15|   0     0     0     0 |   0     0   779     0     0 |   0     0     0     0 
09-05 21:34:29|  6   2  92   0   0| 273B   14k| 215k  215k|   0     0 |1775  2235 | 989     0     0   137   206    60 |1.88 1.72 1.73| 694M 1982M  245M  718M| 215k  215k|  0   0 0.1|0.07  2.63 |   0     0 |1775  2235 |   0 | 3072  47512 |3.0  12 1.0  15|   0     0     0     0 |   0     0   848     0     0 |   0     0     0     0 
09-05 21:35:29|  7   3  91   0   0| 273B   13k| 435k  438k|   0     0 |2144  2350 | 997     0     0   279   387    60 |1.78 1.74 1.73| 694M 1982M  245M  718M| 435k  438k|1.0   0 0.0|0.07  2.23 |   0     0 |2144  2350 |   0 | 3072  47502 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1439     0     0 |   0     0     0     0 
09-05 21:36:29|  7   2  90   0   0| 751B   23k| 391k  394k|   0     0 |2140  2355 |1007     0     0   250   372    60 |1.55 1.67 1.71| 695M 1981M  245M  718M| 391k  394k|1.0   0 0.3|0.13  4.00 |   0     0 |2140  2355 |   0 | 3136  47590 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1422     0     0 |   0     0     0     0 
09-05 21:37:29|  8   2  90   0   0|3891B   87k| 396k  405k|   0     0 |2216  2466 |1013     0     0   258   400    60 |1.55 1.67 1.71| 695M 1979M  245M  719M| 396k  405k|1.0   0 0.2|0.83  6.65 |   0     0 |2216  2466 |   0 | 3168  47671 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1571     0     0 |   0     0     0     0 
09-05 21:38:29|  7   2  91   0   0|1365B   45k| 377k  381k|   0     0 |2147  2394 |1021     0     0   246   365    60 |1.77 1.70 1.72| 695M 1979M  245M  720M| 377k  381k|2.0   0 0.0|0.25  4.83 |   0     0 |2147  2394 |   0 | 3136  47661 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1380     0     0 |   0     0     0     0 
09-05 21:39:29|  6   2  91   0   0| 614B   20k| 414k  418k|   0     0 |2163  2395 | 986     0     0   266   387    60 |1.92 1.78 1.74| 694M 1979M  245M  720M| 414k  418k|  0   0 0.0|0.10  3.90 |   0     0 |2163  2396 |   0 | 3104  47625 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1425     0     0 |   0     0     0     0 
09-05 21:40:29|  6   2  91   0   0|2185B   40k| 368k  373k|   0     0 |2111  2384 | 993     0     0   240   361    60 |2.03 1.81 1.75| 694M 1978M  245M  720M| 369k  374k|1.0   0 0.1|0.53  5.03 |   0     0 |2111  2383 |   0 | 3104  47633 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1368     0     0 |   0     0     0     0 
09-05 21:41:29|  6   2  91   0   0| 410B   23k| 395k  400k|   0     0 |2144  2393 | 995     0     0   254   375    60 |1.53 1.72 1.72| 694M 1978M  245M  721M| 394k  399k|1.0   0 0.0|0.10  3.03 |   0     0 |2145  2393 |   0 | 3072  47643 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1411     0     0 |   0     0     0     0 
09-05 21:42:29|  7   2  91   0   0|3004B   62k| 430k  436k|   0     0 |2274  2515 |1007     0     0   280   419    60 |1.69 1.75 1.73| 694M 1978M  246M  721M| 430k  436k|1.0   0 0.2|0.68  6.53 |   0     0 |2274  2515 |   0 | 3104  47682 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1567     0     0 |   0     0     0     0 
09-05 21:43:29|  6   2  91   0   0| 273B   14k| 404k  403k|   0     0 |2183  2487 |1000     0     0   260   367    60 |1.38 1.64 1.70| 695M 1976M  246M  722M| 404k  403k|1.0   0 0.1|0.07  2.25 |   0     0 |2183  2486 |   0 | 3072  47662 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1357     0     0 |   0     0     0     0 
09-05 21:44:29|  7   3  90   0   0| 751B   31k| 336k  344k|   0     0 |2223  2617 |1115     0     0   214   331    60 |2.05 1.83 1.76| 695M 1975M  246M  723M| 336k  344k|  0   0 0.0|0.10  5.18 |   0     0 |2223  2617 |   0 | 3072  47728 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1332     0     0 |   0     0     0     0 
09-05 21:45:29|  7   2  91   0   0| 273B   55k| 277k  281k|   0     0 |1977  2353 |1021     0     0   180   276    60 |2.43 1.95 1.80| 697M 1972M  246M  722M| 277k  281k|  0   0 0.8|0.07  6.43 |   0     0 |1977  2354 |   0 | 3168  47788 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1193     0     0 |   0     0     0     0 
09-05 21:46:29|  6   2  91   0   0| 273B   30k| 336k  340k|   0     0 |2068  2301 |1004     0     0   234   355    60 |2.00 1.94 1.81| 699M 1971M  246M  722M| 336k  340k|1.0   0 0.1|0.07  5.87 |   0     0 |2068  2301 |   0 | 3168  47779 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1378     0     0 |   0     0     0     0 
09-05 21:47:29|  7   2  91   0   0| 273B   26k| 331k  335k|   0     0 |2077  2290 |1002     0     0   242   364    60 |1.58 1.84 1.78| 698M 1971M  246M  722M| 331k  335k|1.0   0 0.2|0.07  5.38 |   0     0 |2076  2290 |   0 | 3232  47783 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1416     0     0 |   0     0     0     0 
09-05 21:48:29|  6   2  91   0   0| 205B   17k| 317k  320k|   0     0 |2041  2290 |1007     0     0   223   345    60 |1.40 1.74 1.75| 699M 1971M  246M  722M| 317k  320k|  0   0 0.1|0.05  2.80 |   0     0 |2041  2290 |   0 | 3232  47786 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1342     0     0 |   0     0     0     0 
09-05 21:49:29|  6   2  91   0   0| 273B   13k| 415k  419k|   0     0 |2181  2263 | 979     0     0   303   434    60 |1.47 1.71 1.74| 699M 1970M  246M  722M| 415k  420k|2.0   0 0.0|0.07  2.50 |   0     0 |2182  2263 |   0 | 3232  47768 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1550     0     0 |   0     0     0     0 
09-05 21:50:29|  6   2  91   0   0| 273B   13k| 412k  418k|   0     0 |2201  2351 |1000     0     0   287   435    60 |1.99 1.79 1.76| 698M 1971M  246M  722M| 412k  418k|  0   0 0.1|0.07  2.37 |   0     0 |2200  2351 |   0 | 3232  47763 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1673     0     0 |   0     0     0     0 
09-05 21:51:29|  6   3  91   0   0| 273B   13k| 379k  383k|   0     0 |2110  2309 |1001     0     0   255   378    60 |1.97 1.80 1.76| 699M 1970M  246M  722M| 379k  382k|1.0   0 0.0|0.07  2.50 |   0     0 |2110  2309 |   0 | 3232  47757 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1467     0     0 |   0     0     0     0 
09-05 21:52:29|  6   2  91   0   0| 205B   18k| 327k  330k|   0     0 |2057  2313 |1011     0     0   229   346    60 |1.99 1.86 1.79| 699M 1970M  246M  722M| 327k  330k|  0   0 0.2|0.05  3.23 |   0     0 |2057  2313 |   0 | 3200  47759 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1342     0     0 |   0     0     0     0 
09-05 21:53:29|  6   2  92   0   0| 273B   16k| 299k  301k|   0     0 |1950  2260 | 988     0     0   206   307    60 |1.76 1.85 1.79| 699M 1970M  246M  722M| 299k  301k|  0   0 0.1|0.07  2.70 |   0     0 |1949  2259 |   0 | 3200  47760 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1175     0     0 |   0     0     0     0 
09-05 21:54:29|  6   2  91   0   0| 273B   19k| 386k  390k|   0     0 |2107  2283 | 981     0     0   264   389    60 |1.40 1.72 1.75| 699M 1970M  246M  722M| 386k  390k|  0   0 0.0|0.07  3.78 |   0     0 |2107  2283 |   0 | 3200  47760 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1459     0     0 |   0     0     0     0 
09-05 21:55:29|  6   2  91   0   0| 273B   13k| 398k  403k|   0     0 |2141  2304 | 978     0     0   281   410    60 |1.76 1.80 1.78| 699M 1970M  246M  722M| 399k  403k|2.0   0 0.1|0.07  2.42 |   0     0 |2142  2304 |   0 | 3200  47760 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1500     0     0 |   0     0     0     0 
09-05 21:56:29|  6   2  91   0   0| 205B   12k| 415k  422k|   0     0 |2203  2342 | 992     0     0   285   428    60 |1.91 1.86 1.80| 699M 1970M  247M  722M| 415k  422k|  0   0 0.1|0.05  2.38 |   0     0 |2203  2342 |   0 | 3200  47782 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1622     0     0 |   0     0     0     0 
09-05 21:57:29|  6   2  92   0   0| 273B   16k| 306k  309k|   0     0 |2004  2340 | 997     0     0   209   317    60 |1.86 1.85 1.80| 698M 1970M  247M  722M| 306k  310k|1.0   0 0.4|0.07  2.80 |   0     0 |2005  2340 |   0 | 3200  47773 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1240     0     0 |   0     0     0     0 
09-05 21:58:29|  6   2  92   0   0| 273B   28k| 424k  426k|   0     0 |2235  2394 | 988     0     0   297   451    60 |1.76 1.85 1.81| 699M 1970M  247M  722M| 423k  426k|2.0   0 0.1|0.07  5.80 |   0     0 |2235  2394 |   0 | 3200  47770 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1669     0     0 |   0     0     0     0 
09-05 21:59:29|  6   2  92   0   0| 273B   11k| 437k  441k|   0     0 |2263  2372 | 976     0     0   307   464    60 |1.60 1.79 1.79| 699M 1970M  247M  722M| 437k  441k|  0   0 0.0|0.07  2.02 |   0     0 |2263  2372 |   0 | 3168  47766 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1712     0     0 |   0     0     0     0 
09-05 22:00:29|  6   2  92   0   0| 205B   39k| 350k  354k|   0     0 |2094  2333 | 979     0     0   252   377    60 |1.56 1.79 1.79| 701M 1967M  247M  722M| 350k  354k|1.0   0 0.2|0.05  5.72 |   0     0 |2094  2333 |   0 | 3296  47794 |3.0  14 1.0  17|   0     0     0     0 |   0     0  1433     0     0 |   0     0     0     0 
09-05 22:01:29|  6   2  92   0   0| 273B   11k| 245k  248k|   0     0 |1886  2240 | 999     0     0   179   270    60 |1.27 1.66 1.75| 701M 1968M  247M  722M| 245k  248k|2.0   0 0.0|0.07  2.12 |   0     0 |1887  2240 |   0 | 3296  47794 |3.0  14 1.0  17|   0     0     0     0 |   0     0  1081     0     0 |   0     0     0     0 
09-05 22:02:29|  6   2  92   0   0| 273B   14k| 251k  255k|   0     0 |1920  2223 | 971     0     0   198   304    60 |1.30 1.59 1.71| 699M 1969M  247M  722M| 252k  255k|1.0   0 0.2|0.07  2.22 |   0     0 |1922  2224 |   0 | 3200  47761 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1212     0     0 |   0     0     0     0 
09-05 22:03:29|  6   2  91   0   0| 273B   16k| 360k  362k|   0     0 |2119  2317 |1001     0     0   255   381    60 |1.75 1.66 1.73| 699M 1969M  247M  722M| 359k  362k|1.0   0 0.1|0.07  2.82 |   0     0 |2117  2316 |   0 | 3200  47759 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1421     0     0 |   0     0     0     0 
09-05 22:04:29|  6   3  91   0   0| 205B   21k| 358k  362k|   0     0 |2165  2305 |1003     0     0   264   403    60 |1.65 1.66 1.73| 699M 1969M  247M  722M| 358k  362k|1.0   0 0.3|0.05  3.82 |   0     0 |2165  2305 |   0 | 3200  47762 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1722     0     0 |   0     0     0     0 
09-05 22:05:29|  6   2  91   0   0| 410B   15k| 329k  333k|   0     0 |2065  2278 | 983     0     0   237   368    60 |1.86 1.74 1.75| 699M 1970M  247M  722M| 329k  333k|2.0   0 0.0|0.08  1.68 |   0     0 |2065  2278 |   0 | 3200  47767 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1431     0     0 |   0     0     0     0 
09-05 22:06:29|  6   2  91   0   0| 273B   24k| 353k  356k|   0     0 |2114  2356 | 997     0     0   248   374    60 |1.38 1.64 1.71| 699M 1969M  247M  722M| 353k  356k|  0   0 0.1|0.07  4.60 |   0     0 |2114  2356 |   0 | 3200  47762 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1430     0     0 |   0     0     0     0 
09-05 22:07:29|  6   2  91   0   0| 273B   13k| 383k  387k|   0     0 |2184  2346 | 991     0     0   275   430    60 |1.52 1.62 1.71| 699M 1969M  247M  722M| 383k  387k|  0   0 0.2|0.07  2.43 |   0     0 |2186  2347 |   0 | 3200  47766 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1654     0     0 |   0     0     0     0 
09-05 22:08:29|  6   2  91   0   0| 205B   21k| 389k  391k|   0     0 |2144  2335 |1002     0     0   261   386    60 |1.57 1.63 1.70| 699M 1969M  247M  722M| 389k  392k|1.0   0 0.1|0.05  3.90 |   0     0 |2143  2335 |   0 | 3200  47765 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1475     0     0 |   0     0     0     0 
09-05 22:09:29|  6   2  91   0   0| 273B   16k| 445k  450k|   0     0 |2258  2317 | 979     0     0   323   461    60 |1.47 1.58 1.68| 698M 1969M  248M  722M| 445k  449k|  0   0 0.1|0.07  2.53 |   0     0 |2257  2316 |   0 | 3200  47762 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1633     0     0 |   0     0     0     0 
09-05 22:10:29|  6   3  91   0   0| 273B   13k| 427k  431k|   0     0 |2180  2306 | 995     0     0   290   422    60 |1.34 1.52 1.65| 699M 1969M  248M  722M| 427k  431k|  0   0 0.0|0.07  2.23 |   0     0 |2181  2306 |   0 | 3200  47758 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1580     0     0 |   0     0     0     0 
09-05 22:11:29|  6   2  91   0   0| 273B   11k| 427k  432k|   0     0 |2243  2382 |1015     0     0   303   439    60 |1.29 1.47 1.62| 698M 1969M  248M  722M| 427k  432k|1.0   0 0.0|0.07  2.05 |   0     0 |2243  2382 |   0 | 3200  47758 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1624     0     0 |   0     0     0     0 
09-05 22:12:29|  6   2  91   0   0| 205B   16k| 414k  417k|   0     0 |2172  2363 | 995     0     0   280   408    60 |1.78 1.54 1.64| 698M 1969M  248M  722M| 414k  417k|  0   0 0.2|0.05  2.67 |   0     0 |2172  2363 |   0 | 3232  47766 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1524     0     0 |   0     0     0     0 
09-05 22:13:29|  6   2  91   0   0| 273B   26k| 348k  350k|   0     0 |2132  2447 |1029     0     0   231   349    60 |2.19 1.69 1.68| 699M 1969M  248M  722M| 348k  350k|4.0   0 0.3|0.07  2.73 |   0     0 |2132  2447 |   0 | 3232  47772 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1360     0     0 |   0     0     0     0 
09-05 22:14:29|  7   3  90   0   0| 273B   32k| 425k  428k|   0     0 |2435  2787 |1139     0     0   278   410    60 |1.83 1.68 1.68| 700M 1968M  248M  723M| 425k  428k|  0   0 0.1|0.07  6.10 |   0     0 |2435  2787 |   0 | 3232  47770 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1535     0     0 |   0     0     0     0 
09-05 22:15:29|  6   2  92   0   0| 273B   17k| 406k  410k|   0     0 |2227  2427 |1003     0     0   289   436    60 |1.67 1.68 1.68| 699M 1968M  248M  722M| 406k  410k|4.0   0 0.0|0.07  2.83 |   0     0 |2227  2426 |   0 | 3200  47757 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1628     0     0 |   0     0     0     0 
09-05 22:16:29|  6   2  91   0   0| 205B   17k| 449k  454k|   0     0 |2296  2343 | 984     0     0   326   485    60 |1.96 1.75 1.71| 700M 1967M  248M  722M| 449k  454k|1.0   0 0.1|0.05  3.15 |   0     0 |2296  2344 |   0 | 3200  47767 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1794     0     0 |   0     0     0     0 
09-05 22:17:29|  6   2  93   0   0| 273B   50k|  93k   96k|   0     0 |1722  2249 | 980     0     0    99   173    60 |1.69 1.69 1.69| 713M 1954M  248M  723M|  93k   96k|2.0   0 0.8|0.07  7.12 |   0     0 |1722  2249 |   0 | 3424  47875 |3.0  16 2.0  18|   0     0     0     0 |   0     0   752     0     0 |   0     0     0     0 
09-05 22:18:29|  5   1  94   0   0| 273B   43k|  19k   20k|   0     0 |1466  2095 | 940     0     0    29    70    60 |1.58 1.66 1.68| 714M 1953M  248M  723M|  19k   20k|  0   0 0.2|0.07  7.60 |   0     0 |1466  2095 |   0 | 3456  47876 |3.0  16 1.0  19|   0     0     0     0 |   0     0   324     0     0 |   0     0     0     0 
09-05 22:19:29|  5   1  94   0   0| 273B   17k|  53k   54k|   0     0 |1545  2111 | 949     0     0    67   115    60 |1.30 1.57 1.64| 701M 1965M  248M  723M|  53k   54k|1.0   0 0.1|0.07  3.45 |   0     0 |1545  2111 |   0 | 3296  47808 |3.0  13 1.0  16|   0     0     0     0 |   0     0   533     0     0 |   0     0     0     0 
09-05 22:20:29|  5   1  94   0   0| 410B   30k|  38k   41k|   0     0 |1532  2094 | 938     0     0    57   124    60 |1.27 1.51 1.62| 701M 1965M  248M  723M|  38k   41k|  0   0 0.1|0.10  5.35 |   0     0 |1532  2095 |   0 | 3296  47856 |3.0  12 1.0  15|   0     0     0     0 |   0     0   507     0     0 |   0     0     0     0 
09-05 22:21:29|  5   1  94   0   0| 410B   47k|  43k   45k|   0     0 |1608  2171 | 968     0     0    70   143    60 |1.30 1.47 1.60| 701M 1965M  248M  723M|  43k   45k|  0   0 0.1|0.10  5.85 |   0     0 |1607  2170 |   0 | 3264  47826 |3.0  12 1.0  15|   0     0     0     0 |   0     0   585     0     0 |   0     0     0     0 
09-05 22:22:29|  4   1  94   0   0| 273B   17k|  26k   28k|   0     0 |1524  2136 | 961     0     0    52   109    60 |1.34 1.47 1.59| 701M 1965M  248M  723M|  26k   28k|1.0   0 0.2|0.07  3.25 |   0     0 |1524  2136 |   0 | 3232  47800 |3.0  12 1.0  15|   0     0     0     0 |   0     0   493     0     0 |   0     0     0     0 
09-05 22:23:29|  4   1  95   0   0|7373B   19k|4607B 4546B|   0     0 |1398  2113 | 958     0     0     6    24    60 |1.24 1.43 1.57| 701M 1965M  249M  723M|4606B 4545B|  0   0 0.1|0.33  3.67 |   0     0 |1398  2113 |   0 | 3168  48421 |3.0  12 1.0  15|   0     0     0     0 |   0     0   156     0     0 |   0     0     0     0 
09-05 22:24:29|  4   2  94   0   0| 205B   14k|2296B 4417B|   0     0 |1400  2118 | 955     0     0     3    20    60 |1.41 1.44 1.56| 701M 1964M  249M  723M|2296B 4417B|1.0   0 0.1|0.05  2.55 |   0     0 |1401  2118 |   0 | 3168  48431 |3.0  12 1.0  15|   0     0     0     0 |   0     0   198     0     0 |   0     0     0     0 
09-05 22:25:29|  5   2  94   0   0| 273B   24k|5348B 8364B|   0     0 |1426  2115 | 957     0     0     6    30    60 |1.28 1.39 1.54| 701M 1964M  249M  723M|5348B 8364B|  0   0 0.3|0.07  4.60 |   0     0 |1426  2116 |   0 | 3168  48434 |3.0  12 1.0  15|   0     0     0     0 |   0     0   239     0     0 |   0     0     0     0 
09-05 22:26:29|  4   2  94   0   0| 273B   20k|3583B 6141B|   0     0 |1433  2143 | 967     0     0     5    25    60 |1.95 1.56 1.58| 701M 1964M  249M  723M|3583B 6141B|  0   0 0.3|0.07  3.83 |   0     0 |1432  2141 |   0 | 3168  48431 |3.0  12 1.0  15|   0     0     0     0 |   0     0   202     0     0 |   0     0     0     0 
09-05 22:27:29|  5   2  94   0   0| 273B   28k|5557B 8560B|   0     0 |1432  2149 | 969     0     0     7    32    60 |1.79 1.60 1.60| 701M 1964M  249M  723M|5557B 8560B|  0   0 0.5|0.07  5.60 |   0     0 |1432  2148 |   0 | 3168  48444 |3.0  12 1.0  15|   0     0     0     0 |   0     0   258     0     0 |   0     0     0     0 
09-05 22:28:29|  4   2  94   0   0| 205B   20k|3894B 5674B|   0     0 |1398  2115 | 959     0     0     5    24    60 |1.48 1.53 1.57| 701M 1964M  249M  723M|3894B 5674B|3.0   0 0.0|0.05  3.60 |   0     0 |1398  2116 |   0 | 3200  48457 |3.0  12 1.0  15|   0     0     0     0 |   0     0   197     0     0 |   0     0     0     0 
09-05 22:29:29|  4   2  94   0   0| 273B   24k|3262B 5868B|   0     0 |1368  2086 | 936     0     0     4    26    60 |1.41 1.51 1.56| 701M 1963M  249M  723M|3262B 5868B|1.0   0 0.1|0.07  4.82 |   0     0 |1368  2086 |   0 | 3200  48447 |3.0  12 1.0  15|   0     0     0     0 |   0     0   205     0     0 |   0     0     0     0 
09-05 22:30:29|  4   1  94   0   0| 273B   15k|1725B 3757B|   0     0 |1369  2092 | 942     0     0     3    20    60 |1.33 1.47 1.54| 702M 1963M  249M  723M|1725B 3757B|  0   0 0.0|0.07  2.75 |   0     0 |1369  2092 |   0 | 3168  48439 |3.0  12 1.0  15|   0     0     0     0 |   0     0   180     0     0 |   0     0     0     0 
09-05 22:31:29|  4   1  94   0   0| 273B   14k|2256B 4753B|   0     0 |1393  2121 | 953     0     0     3    21    60 |1.32 1.45 1.53| 701M 1963M  249M  723M|2256B 4753B|  0   0 0.1|0.07  2.73 |   0     0 |1393  2121 |   0 | 3168  48430 |3.0  12 1.0  15|   0     0     0     0 |   0     0   183     0     0 |   0     0     0     0 
09-05 22:32:29|  4   2  94   0   0| 205B   16k|2870B 5309B|   0     0 |1361  2123 | 943     0     0     4    22    60 |1.22 1.40 1.51| 701M 1963M  249M  723M|2870B 5309B|1.0   0 0.2|0.05  2.92 |   0     0 |1361  2123 |   0 | 3168  48435 |3.0  12 1.0  15|   0     0     0     0 |   0     0   214     0     0 |   0     0     0     0 
09-05 22:33:29|  4   2  94   0   0| 273B   28k|3633B 5177B|   0     0 |1409  2140 | 963     0     0     4    24    60 |1.20 1.35 1.48| 702M 1963M  249M  723M|3633B 5177B|2.0   0 0.3|0.07  5.75 |   0     0 |1409  2140 |   0 | 3168  48423 |3.0  12 1.0  15|   0     0     0     0 |   0     0   207     0     0 |   0     0     0     0 
09-05 22:34:29|  5   1  94   0   0| 341B   19k|7220B 9814B|   0     0 |1470  2149 | 979     0     0    19    55    60 |1.24 1.35 1.47| 701M 1963M  249M  723M|7223B 9817B|1.0   0 0.0|0.08  3.57 |   0     0 |1470  2149 |   0 | 3168  48420 |3.0  12 1.0  15|   0     0     0     0 |   0     0   316     0     0 |   0     0     0     0 
09-05 22:35:29|  4   1  94   0   0| 273B   21k|7160B 8143B|   0     0 |1412  2088 | 944     0     0    23    56    60 |1.25 1.34 1.46| 702M 1963M  249M  723M|7157B 8140B|  0   0 0.1|0.07  3.70 |   0     0 |1411  2088 |   0 | 3168  48421 |3.0  12 1.0  15|   0     0     0     0 |   0     0   271     0     0 |   0     0     0     0 
09-05 22:36:29|  4   1  94   0   0| 205B   18k|6516B 7387B|   0     0 |1415  2096 | 950     0     0    23    54    60 |1.28 1.34 1.45| 702M 1962M  250M  723M|6516B 7387B|1.0   0 0.0|0.05  3.53 |   0     0 |1415  2097 |   0 | 3168  48426 |3.0  12 1.0  15|   0     0     0     0 |   0     0   253     0     0 |   0     0     0     0 
09-05 22:37:29|  4   1  94   0   0| 273B   16k|6233B 7191B|   0     0 |1443  2125 | 959     0     0    23    52    60 |1.33 1.34 1.44| 702M 1962M  250M  723M|6233B 7191B|  0   0 0.2|0.07  2.97 |   0     0 |1442  2125 |   0 | 3168  48424 |3.0  12 1.0  15|   0     0     0     0 |   0     0   250     0     0 |   0     0     0     0 
09-05 22:38:29|  4   1  95   0   0| 273B   14k|7766B 7752B|   0     0 |1407  2094 | 945     0     0    23    52    60 |1.27 1.32 1.43| 702M 1962M  250M  723M|7766B 7752B|1.0   0 0.0|0.07  2.57 |   0     0 |1408  2095 |   0 | 3168  48412 |3.0  12 1.0  15|   0     0     0     0 |   0     0   242     0     0 |   0     0     0     0 
09-05 22:39:29|  4   1  94   0   0| 273B   13k|5648B 6371B|   0     0 |1419  2090 | 954     0     0    21    50    60 |1.39 1.35 1.44| 702M 1962M  250M  723M|5648B 6371B|  0   0 0.1|0.07  2.42 |   0     0 |1419  2089 |   0 | 3136  48400 |3.0  12 1.0  15|   0     0     0     0 |   0     0   228     0     0 |   0     0     0     0 
09-05 22:40:29|  4   1  95   0   0| 205B   14k|5674B 6349B|   0     0 |1395  2054 | 934     0     0    20    50    60 |1.38 1.36 1.44| 702M 1962M  250M  723M|5674B 6349B|  0   0 0.1|0.05  2.53 |   0     0 |1395  2054 |   0 | 3136  48398 |3.0  12 1.0  15|   0     0     0     0 |   0     0   254     0     0 |   0     0     0     0 
09-05 22:41:29|  4   1  94   0   0| 273B   14k|7080B 8035B|   0     0 |1395  2052 | 930     0     0    21    54    60 |1.21 1.31 1.41| 703M 1961M  250M  723M|7080B 8035B|  0   0 0.0|0.07  2.45 |   0     0 |1395  2052 |   0 | 3136  48397 |3.0  12 1.0  15|   0     0     0     0 |   0     0   269     0     0 |   0     0     0     0 
09-05 22:42:29|  4   1  94   0   0| 273B   19k|7887B 9009B|   0     0 |1425  2106 | 947     0     0    23    58    60 |1.21 1.29 1.40| 703M 1961M  250M  723M|7887B 9009B|  0   0 0.2|0.07  3.85 |   0     0 |1425  2106 |   0 | 3136  48398 |3.0  12 1.0  15|   0     0     0     0 |   0     0   297     0     0 |   0     0     0     0 
09-05 22:43:29|  5   1  94   0   0| 273B   28k|  12k   13k|   0     0 |1481  2138 | 972     0     0    26    63    60 |1.43 1.36 1.42| 702M 1962M  250M  723M|  12k   13k|2.0   0 0.0|0.07  5.23 |   0     0 |1481  2138 |   0 | 3136  48414 |3.0  12 1.0  15|   0     0     0     0 |   0     0   297     0     0 |   0     0     0     0 
09-05 22:44:29|  5   2  93   0   0| 205B   23k|6610B 7208B|   0     0 |1640  2434 |1070     0     0    22    54    60 |1.30 1.34 1.41| 702M 1961M  250M  724M|6610B 7208B|1.0   0 0.1|0.05  3.83 |   0     0 |1640  2434 |   0 | 3168  48423 |3.0  12 1.0  15|   0     0     0     0 |   0     0   312     0     0 |   0     0     0     0 
09-05 22:45:29|  4   1  94   0   0| 273B   20k|6171B 6979B|   0     0 |1461  2175 | 982     0     0    23    53    60 |1.37 1.34 1.40| 702M 1962M  250M  723M|6171B 6979B|  0   0 0.1|0.07  3.73 |   0     0 |1461  2175 |   0 | 3168  48411 |3.0  12 1.0  15|   0     0     0     0 |   0     0   255     0     0 |   0     0     0     0 
09-05 22:46:29|  4   1  95   0   0| 273B   13k|5919B 6625B|   0     0 |1417  2091 | 948     0     0    21    51    60 |1.37 1.34 1.39| 702M 1962M  250M  723M|5919B 6625B|  0   0 0.1|0.07  2.35 |   0     0 |1417  2091 |   0 | 3168  48411 |3.0  12 1.0  15|   0     0     0     0 |   0     0   253     0     0 |   0     0     0     0 
09-05 22:47:29|  5   1  94   0   0| 273B   36k|  11k   13k|   0     0 |1484  2120 | 957     0     0    28    72    60 |1.89 1.48 1.44| 702M 1961M  250M  723M|  11k   13k|1.0   0 0.4|0.07  5.17 |   0     0 |1483  2120 |   0 | 3168  48432 |3.0  12 1.0  15|   0     0     0     0 |   0     0   363     0     0 |   0     0     0     0 
09-05 22:48:29|  5   1  94   0   0| 273B   43k|  14k   16k|   0     0 |1518  2134 | 976     0     0    28    72    60 |1.69 1.48 1.44| 701M 1962M  250M  724M|  14k   16k|  0   0 0.1|0.07  6.75 |   0     0 |1518  2134 |   0 | 3168  48436 |3.0  12 1.0  15|   0     0     0     0 |   0     0   346     0     0 |   0     0     0     0 
09-05 22:49:29|  5   1  94   0   0| 273B   27k|7992B 9302B|   0     0 |1486  2131 | 973     0     0    25    61    60 |1.64 1.50 1.45| 702M 1961M  250M  724M|7992B 9302B|  0   0 0.1|0.07  3.90 |   0     0 |1485  2130 |   0 | 3168  48433 |3.0  12 1.0  15|   0     0     0     0 |   0     0   282     0     0 |   0     0     0     0 
09-05 22:50:29|  5   1  94   0   0| 273B   28k|7980B 9218B|   0     0 |1442  2062 | 944     0     0    25    59    60 |1.67 1.56 1.47| 702M 1961M  251M  724M|7980B 9218B|  0   0 0.1|0.07  5.65 |   0     0 |1442  2062 |   0 | 3168  48433 |3.0  12 1.0  15|   0     0     0     0 |   0     0   278     0     0 |   0     0     0     0 
09-05 22:51:29|  4   1  94   0   0| 273B   14k|6579B 7553B|   0     0 |1455  2113 | 970     0     0    22    55    60 |1.55 1.55 1.47| 702M 1961M  251M  724M|6579B 7553B|  0   0 0.1|0.07  2.78 |   0     0 |1455  2113 |   0 | 3168  48420 |3.0  12 1.0  15|   0     0     0     0 |   0     0   267     0     0 |   0     0     0     0 
09-05 22:52:29|  4   1  94   0   0| 205B   21k|6964B 8150B|   0     0 |1433  2114 | 952     0     0    21    57    60 |1.36 1.49 1.46| 702M 1960M  251M  724M|6964B 8150B|1.0   0 0.4|0.05  3.85 |   0     0 |1433  2114 |   0 | 3168  48421 |3.0  12 1.0  15|   0     0     0     0 |   0     0   300     0     0 |   0     0     0     0 
09-05 22:53:29|  5   1  93   0   0| 410B   43k|  30k   32k|   0     0 |1546  2159 | 981     0     0    38    90    60 |1.51 1.52 1.47| 699M 1963M  251M  725M|  30k   32k|  0   0 0.1|0.08  4.57 |   0     0 |1546  2159 |   0 | 3168  48506 |3.0  12 1.0  15|   0     0     0     0 |   0     0   412     0     0 |   0     0     0     0 
09-05 22:54:29|  5   1  94   0   0| 273B   36k| 112k  114k|   0     0 |1671  2129 | 977     0     0   111   156    60 |1.33 1.47 1.45| 699M 1962M  251M  725M| 112k  114k|  0   0 0.1|0.07  4.63 |   0     0 |1672  2129 |   0 | 3136  48509 |3.0  12 1.0  15|   0     0     0     0 |   0     0   458     0     0 |   0     0     0     0 
09-05 22:55:29|  4   1  94   0   0| 273B   12k|6104B 6904B|   0     0 |1455  2106 | 975     0     0    21    53    60 |1.90 1.57 1.48| 700M 1962M  251M  725M|6106B 6904B|  0   0 0.0|0.07  2.22 |   0     0 |1455  2107 |   0 | 3168  48498 |3.0  12 1.0  15|   0     0     0     0 |   0     0   282     0     0 |   0     0     0     0 
09-05 22:56:29|  5   1  94   0   0| 205B   44k|  10k   11k|   0     0 |1481  2143 | 968     0     0    28    65    60 |1.49 1.53 1.47| 700M 1961M  251M  725M|  10k   11k|  0   0 0.3|0.05  7.38 |   0     0 |1481  2142 |   0 | 3168  48492 |3.0  12 1.0  15|   0     0     0     0 |   0     0   326     0     0 |   0     0     0     0 
09-05 22:57:29|  7   2  91   0   0|2594B   42k| 592k  598k|   0     0 |2552  2565 |1003     0     0   398   581    60 |1.82 1.63 1.51| 698M 1962M  251M  725M| 592k  598k|1.0   0 0.2|0.13  5.67 |   0     0 |2552  2565 |   0 | 3136  48511 |3.0  12 1.0  15|   0     0     0     0 |   0     0  2158     0     0 |   0     0     0     0 
09-05 22:58:29|  8   3  89   0   0| 273B   62k| 758k  521k|   0     0 |2444  2599 |1005     0     0   346   521    60 |2.03 1.70 1.54| 698M 1962M  251M  725M| 758k  521k|2.0   0 0.0|0.07  7.73 |   0     0 |2444  2599 |   0 | 3136  48511 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1981     0     0 |   0     0     0     0 
09-05 22:59:29|  9   3  88   0   0| 273B   64k|1342k 1264k|   0     0 |3639  2938 |1019     0     0   810  1119    60 |1.86 1.72 1.56| 695M 1966M  251M  725M|1342k 1265k|3.0   0 0.0|0.07  5.95 |   0     0 |3640  2938 |   0 | 3040  48473 |3.0  12 1.0  15|   0     0     0     0 |   0     0  3960     0     0 |   0     0     0     0 
09-05 23:00:29|  7   3  91   0   0|3686B   52k| 748k  766k|   0     0 |2791  2628 | 988     0     0   500   702    60 |1.87 1.74 1.58| 697M 1963M  251M  726M| 748k  768k|  0   0 0.2|0.25  6.82 |   0     0 |2793  2630 |   0 | 3136  48500 |3.0  14 1.0  17|   0     0     0     0 |   0     0  2467     0     0 |   0     0     0     0 
09-05 23:01:29|  6   2  91   0   0|3140B   55k| 474k  489k|   0     0 |2348  2493 |1001     0     0   323   469    60 |1.49 1.65 1.55| 697M 1963M  251M  726M| 473k  487k|1.0   0 0.3|0.22  10.4 |   0     0 |2345  2492 |   0 | 3136  48492 |3.0  14 1.0  17|   0     0     0     0 |   0     0  1888     0     0 |   0     0     0     0 
09-05 23:02:29|  6   2  92   0   0|1161B   41k| 301k  255k|   0     0 |1864  2303 | 974     0     0   165   250    60 |1.64 1.67 1.56| 695M 1964M  251M  726M| 301k  255k|1.0   0 0.2|0.13  6.67 |   0     0 |1864  2303 |   0 | 3072  48497 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1018     0     0 |   0     0     0     0 
09-05 23:03:29|  4   1  95   0   0| 273B   17k|3872B 3127B|   0     0 |1384  2111 | 950     0     0     4    21    60 |1.45 1.62 1.55| 696M 1964M  252M  726M|3872B 3127B|  0   0 0.1|0.07  3.03 |   0     0 |1383  2111 |   0 | 3072  48480 |3.0  12 1.0  15|   0     0     0     0 |   0     0   139     0     0 |   0     0     0     0 
09-05 23:04:29|  4   1  94   0   0|1843B   31k|3462B 4119B|   0     0 |1396  2136 | 968     0     0     4    22    60 |1.64 1.64 1.56| 696M 1964M  252M  726M|3462B 4119B|  0   0 0.1|0.40  2.62 |   0     0 |1396  2136 |   0 | 3040  48511 |3.0  12 1.0  15|   0     0     0     0 |   0     0   130     0     0 |   0     0     0     0 
09-05 23:05:29|  5   1  94   0   0| 478B   25k|  57k   59k|   0     0 |1496  2153 | 963     0     0    39    73    60 |2.05 1.76 1.60| 695M 1964M  252M  726M|  57k   59k|  0   0 0.1|0.12  4.50 |   0     0 |1496  2153 |   0 | 3040  48496 |3.0  12 1.0  15|   0     0     0     0 |   0     0   319     0     0 |   0     0     0     0 
09-05 23:06:29|  4   1  94   0   0| 273B   21k|2445B 3104B|   0     0 |1340  2082 | 931     0     0     4    20    60 |1.59 1.68 1.59| 695M 1964M  252M  726M|2445B 3104B|  0   0 0.1|0.07  4.28 |   0     0 |1340  2082 |   0 | 3040  48476 |3.0  12 1.0  15|   0     0     0     0 |   0     0   134     0     0 |   0     0     0     0 
09-05 23:07:29|  4   1  94   0   0| 273B   22k|3203B 4132B|   0     0 |1358  2097 | 938     0     0     4    22    60 |1.47 1.62 1.57| 695M 1964M  252M  726M|3203B 4132B|  0   0 0.1|0.07  4.10 |   0     0 |1358  2097 |   0 | 3008  48473 |3.0  12 1.0  15|   0     0     0     0 |   0     0   149     0     0 |   0     0     0     0 
09-05 23:08:29|  5   2  93   0   0| 205B   19k| 113k   64k|   0     0 |1480  2151 | 942     0     0    43    79    60 |1.66 1.64 1.58| 695M 1964M  252M  726M| 113k   64k|  0   0 0.0|0.05  3.72 |   0     0 |1480  2151 |   0 | 3040  48482 |3.0  12 1.0  15|   0     0     0     0 |   0     0   355     0     0 |   0     0     0     0 
09-05 23:09:29|  4   1  95   0   0| 273B   15k|1737B 2310B|   0     0 |1342  2095 | 940     0     0     3    17    60 |1.35 1.57 1.56| 695M 1964M  252M  726M|1737B 2310B|  0   0 0.1|0.07  2.43 |   0     0 |1342  2095 |   0 | 3008  48465 |3.0  12 1.0  15|   0     0     0     0 |   0     0   126     0     0 |   0     0     0     0 
09-05 23:10:29|  6   1  92   0   0|7168B   35k|  36k   43k|   0     0 |1525  2162 | 969     0     0    32    82    60 |1.68 1.64 1.58| 695M 1963M  252M  727M|  36k   43k|  0   0 0.1|0.40  3.50 |   0     0 |1524  2162 |   0 | 3072  48566 |3.0  12 1.0  15|   0     0     0     0 |   0     0   382     0     0 |   0     0     0     0 
09-05 23:11:29|  6   2  92   0   0| 273B   31k| 286k  156k|   0     0 |1658  2227 | 946     0     0   102   155    60 |1.58 1.60 1.57| 695M 1963M  252M  727M| 286k  156k|1.0   0 0.0|0.07  4.97 |   0     0 |1658  2227 |   0 | 3072  48566 |3.0  12 1.0  15|   0     0     0     0 |   0     0   607     0     0 |   0     0     0     0 
09-05 23:12:29|  5   1  94   0   0| 205B   25k|4029B 4957B|   0     0 |1429  2157 | 976     0     0     6    25    60 |1.55 1.57 1.56| 695M 1962M  252M  727M|4029B 4957B|  0   0 0.2|0.05  4.87 |   0     0 |1428  2157 |   0 | 3040  48517 |3.0  12 1.0  15|   0     0     0     0 |   0     0   187     0     0 |   0     0     0     0 
09-05 23:13:29|  4   1  94   0   0| 273B   17k|3761B 3599B|   0     0 |1383  2077 | 945     0     0     4    20    60 |1.20 1.46 1.52| 696M 1962M  252M  727M|3761B 3599B|1.0   0 0.0|0.07  3.30 |   0     0 |1383  2077 |   0 | 3040  48514 |3.0  12 1.0  15|   0     0     0     0 |   0     0   126     0     0 |   0     0     0     0 
09-05 23:14:29|  6   2  92   0   0| 273B   34k|  62k   36k|   0     0 |1671  2518 |1078     0     0    25    47    60 |1.59 1.53 1.54| 696M 1962M  252M  728M|  62k   36k|  0   0 0.1|0.07  6.50 |   0     0 |1671  2518 |   0 | 3040  48521 |3.0  12 1.0  15|   0     0     0     0 |   0     0   288     0     0 |   0     0     0     0 
09-05 23:15:29|  4   1  94   0   0| 273B   15k|1893B 2491B|   0     0 |1356  2115 | 945     0     0     3    17    60 |1.64 1.54 1.55| 695M 1962M  252M  727M|1893B 2491B|  0   0 0.1|0.07  2.85 |   0     0 |1356  2114 |   0 | 3040  48510 |3.0  12 1.0  15|   0     0     0     0 |   0     0   127     0     0 |   0     0     0     0 
09-05 23:16:29|  4   1  94   0   0| 205B   17k|2161B 2801B|   0     0 |1382  2143 | 967     0     0     3    17    60 |1.44 1.50 1.53| 695M 1962M  252M  727M|2161B 2801B|2.0   0 0.0|0.05  3.22 |   0     0 |1383  2143 |   0 | 3008  48515 |3.0  12 1.0  15|   0     0     0     0 |   0     0   124     0     0 |   0     0     0     0 
09-05 23:17:29|  5   2  93   0   0| 273B   43k|  90k   51k|   0     0 |1524  2265 | 983     0     0    35    63    60 |1.40 1.47 1.52| 708M 1949M  253M  727M|  90k   51k|  0   0 0.6|0.07  7.28 |   0     0 |1524  2265 |   0 | 3200  48617 |3.0  16 2.0  18|   0     0     0     0 |   0     0   347     0     0 |   0     0     0     0 
09-05 23:18:29|  4   1  95   0   0| 273B   26k|5726B 3523B|   0     0 |1364  2112 | 943     0     0     4    19    60 |1.63 1.52 1.54| 709M 1948M  253M  727M|5533B 3375B|  0   0 0.0|0.07  4.53 |   0     0 |1364  2111 |   0 | 3200  48608 |3.0  16 1.0  19|   0     0     0     0 |   0     0   119     0     0 |   0     0     0     0 
09-05 23:19:29|  4   1  95   0   0| 273B   15k|1157B 1514B|   0     0 |1393  2127 | 968     0     0     2    15    60 |1.44 1.49 1.53| 697M 1960M  253M  727M|1157B 1514B|1.0   0 0.0|0.07  3.02 |   0     0 |1393  2128 |   0 | 3072  48528 |3.0  13 1.0  16|   0     0     0     0 |   0     0   164     0     0 |   0     0     0     0 
09-05 23:20:29|  4   1  94   0   0| 205B 7441B|  23k   14k|   0     0 |1373  2049 | 931     0     0    11    28    60 |1.87 1.60 1.56| 695M 1961M  253M  727M|  23k   14k|1.0   0 0.1|0.05  1.32 |   0     0 |1373  2049 |   0 | 2976  48515 |3.0  12 1.0  15|   0     0     0     0 |   0     0   169     0     0 |   0     0     0     0 
09-05 23:21:29|  5   2  93   0   0| 273B   27k| 112k   61k|   0     0 |1502  2184 | 947     0     0    43    76    60 |1.48 1.55 1.54| 695M 1961M  253M  727M| 112k   61k|1.0   0 0.0|0.07  5.15 |   0     0 |1502  2184 |   0 | 2976  48514 |3.0  12 1.0  15|   0     0     0     0 |   0     0   316     0     0 |   0     0     0     0 
09-05 23:22:29|  4   1  95   0   0| 273B   16k|2453B 3243B|   0     0 |1370  2115 | 945     0     0     3    19    60 |1.29 1.47 1.52| 695M 1961M  253M  727M|2453B 3243B|  0   0 0.2|0.07  3.10 |   0     0 |1370  2115 |   0 | 2976  48513 |3.0  12 1.0  15|   0     0     0     0 |   0     0   149     0     0 |   0     0     0     0 
09-05 23:23:29|  4   1  94   0   0| 273B   19k|3570B 3411B|   0     0 |1376  2118 | 948     0     0     4    19    60 |1.27 1.43 1.50| 696M 1961M  253M  727M|3570B 3411B|1.0   0 0.0|0.07  3.70 |   0     0 |1376  2118 |   0 | 2976  48512 |3.0  12 1.0  15|   0     0     0     0 |   0     0   129     0     0 |   0     0     0     0 
09-05 23:24:29|  4   1  95   0   0| 205B   13k|1790B 2370B|   0     0 |1354  2097 | 942     0     0     3    16    60 |1.46 1.46 1.51| 696M 1961M  253M  727M|1790B 2370B|2.0   0 0.1|0.05  2.27 |   0     0 |1354  2097 |   0 | 2976  48515 |3.0  12 1.0  15|   0     0     0     0 |   0     0   116     0     0 |   0     0     0     0 
09-05 23:25:29|  4   1  95   0   0| 273B   12k|1758B 2214B|   0     0 |1370  2079 | 944     0     0     3    16    60 |1.45 1.45 1.50| 696M 1961M  253M  727M|1758B 2214B|1.0   0 0.0|0.07  2.33 |   0     0 |1370  2079 |   0 | 2976  48509 |3.0  12 1.0  15|   0     0     0     0 |   0     0   122     0     0 |   0     0     0     0 
09-05 23:26:29|  4   1  95   0   0| 273B   14k|1696B 2170B|   0     0 |1307  2029 | 907     0     0     2    16    60 |1.19 1.38 1.47| 696M 1960M  253M  727M|1696B 2170B|2.0   0 0.0|0.07  2.28 |   0     0 |1307  2028 |   0 | 2976  48513 |3.0  12 1.0  15|   0     0     0     0 |   0     0   106     0     0 |   0     0     0     0 
09-05 23:27:29|  5   2  94   0   0| 273B   21k| 130k   72k|   0     0 |1506  2186 | 949     0     0    49    77    60 |1.44 1.42 1.48| 694M 1962M  253M  727M| 130k   72k|1.0   0 0.2|0.07  4.05 |   0     0 |1506  2186 |   0 | 2976  48518 |3.0  12 1.0  15|   0     0     0     0 |   0     0   348     0     0 |   0     0     0     0 
09-05 23:28:29|  4   1  94   0   0| 205B   15k|3243B 2130B|   0     0 |1366  2106 | 945     0     0     3    17    60 |1.32 1.39 1.46| 695M 1961M  253M  727M|3243B 2130B|  0   0 0.1|0.05  2.80 |   0     0 |1366  2106 |   0 | 2976  48512 |3.0  12 1.0  15|   0     0     0     0 |   0     0   113     0     0 |   0     0     0     0 
09-05 23:29:29|  5   1  94   0   0| 273B   19k|  94k   53k|   0     0 |1462  2147 | 941     0     0    35    62    60 |1.48 1.43 1.47| 695M 1961M  253M  727M|  94k   53k|1.0   0 0.0|0.07  3.93 |   0     0 |1463  2147 |   0 | 2944  48511 |3.0  12 1.0  15|   0     0     0     0 |   0     0   276     0     0 |   0     0     0     0 
09-05 23:30:29|  4   1  95   0   0| 273B   12k|1936B 2504B|   0     0 |1382  2075 | 948     0     0     3    18    60 |1.89 1.56 1.51| 695M 1961M  253M  727M|1936B 2504B|1.0   0 0.0|0.07  2.42 |   0     0 |1382  2075 |   0 | 2944  48508 |3.0  12 1.0  15|   0     0     0     0 |   0     0   118     0     0 |   0     0     0     0 
09-05 23:31:29|  4   1  94   0   0| 273B   14k|2260B 2850B|   0     0 |1359  2093 | 947     0     0     3    18    60 |1.75 1.57 1.52| 695M 1961M  254M  727M|2260B 2850B|1.0   0 0.0|0.07  2.33 |   0     0 |1359  2093 |   0 | 2976  48507 |3.0  12 1.0  15|   0     0     0     0 |   0     0   119     0     0 |   0     0     0     0 
09-05 23:32:29|  4   1  94   0   0| 205B   20k|3614B 4294B|   0     0 |1342  2091 | 934     0     0     4    21    60 |1.55 1.54 1.51| 696M 1960M  254M  727M|3614B 4294B|  0   0 0.2|0.05  3.95 |   0     0 |1342  2091 |   0 | 2976  48516 |3.0  12 1.0  15|   0     0     0     0 |   0     0   143     0     0 |   0     0     0     0 
09-05 23:33:29|  8   2  90   0   0|2867B  128k|  86k   92k|   0     0 |1760  2330 |1050     0     0    68   151    60 |1.69 1.58 1.53| 706M 1945M  254M  731M|  86k   92k|  0   0 0.1|0.32  6.27 |   0     0 |1760  2330 |   0 | 3136  48918 |3.0  12 1.0  15|   0     0     0     0 |   0     0   714     0     0 |   0     0     0     0 
09-05 23:34:29|  7   2  91   0   0| 273B  107k|  94k   98k|   0     0 |1736  2312 |1040     0     0    71   145    60 |1.63 1.62 1.54| 717M 1934M  254M  732M|  94k   98k|  0   0 0.1|0.07  8.38 |   0     0 |1736  2312 |   0 | 3200  49053 |3.0  12 1.0  15|   0     0     0     0 |   0     0   675     0     0 |   0     0     0     0 
09-05 23:35:29|  4   1  94   0   0| 341B   35k|4850B 5456B|   0     0 |1450  2168 | 991     0     0     6    39    60 |1.44 1.56 1.53| 717M 1933M  254M  732M|4850B 5456B|  0   0 0.0|0.08  4.67 |   0     0 |1450  2168 |   0 | 3072  48924 |3.0  12 1.0  15|   0     0     0     0 |   0     0   216     0     0 |   0     0     0     0 
09-05 23:36:29|  5   1  94   0   0| 205B   41k|3514B 4521B|   0     0 |1437  2165 | 988     0     0     6    26    60 |1.38 1.52 1.52| 718M 1933M  254M  732M|3514B 4521B|  0   0 0.0|0.05  5.98 |   0     0 |1437  2166 |   0 | 3040  48845 |3.0  12 1.0  15|   0     0     0     0 |   0     0   184     0     0 |   0     0     0     0 
09-05 23:37:29|  4   1  94   0   0| 273B   22k|2986B 3956B|   0     0 |1390  2102 | 949     0     0     4    22    60 |1.36 1.50 1.51| 718M 1933M  254M  732M|2986B 3956B|  0   0 0.1|0.07  3.90 |   0     0 |1390  2102 |   0 | 3040  48834 |3.0  12 1.0  15|   0     0     0     0 |   0     0   161     0     0 |   0     0     0     0 
09-05 23:38:29|  7   2  91   1   0| 273B   70k| 127k  134k|   0     0 |1798  2308 |1018     0     0    88   182    60 |1.57 1.53 1.52| 717M 1932M  254M  732M| 127k  134k|  0   0 0.2|0.07  10.6 |   0     0 |1798  2308 |   0 | 3072  48924 |3.0  12 1.0  15|   0     0     0     0 |   0     0   782     0     0 |   0     0     0     0 
09-05 23:39:29|  9   2  89   1   0|9421B  139k| 207k  219k|   0     0 |2108  2425 |1088     0     0   151   316    60 |1.61 1.55 1.53| 719M 1929M  254M  734M| 208k  219k|3.0   0 0.1|0.77  11.2 |   0     0 |2108  2426 |   0 | 3264  49196 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1343     0     0 |   0     0     0     0 
09-05 23:40:29| 11   3  85   1   0|7100B  152k|1074k  863k|   0     0 |3308  2956 |1138     0     0   536   923    60 |2.00 1.67 1.58| 728M 1917M  255M  736M|1074k  863k|1.0   0 0.1|1.30  11.8 |   0     0 |3307  2956 |   0 | 3328  49436 |3.0  12 1.0  15|   0     0     0     0 |   0     0  3450     0     0 |   0     0     0     0 
09-05 23:41:29|  8   2  90   0   0|  32k  182k| 190k  198k|   0     0 |1972  2386 |1038     0     0   116   288    60 |1.81 1.68 1.58| 732M 1904M  255M  744M| 190k  198k|1.0   0 0.1|2.92  11.2 |   0     0 |1972  2386 |   0 | 3264  49738 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1135     0     0 |   0     0     0     0 
09-05 23:42:29|  6   2  91   0   0|9353B  136k|  58k   64k|   0     0 |1696  2330 |1036     0     0    46   141    60 |1.40 1.59 1.56| 737M 1896M  255M  746M|  58k   64k|  0   0 0.2|0.30  7.77 |   0     0 |1695  2330 |   0 | 3200  49732 |3.0  12 1.0  15|   0     0     0     0 |   0     0   664     0     0 |   0     0     0     0 
09-05 23:43:29|  6   2  93   0   0| 273B   46k| 236k  239k|   0     0 |1876  2249 |1014     0     0   174   243    60 |1.56 1.60 1.56| 736M 1898M  255M  746M| 236k  239k|  0   0 0.3|0.07  7.85 |   0     0 |1876  2249 |   0 | 3136  49679 |3.0  12 1.0  15|   0     0     0     0 |   0     0   670     0     0 |   0     0     0     0 
09-05 23:44:29|  6   2  91   0   0| 205B   32k| 313k  317k|   0     0 |2188  2544 |1116     0     0   229   290    60 |1.51 1.58 1.55| 737M 1896M  256M  747M| 313k  317k|1.0   0 0.4|0.05  5.57 |   0     0 |2188  2544 |   0 | 3104  49643 |3.0  12 1.0  15|   0     0     0     0 |   0     0   697     0     0 |   0     0     0     0 
09-05 23:45:29|  7   3  90   0   0| 273B   37k| 425k  275k|   0     0 |1995  2474 |1031     0     0   188   262    60 |1.26 1.50 1.53| 736M 1897M  256M  746M| 426k  275k|4.0   0 0.6|0.07  6.92 |   0     0 |1996  2474 |   0 | 3040  49608 |3.0  12 1.0  15|   0     0     0     0 |   0     0   913     0     0 |   0     0     0     0 
09-05 23:46:29| 11   4  85   0   0| 341B   81k| 982k  535k|   0     0 |2616  2903 |1070     0     0   352   526    60 |1.68 1.60 1.56| 736M 1897M  256M  746M| 982k  536k|2.0   0 0.3|0.08  7.83 |   0     0 |2617  2904 |   0 | 3104  49646 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1874     0     0 |   0     0     0     0 
09-05 23:47:29|  6   2  92   0   0|5393B   73k| 156k   97k|   0     0 |1700  2330 |1014     0     0    64   132    60 |1.28 1.50 1.53| 738M 1894M  256M  746M| 156k   97k|2.0   0 0.5|1.25  11.4 |   0     0 |1699  2329 |   0 | 3072  49673 |3.0  12 1.0  15|   0     0     0     0 |   0     0   548     0     0 |   0     0     0     0 
09-05 23:48:29|  9   2  88   1   0|  12k  328k| 114k  125k|   0     0 |1980  2463 |1098     0     0    93   232    60 |1.64 1.59 1.56| 753M 1873M  257M  751M| 114k  125k|2.0   0 0.1|2.25  11.7 |   0     0 |1980  2463 |   0 | 3328  50264 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1136     0     0 |   0     0     0     0 
09-05 23:49:29|  4   1  94   0   0| 341B   48k|  11k   12k|   0     0 |1473  2198 | 974     0     0    14    59    60 |1.32 1.51 1.53| 754M 1872M  257M  751M|  11k   12k|  0   0 0.1|0.08  7.22 |   0     0 |1473  2197 |   0 | 3264  50200 |3.0  12 1.0  15|   0     0     0     0 |   0     0   305     0     0 |   0     0     0     0 
09-05 23:50:29|  4   1  94   0   0| 273B   21k|6208B 7327B|   0     0 |1461  2174 | 986     0     0    10    41    60 |1.23 1.45 1.51| 754M 1871M  257M  751M|6209B 7328B|  0   0 0.1|0.07  3.93 |   0     0 |1461  2174 |   0 | 3040  49982 |3.0  12 1.0  15|   0     0     0     0 |   0     0   245     0     0 |   0     0     0     0 
09-05 23:51:29|  6   2  92   0   0| 273B   18k| 288k  162k|   0     0 |1679  2204 | 941     0     0   108   175    60 |1.35 1.44 1.50| 752M 1874M  257M  751M| 288k  162k|1.0   0 0.0|0.07  3.25 |   0     0 |1679  2204 |   0 | 3040  49964 |3.0  12 1.0  15|   0     0     0     0 |   0     0   721     0     0 |   0     0     0     0 
09-05 23:52:29|  6   2  92   0   0| 205B  108k|  60k   66k|   0     0 |1692  2396 |1038     0     0    51   107    60 |1.36 1.42 1.49| 753M 1870M  257M  754M|  60k   66k|1.0   0 0.4|0.05  7.20 |   0     0 |1692  2396 |   0 | 3168  50179 |3.0  12 1.0  15|   0     0     0     0 |   0     0   573     0     0 |   0     0     0     0 
09-05 23:53:29| 11   2  87   0   0|  43k  216k| 174k  220k|   0     0 |2220  2578 |1137     0     0   132   360    60 |1.84 1.54 1.53| 761M 1856M  257M  759M| 174k  220k|  0   0 0.0|4.20  9.77 |   0     0 |2220  2578 |   0 | 3360  50581 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1647     0     0 |   0     0     0     0 
09-05 23:54:29|  6   2  92   0   0| 273B  117k| 113k  118k|   0     0 |1753  2281 |1016     0     0    82   170    60 |2.16 1.68 1.58| 760M 1855M  258M  760M| 113k  118k|  0   0 0.1|0.07  10.7 |   0     0 |1753  2281 |   0 | 3328  50566 |3.0  12 1.0  15|   0     0     0     0 |   0     0   752     0     0 |   0     0     0     0 
09-05 23:55:29|  9   2  89   0   0| 410B   57k| 147k  166k|   0     0 |2002  2310 |1085     0     0   125   282    60 |1.82 1.66 1.58| 763M 1851M  258M  761M| 147k  166k|  0   0 0.1|0.10  5.63 |   0     0 |2002  2310 |   0 | 3232  50612 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1285     0     0 |   0     0     0     0 
09-05 23:56:29|  7   2  91   0   0| 205B   83k|  64k   75k|   0     0 |1791  2267 |1044     0     0    68   186    60 |1.92 1.71 1.60| 764M 1849M  258M  762M|  64k   75k|  0   0 0.0|0.05  9.02 |   0     0 |1791  2267 |   0 | 3168  50629 |3.0  12 1.0  15|   0     0     0     0 |   0     0   912     0     0 |   0     0     0     0 
09-05 23:57:29|  7   2  91   0   0| 273B   43k| 319k  185k|   0     0 |1867  2394 |1009     0     0   135   232    60 |1.52 1.63 1.57| 763M 1849M  258M  762M| 319k  185k|  0   0 0.2|0.07  5.62 |   0     0 |1867  2394 |   0 | 3136  50597 |3.0  12 1.0  15|   0     0     0     0 |   0     0   970     0     0 |   0     0     0     0 
09-05 23:58:29|  7   2  91   0   0| 614B  298k| 266k  276k|   0     0 |2047  2342 |1012     0     0   194   334    60 |1.62 1.66 1.59| 762M 1836M  258M  776M| 266k  276k|4.0   0 0.1|0.12  8.02 |   0     0 |2048  2343 |   0 | 3136  50720 |3.0  12 1.0  15|   0     0     0     0 |   0     0  1421     0     0 |   0     0     0     0 
09-05 23:59:29|  5   1  93   0   0| 273B   94k|  80k   83k|   0     0 |1637  2193 | 980     0     0    71   139    60 |1.45 1.60 1.57| 762M 1832M  258M  779M|  80k   83k|  0   0 0.0|0.07  5.13 |   0     0 |1636  2192 |   0 | 3136  50709 |3.0  12 1.0  15|   0     0     0     0 |   0     0   640     0     0 |   0     0     0     0 
10-05 00:00:29|  6   2  92   0   0| 205B   63k| 243k  153k|   0     0 |1822  2356 |1012     0     0   116   200    60 |1.46 1.57 1.56| 764M 1828M  259M  781M| 243k  153k|2.0   0 0.3|0.05  6.75 |   0     0 |1822  2356 |   0 | 3104  50734 |3.0  14 1.0  17|   0     0     0     0 |   0     0   856     0     0 |   0     0     0     0 
10-05 00:01:29|  5   2  93   0   0| 273B   77k| 255k  261k|   0     0 |1899  2246 | 966     0     0   185   290    60 |1.72 1.63 1.58| 764M 1820M  259M  789M| 255k  261k|  0   0 0.0|0.07  3.33 |   0     0 |1899  2246 |   0 | 3104  50730 |3.0  14 1.0  17|   0     0     0     0 |   0     0  1130     0     0 |   0     0     0     0 
10-05 00:02:29|  5   2  93   0   0| 273B  164k|  75k   80k|   0     0 |1596  2128 | 962     0     0    69   134    60 |1.68 1.66 1.60| 763M 1817M  259M  792M|  75k   80k|1.0   0 0.4|0.07  4.15 |   0     0 |1596  2128 |   0 | 3072  50846 |4.0  12 1.0  16|   0     0     0     0 |   0     0   771     0     0 |   0     0     0     0 
10-05 00:03:29|  5   1  94   0   0| 273B   29k|  14k   14k|   0     0 |1488  2129 | 965     0     0    28    72    60 |2.45 1.84 1.66| 764M 1816M  259M  792M|  14k   14k|2.0   0 0.0|0.07  5.05 |   0     0 |1488  2129 |   0 | 3072  50856 |4.0  12 1.0  16|   0     0     0     0 |   0     0   368     0     0 |   0     0     0     0 
10-05 00:04:29|  5   2  93   0   0| 205B   26k|  53k   56k|   0     0 |1575  2180 | 982     0     0    55   111    60 |2.05 1.84 1.67| 762M 1815M  259M  794M|  53k   57k|2.0   0 0.0|0.05  3.63 |   0     0 |1575  2180 |   0 | 3040  50859 |4.0  12 1.0  16|   0     0     0     0 |   0     0   490     0     0 |   0     0     0     0 
10-05 00:05:29|  5   2  93   0   0| 273B   81k|  58k   59k|   0     0 |1548  2134 | 962     0     0    54   111    60 |1.65 1.76 1.65| 764M 1811M  259M  796M|  58k   59k|  0   0 0.1|0.07  3.82 |   0     0 |1548  2134 |   0 | 3040  50878 |4.0  12 1.0  16|   0     0     0     0 |   0     0   507     0     0 |   0     0     0     0 
10-05 00:06:29|  8   2  90   0   0| 341B  150k| 164k  167k|   0     0 |1934  2395 |1054     0     0   128   242    60 |1.55 1.71 1.64| 764M 1804M  260M  803M| 164k  167k|  0   0 0.1|0.08  5.92 |   0     0 |1935  2396 |   0 | 3136  51042 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1058     0     0 |   0     0     0     0 
10-05 00:07:29|  7   2  91   0   0| 498k  120k|  65k   43k|   0     0 |1727  2354 |1030     0     0    55   115    60 |1.78 1.76 1.66| 795M 1753M  261M  821M|  65k   43k|1.0   0 1.8|14.0  13.8 |   0     0 |1727  2353 |   0 | 3328  51712 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1095     0     0 |   0     0     0     0 
10-05 00:08:29|  8   3  89   1   0| 181k  504k| 261k  410k|   0     0 |2105  2511 |1023     0     0   209   318    60 |1.83 1.81 1.68| 768M 1765M  262M  834M| 261k  410k|  0   0 1.4|5.40  12.7 |   0     0 |2105  2511 |   0 | 3136  51550 |4.0  12 1.0  16|   0     0     0     0 |   0     0  2936     0     0 |   0     0     0     0 
10-05 00:09:29|  7   3  90   0   0| 172k  262k| 565k   41k|   0     0 |1815  2607 | 983     0     0   219    89    60 |1.87 1.84 1.70| 813M 1706M  262M  847M| 565k   41k|1.0   0 1.1|1.57  11.1 |   0     0 |1815  2607 |   0 | 3296  51761 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1057     0     0 |   0     0     0     0 
10-05 00:10:29|  7   3  90   0   0| 273B  360k| 624k   40k|   0     0 |1834  2633 | 998     0     0   236    83    60 |1.83 1.83 1.71| 817M 1702M  262M  847M| 624k   40k|3.0   0 0.1|0.07  4.62 |   0     0 |1833  2632 |   0 | 3296  51750 |4.0  12 1.0  16|   0     0     0     0 |   0     0   745     0     0 |   0     0     0     0 
10-05 00:11:29|  8   3  88   0   0| 819B  263k| 926k  191k|   0     0 |2116  2770 |1001     0     0   343   208    60 |1.89 1.88 1.74| 819M 1702M  262M  846M| 926k  191k|2.0   0 0.0|0.10  4.13 |   0     0 |2116  2770 |   0 | 3296  51774 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1206     0     0 |   0     0     0     0 
10-05 00:12:29|  9   3  88   0   0|1570B  483k| 897k  101k|   0     0 |2177  2990 |1050     0     0   351   142    60 |2.48 2.02 1.79| 823M 1690M  263M  853M| 897k  101k|  0   0 0.2|0.28  9.53 |   0     0 |2178  2990 |   0 | 3296  51800 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1096     0     0 |   0     0     0     0 
10-05 00:13:29|  5   2  93   0   0| 341B  229k| 233k   16k|   0     0 |1609  2312 | 984     0     0   101    62    60 |1.92 1.95 1.78| 824M 1693M  263M  849M| 233k   16k|1.0   0 0.1|0.08  4.98 |   0     0 |1609  2312 |   0 | 3296  51760 |4.0  12 1.0  16|   0     0     0     0 |   0     0   478     0     0 |   0     0     0     0 
10-05 00:14:29|  6   2  91   0   0|  42k   73k|  65k   98k|   0     0 |1844  2573 |1094     0     0    58   144    60 |1.92 1.93 1.79| 826M 1686M  263M  855M|  65k   98k|  0   0 0.1|2.18  6.50 |   0     0 |1844  2573 |   0 | 3264  51852 |4.0  12 1.0  16|   0     0     0     0 |   0     0   654     0     0 |   0     0     0     0 
10-05 00:15:29|  5   1  93   0   0| 273B   55k|  14k   15k|   0     0 |1526  2209 | 994     0     0    29    71    60 |1.50 1.82 1.76| 825M 1686M  263M  854M|  14k   15k|  0   0 0.1|0.07  7.45 |   0     0 |1526  2209 |   0 | 3232  51854 |4.0  12 1.0  16|   0     0     0     0 |   0     0   372     0     0 |   0     0     0     0 
10-05 00:16:29|  9   2  88   0   0|9421B   98k| 323k  208k|   0     0 |2109  2595 |1085     0     0   159   287    60 |1.88 1.89 1.79| 824M 1686M  263M  855M| 323k  208k|1.0   0 0.0|1.07  7.92 |   0     0 |2109  2595 |   0 | 3360  51984 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1196     0     0 |   0     0     0     0 
10-05 00:17:29|  6   2  93   0   0| 273B   94k|  83k   86k|   0     0 |1671  2257 | 995     0     0    74   143    60 |1.73 1.88 1.79| 837M 1670M  264M  858M|  83k   86k|1.0   0 0.6|0.07  6.27 |   0     0 |1671  2257 |   0 | 3520  52083 |4.0  16 2.0  19|   0     0     0     0 |   0     0   663     0     0 |   0     0     0     0 
10-05 00:18:29|  4   1  94   0   0| 273B   51k|8030B 8249B|   0     0 |1474  2134 | 971     0     0    23    61    60 |1.87 1.86 1.79| 838M 1668M  264M  858M|8032B 8251B|1.0   0 0.0|0.07  8.90 |   0     0 |1474  2134 |   0 | 3488  52033 |4.0  16 1.0  20|   0     0     0     0 |   0     0   308     0     0 |   0     0     0     0 
10-05 00:19:29|  5   1  94   0   0| 273B   18k|  50k   12k|   0     0 |1450  2091 | 944     0     0    38    63    60 |1.76 1.84 1.79| 826M 1680M  264M  859M|  50k   12k|1.0   0 0.0|0.07  3.12 |   0     0 |1450  2091 |   0 | 3296  51940 |4.0  13 1.0  17|   0     0     0     0 |   0     0   376     0     0 |   0     0     0     0 
10-05 00:20:29|  7   2  90   1   0| 200k  361k|  93k  240k|   0     0 |1781  2370 | 985     0     0   105   149    60 |1.56 1.77 1.77| 803M 1689M  264M  871M|  93k  240k|2.0   0 1.0|5.77  11.8 |   0     0 |1781  2370 |   0 | 3328  52098 |4.0  12 1.0  16|   0     0     0     0 |   0     0  2257     0     0 |   0     0     0     0 
10-05 00:21:29|  7   3  89   1   0|1352k  332k|  81k  138k|   0     0 |1887  2481 |1046     0     0    86   178    60 |1.66 1.78 1.78| 768M 1724M  265M  871M|  81k  138k|  0   0 1.8|12.6  11.2 |   0     0 |1887  2481 |   0 | 3168  51945 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1846     0     0 |   0     0     0     0 
10-05 00:22:29|  7   2  90   0   0| 683B   88k| 254k  154k|   0     0 |1853  2347 |1018     0     0   116   200    60 |1.50 1.73 1.76| 767M 1724M  265M  871M| 254k  154k|  0   0 1.3|0.15  14.3 |   0     0 |1853  2347 |   0 | 3104  51976 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1361     0     0 |   0     0     0     0 
10-05 00:23:29|  5   1  94   0   0| 273B   24k|  33k   32k|   0     0 |1530  2154 | 987     0     0    40    88    60 |2.07 1.82 1.78| 767M 1723M  265M  872M|  33k   32k|  0   0 0.0|0.07  3.40 |   0     0 |1530  2154 |   0 | 3104  52001 |4.0  12 1.0  16|   0     0     0     0 |   0     0   390     0     0 |   0     0     0     0 
10-05 00:24:29|  6   1  93   0   0|2253B   64k|  29k   32k|   0     0 |1566  2173 | 991     0     0    40   102    60 |2.55 1.99 1.84| 767M 1723M  265M  872M|  29k   32k|1.0   0 0.1|0.55  4.93 |   0     0 |1566  2173 |   0 | 3136  52103 |4.0  12 1.0  16|   0     0     0     0 |   0     0   486     0     0 |   0     0     0     0 
10-05 00:25:29|  6   2  92   0   0|5052B  173k| 188k  196k|   0     0 |1901  2301 |1002     0     0   145   273    60 |1.71 1.85 1.80| 768M 1715M  265M  879M| 188k  196k|1.0   0 0.1|0.88  5.90 |   0     0 |1902  2304 |   0 | 3136  52314 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1105     0     0 |   0     0     0     0 
10-05 00:26:29|  6   2  92   0   0|  21k   42k| 277k  214k|   0     0 |1866  2270 | 964     0     0   144   266    60 |2.46 2.00 1.86| 770M 1711M  266M  880M| 277k  214k|  0   0 0.1|1.15  6.37 |   0     0 |1865  2267 |   0 | 3200  52338 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1038     0     0 |   0     0     0     0 
10-05 00:27:29|  5   2  93   0   0| 273B   21k| 215k  220k|   0     0 |1848  2234 | 963     0     0   160   253    60 |1.63 1.84 1.81| 771M 1710M  266M  880M| 215k  220k|  0   0 0.3|0.07  3.08 |   0     0 |1848  2234 |   0 | 3168  52343 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1019     0     0 |   0     0     0     0 
10-05 00:28:29|  5   1  94   0   0| 205B   70k|  68k   70k|   0     0 |1593  2151 | 973     0     0    60   126    60 |1.32 1.72 1.77| 771M 1707M  266M  883M|  68k   70k|2.0   0 0.1|0.05  4.13 |   0     0 |1593  2151 |   0 | 3168  52361 |4.0  12 1.0  16|   0     0     0     0 |   0     0   565     0     0 |   0     0     0     0 
10-05 00:29:29|  8   2  90   0   0|7851B  119k|  73k   84k|   0     0 |1834  2304 |1067     0     0    74   191    60 |1.94 1.82 1.80| 775M 1699M  266M  886M|  73k   84k|  0   0 0.0|1.52  7.22 |   0     0 |1834  2304 |   0 | 3104  52593 |4.0  12 1.0  16|   0     0     0     0 |   0     0   877     0     0 |   0     0     0     0 
10-05 00:30:29|  5   1  94   0   0| 410B   30k|  28k   29k|   0     0 |1528  2163 | 982     0     0    36    84    60 |1.58 1.74 1.78| 774M 1699M  267M  886M|  28k   29k|2.0   0 0.1|0.10  4.08 |   0     0 |1528  2163 |   0 | 3072  52622 |4.0  12 1.0  16|   0     0     0     0 |   0     0   413     0     0 |   0     0     0     0 
10-05 00:31:29|  7   2  91   0   0| 273B   55k| 391k  271k|   0     0 |2010  2372 |1001     0     0   192   316    60 |1.86 1.80 1.80| 774M 1697M  267M  888M| 391k  271k|4.0   0 0.1|0.07  3.48 |   0     0 |2010  2372 |   0 | 3136  52652 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1256     0     0 |   0     0     0     0 
10-05 00:32:29|  7   2  91   0   0| 205B   46k| 264k  266k|   0     0 |2059  2413 |1028     0     0   193   318    60 |1.77 1.79 1.80| 778M 1693M  267M  888M| 264k  266k|2.0   0 1.1|0.05  6.70 |   0     0 |2060  2414 |   0 | 3264  52820 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1387     0     0 |   0     0     0     0 
10-05 00:33:29|  7   2  91   0   0|1365B  116k| 225k  229k|   0     0 |1978  2379 |1005     0     0   170   285    60 |1.47 1.71 1.77| 778M 1689M  267M  891M| 225k  229k|  0   0 0.8|0.22  8.50 |   0     0 |1978  2378 |   0 | 3232  52853 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1221     0     0 |   0     0     0     0 
10-05 00:34:29|  7   3  90   1   0|1426k  105k|  97k  101k|   0     0 |1892  2449 |1035     0     0    89   217    60 |1.64 1.71 1.77| 884M 1498M  267M  976M|  97k  101k|  0   0 1.1|11.4  10.0 |   0     0 |1892  2449 |   0 | 3424  53133 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1183     0     0 |   0     0     0     0 
10-05 00:35:29|  8   2  90   0   0|1229B  170k| 195k  207k|   0     0 |2068  2478 |1049     0     0   153   312    60 |1.61 1.69 1.76| 892M 1487M  268M  979M| 195k  207k|1.0   0 0.0|0.20  16.1 |   0     0 |2068  2478 |   0 | 3552  53543 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1316     0     0 |   0     0     0     0 
10-05 00:36:29|  8   3  89   0   0| 205B  119k| 460k  316k|   0     0 |2242  2549 |1046     0     0   225   396    60 |1.47 1.64 1.73| 900M 1476M  268M  980M| 460k  316k|  0   0 0.0|0.05  10.2 |   0     0 |2242  2549 |   0 | 3584  53701 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1552     0     0 |   0     0     0     0 
10-05 00:37:29|  5   2  93   0   0|  12k   95k| 100k   88k|   0     0 |1698  2241 | 994     0     0    85   152    60 |2.06 1.78 1.78| 916M 1455M  268M  985M| 100k   88k|  0   0 0.2|0.15  7.80 |   0     0 |1698  2241 |   0 | 3488  53648 |4.0  12 1.0  16|   0     0     0     0 |   0     0   673     0     0 |   0     0     0     0 
10-05 00:38:29|  5   1  94   0   0| 273B   63k|  45k   25k|   0     0 |1540  2156 | 972     0     0    50    89    60 |1.47 1.67 1.73| 918M 1452M  268M  986M|  45k   25k|1.0   0 0.0|0.07  4.00 |   0     0 |1541  2157 |   0 | 3456  53646 |4.0  12 1.0  16|   0     0     0     0 |   0     0   427     0     0 |   0     0     0     0 
10-05 00:39:29|  5   2  93   0   0| 273B   83k| 117k   99k|   0     0 |1662  2190 | 973     0     0    94   149    60 |1.43 1.62 1.71| 920M 1446M  268M  990M| 117k   99k|  0   0 0.1|0.07  4.92 |   0     0 |1662  2190 |   0 | 3424  53635 |4.0  12 1.0  16|   0     0     0     0 |   0     0   634     0     0 |   0     0     0     0 
10-05 00:40:29|  6   2  91   0   0| 205B  170k| 274k   74k|   0     0 |1767  2336 |1000     0     0   162   126    60 |1.75 1.65 1.72| 930M 1438M  268M  988M| 274k   74k|3.0   0 0.0|0.05  4.93 |   0     0 |1768  2337 |   0 | 3424  53675 |4.0  12 1.0  16|   0     0     0     0 |   0     0   724     0     0 |   0     0     0     0 
10-05 00:41:29|  6   2  92   0   0| 273B   32k| 244k  172k|   0     0 |1812  2253 | 982     0     0   128   220    60 |1.49 1.60 1.70| 930M 1437M  268M  989M| 244k  172k|  0   0 0.0|0.07  3.25 |   0     0 |1812  2253 |   0 | 3424  53718 |4.0  12 1.0  16|   0     0     0     0 |   0     0   937     0     0 |   0     0     0     0 
10-05 00:42:29|  6   2  92   0   0| 478B   96k| 104k   94k|   0     0 |1727  2253 | 996     0     0    92   163    60 |1.48 1.57 1.68| 932M 1431M  269M  992M| 104k   94k|1.0   0 0.2|0.12  6.22 |   0     0 |1727  2253 |   0 | 3424  53786 |4.0  12 1.0  16|   0     0     0     0 |   0     0   737     0     0 |   0     0     0     0 
10-05 00:43:29|  6   2  92   0   0|1638B  121k| 133k   59k|   0     0 |1732  2316 |1011     0     0   109   126    60 |2.15 1.73 1.73| 935M 1428M  269M  992M| 133k   59k|  0   0 0.0|0.15  8.43 |   0     0 |1732  2316 |   0 | 3488  53837 |4.0  12 1.0  16|   0     0     0     0 |   0     0   658     0     0 |   0     0     0     0 
10-05 00:44:29|  8   3  89   0   0| 273B  195k| 174k   19k|   0     0 |1943  2677 |1136     0     0   138    72    60 |2.23 1.85 1.77| 945M 1419M  269M  992M| 174k   20k|1.0   0 0.1|0.07  6.95 |   0     0 |1943  2678 |   0 | 3488  53833 |4.0  12 1.0  16|   0     0     0     0 |   0     0   582     0     0 |   0     0     0     0 
10-05 00:45:29|  6   3  91   0   0| 273B  179k| 202k  124k|   0     0 |1842  2402 |1024     0     0   158   169    60 |2.21 1.95 1.81| 949M 1411M  269M  995M| 201k  123k|1.0   0 0.0|0.07  4.12 |   0     0 |1841  2402 |   0 | 3456  53824 |4.0  12 1.0  16|   0     0     0     0 |   0     0   797     0     0 |   0     0     0     0 
10-05 00:46:29|  6   2  92   0   0| 273B  161k| 232k  100k|   0     0 |1774  2265 | 979     0     0   169   151    60 |1.80 1.88 1.80| 956M 1401M  269M  997M| 232k  100k|  0   0 0.0|0.07  5.20 |   0     0 |1774  2265 |   0 | 3456  53838 |4.0  12 1.0  16|   0     0     0     0 |   0     0   778     0     0 |   0     0     0     0 
10-05 00:47:29|  5   2  93   0   0| 273B   82k| 105k   15k|   0     0 |1577  2192 | 972     0     0    90    72    60 |1.53 1.78 1.77| 961M 1396M  269M  997M| 105k   15k|1.0   0 0.2|0.07  5.60 |   0     0 |1577  2192 |   0 | 3424  53835 |4.0  12 1.0  16|   0     0     0     0 |   0     0   462     0     0 |   0     0     0     0 
10-05 00:48:29|  8   3  88   1   0| 761k 1855k|  40k   17k|   0     0 |2525  3447 |1021     0     0    49   436    60 |1.56 1.74 1.76| 881M 1445M  296M  995M|  40k   17k|  0   0 0.0| 117  8.93 |   0     0 |2525  3447 |   0 | 3392  56522 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1743     0     0 |   0     0     0     0 
10-05 00:49:29|  8   2  90   0   0| 166k   79k| 300k  389k|   0     0 |2161  2376 |1021     0     0   257   339    60 |1.62 1.72 1.75| 918M 1394M  297M 1009M| 300k  390k|2.0   0 0.9|1.97  12.9 |   0     0 |2162  2377 |   0 | 3424  56632 |4.0  12 1.0  16|   0     0     0     0 |   0     0  2448     0     0 |   0     0     0     0 
10-05 00:50:29|  6   2  92   0   0| 187k   91k| 284k  418k|   0     0 |2023  2138 | 956     0     0   301   316    60 |1.77 1.73 1.75| 918M 1382M  297M 1020M| 284k  418k|2.0   0 0.4|2.27  3.03 |   0     0 |2022  2138 |   0 | 3392  56622 |4.0  12 1.0  16|   0     0     0     0 |   0     0   770     0     0 |   0     0     0     0 
10-05 00:51:29|  7   2  91   0   0| 193k   50k|  66k  178k|   0     0 |1674  2189 | 978     0     0   101   118    60 |2.08 1.79 1.76| 920M 1369M  297M 1031M|  66k  178k|  0   0 0.0|2.15  8.57 |   0     0 |1674  2189 |   0 | 3456  56635 |4.0  12 1.0  16|   0     0     0     0 |   0     0   673     0     0 |   0     0     0     0 
10-05 00:52:29|  7   2  91   0   0| 239k   36k|  89k  222k|   0     0 |1722  2250 |1002     0     0   110   119    60 |1.96 1.80 1.76| 919M 1360M  297M 1042M|  89k  222k|1.0   0 0.2|2.80  4.95 |   0     0 |1722  2250 |   0 | 3488  56677 |4.0  12 1.0  16|   0     0     0     0 |   0     0   602     0     0 |   0     0     0     0 
10-05 00:53:29|  6   2  92   0   0| 244k   43k|  49k  179k|   0     0 |1624  2231 | 989     0     0    83    77    60 |1.73 1.76 1.75| 918M 1362M  297M 1040M|  49k  179k|1.0   0 0.0|3.40  7.12 |   0     0 |1624  2231 |   0 | 3456  56679 |4.0  12 1.0  16|   0     0     0     0 |   0     0   456     0     0 |   0     0     0     0 
10-05 00:54:29|  6   2  92   0   0| 186k   76k|  68k  202k|   0     0 |1612  2188 | 966     0     0    97    88    60 |1.63 1.76 1.75| 918M 1357M  297M 1045M|  68k  202k|  0   0 0.1|2.37  4.10 |   0     0 |1612  2188 |   0 | 3456  56685 |4.0  12 1.0  16|   0     0     0     0 |   0     0   474     0     0 |   0     0     0     0 
10-05 00:55:29|  7   2  91   0   0| 166k   27k|  81k  211k|   0     0 |1676  2210 | 988     0     0   105   103    60 |1.75 1.76 1.75| 919M 1349M  297M 1052M|  81k  211k|  0   0 0.0|1.80  4.08 |   0     0 |1676  2210 |   0 | 3456  56691 |4.0  12 1.0  16|   0     0     0     0 |   0     0   539     0     0 |   0     0     0     0 
10-05 00:56:29|  8   2  90   0   0| 342k   52k|  32k  168k|   0     0 |1769  2411 |1053     0     0    75    77    60 |1.66 1.75 1.75| 947M 1311M  298M 1060M|  32k  168k|  0   0 1.3|14.2  6.52 |   0     0 |1769  2411 |   0 | 3520  57573 |4.0  12 1.0  16|   0     0     0     0 |   0     0   706     0     0 |   0     0     0     0 
10-05 00:57:29|  8   3  89   0   0| 334k   25k| 199k  321k|   0     0 |2075  2519 |1051     0     0   179   245    60 |1.51 1.69 1.73| 946M 1296M  298M 1076M| 200k  321k|5.0   0 1.4|7.62  4.05 |   0     0 |2076  2520 |   0 | 3488  57640 |4.0  12 1.0  16|   0     0     0     0 |   0     0  1288     0     0 |   0     0     0     0 
10-05 00:58:29| 12   4  78   6   0|   -     - |  16E   16E|   0     0 |2031  2614 |1158     0     0    61   110    60 |2.57 2.00 1.83| 856M 1281M  298M 1177M|  16E   16E|2.0   0 2.9|   -     - |   0     0 |2029  2614 |   0 | 3008  57141 |4.0  12 1.0  16|   0     0     0     0 |   0     0  3672     0     0 |   0     0     0     0 
10-05 00:59:29| 18   6  75   2   0|1789k 3031k|4341B   76k|   0     0 |2593  3532 |1373     0     0    22     0    60 |2.44 2.08 1.88| 801M 1210M  335M 1177M|4341B   76k|1.0   0 1.9| 264  33.2 |   0     0 |2592  3532 |   0 | 3072    127k|4.0  12 1.0  16|   0     0     0     0 |   0     0    15k    0     0 |   0     0     0     0 


#### crash
May 10 00:57:54 unreliable-N2 kernel: [33472.708404@3] xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.
May 10 00:57:54 unreliable-N2 kernel: [33472.708446@3] xhci-hcd xhci-hcd.0.auto: Assuming host is dying, halting host.
May 10 00:57:54 unreliable-N2 kernel: [33472.735466@3] xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
May 10 00:57:54 unreliable-N2 kernel: [33472.735613@0] hub 2-1:1.0: hub_ext_port_status failed (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735633@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735660@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735667@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735672@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735678@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735680@0] usb 2-1-port3: Cannot enable. Maybe the USB cable is bad?
May 10 00:57:54 unreliable-N2 kernel: [33472.735692@0] usb 2-1-port3: cannot disable (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735704@0] hub 2-1:1.0: hub_ext_port_status failed (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735710@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735715@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735721@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735726@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735732@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735734@0] usb 2-1-port3: Cannot enable. Maybe the USB cable is bad?
May 10 00:57:54 unreliable-N2 kernel: [33472.735740@0] usb 2-1-port3: cannot disable (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735749@0] hub 2-1:1.0: hub_ext_port_status failed (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735755@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735761@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735767@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735772@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735778@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735780@0] usb 2-1-port3: Cannot enable. Maybe the USB cable is bad?
May 10 00:57:54 unreliable-N2 kernel: [33472.735786@0] usb 2-1-port3: cannot disable (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735796@0] hub 2-1:1.0: hub_ext_port_status failed (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735801@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735807@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735812@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735818@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735823@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735826@0] usb 2-1-port3: Cannot enable. Maybe the USB cable is bad?
May 10 00:57:54 unreliable-N2 kernel: [33472.735831@0] usb 2-1-port3: cannot disable (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735838@0] usb 2-1-port3: cannot disable (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735932@0] hub 2-1:1.0: hub_ext_port_status failed (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.918869@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0000: -110
May 10 00:57:54 unreliable-N2 kernel: [33472.918879@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0001: -19
May 10 00:57:54 unreliable-N2 kernel: [33472.918884@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0009: -19
May 10 00:57:54 unreliable-N2 kernel: [33472.918889@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x000a: -19
May 10 00:57:54 unreliable-N2 kernel: [33472.918893@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0004: -19
May 10 00:57:54 unreliable-N2 kernel: [33472.918897@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0005: -19
May 10 00:57:54 unreliable-N2 kernel: [33472.920025@3] usb 1-1: USB disconnect, device number 2
May 10 00:57:54 unreliable-N2 kernel: [33472.921869@3] usb 2-1: USB disconnect, device number 2
May 10 00:57:54 unreliable-N2 kernel: [33472.921877@3] usb 2-1.2: USB disconnect, device number 3
May 10 00:57:54 unreliable-N2 kernel: [33472.921973@3] ax88179_178a 2-1.2:1.0 lan: unregister 'ax88179_178a' usb-xhci-hcd.0.auto-1.2, ASIX AX88179 USB 3.0 Gigabit Ethernet
May 10 00:57:54 unreliable-N2 kernel: [33472.922017@3] ax88179_178a 2-1.2:1.0 lan: Failed to read reg index 0x0002: -19
May 10 00:57:54 unreliable-N2 kernel: [33472.922022@3] ax88179_178a 2-1.2:1.0 lan: Failed to write reg index 0x0002: -19
May 10 00:57:54 unreliable-N2 kernel: [33472.944308@3] ax88179_178a 2-1.2:1.0 lan (unregistered): Failed to write reg index 0x0002: -19
May 10 00:57:54 unreliable-N2 kernel: [33472.944316@3] ax88179_178a 2-1.2:1.0 lan (unregistered): Failed to write reg index 0x0001: -19
May 10 00:57:54 unreliable-N2 kernel: [33472.944320@3] ax88179_178a 2-1.2:1.0 lan (unregistered): Failed to write reg index 0x0002: -19
May 10 00:57:54 unreliable-N2 kernel: [33472.944643@3] usb 2-1.3: USB disconnect, device number 4
May 10 00:57:54 unreliable-N2 kernel: [33472.945375@3] sd 0:0:0:0: [sda] Synchronizing SCSI cache
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       
 10:          0          0          0          0          0          0     GIC-0  29 Level     arch_timer
 11:   19610052    6941173    1430196     410278    4576853     375253     GIC-0  30 Level     arch_timer
 14:          0          0          0          0          0          0     GIC-0  92 Edge      Meson TimerF
 15:      31456          0          0          0          0          0     GIC-0 192 Level     ffe40000.bifrost
 16:          1          0          0          0          0          0     GIC-0 193 Level     ffe40000.bifrost
 17:       7765          0          0          0          0          0     GIC-0 194 Level     ffe40000.bifrost
 19:          3          0          0          0          0          0     GIC-0 241 Edge    
 20:          1          0          0          0          0          0     GIC-0 242 Edge    
 21:    2592116          0          0          0          0          0     GIC-0  40 Edge      ext
 22:       1558          0    4523638          0          0          0     GIC-0  62 Level     xhci-hcd:usb1
 23:          1          0          0          0          0          0     GIC-0  48 Level     amlogic_botg_detect
 24:          0          0          0          0          0          0     GIC-0  63 Level     dwc_otg, dwc_otg_pcd
 25:          0          0          0          0          0          0     GIC-0 232 Edge      meson-g12a-saradc
 26:          0          0          0          0          0          0     GIC-0  67 Edge      ff634800.p_tsensor
 27:          0          0          0          0          0          0     GIC-0  68 Edge      ff634c00.d_tsensor
 28:          0          0          0          0          0          0     GIC-0 247 Edge      ffd1d000.i2c
 30:       1197          0          0          0          0          0     GIC-0  71 Edge      ffd1c000.i2c
 32:        675          0          0          0          0          0     GIC-0 225 Edge      meson_uart
 36:       1938          0          0          0          0          0     GIC-0 184 Edge    
 41:          2          0          0          0          0          0     GIC-0  89 Edge      hdmitx
 42:          0          0          0          0          0          0     GIC-0 235 Edge      hdmi_aocecb
 44:          0          0          0          0          0          0     GIC-0 178 Edge      ge2d
 45:    2029887          0          0          0          0          0     GIC-0  35 Edge      vsync, osd-vsync
 46:          0          0          0          0          0          0     GIC-0 176 Edge      gdc
 49:          0          0          0          0          0          0     GIC-0  75 Edge      vdec-0
 50:          0          0          0          0          0          0     GIC-0  76 Edge      vdec-1
 53:    2029886          0          0          0          0          0     GIC-0 121 Edge      rdma
 54:          0          0          0          0          0          0     GIC-0  88 Edge      osd-vsync-viu2
 55:          5          0          0          0          0          0     GIC-0 223 Edge      meson-aml-mmc
 56:     167757          0          0          0          0          0     GIC-0 222 Edge      meson-aml-mmc
 58:          0          0          0          0          0          0     GIC-0  83 Edge      dmc_monitor
 68:          0          0          0          0          0          0     GIC-0  33 Edge      audiolocker
 69:          0          0          0          0          0          0     GIC-0  78 Edge      pre_di
 70:          0          0          0          0          0          0     GIC-0  72 Edge      post_di
IPI0:   3696745    1875952     798668     378467     224029     208140       Rescheduling interrupts
IPI1:      1362        961       3148       1279       1030        867       Function call interrupts
IPI2:         0          0          0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0          0          0       Timer broadcast interrupts
IPI4:   1596083    1042854     780017     130355     713413     111450       IRQ work interrupts
IPI5:         0          0          0          0          0          0       CPU wake-up interrupts
IPI6:         0          0          0          0          0          0       AML pmu cross interrupts
Err:          0
##### slabinfo
slabinfo - version: 2.1
# name                        <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> <total bytes> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
bio-1                              651    651    768   21    4   499968 : tunables    0    0    0 : slabdata     31     31      0
kcopyd_job                           0      0   3312    9    8        0 : tunables    0    0    0 : slabdata      0      0      0
dm_uevent                            0      0   2632   12    8        0 : tunables    0    0    0 : slabdata      0      0      0
nfs_direct_cache                     0      0    360   22    2        0 : tunables    0    0    0 : slabdata      0      0      0
nfs_commit_data                     23     23    704   23    4    16192 : tunables    0    0    0 : slabdata      1      1      0
nfs_read_data                       36     36    896   18    4    32256 : tunables    0    0    0 : slabdata      2      2      0
nfs_inode_cache                     64     64   1024   16    4    65536 : tunables    0    0    0 : slabdata      4      4      0
fscache_cookie_jar                  92     92     88   46    1     8096 : tunables    0    0    0 : slabdata      2      2      0
squashfs_inode_cache                25     25    640   25    4    16000 : tunables    0    0    0 : slabdata      1      1      0
nf_conntrack                       413    550    320   25    2   176000 : tunables    0    0    0 : slabdata     22     22      0
nfsd_drc                             0      0     96   42    1        0 : tunables    0    0    0 : slabdata      0      0      0
nfsd4_delegations                    0      0    232   17    1        0 : tunables    0    0    0 : slabdata      0      0      0
nfsd4_stateids                       0      0    176   23    1        0 : tunables    0    0    0 : slabdata      0      0      0
nfsd4_files                          0      0    288   28    2        0 : tunables    0    0    0 : slabdata      0      0      0
nfsd4_lockowners                     0      0    400   20    2        0 : tunables    0    0    0 : slabdata      0      0      0
nfsd4_openowners                     0      0    440   18    2        0 : tunables    0    0    0 : slabdata      0      0      0
rpc_inode_cache                    125    125    640   25    4    80000 : tunables    0    0    0 : slabdata      5      5      0
scsi_cmd_cache                      21     21    384   21    2     8064 : tunables    0    0    0 : slabdata      1      1      0
ext4_groupinfo_4k                 1120   1260    144   28    1   181440 : tunables    0    0    0 : slabdata     45     45      0
cfq_io_cq                            0      0    112   36    1        0 : tunables    0    0    0 : slabdata      0      0      0
bsg_cmd                              0      0    312   26    2        0 : tunables    0    0    0 : slabdata      0      0      0
mqueue_inode_cache                  36     36    896   18    4    32256 : tunables    0    0    0 : slabdata      2      2      0
discard_entry                        0      0     88   46    1        0 : tunables    0    0    0 : slabdata      0      0      0
f2fs_inode_cache                     0      0   1096   29    8        0 : tunables    0    0    0 : slabdata      0      0      0
sdcardfs_inode_data_cache            0      0     24  170    1        0 : tunables    0    0    0 : slabdata      0      0      0
sdcardfs_inode_cache                 0      0    608   26    4        0 : tunables    0    0    0 : slabdata      0      0      0
fat_inode_cache                     23     23    712   23    4    16376 : tunables    0    0    0 : slabdata      1      1      0
fat_cache                            0      0     40  102    1        0 : tunables    0    0    0 : slabdata      0      0      0
jbd2_journal_head                 2040   2176    120   34    1   261120 : tunables    0    0    0 : slabdata     64     64      0
jbd2_revoke_table_s                512    512     16  256    1     8192 : tunables    0    0    0 : slabdata      2      2      0
ext2_inode_cache                     0      0    760   21    4        0 : tunables    0    0    0 : slabdata      0      0      0
ext4_inode_cache                223568 226380   1064   30    8 240868320 : tunables    0    0    0 : slabdata   7546   7546      0
ext4_allocation_context            192    192    128   32    1    24576 : tunables    0    0    0 : slabdata      6      6      0
ext4_system_zone                  1122   1122     40  102    1    44880 : tunables    0    0    0 : slabdata     11     11      0
ext4_io_end                       1024   1024     64   64    1    65536 : tunables    0    0    0 : slabdata     16     16      0
ext4_extent_status               34773  44778     40  102    1  1791120 : tunables    0    0    0 : slabdata    439    439      0
mbcache                             73     73     56   73    1     4088 : tunables    0    0    0 : slabdata      1      1      0
fscrypt_info                       768    768     32  128    1    24576 : tunables    0    0    0 : slabdata      6      6      0
fscrypt_ctx                        510    510     48   85    1    24480 : tunables    0    0    0 : slabdata      6      6      0
dio                                100    100    640   25    4    64000 : tunables    0    0    0 : slabdata      4      4      0
pid_namespace                       14     14   2232   14    8    31248 : tunables    0    0    0 : slabdata      1      1      0
posix_timers_cache                  16     16    248   16    1     3968 : tunables    0    0    0 : slabdata      1      1      0
ip4-frags                           17     17    240   17    1     4080 : tunables    0    0    0 : slabdata      1      1      0
UDP                                510    510    960   17    4   489600 : tunables    0    0    0 : slabdata     30     30      0
tw_sock_TCP                        600    600    272   30    2   163200 : tunables    0    0    0 : slabdata     20     20      0
request_sock_TCP                   144    144    328   24    2    47232 : tunables    0    0    0 : slabdata      6      6      0
TCP                                309    368   1984   16    8   730112 : tunables    0    0    0 : slabdata     23     23      0
dquot                              224    224    256   16    1    57344 : tunables    0    0    0 : slabdata     14     14      0
scsi_data_buffer                     0      0     24  170    1        0 : tunables    0    0    0 : slabdata      0      0      0
request_queue                       45     45   2080   15    8    93600 : tunables    0    0    0 : slabdata      3      3      0
blkdev_requests                    444    528    360   22    2   190080 : tunables    0    0    0 : slabdata     24     24      0
blkdev_ioc                         273    273    104   39    1    28392 : tunables    0    0    0 : slabdata      7      7      0
user_namespace                       0      0    432   18    2        0 : tunables    0    0    0 : slabdata      0      0      0
sock_inode_cache                  4925   4925    640   25    4  3152000 : tunables    0    0    0 : slabdata    197    197      0
file_lock_cache                    114    114    208   19    1    23712 : tunables    0    0    0 : slabdata      6      6      0
net_namespace                        5      5   6208    5    8    31040 : tunables    0    0    0 : slabdata      1      1      0
shmem_inode_cache                 5716   5880    664   24    4  3904320 : tunables    0    0    0 : slabdata    245    245      0
taskstats                          144    144    328   24    2    47232 : tunables    0    0    0 : slabdata      6      6      0
proc_inode_cache                  8746   8975    648   25    4  5815800 : tunables    0    0    0 : slabdata    359    359      0
sigqueue                           150    150    160   25    1    24000 : tunables    0    0    0 : slabdata      6      6      0
bdev_cache                         114    114    832   19    4    94848 : tunables    0    0    0 : slabdata      6      6      0
kernfs_node_cache                78574  78574    120   34    1  9428880 : tunables    0    0    0 : slabdata   2311   2311      0
mnt_cache                          936    936    448   18    2   419328 : tunables    0    0    0 : slabdata     52     52      0
inode_cache                      21624  21952    576   28    4 12644352 : tunables    0    0    0 : slabdata    784    784      0
dentry                          299626 304479    192   21    1 58459968 : tunables    0    0    0 : slabdata  14499  14499      0
iint_cache                           0      0    128   32    1        0 : tunables    0    0    0 : slabdata      0      0      0
avc_xperms_data                    256    256     32  128    1     8192 : tunables    0    0    0 : slabdata      2      2      0
avc_node                          2184   2184     72   56    1   157248 : tunables    0    0    0 : slabdata     39     39      0
selinux_file_security             5376   5376     16  256    1    86016 : tunables    0    0    0 : slabdata     21     21      0
selinux_inode_security          248413 256734     80   51    1 20538720 : tunables    0    0    0 : slabdata   5034   5034      0
buffer_head                      97106 123903    104   39    1 12885912 : tunables    0    0    0 : slabdata   3177   3177      0
nsproxy                           2847   2847     56   73    1   159432 : tunables    0    0    0 : slabdata     39     39      0
vm_area_struct                   32828  33373    176   23    1  5873648 : tunables    0    0    0 : slabdata   1451   1451      0
mm_struct                         2052   2052    832   19    4  1707264 : tunables    0    0    0 : slabdata    108    108      0
files_cache                       2116   2116    704   23    4  1489664 : tunables    0    0    0 : slabdata     92     92      0
signal_cache                      3039   3240   1088   30    8  3525120 : tunables    0    0    0 : slabdata    108    108      0
sighand_cache                     1575   1575   2112   15    8  3326400 : tunables    0    0    0 : slabdata    105    105      0
task_struct                       1616   1647   3584    9    8  5902848 : tunables    0    0    0 : slabdata    183    183      0
cred_jar                         12585  12810    192   21    1  2459520 : tunables    0    0    0 : slabdata    610    610      0
anon_vma_chain                   44449  44864     64   64    1  2871296 : tunables    0    0    0 : slabdata    701    701      0
anon_vma                         29495  29733     80   51    1  2378640 : tunables    0    0    0 : slabdata    583    583      0
pid                               4384   4384    128   32    1   561152 : tunables    0    0    0 : slabdata    137    137      0
radix_tree_node                  11368  13412    584   28    4  7832608 : tunables    0    0    0 : slabdata    479    479      0
trace_event_file                  1886   1886     88   46    1   165968 : tunables    0    0    0 : slabdata     41     41      0
ftrace_event_field                6334   8415     48   85    1   403920 : tunables    0    0    0 : slabdata     99     99      0
idr_layer_cache                    525    525   2096   15    8  1100400 : tunables    0    0    0 : slabdata     35     35      0
dma-kmalloc-8192                     0      0   8192    4    8        0 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-4096                     0      0   4096    8    8        0 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-2048                     0      0   2048   16    8        0 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-1024                     0      0   1024   16    4        0 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-512                      0      0    512   16    2        0 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-256                      0      0    256   16    1        0 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-128                      0      0    128   32    1        0 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-64                       0      0     64   64    1        0 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-192                      0      0    192   21    1        0 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-8192                        88     88   8192    4    8   720896 : tunables    0    0    0 : slabdata     22     22      0
kmalloc-4096                       387    408   4096    8    8  1671168 : tunables    0    0    0 : slabdata     51     51      0
kmalloc-2048                      1735   1824   2048   16    8  3735552 : tunables    0    0    0 : slabdata    114    114      0
kmalloc-1024                      4794   4832   1024   16    4  4947968 : tunables    0    0    0 : slabdata    302    302      0
kmalloc-512                       4816   4816    512   16    2  2465792 : tunables    0    0    0 : slabdata    301    301      0
kmalloc-256                       9104  10896    256   16    1  2789376 : tunables    0    0    0 : slabdata    681    681      0
kmalloc-192                       6112   6531    192   21    1  1253952 : tunables    0    0    0 : slabdata    311    311      0
kmalloc-128                       9679   9920    128   32    1  1269760 : tunables    0    0    0 : slabdata    310    310      0
kmalloc-64                       64057  66880     64   64    1  4280320 : tunables    0    0    0 : slabdata   1045   1045      0
kmem_cache_node                   1856   1856     64   64    1   118784 : tunables    0    0    0 : slabdata     29     29      0
kmem_cache                        1808   1808    256   16    1   462848 : tunables    0    0    0 : slabdata    113    113      0

The only outstanding element from stats level I can see is that system memory used/free/cached/buffers was:
946M 1296M 298M 1076M
And that was the highest usage.

Also the xHCI host not responding is interesting. If it would be power surcharge one could expect different type of message.
May 10 00:57:54 unreliable-N2 kernel: [33472.708404@3] xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.

Code: Select all

May 10 00:57:54 unreliable-N2 kernel: [33472.708446@3] xhci-hcd xhci-hcd.0.auto: Assuming host is dying, halting host.
May 10 00:57:54 unreliable-N2 kernel: [33472.735466@3] xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
May 10 00:57:54 unreliable-N2 kernel: [33472.735613@0] hub 2-1:1.0: hub_ext_port_status failed (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735633@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735660@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735667@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735672@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735678@0] usb 2-1-port3: cannot reset (err = -22)
May 10 00:57:54 unreliable-N2 kernel: [33472.735680@0] usb 2-1-port3: Cannot enable. Maybe the USB cable is bad?
And then that Cannot enable. Maybe the USB cable is bad?. But that is maybe a result of system not being able to access USB onboard hub? There is nothing but the ethernet and one HDD - connected. (cable is all good, etc.).

joshua.yang
Posts: 304
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 4 times
Been thanked: 50 times
Contact:

Re: USB disconnects randomly

Unread post by joshua.yang » Fri May 10, 2019 10:59 am

Soleil wrote:
Fri May 10, 2019 4:20 am
Other idea, how about encrypting these drives for next round copy? And maybe a large number of small files? Maybe it has something with memory map issue and pointer getting data from wrong mem area?
Thanks for your idea.

Did you mean LUKS?
Right, I think it'd be better to encrypt it as you said. I'll encrypt the drives using LUKS since it affects the HDD and CPU load.

And I will add a test for copying a large number of small files. I will write a script, that would create 100000, 1MB files and copy them from SSD to HDD infinitely. And will run it concurrently.

About the memory, it seems the other test needed. :)

joshua.yang
Posts: 304
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 4 times
Been thanked: 50 times
Contact:

Re: USB disconnects randomly

Unread post by joshua.yang » Fri May 10, 2019 11:01 am

During 23 hours, I've not seen any error yet. Sad. Definitely, some changes needed for the test. :)

joshua.yang
Posts: 304
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 4 times
Been thanked: 50 times
Contact:

Re: USB disconnects randomly

Unread post by joshua.yang » Fri May 10, 2019 3:32 pm

Now the test drives are LUKS encrypted.

Code: Select all

root@odroid:~# cryptsetup -v status hdd
/dev/mapper/hdd is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 256 bits
  key location: dm-crypt
  device:  /dev/sdb1
  sector size:  512
  offset:  4096 sectors
  size:    976766991 sectors
  mode:    read/write
Command successful.
root@odroid:~# cryptsetup -v status ssd
/dev/mapper/ssd is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 256 bits
  key location: dm-crypt
  device:  /dev/sda1
  sector size:  512
  offset:  4096 sectors
  size:    500112015 sectors
  mode:    read/write
Command successful.
root@odroid:~#
I wrote a new test script using Python3 to perform async tasks concurrently, which are 10 cp -v commands.
It creates 100MB 1000 files and 1MB 1000 files, and copy them from SSD to HDD infinitely.
Attachments
스크린샷 2019-05-10 오후 3.18.58.png
스크린샷 2019-05-10 오후 3.18.58.png (1.05 MiB) Viewed 5493 times

Soleil
Posts: 38
Joined: Tue Apr 30, 2019 9:20 am
languages_spoken: english
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: USB disconnects randomly

Unread post by Soleil » Fri May 10, 2019 5:21 pm

If tests progress is not too advanced, maybe switching to LUKS format as following?
cryptsetup -v --cipher twofish-xts-essiv:sha256 --hash sha512 --use-random --key-size 512 --iter-time=100000 luksFormat /dev/<device>

This would bring it closer to what I've.

One detail at my end - after yesterday crash, after reboot, I've left HDD connected but LUKS hasn't been opened and filesystem hasn't been mounted.
I'll observe memory consumption and time will show system stability.

Memory tests is there equivalent of Memtest86 for Arm platform? This could be a quicker way to rule out physical memory error whilst wouldn't rule out memory leak.

rosenrot
Posts: 33
Joined: Wed Apr 10, 2019 2:37 am
languages_spoken: english
ODROIDs: n2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: USB disconnects randomly

Unread post by rosenrot » Sat May 11, 2019 6:57 pm

For me after around 5 days the USB HDD crashed again with the same error as stated above. So even though there is no error for several days, it is still an open issue.

I switched to another USB port to check if the error occurs more likely for other ports. I know that this is quite unlikely but hey, I don't have a better idea.

elatllat
Posts: 1592
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 25 times
Been thanked: 75 times
Contact:

Re: USB disconnects randomly

Unread post by elatllat » Sat May 11, 2019 7:58 pm

Did you try a faraday cage?
That Wi-Fi story reminds me of when a camera flash was interfering with a package on the RPI (link).
Last edited by elatllat on Sat May 11, 2019 9:56 pm, edited 1 time in total.

rosenrot
Posts: 33
Joined: Wed Apr 10, 2019 2:37 am
languages_spoken: english
ODROIDs: n2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: USB disconnects randomly

Unread post by rosenrot » Sat May 11, 2019 9:40 pm

elatllat wrote:
Sat May 11, 2019 7:58 pm
Did you try a faraday cage?
That Wi-Fi story reminds me of when a camera flash was interfering with a package on the RPI.
Honestly, I hope that I don't have to build a faraday cage around my n2. Do you think this might be a problem for real?

Maybe all the people testing here should state how close their wifi router is. Mine is about 40cm away.

elatllat
Posts: 1592
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 25 times
Been thanked: 75 times
Contact:

Re: USB disconnects randomly

Unread post by elatllat » Sat May 11, 2019 9:55 pm

Other equipment such as microwaves ovens have longer reach and interfere at those frequencies. Reminds me of that Astro physics experiment that was ruined by microwave oven (link).

User avatar
mad_ady
Posts: 6894
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 250 times
Been thanked: 183 times
Contact:

Re: USB disconnects randomly

Unread post by mad_ady » Sat May 11, 2019 10:14 pm

Though not sure it's related to your problem I can attest that baby monitors can rage havoc on 2.4G wifi. Also, if needed a microwave oven makes a great faraday cage while unplugged and with the door closed. But you'll need a battery pack inside the oven

BadPritt
Posts: 161
Joined: Wed Jan 17, 2018 4:08 am
languages_spoken: english dutch french spanish german
ODROIDs: Odroid C2
Odroid XU4Q
Has thanked: 18 times
Been thanked: 17 times
Contact:

Re: USB disconnects randomly

Unread post by BadPritt » Sun May 12, 2019 3:48 am

Hi all.
I'm the poster of the first thread about this problem.
The last weeks I've been using the OTG microUSB for all peripherals like wifi/mouse/keyboard/...
Only my USB3 devices I connect to the USB3 ports.
Like this I've never had the USB's crashing. But I've also transfered to Armbian since I've only been using the microUSB.

So either Armbian doesn't have this problem but Ubuntu Mate, Meveric's Stretch and CoreElec do. Or the microUSB port is more stable than the USB3 ports.

I'm using my N2 for productivity now so I can't do any tests on it. When I can I'll do more tests with Ubuntu Mate, and also connect all my devices to USB3 in Armbian.
I am happy it's stable now. In more than a week haven't had any issue's. While the first weeks all my Blender renders of +10 hours failed because all USB ports failed.
I'll keep an eye on this thread to see if someone finds a good fix.

The disconnecting problem of my wifi seems also fixed. But I'm using microUSB over USB2 for this. So for a hard drive this isn't an option.

I had these problems when using a USB2 hub with keyboard + mouse 2.4Ghz wifi and a 2.4Ghz wifi dongle. And also my USB3 (to SATA) SSD connected to USB3.
If my humble memory serves me right I've only had this when both USB2 + USB3 devices are connected at the same time. Never with only USB2 devices, and never with only USB3 devices. Could be coincidence
Good luck to all.

P.S.: It doesn't seem a power issue. I think it's interference of 2.4Ghz that shuts down the USB, and it isn't restarted after that. USB voltage is always stable. And I've been powering my display the last days with it and haven't had issue's. It's 1.2A(I know, a bit over spec)

elatllat
Posts: 1592
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 25 times
Been thanked: 75 times
Contact:

Re: USB disconnects randomly

Unread post by elatllat » Sun May 12, 2019 6:32 am

What dispay are you using?

BadPritt
Posts: 161
Joined: Wed Jan 17, 2018 4:08 am
languages_spoken: english dutch french spanish german
ODROIDs: Odroid C2
Odroid XU4Q
Has thanked: 18 times
Been thanked: 17 times
Contact:

Re: USB disconnects randomly

Unread post by BadPritt » Sun May 12, 2019 8:26 am

elatllat wrote:
Sun May 12, 2019 6:32 am
What dispay are you using?
This one. It's 1080p and can be powered with 5V USB.
https://www.aliexpress.com/item/13-3-2K ... 4c4dzYMK9P
You can see it here.
https://www.youtube.com/watch?v=Kl60RUdLvDQ&t
Cheers.
NicoD

rosenrot
Posts: 33
Joined: Wed Apr 10, 2019 2:37 am
languages_spoken: english
ODROIDs: n2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: USB disconnects randomly

Unread post by rosenrot » Sun May 12, 2019 7:07 pm

I'm not so sure about the 2.4GHz interference. I do have a 2.4GHz mouse dongle connected to another N2 and it fails seldom when compared to my other N2 having a HDD connected. But hey, I'm open for everything that get's this issue out of my way without having to put my N2 in a Faraday cage.

Anyone here that can confirm that the USB devices don't fail completely? I think the problem is that single USB endpoints suddenly stop working. But this is just my wild guess.

crashoverride
Posts: 4554
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 85 times
Contact:

Re: USB disconnects randomly

Unread post by crashoverride » Sun May 12, 2019 9:21 pm

There are only a couple of USB to SATA bridge chips that are known to work with linux UAS. If its not a JMicron controller, then it likely will not work reliably.
rosenrot wrote:
Tue May 07, 2019 2:28 pm
It is a Seagate Backup Plus 4TB 3.0 which is capable of UASP.
Linux UAS is not compatible with any Seagate devices. You should black-list UAS for the device. Unless you know for certain you are using a JMicron controller, you should just delete the UAS module to prevent data corruption when attaching drives. This is a linux issue, not a hardware issue, and is reproducible on any device running linux.

Note that in the pictures @joshua.yang posted, the Hardkernel USB to SATA interface is used. This device has a JMicron chipset.
https://www.hardkernel.com/shop/usb3-0- ... oard-plus/
This board contains a USB3.0 to SATA3 controller (JMicron JMS578) allowing interfacing of a single 2.5″ SATA HDD/SDD. Archive over ~260MB/sec transfer speed with UAS capability.

rosenrot
Posts: 33
Joined: Wed Apr 10, 2019 2:37 am
languages_spoken: english
ODROIDs: n2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: USB disconnects randomly

Unread post by rosenrot » Sun May 12, 2019 9:50 pm

crashoverride wrote:
Sun May 12, 2019 9:21 pm
There are only a couple of USB to SATA bridge chips that are known to work with linux UAS. If its not a JMicron controller, then it likely will not work reliably.
rosenrot wrote:
Tue May 07, 2019 2:28 pm
It is a Seagate Backup Plus 4TB 3.0 which is capable of UASP.
Linux UAS is not compatible with any Seagate devices. You should black-list UAS for the device. Unless you know for certain you are using a JMicron controller, you should just delete the UAS module to prevent data corruption when attaching drives. This is a linux issue, not a hardware issue, and is reproducible on any device running linux.

Note that in the pictures @joshua.yang posted, the Hardkernel USB to SATA interface is used. This device has a JMicron chipset.
https://www.hardkernel.com/shop/usb3-0- ... oard-plus/
This board contains a USB3.0 to SATA3 controller (JMicron JMS578) allowing interfacing of a single 2.5″ SATA HDD/SDD. Archive over ~260MB/sec transfer speed with UAS capability.

Thanks for letting me know, I was not aware of that. So at least I know now that the USB HDD issue might have another cause.

gahabana
Posts: 29
Joined: Wed Sep 07, 2016 1:47 am
languages_spoken: english
ODROIDs: odroid-c2
odroid-n2
odroid-xu4
Has thanked: 2 times
Been thanked: 0
Contact:

Re: USB disconnects randomly

Unread post by gahabana » Mon May 13, 2019 6:42 am

I have had problems with most USB 3.0/3.1 attached drives on N2.
I have disabled UAS (via usb-quirks method but also by recompiling kernel to remove UAS support for any drive) as I thought that was the culprit.
I do manage to get (with no UAS) working USB-SATA (Sandisk SSD drive connected to VIA 715 bridge chipset) but none of the three USB-C/USB-3.1 to NVME drives work reliably.
I have two with JMicron JMS-583 chipsets and one with ASMedia 2362 chipset. Drives I have tried with all these 3 are Intel 660P 2TB as well as WD 512 GB Black PCIe.
Both drives (NVME) work well in all 3 enclosures connect to both MacBookPro (USB-C ports) as well as to regular PC running Linux (USB 3.1 ports). On Mac and PC i get close to 1GBytes/second transfer speeds and have tried copying 500GB of data either way with no issues.

However on N2 no matter which of the 3 enclosures and which of the two drives I use - IF i transfer data via network (copy from Samba share or via rsync) - i always get device disconnected message (sometimes very quickly sometimes after 50-100GB have been copied) after which USB drive is producing errors or is inaccessible or entire Odroid-N2 freezes (blue light blinks very fast/kernel budder full with msgs i guess).

I have succeeded running tests (iozone) and/or copying files from/to same drive - but whenever it arrives via network - it produces disconnects.

I have removed EVERYTHING else from USB (have only keyboard/mouse in addition to other drives) and did testing with one drive only and i can not get it to work even semi-reliably.

Not sure if it is a SW problem that could be fixed or HW issue that is preventing N2.

(BTW i have tried changing various cables and it did not help).

here is the typical kernel log :

Code: Select all

[    4.098731] rtc-pcf8563 3-0051: low voltage detected, date/time is not reliable.
[    4.099080] rtc-pcf8563 3-0051: rtc core: registered rtc-pcf8563 as rtc0
[    5.515127] nr_pdflush_threads exported in /proc is scheduled for removal
[    5.880891] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    5.963525] meson6-dwmac ff3f0000.ethernet eth0: fail to init PTP.
[    5.963921] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    6.303342] scsi 0:0:0:0: Direct-Access     ASMT     M.2 NVME SSD     0    PQ: 0 ANSI: 6
[    6.306535] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    9.180104] sd 0:0:0:0: [sda] 4000797360 512-byte logical blocks: (2.05 TB/1.86 TiB)
[    9.180320] sd 0:0:0:0: [sda] Write Protect is off
[    9.180324] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[    9.180587] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    9.188654]  sda: sda1
[    9.189727] sd 0:0:0:0: [sda] Attached SCSI disk
[   10.059779] meson6-dwmac ff3f0000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[   10.059799] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   10.218799] FS-Cache: Loaded
[   10.236827] FS-Cache: Netfs 'cifs' registered for caching
[   10.237127] Key type cifs.spnego registered
[   10.237134] Key type cifs.idmap registered
[   10.273772] CIFS VFS: error -95 on ioctl to get interface list
[   11.211477] vout: aml_tvout_mode_work: monitor_timeout
[   11.623816] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[   15.368408] meson_uart ff803000.serial: ttyS0 use xtal(24M) 24000000 change 115200 to 115200
[   62.475504] fb: mem_free_work, free memory: addr:800000
[  570.991212] usb 2-1.3: USB disconnect, device number 3
[  570.993825] blk_update_request: I/O error, dev sda, sector 1700022272
[  570.996650] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 117836261 (offset 0 size 2101248 starting block 212503040)
[  570.996655] Buffer I/O error on device sda1, logical block 212502528
thank you !

elatllat
Posts: 1592
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 25 times
Been thanked: 75 times
Contact:

Re: USB disconnects randomly

Unread post by elatllat » Mon May 13, 2019 7:53 am

May also be worth trying the 5.2 kernel whenever it becomes available. (It is a lot closer to upstream than the 4.9 branch and thus much more tested)

armfreedom
Posts: 3
Joined: Mon Oct 15, 2018 4:49 pm
languages_spoken: english
Has thanked: 1 time
Been thanked: 0
Contact:

Re: USB disconnects randomly

Unread post by armfreedom » Mon May 13, 2019 12:19 pm

Try to configure powertop as a service https://wiki.archlinux.org/index.php/powertop
with powertop --auto-tune it will auto tune the power source of the usb bus.

gahabana
Posts: 29
Joined: Wed Sep 07, 2016 1:47 am
languages_spoken: english
ODROIDs: odroid-c2
odroid-n2
odroid-xu4
Has thanked: 2 times
Been thanked: 0
Contact:

Re: USB disconnects randomly

Unread post by gahabana » Mon May 13, 2019 2:51 pm

armfreedom wrote:
Mon May 13, 2019 12:19 pm
Try to configure powertop as a service https://wiki.archlinux.org/index.php/powertop
with powertop --auto-tune it will auto tune the power source of the usb bus.
tx @armfreedom - did try that too.... powertop --auto-tune requires calibration to be done and running powertop --calibrate for 5mins did not produce output file required by the --auto-tune.

Code: Select all

Cannot load from file /var/cache/powertop/saved_parameters.powertop
It seems to need to run on battery power to do that.
How did you manage to run powertop --auto-tune on your H2 board ?

thank you !

User avatar
odroid
Site Admin
Posts: 32743
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 223 times
Been thanked: 369 times
Contact:

Re: USB disconnects randomly

Unread post by odroid » Mon May 13, 2019 3:07 pm

Can you guys check lsusb and lsusb -t outputs when you meet the disconnection problem?
Does still exist the Genesys Logic USB hub controller?

Code: Select all

root@odroid:~# lsusb
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

root@odroid:~# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M 
I've just saw it disappeared once with a couple of cameras. But I can't reproduce the issue so far.

gahabana
Posts: 29
Joined: Wed Sep 07, 2016 1:47 am
languages_spoken: english
ODROIDs: odroid-c2
odroid-n2
odroid-xu4
Has thanked: 2 times
Been thanked: 0
Contact:

Re: USB disconnects randomly

Unread post by gahabana » Mon May 13, 2019 11:51 pm

odroid wrote:
Mon May 13, 2019 3:07 pm
Can you guys check lsusb and lsusb -t outputs when you meet the disconnection problem?
Does still exist the Genesys Logic USB hub controller?

Code: Select all

root@odroid:~# lsusb
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

root@odroid:~# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M 
I've just saw it disappeared once with a couple of cameras. But I can't reproduce the issue so far.
hi, yes, hubs are still showing though drive is producing errors. Here it is: (Device 4 is USB-SATA-SSD Via chipset drive that works and Device 3 is ASM USB3.1-NVME SSD that produces an error). All i needed to do (machine was rebooted fresh few hours earlier is to do 'ls -ltr' on that drive and i logged the errors and drive is not accessible.
You can see below after the drive fails.

1. Kernel log
2. lsusb
3. lsusb -t

Code: Select all

[  611.319184] fb: osd[0] enable: 0 (kworker/0:1)
[ 1054.958634] CIFS VFS: Free previous auth_key.response = ffffffc0c9292080
[27853.818112] usb 2-1.1: reset SuperSpeed USB device number 3 using xhci-hcd
[27856.958282] sd 0:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[27856.958291] sd 0:0:0:0: [sda] tag#0 Sense Key : Not Ready [current]
[27856.958296] sd 0:0:0:0: [sda] tag#0 Add. Sense: Medium not present
[27856.958303] sd 0:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 01 30 80 00 00 08 00
[27856.958307] blk_update_request: I/O error, dev sda, sector 77952
[27856.965451] EXT4-fs warning (device sda1): htree_dirblock_to_tree:963: inode #2: lblock 0: comm ls: error -5 reading directory block


zh@odroidn2:~$ sudo lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
        |__ Port 3: Dev 4, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M


zh@odroidn2:~$ sudo lsusb
Bus 002 Device 004: ID 2109:0715 VIA Labs, Inc.
Bus 002 Device 003: ID 174c:2362 ASMedia Technology Inc.
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

User avatar
odroid
Site Admin
Posts: 32743
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 223 times
Been thanked: 369 times
Contact:

Re: USB disconnects randomly

Unread post by odroid » Tue May 14, 2019 9:47 am

@gahabana,
Thank you for sharing your test result.
Can you swap the host ports something like Device 3 can be the VIA drive and Device 4 can be ASMedia drive?
Is the ASmedia one always causing the reset SuperSpeed USB device number 3 using xhci-hcd issue?

Also try to clean USB 3.0 super-speed signal pads in the host connectors too.
viewtopic.php?f=147&t=30446&p=218401#p218401

gahabana
Posts: 29
Joined: Wed Sep 07, 2016 1:47 am
languages_spoken: english
ODROIDs: odroid-c2
odroid-n2
odroid-xu4
Has thanked: 2 times
Been thanked: 0
Contact:

Re: USB disconnects randomly

Unread post by gahabana » Tue May 14, 2019 1:57 pm

odroid wrote:
Tue May 14, 2019 9:47 am
@gahabana,
Thank you for sharing your test result.
Can you swap the host ports something like Device 3 can be the VIA drive and Device 4 can be ASMedia drive?
Is the ASmedia one always causing the reset SuperSpeed USB device number 3 using xhci-hcd issue?

Also try to clean USB 3.0 super-speed signal pads in the host connectors too.
viewtopic.php?f=147&t=30446&p=218401#p218401
@odroid,
I have already tried all 4 ports (and 3 different cables that came with 3 different housings (via for sata and asm&jmicron for nvme drives) ... so i can confirm it happens with device 3 and 4 and 1 and 2.
as device (N2) is new i have not yet tried cleaning contacts ... am sceptical but will try that this weekend when i am back as am traveling this week .

PS. these USB - NVME adaptres are cheap these days (20-50$) and nvme drives ubiquitous - is there anyone who is using them and have no issues ?

tx

User avatar
odroid
Site Admin
Posts: 32743
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 223 times
Been thanked: 369 times
Contact:

Re: USB disconnects randomly

Unread post by odroid » Tue May 14, 2019 2:28 pm

@gahabana,
I see. We have just ordered a USB-to-NVME bridge enclosure since we probably have no problem with USB-to-SATA.
I hope we can share the test result early next week.

joshua.yang
Posts: 304
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 4 times
Been thanked: 50 times
Contact:

Re: USB disconnects randomly

Unread post by joshua.yang » Tue May 14, 2019 3:01 pm

My test runs without any issues for 3 days 19 hours.
So to give it more stress I just have changed LUKS options to @Soleil suggested. Also changed HDD from WD 500GB to Seagate 5TB.

Code: Select all

root@odroid:~# cryptsetup -v status ssd
/dev/mapper/ssd is active and is in use.
  type:    LUKS1
  cipher:  twofish-xts-essiv:sha256
  keysize: 512 bits
  key location: dm-crypt
  device:  /dev/sda1
  sector size:  512
  offset:  4096 sectors
  size:    500112015 sectors
  mode:    read/write
Command successful.
root@odroid:~# cryptsetup -v status hdd
/dev/mapper/hdd is active and is in use.
  type:    LUKS1
  cipher:  twofish-xts-essiv:sha256
  keysize: 512 bits
  key location: dm-crypt
  device:  /dev/sdb1
  sector size:  512
  offset:  4096 sectors
  size:    9767534991 sectors
  mode:    read/write
Command successful.

etcetera
Posts: 10
Joined: Fri Feb 15, 2019 11:31 pm
languages_spoken: english, german
ODROIDs: 3 x N2
Location: Switzerland
Has thanked: 11 times
Been thanked: 1 time
Contact:

Re: USB disconnects randomly

Unread post by etcetera » Tue May 14, 2019 11:05 pm

Dear readers


I have some questions about how to blacklist in order to force usb-storage and ignore uas (mostly for external USB 3.0 Seagate drives and smartmontools). This may not be the appropriate topic, but blacklisting has been mentioned above. And the wiki, forum and google do not seem to help further.

So far I am more used to x86 systems running Debian and Ubuntu. There I do successfully the following:
  1. Use lsusb to find out VendorId and ProductId
  2. Put the string options usb-storage quirks=<VendorId>:<ProductId>:u in the file /etc/modprobe.d/blacklist-uas.conf (filename does not matter, only extension).
  3. update-initramfs -u
  4. reboot
On Odroid-N2 and Ubuntu Mate I tried unsuccessfully (sic!) the first two steps from above and then:
  1. update-initramfs -c -k `uname -r`
  2. mkimage -A arm64 -O linux -T ramdisk -C none -a 0 -e 0 -n initrd-`uname -r` -d /boot/initrd.img-`uname -r` /boot/uInitrd-`uname -r`
  3. cp /boot/uInitrd-`uname -r` /media/boot/uInitrd
  4. reboot
On x86 systems running Debian and Ubuntu the command modinfo usb-storage is working, on Odroid-N2 and Ubuntu Mate it is not working as expected. Something must be different, but I do not find out what. This applies to the other kernel module commands, too.

Last I tried unsuccessfully module_blacklist=uas to bootargs in /media/boot/boot.ini.

I would be very happy if I get some hints regarding blacklisting on arm. Any pointers greatly appreciated. Thank you.


Regards
etcetera
Last edited by etcetera on Tue May 14, 2019 11:20 pm, edited 2 times in total.

elatllat
Posts: 1592
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 25 times
Been thanked: 75 times
Contact:

Re: USB disconnects randomly

Unread post by elatllat » Tue May 14, 2019 11:16 pm

etcetera wrote:
Tue May 14, 2019 11:05 pm
...blacklisting...
Try

Code: Select all

echo -n 0bc2:3312:u,0bc2:ab38:u > /sys/module/usb_storage/parameters/quirks
Then re-plug the drives to find out if the quirks or the boot parameters are not working.
These users thanked the author elatllat for the post:
etcetera (Tue May 14, 2019 11:41 pm)

etcetera
Posts: 10
Joined: Fri Feb 15, 2019 11:31 pm
languages_spoken: english, german
ODROIDs: 3 x N2
Location: Switzerland
Has thanked: 11 times
Been thanked: 1 time
Contact:

Re: USB disconnects randomly

Unread post by etcetera » Wed May 15, 2019 12:20 am

Dear @elatllat


Thank you @elatllat. I tried your solution successfully.

Code: Select all

[ 2371.718214] usb 2-1.3: UAS is blacklisted for this device, using usb-storage instead
[ 2371.718373] usb 2-1.3: UAS is blacklisted for this device, using usb-storage instead
Now I am looking for a more permanent solution that survives rebooting.


Regards
etcetera

crashoverride
Posts: 4554
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 85 times
Contact:

Re: USB disconnects randomly

Unread post by crashoverride » Wed May 15, 2019 12:43 am

etcetera wrote:
Wed May 15, 2019 12:20 am
Now I am looking for a more permanent solution that survives rebooting.
It should be the same procedure as for XU4 (add it to the kernel command line):
https://wiki.odroid.com/odroid-xu4/os_i ... s_and_tips
If your external HDD is not working properly, add its VID/PID in boot.ini Boot arguments to disable the UAS function something like this.
usb-storage.quirks=0x0bc2:0x2322:u
These users thanked the author crashoverride for the post:
etcetera (Wed May 15, 2019 2:07 am)

elatllat
Posts: 1592
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 25 times
Been thanked: 75 times
Contact:

Re: USB disconnects randomly

Unread post by elatllat » Wed May 15, 2019 1:26 am

crashoverride wrote:
Wed May 15, 2019 12:43 am
...wiki.odroid...
That could be improved... something like

Code: Select all

echo "user_quirks=\" usb-storage.quirks=0bc2:3312:u,0bc2:ab38:u,0bc2:2322:u\"" >>  /media/boot/boot.ini.default
getBootIni.sh
Would be more robust because disable_vu7 is not appending to hid_quirks and /media/boot/boot.ini gets replaced once a month (unless they fixed that)
These users thanked the author elatllat for the post:
etcetera (Wed May 15, 2019 2:07 am)

joshua.yang
Posts: 304
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 4 times
Been thanked: 50 times
Contact:

Re: USB disconnects randomly

Unread post by joshua.yang » Thu May 16, 2019 12:25 pm

The one-week test has done.

I thought that it is needed to figure out the difference between uas and usb-storage driver. So I have also tested with uas, usb-storage (blacklisted uas driver following @elatllat mentioned) for one day each.

Here the total results.

Environments.
- ubuntu-18.04.2-4.9-minimal-odroid-n2-20190329.img
- Connect one SSD and one HDD to N2 using USB-SATA bridge (UAS capable)
- Format them EXT4
- Copy 100MB 1000, 1MB 1000 files from SSD to HDD, 10 cp commands run asynchronously

Results.
- General: 1 day, OK
- LUKS Default option: 3 days 19 hours, OK
- LUKS @Soleil option: 20 hours, some messages repeat on dmesg (seems like it relates to memory allocation) but it kept on testing. Not fatal.
- LUKS @Soleil option + usb-storage driver: 1 day, OK

No fatal issues found for these tests, such as disconnecting drives, disappearing USB hub, ...

Today we got a USB-NVMe bridge device. So we are plan to test with this, the test is that copying files from SSD to NVMe.
I will encrypt this NVMe drive too and will use usb-storage driver. This test starts today.

Any ideas welcome.
Last edited by joshua.yang on Thu May 16, 2019 2:13 pm, edited 1 time in total.

gahabana
Posts: 29
Joined: Wed Sep 07, 2016 1:47 am
languages_spoken: english
ODROIDs: odroid-c2
odroid-n2
odroid-xu4
Has thanked: 2 times
Been thanked: 0
Contact:

Re: USB disconnects randomly

Unread post by gahabana » Thu May 16, 2019 1:57 pm

joshua.yang wrote:
Thu May 16, 2019 12:25 pm
The one-week test has done.

I thought that it is needed to figure out the difference between uas and usb-storage driver. So I have also tested with uas, usb-storage (blacklisted uas driver following @elatllat mentioned) for one day each.

Here the total results.

Environments.
- ubuntu-18.04.2-4.9-minimal-odroid-n2-20190329.img
- Connect one SSD and one HDD to N2 using USB-SATA bridge (UAS capable)
- Format them EXT4
- Copy 100MB 1000, 1MB 1000 files from SSD to HDD, 10 cp commands run asynchronously

Results.
- General: 1 day, OK
- LUKS Default option: 3 days 19 hours, OK
- LUKS @Soleil option: 20 hours, some messages repeat on dmesg (seems like it relates to memory allocation) but it kept on testing. Not fatal.
- LUKS @Soleil option + usb-storage driver: 1 day, OK

No fatal issues found for these tests, such as disconnecting drives, disappearing USB hub, ...

Today we get a USB-NVMe bridge device. So we are plan to test with this, the test is that copying files from SSD to NVMe.
I will encrypt this NVMe drive too and will use usb-storage driver. This test starts today.

Any ideas welcome.
hi @joshua - looking forward to your tests !
My experience is that sometimes i could reliable copy 10s of GB to the drive (USB-NVME) - from/to the same drive itself or run 5 min iozone tests that often it was ennough to copy from samba share to the drive itself to get it to disconnect fairly quickly. rsync from the server to the N2 the same.

My test is portion of my music collection - .flac files average size of 20-40MBytes each - collection has about 350 GB so about 10,000 files ... sometimes it fails very quickly sometimes it copies 50-100GB and then it fails (USB disconnect). Of course all with usb-storage drivers as uas does not work well.

That would be my suggestion (and it mimics real-world use case :)

thank you !!!

joshua.yang
Posts: 304
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 4 times
Been thanked: 50 times
Contact:

Re: USB disconnects randomly

Unread post by joshua.yang » Thu May 16, 2019 3:28 pm

Tested USB-NVMe bridge device uses JMS583 chipset and it is UASP capable. But will test with usb-storage driver.

The product is: http://www.revuahn.com/UX970EVO (Sorry for this information is Korean)
Install Samsung 960 EVO 250GB into it.

And here is the lsusb results.

Code: Select all

root@odroid:~# lsusb
Bus 002 Device 004: ID 152d:0583 JMicron Technology Corp. / JMicron USA Technology Corp.
Bus 002 Device 003: ID 152d:0578 JMicron Technology Corp. / JMicron USA Technology Corp.
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@odroid:~# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
        |__ Port 4: Dev 4, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
This also LUKS encrypted using @Soleil option, and the test is just started.
스크린샷 2019-05-16 오후 3.28.11.png
스크린샷 2019-05-16 오후 3.28.11.png (1.12 MiB) Viewed 4686 times
gahabana wrote:
Thu May 16, 2019 1:57 pm
hi @joshua - looking forward to your tests !
My experience is that sometimes i could reliable copy 10s of GB to the drive (USB-NVME) - from/to the same drive itself or run 5 min iozone tests that often it was ennough to copy from samba share to the drive itself to get it to disconnect fairly quickly. rsync from the server to the N2 the same.

My test is portion of my music collection - .flac files average size of 20-40MBytes each - collection has about 350 GB so about 10,000 files ... sometimes it fails very quickly sometimes it copies 50-100GB and then it fails (USB disconnect). Of course all with usb-storage drivers as uas does not work well.

That would be my suggestion (and it mimics real-world use case :)

thank you !!!
Hope this could reproduce the issue. :)

Soleil
Posts: 38
Joined: Tue Apr 30, 2019 9:20 am
languages_spoken: english
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: USB disconnects randomly

Unread post by Soleil » Fri May 17, 2019 11:21 pm

Further test is pretty interesting. System was find for +24h with USB drive connected, but not mounted (luks wasn't opened).
Then luks was opened and filesystem (2TB) mounted.
Slow write + some read was fine. In fact ~7GB was written + some other random data as backup processes were running over night. All was good outside of some USB related messages from GigabitEthernet USB adapter.

That ~7GB of written data was over the course of 5h rsync from remote system.

When system crashed and crashed instantly was when rsync -c was run - what it does it runs checksum on sliding window of the file. Literally couple of seconds after that USB issues occurred - nothing connected to USB is visible.
Looking through UAS problems, it doesn't seem to be related - no UAS specific type of issues are visible.

See below crash and note how quickly after fresh boot was it:

Code: Select all

[  239.146068] device-mapper: uevent: version 1.0.3
[  239.148247] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
[  353.300092] EXT4-fs (dm-0): recovery complete
[  353.375350] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
[  546.288182] xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.
[  546.288196] xhci-hcd xhci-hcd.0.auto: Assuming host is dying, halting host.
[  546.307633] xhci-hcd xhci-hcd.0.auto: Host not halted after 16000 microseconds.
[  546.307652] xhci-hcd xhci-hcd.0.auto: Non-responsive xHCI host is not halting.
[  546.307656] xhci-hcd xhci-hcd.0.auto: Completing active URBs anyway.
[  546.307851] ax88179_178a 2-1.1:1.0 lan: Failed to read reg index 0x0000: -110
[  546.307973] ax88179_178a 2-1.1:1.0 lan: Failed to read reg index 0x0001: -108
[  546.307986] ax88179_178a 2-1.1:1.0 lan: Failed to read reg index 0x0009: -108
[  546.308001] ax88179_178a 2-1.1:1.0 lan: Failed to read reg index 0x000a: -108
[  546.308020] ax88179_178a 2-1.1:1.0 lan: Failed to read reg index 0x0004: -108
[  546.308038] ax88179_178a 2-1.1:1.0 lan: Failed to read reg index 0x0005: -108
[  546.309808] xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
[  546.309928] hub 2-1:1.0: hub_ext_port_status failed (err = -108)
[  546.309983] usb 2-1-port2: cannot reset (err = -108)
[  546.310023] usb 2-1-port2: cannot reset (err = -108)
[  546.310057] usb 2-1-port2: cannot reset (err = -108)
[  546.310092] usb 2-1-port2: cannot reset (err = -108)
[  546.310124] usb 2-1-port2: cannot reset (err = -108)
[  546.310139] usb 2-1-port2: Cannot enable. Maybe the USB cable is bad?
[  546.310179] usb 2-1-port2: cannot disable (err = -108)
[  546.310239] hub 2-1:1.0: hub_ext_port_status failed (err = -108)
[  546.310273] usb 2-1-port2: cannot reset (err = -108)
[  546.310304] usb 2-1-port2: cannot reset (err = -108)
[  546.310336] usb 2-1-port2: cannot reset (err = -108)
[  546.310365] usb 2-1-port2: cannot reset (err = -108)
[  546.310396] usb 2-1-port2: cannot reset (err = -108)
[  546.310410] usb 2-1-port2: Cannot enable. Maybe the USB cable is bad?
[  546.310444] usb 2-1-port2: cannot disable (err = -108)
[  546.310497] hub 2-1:1.0: hub_ext_port_status failed (err = -108)
[  546.310528] usb 2-1-port2: cannot reset (err = -108)
[  546.310559] usb 2-1-port2: cannot reset (err = -108)
[  546.310589] usb 2-1-port2: cannot reset (err = -108)
[  546.310619] usb 2-1-port2: cannot reset (err = -108)
[  546.310649] usb 2-1-port2: cannot reset (err = -108)
[  546.310663] usb 2-1-port2: Cannot enable. Maybe the USB cable is bad?
[  546.310695] usb 2-1-port2: cannot disable (err = -108)
[  546.310745] hub 2-1:1.0: hub_ext_port_status failed (err = -108)
[  546.310776] usb 2-1-port2: cannot reset (err = -108)
[  546.310807] usb 2-1-port2: cannot reset (err = -108)
[  546.310837] usb 2-1-port2: cannot reset (err = -108)
[  546.310867] usb 2-1-port2: cannot reset (err = -108)
[  546.310897] usb 2-1-port2: cannot reset (err = -108)
[  546.310910] usb 2-1-port2: Cannot enable. Maybe the USB cable is bad?
[  546.310941] usb 2-1-port2: cannot disable (err = -108)
[  546.310974] usb 2-1-port2: cannot disable (err = -108)
[  546.311181] hub 2-1:1.0: hub_ext_port_status failed (err = -108)
[  546.404239] sd 0:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
[  546.404261] sd 0:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 4c e3 ef 00 00 01 00 00
[  546.404268] blk_update_request: I/O error, dev sda, sector 1290006272
[  546.488231] sd 0:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
[  546.488252] sd 0:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 4c e3 f0 00 00 01 00 00
[  546.488261] blk_update_request: I/O error, dev sda, sector 1290006528
[  546.511920] usb 1-1: USB disconnect, device number 2
[  546.513385] usb 2-1: USB disconnect, device number 2
[  546.513423] usb 2-1.1: USB disconnect, device number 3
[  546.513918] ax88179_178a 2-1.1:1.0 lan: unregister 'ax88179_178a' usb-xhci-hcd.0.auto-1.1, ASIX AX88179 USB 3.0 Gigabit Ethernet
[  546.514387] ax88179_178a 2-1.1:1.0 lan: Failed to read reg index 0x0002: -19
[  546.514406] ax88179_178a 2-1.1:1.0 lan: Failed to write reg index 0x0002: -19
[  546.556268] ax88179_178a 2-1.1:1.0 lan (unregistered): Failed to write reg index 0x0002: -19
[  546.556277] ax88179_178a 2-1.1:1.0 lan (unregistered): Failed to write reg index 0x0001: -19
[  546.556282] ax88179_178a 2-1.1:1.0 lan (unregistered): Failed to write reg index 0x0002: -19
[  546.556682] usb 2-1.2: USB disconnect, device number 4
[  546.557606] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[  546.560209] blk_update_request: I/O error, dev sda, sector 1950617632
[  546.561388] blk_update_request: I/O error, dev sda, sector 1290006272
[  546.568010] blk_update_request: I/O error, dev sda, sector 1950617640
[  546.574630] Aborting journal on device dm-0-8.
[  546.580328] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[  546.580343] blk_update_request: I/O error, dev sda, sector 1950617600
[  546.585762] blk_update_request: I/O error, dev sda, sector 1950617600
[  546.592226] Buffer I/O error on dev dm-0, logical block 243826688, lost sync page write
[  546.600408] JBD2: Error -5 detected when updating journal superblock for dm-0-8.
[  546.600472] scsi 0:0:0:0: rejecting I/O to dead device
[  546.600477] blk_update_request: I/O error, dev sda, sector 1290006272
lsusb after crash"

Code: Select all

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 Unused
  bDeviceProtocol         3 
  bMaxPacketSize0         9
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0003 3.0 root hub
  bcdDevice            4.09
  iManufacturer           3 
  iProduct                2 
  iSerial                 1 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           31
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
        bMaxBurst               0

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 Unused
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0002 2.0 root hub
  bcdDevice            4.09
  iManufacturer           3 
  iProduct                2 
  iSerial                 1 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           25
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
And lsusb shortly after boot (all devices visible):

Code: Select all

Bus 002 Device 004: ID 0480:0900 Toshiba America Inc 
Bus 002 Device 003: ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc. 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

It all suggests that it is the Genesys Logic 4-port hub which fails here.

Soleil
Posts: 38
Joined: Tue Apr 30, 2019 9:20 am
languages_spoken: english
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: USB disconnects randomly

Unread post by Soleil » Sat May 18, 2019 1:13 am

Another simplified test.
Fresh boot and simple dd from device to /dev/null.
Crashes USB hub sometimes after ~300MB sometimes later:

Code: Select all

# dd if=/dev/sda bs=1024k count=7000 status=progress >/dev/null
5143265280 bytes (5.1 GB, 4.8 GiB) copied, 96 s, 53.6 MB/s
# after this Genesys hub disappeared from the system
Please suggest what other tests are to be done.
Happy to use debug mode kernel if required to get more details for you guys.

User avatar
tobetter
Posts: 4265
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 59 times
Been thanked: 235 times
Contact:

Re: USB disconnects randomly

Unread post by tobetter » Sat May 18, 2019 1:46 am

Soleil wrote:
Sat May 18, 2019 1:13 am
Another simplified test.
Fresh boot and simple dd from device to /dev/null.
Crashes USB hub sometimes after ~300MB sometimes later:

Code: Select all

# dd if=/dev/sda bs=1024k count=7000 status=progress >/dev/null
5143265280 bytes (5.1 GB, 4.8 GiB) copied, 96 s, 53.6 MB/s
# after this Genesys hub disappeared from the system
Please suggest what other tests are to be done.
Happy to use debug mode kernel if required to get more details for you guys.
You are not able to see this once USB is crashed?

Code: Select all

Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.

Soleil
Posts: 38
Joined: Tue Apr 30, 2019 9:20 am
languages_spoken: english
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: USB disconnects randomly

Unread post by Soleil » Sat May 18, 2019 6:09 am

Exactly, all what I see is:

Code: Select all

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Full extract in earlier post (lsusb -v)

Happy to run additional debug kernel to dump some USB registers. I've feeling that it is somehow related to MB/s pumped through USB hub, as ~7GB file was successfully stored on this very same HDD on encrypted FS, just slowly as was transferred from remote location.
Or maybe it is not about speed but ok to "write" but not ok to "read" from the connected device?
Will re-test it.

Soleil
Posts: 38
Joined: Tue Apr 30, 2019 9:20 am
languages_spoken: english
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: USB disconnects randomly

Unread post by Soleil » Sat May 18, 2019 6:21 am

Very interesting observation.
7GB write to encrypted disk went perfectly fine:

Code: Select all

# dd if=/dev/zero of=./zeros.dd bs=1024k count=7000 status=progress 
7337934848 bytes (7.3 GB, 6.8 GiB) copied, 90 s, 81.5 MB/s
7000+0 records in
7000+0 records out
7340032000 bytes (7.3 GB, 6.8 GiB) copied, 90.9651 s, 80.7 MB/s
Read of that zero filled file and dumping it to /dev/null caused crash.

Code: Select all

# dd if=./zeros.dd  of=/dev/zero bs=1024k status=progress 
1184890880 bytes (1.2 GB, 1.1 GiB) copied, 30 s, 39.5 MB/s 
Update then hung up as, yes, Genosys hub disappeared.

Interestingly first one to raise issues is this GigabiEthernet USB3 adapter

Code: Select all

[18302.603828] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18302.605136] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18302.611707] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18302.622074] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18302.624971] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18302.687678] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18302.688806] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18302.695826] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18302.706001] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18302.916354] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18313.681290] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18334.096039] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18334.097214] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18334.103999] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18334.110709] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18343.219360] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18343.868587] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18343.869759] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18343.880190] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18343.883054] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18343.893773] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18344.017117] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
[18344.018252] ax88179_178a 2-1.1:1.0 lan: kevent 2 may have been dropped
shortly after followed by

Code: Select all

[18419.860503] xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.
[18419.860545] xhci-hcd xhci-hcd.0.auto: Assuming host is dying, halting host.
[18419.887171] xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
[18419.887333] hub 2-1:1.0: hub_ext_port_status failed (err = -22)
[18419.887369] usb 2-1-port2: cannot reset (err = -22)
[18419.887392] usb 2-1-port2: cannot reset (err = -22)
[18419.887413] usb 2-1-port2: cannot reset (err = -22)
[18419.887433] usb 2-1-port2: cannot reset (err = -22)
[18419.887452] usb 2-1-port2: cannot reset (err = -22)
[18419.887462] usb 2-1-port2: Cannot enable. Maybe the USB cable is bad?
Important - write was more than double the read rate which goes against any logic. Encryption is usually slower than decryption. It is not CPU limitation as CPU was idle and doesn't seem to be HDD related.
It all points down to something within kernel module or Genesys HUB.

And some links:
- no solution, though similar symptoms - https://forum.turris.cz/t/how-to-reset-usb-bus/5248 xHCI host not responding to stop endpoint command
- no solution, somehow similar - https://bbs.archlinux.org/viewtopic.php?id=236536

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 1 guest