I2C driver floods serial console with errors

Moderators: mdrjr, odroid

I2C driver floods serial console with errors

Unread postby enter_the_bot » Tue Apr 21, 2015 7:36 pm

I've been trying to hook up some I2C sensors.

After a while serial console gets flooded with error messages like this
Code: Select all
i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x76

It also shuts network connection.

I've been trying to fix it myself but it hasn't turned out to be feasible at the moment. I'd stumbled upon serious issues with the kernel module. One of them has been fixed with the following PR https://github.com/hardkernel/linux/pull/102 that fixes an Oops upon I2C kernel module removal. By the way, thank you for the quick response on that issue.

I've debugged I2C for several days and the I've noticed that
Code: Select all
aml_i2c_wait_ack(struct aml_i2c *i2c)
fails from to time.

ctrl->error byte in control register seems to be the cause of the problem. Unfortunately, not always. Sometimes the function fails due to timeout.

Once it's corrupted I2C cannot restore its state. My first guess was incorrect I2C clock stretching implementation but I failed to grasp if that's true.

I've tried to make this driver work in the I2C_DELAY_MODE mostly but also tried to switch to other modes like I2C_POLLING_MODE and I2C_INTERRUPT_MODE.

Code: Select all
echo 2 > /sys/class/i2c1/mode


I2C_INTERRUPT_MODE is not supported at all at the moment and I2C_POLLING_MODE fails not gracefully (but I haven't really looked into why).

I'm looking forward to the feedback.
enter_the_bot
 
Posts: 3
Joined: Thu Apr 09, 2015 8:58 pm
languages_spoken: english
ODROIDs: Odroid-C1

Re: I2C driver floods serial console with errors

Unread postby ralfz » Wed May 06, 2015 8:41 pm

Hi there,

I've a similar behavior on the I2C bus.
After a time (several hours) the bus crashes entirely.

Code: Select all
i2c i2c-1: [aml_i2c_xfer] error ret = -110 (-ETIMEOUT) token 1, master_no(1) 100K addr 0x28


To write data to the bus I use i2c_smbus_write_byte_data.

I hope somebody could give me a hint.

Best regards
ralfz
 
Posts: 65
Joined: Wed Mar 11, 2015 12:45 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby odroid » Thu May 07, 2015 11:57 am

Does that issues occur when the I2C device is not responding?
We couldn't reproduce the issue with our weather board examples over 24 hours test.
http://odroid.com/dokuwiki/doku.php?id=en:c1_wb_16x2lcd
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby ralfz » Wed Jul 08, 2015 5:31 pm

Hello,

now I figured out the first error which occurs is always

Code: Select all
<3>[40867.812364@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x38


This error occurs device independent.

I also tried to simulate a transmission error by pulling the data signal to ground during a transmission.
If I do this the error is printed to dmesg. After the disruption the system works fine. (So I have no idea how the produce this behavior)

The error message comes out from aml_i2c.c::aml_i2c_xfer near line 616 (if (ret) {...} ). I set a call to the debug method within this block and got the following:
<4>[59250.113915@3] addr [8c70038] token tag : 0,0,0,0,0,0,0,0,clk_delay 31c, clk is 50K
<4>[59250.113964@3] w0 3, w1 0, r0 762002ff, r1 e03886b, cur_token 1, rd cnt 0, status 0,error 1, ack_ignore 0,start 1
<4>[59250.113985@3] [aml_i2c_dbg] manual_en, why?

GPIO_TRIGGER:
<3>[59250.114001@3] i2c i2c-1: MY_DBG : GPIO - trig_gpio:(-1).

Usual statement:
<3>[59250.114020@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x38


Is this information usefull to help?
Or which further information would be useful?

EDIT:
=====================================================================================
Well after further debugging it seems to be exactly the same problem as enter_the_bot.
The bus is running at 100kHz. In the log I saw the following: clk is 50K
But this shouldn't disturp the bus...
=====================================================================================

EDIT_2:
=====================================================================================

I added a further dbg information.
In the log I found a "division by zero" similar to the blue colored line:

<4>[ 3391.967978@1] addr [8c70028] token tag : 0,0,0,0,0,0,0,0,Division by zero in kernel.


<4>[ 4960.623210@3] addr [8c70028] token tag : 0,0,0,0,0,0,0,0,clk_delay 8, clk is 4980K
<4>[ 4960.623270@3] w0 2605, w1 0, r0 72200200, r1 f02c41c, cur_token 0, rd cnt 0, status 1,error 0, ack_ignore 0,start 1
<4>[ 4960.623292@3] [aml_i2c_dbg] manual_en, why?
<3>[ 4960.623310@3] i2c i2c-1: MY DBG : GPIO - trig_gpio: (-1), 3-Read / 7-Write: (7) .
<3>[ 4960.623334@3] i2c i2c-1: [aml_i2c_xfer] error ret = -110 (-ETIMEOUT) token 1, master_no(1) 100K addr 0x28
<6>[ 4960.623380@3] nfc_clrc663 1-0028: write_byte: WARNING: rewrite REG <5>, VAL <38>, RETURN-VALUE: <-11>
<4>[ 4961.027236@3] addr [8c70028] token tag : 0,0,0,0,0,0,0,0,clk_delay 31c, clk is 50K
<4>[ 4961.027293@3] w0 2605, w1 0, r0 72200200, r1 f02c41c, cur_token 0, rd cnt 0, status 1,error 0, ack_ignore 0,start 1
<4>[ 4961.027315@3] [aml_i2c_dbg] manual_en, why?
<3>[ 4961.027332@3] i2c i2c-1: MY DBG: GPIO - trig_gpio: (-1), 3-Read / 7-Write: (7) .
<3>[ 4961.027355@3] i2c i2c-1: [aml_i2c_xfer] error ret = -110 (-ETIMEOUT) token 1, master_no(1) 100K addr 0x28


Further the crash seems to occur only in the write part of the aml_i2c driver. (function: aml_i2c_write() ) The "-ETIMEOUT" is
reported. That means the ctrl-status flag isn't IDLE and will never be IDLE again after the error occured.

I compiled the driver as module. insmod driver - rmmod driver doesnt work. The system will freeze instantly.

The unchangeable state comes from : i2c->master_regs->i2c_ctrl. I think the used struct is just used for reading a specific register in the cpu.
Which mechanisms are able to change the status value? Is it only the I2C hardware controller itselfe?
The struct can be found in aml_i2c.h
Code: Select all
/*I2C_CONTROL_REG   0x2140*/
struct aml_i2c_reg_ctrl { ...

...
unsigned int status:1;
...


Also the scarry manual_enable flag can be found there.
=====================================================================================

Thanks, ralfz
ralfz
 
Posts: 65
Joined: Wed Mar 11, 2015 12:45 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby ralfz » Thu Jul 16, 2015 9:13 pm

Hey together,

cause the i2c hangs from time to time in the "finit infinity timeout mode" (the status bit [Bit: 2] in register 2140 at address 0xfe108500 will never set to IDLE[0] again) and the
value of the aml_i2c_reg_ctrl bit field looks strange, I did something weird.

Example:
Code: Select all
bit representation:
           | 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 | 23 | ME | 21 | 20 | 19 | 18 | 17 | 16 | 15 | 14 | 13 | 12 | 11 | 10 |  9 |  8 |  7 |  6 |  5 |  4 |  3 |Stat|  1 |Star|
           | 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 | 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 | 15 | 14 | 13 | 12 | 11 | 10 |  9 |  8 |  7 |  6 |  5 |  4 |  3 |  2 |  1 |  0 |
0x00408005    0    0    0    0    0    0    0    0    0    1    0    0    0    0    0    0    1    0    0    0    0    0    0    0    0    0    0    0    0    1    0    1

As you can see, the Bit 22 (Manual enable) is set. I think this leads to the strange behavior.

What I did:
If the i2c-driver hangs in the endless timeout a method will be triggered, which was implemented by me within the aml i2c driver context.
Well I know the following code isn't the best way to solve this!!
Code: Select all
static void my_i2c_reset(struct aml_i2c *i2c) {

   struct aml_i2c_reg_ctrl* ctrl;

   ctrl = (struct aml_i2c_reg_ctrl*) &(i2c->master_regs->i2c_ctrl);

   dev_err(i2c->dev, "MY DBG : -- I2C RESET-- current CTRL   ADDR: (0x%8x) .\n", (unsigned int) ctrl);

   dev_err(i2c->dev, "MY DBG : -- I2C RESET-- before stop,   CTRL: (0x%8x) .\n", i2c->master_regs->i2c_ctrl);
   
        aml_i2c_stop(i2c);

   dev_err(i2c->dev, "MY DBG : -- I2C RESET-- aml stopped,   CTRL: (0x%8x) .\n", i2c->master_regs->i2c_ctrl);
   ctrl->ack_ignore = 0;
   ctrl->manual_en  = 0;
   ctrl->start     = 0;
   ctrl->start     = 0;
   udelay(i2c->wait_xfer_interval);
   ctrl->start     = 1;
   udelay(i2c->wait_xfer_interval);
   ctrl->start     = 1;
   udelay(i2c->wait_xfer_interval);
   ctrl->start     = 1;
   dev_err(i2c->dev, "MY DBG : -- I2C RESET-- after man rst, CTRL: (0x%8x) .\n", i2c->master_regs->i2c_ctrl);
}

Since I made a call to this method the driver came back to live after the timeouts.

I made just one change to the original code checked out from hardkernels repository. I added two clients in the DTS:
Code: Select all
...
i2c@c1108500{
        compatible = "amlogic,aml_i2c";
        #address-cells = <1>;
        #size-cells = <0>;
        dev_name = "i2c-A";
        status = "ok";
        reg = <0xc1108500 0x20>;
        device_id = <1>;
        pinctrl-names="default";
        pinctrl-0 = <&a_i2c_master>;
        use_pio = <0>;
        master_i2c_speed = <100000>;
       
         my_i2c_dev1: my_i2c_dev1@28 {
            compatible = "hardkernel,nfc_clrc663";
            dev_name = "nfc_clrc663";
            
            interrupt-controller;
            interrupt-parent = <&gic>;
            interrupts = <0 64 0>;
         
            gpio_irq = "GPIOX_17";   // pin 114   
            reg = <0x28>;
         };
       
         atmel_mxt_ts: atmel_mxt_ts@4c {
            compatible = "atmel,maxtouch";
            dev_name = "atmel_mxt_ts";
            
            interrupt-controller;
            interrupt-parent = <&gic>;
            interrupts = <0 65 0>; 
            
            reg = <0x4c>;      
         };                   
    };
...

Could somebody confirm I did this in a right way?

But also without loading the corresponding drivers and just using the userspace i2c interface I got into the timeout trouble.

How can I find out what leads to the confusion of the i2c controller?

regards
ralfz
 
Posts: 65
Joined: Wed Mar 11, 2015 12:45 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby danneman » Tue Jul 28, 2015 6:56 am

Hi.

I am so glad I found this thread. I have been in to a problem with i2c last couple of weeks. I am using a script with python WiringPi and it works perfect with Raspberry Pi but running on Odroid it works for a couple of hours and then it fails completly and a sometimes even reboot is necessary. I haven't even thought about a module aml_i2c problem so now I will look it to some logfiles.

Has anybody found the bug?

/Daniel
danneman
 
Posts: 7
Joined: Tue Jul 28, 2015 6:48 am
languages_spoken: english
ODROIDs: C1

Re: I2C driver floods serial console with errors

Unread postby danneman » Sun Aug 02, 2015 8:51 am

Hi.

Ralfz. Is your workaround stable? How fast will your i2c driver return to normal state after timeout?

We are using i2c in a time sensitive application and can't have more than a couple of 100 of milliseconds delay on i2c bus so this is a big issue for us. Now during a 24 hour endurancy test the bus is non working for total of 30 minutes. So the bus is obviously returning to normal state but what is causing the bus to stop working?

I have all latest softwares installed and using Ubuntu.

/Daniel
danneman
 
Posts: 7
Joined: Tue Jul 28, 2015 6:48 am
languages_spoken: english
ODROIDs: C1

Re: I2C driver floods serial console with errors

Unread postby ralfz » Mon Aug 03, 2015 4:11 pm

Hi,

I assume that this workaround isn't stable . For further debugging I'd like to get KGDB to work. (http://forum.odroid.com/viewtopic.php?f=113&t=14941)
(I have to know which part of the whole software is responsible for the "manual mode")
My application polls I2C data every 0,5 sec. If an error occurs on the I2C bus the app stops polling to restore the driver/controller.

In total - I haven't solved the problem yet :/ :/

Let me know, if you have some news.


Best regards
ralfz
 
Posts: 65
Joined: Wed Mar 11, 2015 12:45 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby danneman » Tue Oct 06, 2015 4:46 am

Any update on i2c issue.

Is this problem fixed in new kernel?
danneman
 
Posts: 7
Joined: Tue Jul 28, 2015 6:48 am
languages_spoken: english
ODROIDs: C1

Re: I2C driver floods serial console with errors

Unread postby puccini » Tue Oct 13, 2015 11:19 pm

I have been struggling with that issue for quite some time now and nothing worked so far. In some of the cases the hang disappears after some minutes while in others it takes half a day and more to resolve by itself (in case I decide not to restart it)
What I have noticed is that the odroid C1 holds the SCL line down when that happens, that is not a standard condition for I2C protocol as far as I have checked.
I was thinking to experiment with the module a bit but could not find how to compile it without the rest of the image.

rgrds
puccini
 
Posts: 4
Joined: Tue Oct 13, 2015 10:37 pm
languages_spoken: english
ODROIDs: Odroid C1

Re: I2C driver floods serial console with errors

Unread postby danneman » Wed Oct 14, 2015 7:11 am

I tested latest kernel and software and the bug is still crawling.

Where can I find source code for current i2c driver? I would like to dig in to this issue as well...

/Daniel
danneman
 
Posts: 7
Joined: Tue Jul 28, 2015 6:48 am
languages_spoken: english
ODROIDs: C1

Re: I2C driver floods serial console with errors

Unread postby odroid » Wed Oct 14, 2015 10:49 am

We've connected a TCA6416 I2C IO Expander IC and ran a small application software whick keep writing data to toggle a few LEDs for 3 days.
But we couldn't find any issue. When we shorted the SDA or SCL to ground, we could see the error message.
Once we open it, the application keeps working well withour restart.

Anyway, I will post the source code location soon.

Update: The I2C driver is placed on
https://github.com/hardkernel/linux/blob/odroidc-3.10.y/drivers/amlogic/i2c/aml_i2c.c
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby puccini » Wed Oct 14, 2015 3:34 pm

Could it be because of the concurrency?

My setup uses i2c-1

I have owfs installed that connects to an (I2C to 1Wire interface DS2482S-100) and at the same time have a I2C port expander MCP23017 that is being polled every 300ms or so from a python script
The I2C line is connected thru voltage translator TCA9406 to the rest of the chips.

I have not tried to disable the owfs for some days to see if that makes a difference.

When I mentioned the SCL line was being held down by the odroid I meant that odroid was keeping it low. I cut the line for a while and it was still low on odroid side and high on the other side. In any way that did not restore the state of the communication. So something on the I2C implementation on odroid side was keeping it low for some reason.

Thanks for the link to the sources but are there some instructions on the compilation. I have tried running make against that directory but it was missing something that I guess is coming from the outer scripts.

rgrds
puccini
 
Posts: 4
Joined: Tue Oct 13, 2015 10:37 pm
languages_spoken: english
ODROIDs: Odroid C1

Re: I2C driver floods serial console with errors

Unread postby odroid » Wed Oct 14, 2015 3:42 pm

We couldn't see the SCL line issue you observed. It seems other I2C slaves kept the line to low probably.

Anyway, you need to build the whole source of kernel.
If you want to cross-build only your host PC, refer this guide.
http://odroid.com/dokuwiki/doku.php?id= ... ing_kernel
Refer this link for on-board native compile.
http://odroid.com/dokuwiki/doku.php?id= ... o_checkout
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby ralfz » Thu Oct 15, 2015 9:07 pm

Hi,

I'am also still interested to solve this issue. As I wrote I observed that in the issue case the control register is set with wrong values.

Regarding this I wrote a workarround.

The bus balks in the aml_i2c_wait_ack(...) method. This method will never be leaved. The reason of this behavior is the STATE of the bus cannot be IDLE again (cause manual mod is enabled now).
I'd like to offer a patch.

There are two important parts within the patch:

1.) If the aml_i2c_wait_ack(...) balks - the ctrl register will be reset
2.) To reproduce this behavior I added some code to rw_special_reg(...)
To trigger the issue use: root@odroidc: # echo "w 2" > /sys/class/i2c1/customize (This will set the manuale enable bit manually ;-) )

For a better understanding of the control mechanisms I left the dbg messages within the patch.

Attention: If anyone will use the provided diff as a patch be careful!


Edit:
I belive that somewhere in the whole code (Android,Kernel) "something" will write to a wrong register cause of a wrong determined offset!
Think about that:

What mechanism is able to set this ME bit??

It would be nice to get some feedback!

Best regards
Attachments
aml_i2c_c1_workaround_01_svn_trac.txt
i2c workarround
(4.29 KiB) Downloaded 66 times
ralfz
 
Posts: 65
Joined: Wed Mar 11, 2015 12:45 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby puccini » Fri Oct 16, 2015 1:57 am

Hi,

Thanks for the details, ralfz. I can confirm that the case is the same for me as well.

It started as
Oct 15 18:02:03 odroid kernel: [74902.583952@0] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x20

followed by
Oct 15 18:03:18 odroid kernel: [74978.096892@2] i2c i2c-1: [aml_i2c_xfer] error ret = -110 (-ETIMEOUT) token 1, master_no(1) 100K addr 0x18


the i2c registers dump (
Code: Select all
cat /sys/class/i2c1/info
) shows that the manual mode has been enabled (bit22) i2c_ctrl: 0x571c005 (‭0101011100011100000000000101‬) and also the SCL line is down as observed previously on HW level.

Oct 15 18:44:23 odroid kernel: [77442.730292@1] i2c master_no(1) current slave addr is 0x20
Oct 15 18:44:23 odroid kernel: [77442.733248@1] wait ack timeout is 0x3d090
Oct 15 18:44:23 odroid kernel: [77442.762338@1] master regs base is 0xfe108500
Oct 15 18:44:23 odroid kernel: [77442.765210@1] i2c_ctrl: 0x571c005
Oct 15 18:44:23 odroid kernel: [77442.768023@1] ctrl.rdsda 0x1
Oct 15 18:44:23 odroid kernel: [77442.770824@1] ctrl.rdscl 0x0
Oct 15 18:44:23 odroid kernel: [77442.812329@1] ctrl.wrsda 0x1
Oct 15 18:44:23 odroid kernel: [77442.815177@1] ctrl.wrscl 0x0
Oct 15 18:44:23 odroid kernel: [77442.817988@1] ctrl.manual_en 0x1
Oct 15 18:44:23 odroid kernel: [77442.820877@1] ctrl.clk_delay 0x31c
Oct 15 18:44:23 odroid kernel: [77442.852327@1] ctrl.rd_data_cnt 0x0
Oct 15 18:44:23 odroid kernel: [77442.855098@1] ctrl.cur_token 0x0
Oct 15 18:44:23 odroid kernel: [77442.857865@1] ctrl.error 0x0
Oct 15 18:44:23 odroid kernel: [77442.860630@1] ctrl.status 0x1
Oct 15 18:44:23 odroid kernel: [77442.892326@1] ctrl.ack_ignore 0x0
Oct 15 18:44:23 odroid kernel: [77442.895126@1] ctrl.start 0x1
Oct 15 18:44:23 odroid kernel: [77442.897893@1] i2c_slave_addr: 0x118e0040
Oct 15 18:44:23 odroid kernel: [77442.900679@1] i2c_token_list_0: 0x421
Oct 15 18:44:23 odroid kernel: [77442.942330@1] i2c_token_list_1: 0x0
Oct 15 18:44:23 odroid kernel: [77442.945173@1] i2c_token_wdata_0: 0x13
Oct 15 18:44:23 odroid kernel: [77442.947984@1] i2c_token_wdata_1: 0x0
Oct 15 18:44:23 odroid kernel: [77442.950779@1] i2c_token_rdata_0: 0xa


I have changed the bit 22 in 0x571c005, and the changed resulting value 0x531C005 set back to the register that enabled the i2c routines.

Code: Select all
echo w 0x2140 0x531C005  > /sys/class/i2c1/cbus_reg


there was some error reported on the console but the message in the syslog showed success and the i2c was enabled.

Oct 15 18:57:18 odroid kernel: [78218.135291@1] write cbus reg 0x2140 value 531c005


so a direct work around on OS level would be for the register to be changed to some "reset" value. I just changed the bit 22 to disable the manual mode in my case, but it may work equally well with some standard value to be put in a script (
Code: Select all
echo w 0x2140 0x7B1C002 > /sys/class/i2c1/cbus_reg
).

(I played with the bits and came with the number 0x7B1C002 for the initial state of that register with the speed set and the lines up)

rgrds
puccini
 
Posts: 4
Joined: Tue Oct 13, 2015 10:37 pm
languages_spoken: english
ODROIDs: Odroid C1

Re: I2C driver floods serial console with errors

Unread postby odroid » Fri Oct 16, 2015 11:00 am

Can you guys try this patch?
https://github.com/hardkernel/linux/com ... e13512253d

Once we applied the patch, we could use the i2cdetect command.
It seems to be also related to what you are discussing.
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby ralfz » Fri Oct 16, 2015 5:37 pm

Hi,

thanks for that. I'll have a closer look on the patch next monday. I'll also install it on our C1s for testing.

regards
ralfz
 
Posts: 65
Joined: Wed Mar 11, 2015 12:45 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby puccini » Sun Oct 18, 2015 3:03 am

hi,
I have enabled the debug mode and made a script to copy the problematic syslog files only. Now I have the debug for number of short outages, where the i2c resolves by itself after some seconds and one full hang of i2c that I resolved by setting the manual enable bit to 0.
I am putting below just few extracts from the syslog of the time before the failure. It can clearly be seen that somethin either in the i2c driver or in the kernel is messing up with the (I2C_M_0_CONTROL_REG 0x2140) register.

1. i2c hang resolves by itself. It is not really a hang as only the clock is messed up as seen below so its value is being set correctly upon next communication.
Oct 17 12:23:27 odroid kernel: [67068.388507@3] addr [8c70018] token tag : 1,2,4,4,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 12:23:27 odroid kernel: [67068.388529@3] w0 e1e1, w1 0, r0 a, r1 0, cur_token 1, rd cnt 0, status 0,error 0, ack_ignore 0,start 1
Oct 17 12:23:27 odroid kernel: [67068.388843@3] [aml_i2c_adap1]:writee1-e1-
Oct 17 12:23:27 odroid kernel: [67068.388858@3] addr [8c70018] token tag : 1,3,0,0,0,0,0,0,clk_delay 2, clk is 19921K
Oct 17 12:23:27 odroid kernel: [67068.388880@3] w0 e1e1, w1 0, r0 a, r1 0, cur_token 1, rd cnt 0, status 0,error 0, ack_ignore 0,start 1
Oct 17 12:23:27 odroid kernel: [67068.388933@3] [aml_i2c_adap1]:read0-
Oct 17 12:23:27 odroid kernel: [67068.388947@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x18
Oct 17 12:23:27 odroid kernel: [67068.393914@3] addr [8c70018] token tag : 1,3,0,0,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 12:23:27 odroid kernel: [67068.393945@3] w0 e1e1, w1 0, r0 a, r1 0, cur_token 1, rd cnt 0, status 0,error 1, ack_ignore 0,start 1
Oct 17 12:23:27 odroid kernel: [67068.394054@3] addr [8c70018] token tag : 5,0,0,0,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 12:23:27 odroid kernel: [67068.394076@3] w0 e1e1, w1 0, r0 a, r1 0, cur_token 2, rd cnt 0, status 0,error 0, ack_ignore 0,start 1
Oct 17 12:23:27 odroid kernel: [67068.394200@3] [aml_i2c_adap1]:readc-


2. the same as the first one only the clk_delay here is set to 0 hence the division by zero error reported during the debug
Oct 17 11:56:38 odroid kernel: [65459.363867@1] addr [8c70018] token tag : 1,2,4,4,0,0,0,0,Division by zero in kernel.
Oct 17 11:56:38 odroid kernel: [65459.363966@1] CPU: 1 PID: 2368 Comm: owfs Not tainted 3.10.80-128 #1
Oct 17 11:56:38 odroid kernel: [65459.363998@1] [<c0014e3c>] (unwind_backtrace+0x0/0xec) from [<c0011e80>] (show_stack+0x10/0x14)
Oct 17 11:56:38 odroid kernel: [65459.364023@1] [<c0011e80>] (show_stack+0x10/0x14) from [<c0235410>] (Ldiv0+0x8/0x10)
Oct 17 11:56:38 odroid kernel: [65459.364055@1] [<c0235410>] (Ldiv0+0x8/0x10) from [<bf005d34>] (aml_i2c_dbg+0x88/0x130 [aml_i2c])
Oct 17 11:56:38 odroid kernel: [65459.364084@1] [<bf005d34>] (aml_i2c_dbg+0x88/0x130 [aml_i2c]) from [<bf007024>] (aml_i2c_write+0xe0/0x160 [aml_i2c])
Oct 17 11:56:38 odroid kernel: [65459.364113@1] [<bf007024>] (aml_i2c_write+0xe0/0x160 [aml_i2c]) from [<bf006c24>] (aml_i2c_xfer+0x1d0/0x358 [aml_i2c])
Oct 17 11:56:38 odroid kernel: [65459.364141@1] [<bf006c24>] (aml_i2c_xfer+0x1d0/0x358 [aml_i2c]) from [<c034b454>] (__i2c_transfer+0x48/0x7c)
Oct 17 11:56:38 odroid kernel: [65459.364163@1] [<c034b454>] (__i2c_transfer+0x48/0x7c) from [<c034c9e8>] (i2c_transfer+0xa4/0xcc)
Oct 17 11:56:38 odroid kernel: [65459.364185@1] [<c034c9e8>] (i2c_transfer+0xa4/0xcc) from [<c034d014>] (i2c_smbus_xfer+0x568/0x5b8)
Oct 17 11:56:38 odroid kernel: [65459.364208@1] [<c034d014>] (i2c_smbus_xfer+0x568/0x5b8) from [<c034dfe4>] (i2cdev_ioctl_smbus+0x184/0x218)
Oct 17 11:56:38 odroid kernel: [65459.364231@1] [<c034dfe4>] (i2cdev_ioctl_smbus+0x184/0x218) from [<c034e5b4>] (i2cdev_ioctl+0x1e4/0x230)
Oct 17 11:56:38 odroid kernel: [65459.364255@1] [<c034e5b4>] (i2cdev_ioctl+0x1e4/0x230) from [<c0102200>] (do_vfs_ioctl+0x560/0x5b4)
Oct 17 11:56:38 odroid kernel: [65459.364273@1] [<c0102200>] (do_vfs_ioctl+0x560/0x5b4) from [<c01022a4>] (SyS_ioctl+0x50/0x7c)
Oct 17 11:56:38 odroid kernel: [65459.364297@1] [<c01022a4>] (SyS_ioctl+0x50/0x7c) from [<c000dfc0>] (ret_fast_syscall+0x0/0x38)
Oct 17 11:56:38 odroid kernel: [65459.364309@1] clk_delay 0, clk is 0K
Oct 17 11:56:38 odroid kernel: [65459.364318@1] w0 ffa5, w1 0, r0 10, r1 0, cur_token 1, rd cnt 0, status 0,error 0, ack_ignore 0,start 0
Oct 17 11:56:38 odroid kernel: [65459.364369@1] [aml_i2c_adap1]:writea5-ff-
Oct 17 11:56:38 odroid kernel: [65459.364384@1] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x18
Oct 17 11:56:38 odroid kernel: [65459.369452@1] addr [8c70018] token tag : 1,2,4,4,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 11:56:38 odroid kernel: [65459.369480@1] w0 2000, w1 0, r0 10, r1 0, cur_token 1, rd cnt 0, status 0,error 1, ack_ignore 0,start 1
Oct 17 11:56:38 odroid kernel: [65459.369700@1] [aml_i2c_adap1]:write0-20-
Oct 17 11:56:38 odroid kernel: [65459.369714@1] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 2, master_no(1) 100K addr 0x18
Oct 17 11:56:38 odroid kernel: [65459.378289@1] addr [8c70018] token tag : 1,2,4,4,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 11:56:38 odroid kernel: [65459.378322@1] w0 98e5, w1 0, r0 10, r1 0, cur_token 2, rd cnt 0, status 0,error 1, ack_ignore 0,start 1
Oct 17 11:56:38 odroid kernel: [65459.378541@1] [aml_i2c_adap1]:writee5-98-
Oct 17 11:56:38 odroid kernel: [65459.378555@1] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 2, master_no(1) 100K addr 0x18
Oct 17 11:56:38 odroid kernel: [65459.387421@1] addr [8c70018] token tag : 1,2,4,0,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 11:56:38 odroid kernel: [65459.387525@1] w0 b4, w1 0, r0 10, r1 0, cur_token 2, rd cnt 0, status 0,error 1, ack_ignore 0,start 1
Oct 17 11:56:38 odroid kernel: [65459.387751@1] [aml_i2c_adap1]:writeb4-
Oct 17 11:56:38 odroid kernel: [65459.387773@1] addr [8c70020] token tag : 1,2,4,0,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 11:56:38 odroid kernel: [65459.387796@1] w0 13, w1 0, r0 10, r1 0, cur_token 1, rd cnt 0, status 0,error 0, ack_ignore 0,start 1
Oct 17 11:56:38 odroid kernel: [65459.387995@1] addr [8c70020] token tag : 1,3,0,0,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 11:56:38 odroid kernel: [65459.388017@1] w0 13, w1 0, r0 10, r1 0, cur_token 3, rd cnt 0, status 0,error 0, ack_ignore 0,start 1
Oct 17 11:56:38 odroid kernel: [65459.388134@1] addr [8c70020] token tag : 5,0,0,0,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 11:56:38 odroid kernel: [65459.388156@1] w0 13, w1 0, r0 10, r1 0, cur_token 2, rd cnt 0, status 0,error 0, ack_ignore 0,start 1
Oct 17 11:56:38 odroid kernel: [65459.388276@1] [aml_i2c_adap1]:read4-


3. i2c hang. Obviously the change affected the bit22 that is just next to the clk_delay bits 21:12 and since that bit is not being reset by the driver the i2c is in hang until the next mess up with the ctrl register that would set the bit 22 back to 0. In my case it takes between 20 000 and 30 000 seconds for the i2c state to be resolved by itself. The clock is also changed as seen below
Oct 17 13:30:08 odroid kernel: [71069.755574@1] addr [8c70018] token tag : 1,3,0,0,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 13:30:08 odroid kernel: [71069.755597@1] w0 e1e1, w1 0, r0 a, r1 0, cur_token 1, rd cnt 0, status 0,error 0, ack_ignore 0,start 1
Oct 17 13:30:08 odroid kernel: [71069.755706@1] addr [8c70018] token tag : 5,0,0,0,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 13:30:08 odroid kernel: [71069.755728@1] w0 e1e1, w1 0, r0 a, r1 0, cur_token 2, rd cnt 0, status 0,error 0, ack_ignore 0,start 1
Oct 17 13:30:08 odroid kernel: [71069.755850@1] [aml_i2c_adap1]:readff-
Oct 17 13:30:08 odroid kernel: [71069.755903@1] addr [8c70018] token tag : 1,2,4,4,0,0,0,0,clk_delay 2b4, clk is 57K
Oct 17 13:30:08 odroid kernel: [71069.755933@1] w0 ffa5, w1 0, r0 ff, r1 0, cur_token 1, rd cnt 0, status 0,error 0, ack_ignore 0,start 0
Oct 17 13:30:08 odroid kernel: [71069.755949@1] [aml_i2c_dbg] manual_en, why?
Oct 17 13:30:08 odroid kernel: [71069.756045@1] [aml_i2c_adap1]:writea5-ff-
Oct 17 13:30:08 odroid kernel: [71069.756087@1] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x18
Oct 17 13:30:08 odroid kernel: [71069.761064@1] addr [8c70018] token tag : 1,2,4,4,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 13:30:08 odroid kernel: [71069.761090@1] w0 2000, w1 0, r0 ff, r1 0, cur_token 1, rd cnt 0, status 0,error 1, ack_ignore 0,start 1
Oct 17 13:30:08 odroid kernel: [71069.761095@1] [aml_i2c_dbg] manual_en, why?
Oct 17 13:30:08 odroid kernel: [71069.761198@1] [aml_i2c_adap1]:write0-20-
Oct 17 13:30:08 odroid kernel: [71069.761212@1] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x18
Oct 17 13:30:08 odroid kernel: [71069.770042@1] addr [8c70018] token tag : 1,2,4,4,0,0,0,0,clk_delay 31c, clk is 50K
Oct 17 13:30:08 odroid kernel: [71069.770068@1] w0 98e5, w1 0, r0 ff, r1 0, cur_token 1, rd cnt 0, status 0,error 1, ack_ignore 0,start 1
Oct 17 13:30:08 odroid kernel: [71069.770073@1] [aml_i2c_dbg] manual_en, why?


I have checked the source code once again for a potential logic that may slip and set the wrong register but could not find anything, so it must be something outside of the driver that sets the 0x2140 register. If this is the case may be the bit22 and generally the important bits like the clock as well should be set to the correct value prior to every start operation. In any way the driver does not work in a manual mode.

I have tried to compile the module but the resulting .ko file is a way bigger than the one from the repository, so I cannot suggest a working code change here, but it should be just a few lines added in the aml_i2c_start_token_xfer function.

rgrds
puccini
 
Posts: 4
Joined: Tue Oct 13, 2015 10:37 pm
languages_spoken: english
ODROIDs: Odroid C1

Re: I2C driver floods serial console with errors

Unread postby ralfz » Mon Nov 23, 2015 8:05 pm

Hi,

I tested the patch. But the "manual enable" problem isn't solved yet. Further more the drivers hangs in the infinity loop...

regards
ralfz
 
Posts: 65
Joined: Wed Mar 11, 2015 12:45 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby odroid » Wed Nov 25, 2015 7:06 pm

Please try another update.
https://github.com/hardkernel/linux/com ... e00c5449c5

The latest kernel update 3.10.80-134 also inlcudes this stability improvement.
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby ralfz » Wed Nov 25, 2015 9:24 pm

Hi,

thx for this I'll try it.

But this seems to solve the symptoms, not the reasons - am I right?
Code: Select all
+   struct aml_i2c_reg_ctrl* ctrl =
+      (struct aml_i2c_reg_ctrl*)&(i2c->master_regs->i2c_ctrl);
+
+   if(ctrl->manual_en || ctrl->ack_ignore)   {
+      dev_err(&i2c->adap.dev, "Controller Error! manual_en = %d, ack_ignore = %d\n",
+                        ctrl->manual_en, ctrl->ack_ignore);
+      // Controller Error Fix.
+      ctrl->manual_en = 0;   ctrl->ack_ignore = 0;
+   }
+


What could be the cause for ctrl->manual_en = 1 ? Is it an error within the controller itselfe? Or "other software" which will accidently set the bit?

Best regards
ralfz
 
Posts: 65
Joined: Wed Mar 11, 2015 12:45 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby odroid » Wed Nov 25, 2015 9:31 pm

We are not sure it is a silicon bug or other driver's side effect.
But it helped to pass the 24hours of the intensive I2C stress test.
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby ralfz » Fri Feb 19, 2016 1:39 am

Hi,

I tried it.
The "manual enable" error seems to be catched. But the bus still strugles from time to time with the same symptoms. :/ :/

br
ralfz
 
Posts: 65
Joined: Wed Mar 11, 2015 12:45 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby danneman » Sun Feb 21, 2016 6:52 am

I have also tried it. Having an intense python i2c application running fine for month on Rpi and BBB but not on C1+. It just stops working and I have to reset the i2c connection.

But I have to say it is working for a longer period now than before. After the update it will work a couple of days instead of hours.
/Daniel
danneman
 
Posts: 7
Joined: Tue Jul 28, 2015 6:48 am
languages_spoken: english
ODROIDs: C1

Re: I2C driver floods serial console with errors

Unread postby HABOT » Mon Mar 07, 2016 4:36 am

Hello, we are suffering from this issue on the I2C bus and since the Amls805 "mock" document provided by hardkernel lacks lots of information regarding the SoC, so we the community could find a workaround to the flaws, could you please provide a compiled version of the driver module (aml_i2c) so we just need to replace the flawed version??

Currently we do require to run a very intensive process on the I2C bus that updates many devices on a 10 millisecond basis but even that this is not as fast as many other processes on a hardware bus, it completely crashes the ODROID-C1 to the point that a full reset is needed! It takes a few minutes to stop working and now we are just looking out for another options since we have a deadline to accomplish and the solution to this problem is stopping us from doing our job. We don't have enough time to cope with the search of required libraries to compile and fix the device driver by ourselves! No doubt this is a really advanced and nice open platform, but providing the solution to this kind of bugs is entirely your job, if you are not willing to provide all of the details of the hardware so we can freely and fully rewrite a driver.

by writing to cbus_reg attrib of the i2c1 sysfs mapped ko, and reseting bit 22 (manual mode) as it has been widely explained here, it works for a while again, but all of the sudden it shuts down and a full reset is required since the I2C port doesn't respond anymore. The error reported is:

kernel: [ 1047.181941@1] i2c i2c-1: [aml_i2c_xfer] error ret = -110 (-ETIMEOUT) token 1, master_no(1) 100K addr 0x5a

The source code that you provided for the driver in order to be compiled asks for several dependencies not available in my development platform, so please provide the compiled new version of the driver to keep going.
HABOT
 
Posts: 11
Joined: Thu Feb 12, 2015 1:35 am
languages_spoken: english, spanish, some french
ODROIDs: XU3 Lite, C1

Re: I2C driver floods serial console with errors

Unread postby odroid » Mon Mar 07, 2016 11:36 am

If your current kernel version is 3.10.80-134 or higher, the patched driver should be already included.

What is your kernel version?
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby HABOT » Mon Mar 07, 2016 12:05 pm

Hello, by further debugging the driver, I've found these weird report lines mixed within several lines of the original error:

kernel: [ 1371.805661@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 50K addr 0x5a
kernel: [ 1371.844936@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 50K addr 0x5a
kernel: [ 1371.848368@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 50K addr 0x5a
kernel: [ 1371.857949@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 50K addr 0x5a
kernel: [ 1371.866906@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 50K addr 0x5a
kernel: [ 1371.874651@0] Uart 0 Driver: Overflow Error while receiving a character
kernel: [ 1371.874736@0] Uart 0 Driver: Overflow Error while receiving a character[/b]
kernel: [ 1375.330550@1] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 50K addr 0x5a

Seems like the Uart driver overflowed..., does it ring any bell to you? Counted five of these lines for thousand of the normal i2c-1 error. Maybe the overflow of the UART FIFO overwrites some I2C registers since the address ranges are neighboring each other (UART0 address range 0x2130 ~ 0x2135 vs I2C address range 0x2140 - 0x2146 starting with the I2C_M_0_CONTROL_REG)
Last edited by HABOT on Mon Mar 07, 2016 1:38 pm, edited 3 times in total.
HABOT
 
Posts: 11
Joined: Thu Feb 12, 2015 1:35 am
languages_spoken: english, spanish, some french
ODROIDs: XU3 Lite, C1

Re: I2C driver floods serial console with errors

Unread postby HABOT » Mon Mar 07, 2016 12:09 pm

My kernel version is 3.10.43
HABOT
 
Posts: 11
Joined: Thu Feb 12, 2015 1:35 am
languages_spoken: english, spanish, some french
ODROIDs: XU3 Lite, C1

Re: I2C driver floods serial console with errors

Unread postby odroid » Mon Mar 07, 2016 12:38 pm

Update your kernel with "sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade".
You should have 3.10.9x of kernel after update.
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby HABOT » Mon Mar 07, 2016 1:36 pm

After update/upgrade kernel version remains the same: 3.10.43. I tried the I2C communication process again and still crashes after a while. Error report:

kernel: [ 337.292766@2] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
kernel: [ 337.296197@2] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
kernel: [ 337.301569@0] Uart 0 Driver: Overflow Error while receiving a character
kernel: [ 337.301774@0] Uart 0 Driver: Overflow Error while receiving a character
kernel: [ 337.308672@0] Uart 0 Driver: Overflow Error while receiving a character
kernel: [ 337.320067@0] Uart 0 Driver: Overflow Error while receiving a character
kernel: [ 337.332146@2] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
kernel: [ 337.341195@2] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
kernel: [ 337.350554@2] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
kernel: [ 337.353293@0] Uart 0 Driver: Overflow Error while receiving a character
kernel: [ 337.363037@0] Uart 0 Driver: Overflow Error while receiving a character
kernel: [ 337.373213@2] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
kernel: [ 337.375191@0] Uart 0 Driver: Overflow Error while receiving a character
kernel: [ 337.389225@2] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
HABOT
 
Posts: 11
Joined: Thu Feb 12, 2015 1:35 am
languages_spoken: english, spanish, some french
ODROIDs: XU3 Lite, C1

Re: I2C driver floods serial console with errors

Unread postby HABOT » Tue Mar 08, 2016 1:02 am

After kernel update to 3.10.80-142 I've been running the process on the I2C port and it seems to be stable. It's worth to notice that the internal error is still there since the patch you applied to the driver validates the manual_en bit and resets it in case it is set. The manual bit [22] at 0x2140 is being changed by another driver, DMA access or a binary ghost.

Code: Select all
if(ctrl->manual_en || ctrl->ack_ignore)   {
      dev_err(&i2c->adap.dev, "Controller Error! manual_en = %d, ack_ignore = %d\n",
                        ctrl->manual_en, ctrl->ack_ignore);
      // Controller Error Fix.
      ctrl->manual_en = 0;   ctrl->ack_ignore = 0;
}


kernel log shows:

kernel: [ 1492.780243@0] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
kernel: [ 1494.893468@0] i2c i2c-1: Controller Error! manual_en = 1, ack_ignore = 0
kernel: [ 1494.895071@0] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
kernel: [ 1512.104890@2] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
kernel: [ 1515.889458@3] i2c i2c-1: Controller Error! manual_en = 1, ack_ignore = 1
kernel: [ 1520.981302@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5a
HABOT
 
Posts: 11
Joined: Thu Feb 12, 2015 1:35 am
languages_spoken: english, spanish, some french
ODROIDs: XU3 Lite, C1

Re: I2C driver floods serial console with errors

Unread postby odroid » Tue Mar 08, 2016 12:35 pm

Sometimes the manual_en bit is set accidentally (by IP issue or silicon bug or whatever), we needed to clear/reset it to keep using the I2C bus.
Otherwise, the I2C access was totally locked.
It is a sort of dirty & quick workaround. But we still don't know how to avoid that locking situation.
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby voodoo » Wed Mar 09, 2016 8:40 pm

I'm also having the same problem with Odroid C1.

kernel: [40666.751082@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x76

It's quite curious, the error appears each time that the weather board is accessed, but the data/measures obtained from the weather board are always correct (even if the error is there). Fortunately the data is stored in a local mongodb, that's why i could check that fact.

Anyway, Odroid C1 becomes unaccessible (through IP) once the error persists for several hours... Odroid doesn't even react to a simple ping.

Hope you find a solution, it's a big bug.
voodoo
 
Posts: 15
Joined: Mon Mar 07, 2016 6:01 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby odroid » Wed Mar 09, 2016 9:10 pm

Do you mean the several hours of continuously repeated errors could cause the Ethernet network problem?
In that case we need to look into this issue more carefully again. What is your kernel version?
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby voodoo » Wed Mar 09, 2016 9:38 pm

My kernel version is 3.10.80-141

I've associated this i2c error to the ethernet network problem due to after a year working with Odroid C1 (without any networking problem btw), this connectivity problem has appeared after using weather board. No more errors are showed in /var/log/kernel.log, /var/log/syslog and /var/log/auth.log.

Any suggestion would be welcome. Thanks.
voodoo
 
Posts: 15
Joined: Mon Mar 07, 2016 6:01 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby odroid » Wed Mar 09, 2016 9:47 pm

Since there is no clue in the log files, we need to start a similar test hear.

Let us know how your software frequently accessed the weather board.
We will run a program to read the weather data at same frequency and keep pinging the C1 board from other ODROID every couple of minutes.
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby voodoo » Wed Mar 09, 2016 10:05 pm

The weather board is accessed by software every 5 minutes. As i mentioned, the data provided by the weather board is correct (actually it's stored locally without any problem).

Thanks for your support and tests, good job guys! I'll wait for news.
voodoo
 
Posts: 15
Joined: Mon Mar 07, 2016 6:01 am
languages_spoken: english

Re: I2C driver floods serial console with errors

Unread postby odroid » Sun Apr 17, 2016 11:33 am

We've run below script for 40+ hours to reproduce your issue. But the network was okay.

Code: Select all
#
# ODROID-C1/C2 I2C Test Script
#

#
# Setup test condition
#

# log file name
logfile="log.txt"

# total sec
tick_count=1

# Sensor data check time delay
sensor_read_sec=5

# Ethernet ping check time delay
eth_check_sec=60

# Current Status save to file time delay
status_save_sec=30

#
# Setup test condition end
#

if [ -f "$logfile" ] ; then
   echo backup $logfile to $logfile.$(date +%H%M%S)
   cp $logfile $logfile.$(date +%H%M%S)
   rm -f $logfile
fi

# I2C Driver remove
modprobe -r aml_i2c

# I2C Driver Porbe
modprobe aml_i2c

# Weather Board Sensor driver Probe
modprobe bmp085-i2c
echo bmp085 0x77 > /sys/bus/i2c/devices/i2c-1/new_device
modprobe si702x
echo si702x 0x40 > /sys/bus/i2c/devices/i2c-1/new_device

# run_flag
run_flag=0

# init variable
ping -c 1 -w 5 192.168.20.4 &> /dev/null
if [ "$?" == "0" ] ; then
   status="OK"
   echo Ethernet ping test is OK
else
   status="FAIL"
   echo Ethernet ping test is FAIL
fi

time=$(date +%Y\/%m\/%d-\%H:\%M:\%S)
press=$(cat /sys/bus/i2c/drivers/bmp085/1-0077/pressure0_input)
temp=$(cat /sys/bus/i2c/drivers/si702x/1-0040/temperature)
humi=$(cat /sys/bus/i2c/drivers/si702x/1-0040/humidity)

echo $time,pressure=$press,temperature=$temp,humidity=$humi

#
# start script
#
while true

do

time=$(date +%Y\/%m\/%d-\%H:\%M:\%S)

let run_flag=$tick_count%$sensor_read_sec

if [ $run_flag == 0 ] ; then
   press=$(cat /sys/bus/i2c/drivers/bmp085/1-0077/pressure0_input)
   temp=$(cat /sys/bus/i2c/drivers/si702x/1-0040/temperature)
   humi=$(cat /sys/bus/i2c/drivers/si702x/1-0040/humidity)
   
   echo $time,pressure=$press,temperature=$temp,humidity=$humi
fi

let run_flag=$tick_count%$eth_check_sec

if [ $run_flag == 0 ] ; then
   ping -c 1 -w 5 192.168.20.4 &> /dev/null
   if [ "$?" == "0" ] ; then
      status="OK"
      echo $time Ethernet ping test is OK
   else
      status="FAIL"
      echo $time Ethernet ping test is FAIL
   fi
fi

let run_flag=$tick_count%$status_save_sec

if [ $run_flag == 0 ] ; then
   echo ETH Status=$status : $time $press $temp $humi >> $logfile
fi

let tick_count=tick_count+1
sleep 1

done



That script do below things.
Read the weather board every 5 seconds
Check the Ethernet connection every 60 seconds
Save the status every 30 seconds
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby HABOT » Fri May 06, 2016 10:17 am

Hello, as the peer voodoo mentioned, I've been experiencing the same problem for some days now. The ethernet connection suddenly stop working for several minutes or even forever (requiring reboot). The I2C port also stop working and the kernel log is flooded with messages/errors like these:

May 5 19:52:50 [node-name] kernel: [ 153.400139@1] [adjust link] -> eth: phy_speed <> priv_speed)
May 5 19:52:50 [node-name] kernel: [ 153.400145@1] [adjust link] -> eth: switching to RGMII 100
May 5 19:52:50 [node-name] kernel: [ 153.400158@1] [adjust link -> eth: am_adjust_link state change (new_state=true)
May 5 19:52:50 [node-name] kernel: [ 153.400165@1] libphy: 0:01 - Link is Up - 100/Full
May 5 19:53:23 [node-name] ntpdate[749]: Can't find host 3.ubuntu.pool.ntp.org: Name or service not known (-2)
May 5 19:54:03 [node-name] ntpdate[749]: Can't find host ntp.ubuntu.com: Name or service not known (-2)
May 5 19:54:03 [node-name] ntpdate[749]: no servers can be used, exiting
May 5 19:54:03 [node-name] ntpd[1178]: ntpd 4.2.6p5@1.2349-o Thu Feb 11 18:34:32 UTC 2016 (1)
May 5 19:54:03 [node-name] ntpd[1179]: proto: precision = 1.000 usec
May 5 19:54:03 [node-name] ntpd[1179]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
May 5 19:54:03 [node-name] ntpd[1179]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
May 5 19:54:03 [node-name] ntpd[1179]: Listen and drop on 1 v6wildcard :: UDP 123
May 5 19:54:03 [node-name] ntpd[1179]: Listen normally on 2 lo 127.0.0.1 UDP 123
May 5 19:54:03 [node-name] ntpd[1179]: Listen normally on 3 eth1 192.168.0.26 UDP 123
May 5 19:54:03 [node-name] ntpd[1179]: Listen normally on 4 lo ::1 UDP 123
May 5 19:54:03 [node-name] ntpd[1179]: Listen normally on 5 eth1 fe80::21e:6ff:fecc:bc33 UDP 123
May 5 19:54:03 [node-name] ntpd[1179]: peers refreshed
May 5 19:54:03 [node-name] ntpd[1179]: Listening on routing socket on fd #22 for interface updates
May 5 19:54:28 [node-name] ntpd[1179]: ntpd exiting on signal 15
May 5 19:54:29 [node-name] ntpd[1215]: ntpd 4.2.6p5@1.2349-o Thu Feb 11 18:34:32 UTC 2016 (1)
May 5 19:54:29 [node-name] ntpd[1216]: proto: precision = 1.000 usec
May 5 19:54:29 [node-name] ntpd[1216]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
May 5 19:54:29 [node-name] ntpd[1216]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
May 5 19:54:29 [node-name] ntpd[1216]: Listen and drop on 1 v6wildcard :: UDP 123
May 5 19:54:29 [node-name] ntpd[1216]: Listen normally on 2 lo 127.0.0.1 UDP 123
May 5 19:54:29 [node-name] ntpd[1216]: Listen normally on 3 eth1 192.168.0.26 UDP 123
May 5 19:54:29 [node-name] ntpd[1216]: Listen normally on 4 lo ::1 UDP 123
May 5 19:54:29 [node-name] ntpd[1216]: Listen normally on 5 eth1 fe80::21e:6ff:fecc:bc33 UDP 123
May 5 19:54:29 [node-name] ntpd[1216]: peers refreshed
May 5 19:54:29 [node-name] ntpd[1216]: Listening on routing socket on fd #22 for interface updates
May 5 19:54:29 [node-name] kernel: [ 252.177919@1] init: plymouth-upstart-bridge main process ended, respawning
May 5 19:54:59 [node-name] kernel: [ 282.736185@3] i2c i2c-1: Controller Error! manual_en = 1, ack_ignore = 0
May 5 19:55:02 [node-name] kernel: [ 285.174798@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5d
May 5 19:55:04 [node-name] kernel: [ 286.836774@3] i2c i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr 0x5d
May 5 19:55:04 [node-name] kernel: [ 287.452825@3] i2c i2c-1: Controller Error! manual_en = 1, ack_ignore = 0
May 5 19:55:08 [node-name] ntpdate[1210]: Can't find host 0.ubuntu.pool.ntp.org: Name or service not known (-2)
May 5 19:55:09 [node-name] ntpd[1216]: Deferring DNS for 0.ubuntu.pool.ntp.org 1


The ethernet port stops working and the board becomes unresponsive (no ping, no sockets, no ssh ...).

I have the Odroid-C1 in a network with an Odroid XU3 Lite and other two boards but the C1 is the only one showing this instability. At this point I don't know what to do with the C1 since it has exposed many problems that stop us from advancing.
HABOT
 
Posts: 11
Joined: Thu Feb 12, 2015 1:35 am
languages_spoken: english, spanish, some french
ODROIDs: XU3 Lite, C1

Re: I2C driver floods serial console with errors

Unread postby odroid » Sat May 07, 2016 4:42 pm

What is your kernel version?
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby HABOT » Tue May 10, 2016 8:33 am

odroid wrote:What is your kernel version?


It is 3.10.80-142

By making lots of tests, I've found that the C1 is pretty sensitive to voltage surges. I have some other SBCs in a network but the C1 is the only one that crashes the ethernet port since I can keep SSH the other boards. I've changed some elements in the electrical topology and now it is back working stable. Anyway, if you consider that a kernel upgrade is required, please advice thx.
HABOT
 
Posts: 11
Joined: Thu Feb 12, 2015 1:35 am
languages_spoken: english, spanish, some french
ODROIDs: XU3 Lite, C1

Re: I2C driver floods serial console with errors

Unread postby tve » Mon Nov 21, 2016 3:49 am

I have run into the "[aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr" issue with kernel 3.10.103-168 when the device attached to the i2c bus was not responding correctly. In my case there was a floating ground. Hope this helps someone...
tve
 
Posts: 17
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+

Re: I2C driver floods serial console with errors

Unread postby wittend » Sat May 06, 2017 9:39 am

tve wrote:I have run into the "[aml_i2c_xfer] error ret = -5 (-EIO) token 1, master_no(1) 100K addr" issue with kernel 3.10.103-168 when the device attached to the i2c bus was not responding correctly. In my case there was a floating ground. Hope this helps someone...


I am having a terrible problem with this error and also with my ODroid C0's simply locking up altogether. It has gotten so bad that I was only just barely able to bring up one to an HDMI-desktop long enough to open a terminal and type uname -r to get my kernel version: 3.10.104-186. I don't know if something got overwritten or what. I have moved the Flash to more than one C0 and it behaves the same on all. I have removed all I2C hardware that I was working with and disabled the service I was making. It just keeps getting worse!

I have been working on this for a year and I do not know where to go from here.

Dave
wittend
 
Posts: 23
Joined: Tue Jan 05, 2016 4:49 am
languages_spoken: english
ODROIDs: ODROID-U2, C0, C1+, XU4

Re: I2C driver floods serial console with errors

Unread postby wittend » Sun May 07, 2017 10:47 pm

I appear to have been conflating two or more problems, one with some desktop processes that were making operation steadily worse. But for me the I2C error problem remains and undermines my comfort using this platform.

After close reading of all that was written on this subject, I am still unclear whether this problem is something that has been:

1) fixed outright,
2) fixed in later kernels, but still plaguing me for some reason
3) is fundamental to the hardware, but can in some cases be worked-around with patches,
4) can't really be fixed so people have just given up on it.

For me, it always eventually kicks in when reading the LSM9DS1 imu by I2C. sometimes quickly, sometimes after an extended (~4 hr +) run.

Dave
wittend
 
Posts: 23
Joined: Tue Jan 05, 2016 4:49 am
languages_spoken: english
ODROIDs: ODROID-U2, C0, C1+, XU4

Re: I2C driver floods serial console with errors

Unread postby odroid » Mon May 08, 2017 9:26 am

We connected our Weather-board to C1+ i2c port and ran a test program(reading the sensors values every 1 second with 400Khz mode) on Kernel 3.10 over three weeks.
But we couldn't see any error.

Can you share your test source code which reads the IMU sensor values continuously for our review?
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby rudasi » Fri Sep 08, 2017 7:56 am

I am noticing the same issue where my odroid c2 gets really slow and I loose network connection, I get the following error,

i2c-1: [aml_i2c_xfer] error ret = -5 (-EIO)

In my case I noticed this error when I didnt have any of my i2c devices connected to the odroid but I tried to do a read and write using standard linux write/reads on an i2c file handle. Is there some check I should do before to make sure I have an i2c device on the bus or should the read/write fail but not cause the device to lock up?

I am running on kernel 3.14.79-102.

Thanks!
rudasi
 
Posts: 4
Joined: Mon May 02, 2016 4:10 pm
languages_spoken: english
ODROIDs: U3

Re: I2C driver floods serial console with errors

Unread postby odroid » Fri Sep 08, 2017 9:55 am

Normally, we have to check I2C device ID before reading/writing the I2C node.

Can you share your I2C read/write code to reproduce the system slowness and network disconnection issue?
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: I2C driver floods serial console with errors

Unread postby rudasi » Fri Sep 08, 2017 10:33 am

Code: Select all
#include <drivers/i2c.h>
#include <stdio.h>
#include <linux/i2c-dev.h>
#include <sys/ioctl.h>
#include <fcntl.h>
#include <errno.h>

unsigned char slave;
I2CDriver::I2CDriver(){
  tx_msg = new char[BUFFER_SIZE];
  rx_msg = new char[BUFFER_SIZE];

  file = open(i2c_dev_name, O_RDWR);
  if (file == 1){
    fprintf(stderr, "Bad device name %s\r\n", i2c_dev_name);
    exit(1);
  } else{
    printf("*** Device '%s' opened successfully\r\n", i2c_dev_name);
  }
}

void I2CDriver::writeData(void){
  if (write(file, tx_msg, BUFFER_SIZE*sizeof(char)) != BUFFER_SIZE*sizeof(char)){
    int ackerr = errno;
    printf("Failed to write i2c data with error: %d:%d\n", ackerr, slave);
  }
}

void I2CDriver::readData(void){
  memset(rx_msg, 0, sizeof(char)*BUFFER_SIZE);
  if (read(file, rx_msg, BUFFER_SIZE*sizeof(char)) != BUFFER_SIZE*sizeof(char)){
    int ackerr = errno;
    printf("Failed to read i2c data with error: %d\n", ackerr);
  }
}

void I2CDriver::writeMotorData(motor_data_t* data){
  memset(tx_msg, 0, sizeof(char)*BUFFER_SIZE);
  tx_msg[0] = CMD_CTRL_WRITE;
  memcpy(tx_msg+1, data, sizeof(motor_data_t));
  writeData();
}

void I2CDriver::readMotorData(motor_data_t* data){
  memset(tx_msg, 0, sizeof(char)*BUFFER_SIZE);
  tx_msg[0] = CMD_CTRL_READ;
  writeData();

  readData();
  memcpy(data, rx_msg+1, sizeof(motor_data_t));
}

void I2CDriver::switchSlave(unsigned int index){
//  unsigned char slave;
  if (index == 1){
    slave = SLAVE_ADDRESS_1;
  } else if (index == 2){
    slave = SLAVE_ADDRESS_2;
  } else if (index == 3){
    slave = SLAVE_ADDRESS_3;
  } else if (index == 4){
    slave = SLAVE_ADDRESS_4;
  } else {
    printf("incorrect slave address\n");
    return;
  }
  //ros::Duration(0.01).sleep();
  /* Specify the slave address */
  if (ioctl(file, I2C_SLAVE, slave) < 0){
    int switcherr = errno;
    fprintf(stderr, "Failed to acquire bus access and/or talk to slave, ERR:%d\r\n", switcherr);
  }
}


i2c_dev_name is = "/dev/i2c-1"

Even when I don't have any slave devices on the bus the switch slave call, ioctl(file, I2C_SLAVE, slave) < 0 succeeds.
Last edited by rudasi on Wed Jul 25, 2018 12:15 am, edited 1 time in total.
rudasi
 
Posts: 4
Joined: Mon May 02, 2016 4:10 pm
languages_spoken: english
ODROIDs: U3

Re: I2C driver floods serial console with errors

Unread postby odroid » Fri Sep 08, 2017 1:16 pm

It is very normal because there is no "acknowledge" signal from slave device.

I think you can add a device detection code in I2CDriver::I2CDriver().
Which slave device do you use?
User avatar
odroid
Site Admin
 
Posts: 28610
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Next

Return to Hardware and peripherals

Who is online

Users browsing this forum: No registered users and 1 guest