UDP logging adds 0x00 chars

Post Reply
rpgfun
Posts: 7
Joined: Wed Dec 29, 2021 4:10 am
languages_spoken: english
ODROIDs: HC1, HC2, C1, SP3
Has thanked: 1 time
Been thanked: 3 times
Contact:

UDP logging adds 0x00 chars

Post by rpgfun »

Hi,

I installed the new firmware and started to test UDP logging, but discovered two problems:
  • I have hex-zero chars in the datastream (one additional byte in columns 1, 6, 11,16). This is specific to UDP, serial logging does not have this problem.
  • the on/off columns (9,14) change from 0 -> 3 -> 1 -> (2) -> 0 when switching output-channels on and off. The transition value 3 (always) and 2 (only sometimes) is only visible once. Serial logging has the same issue.
I can work around the first issue (also removing the CR) with

Code: Select all

netcat -u -l 6000 | tr -d '\015\000' | tee output.csv
The second issue needs some unnecessary logic in data-analysis.

BTW: the firmware (and wiki) talks about the UDP client, but in fact the SP3 is the client and you need to enter the IP of the UDP-server (that is where netcat listens).

Otherwise: very good work, thank you!

lsc1117
Posts: 280
Joined: Thu Aug 22, 2013 12:46 am
languages_spoken: english
Location: South Korea
Has thanked: 9 times
Been thanked: 37 times
Contact:

Re: UDP logging adds 0x00 chars

Post by lsc1117 »

Hello, Thank you for the issues.

Could you share the output.csv without removing the CR?
I can't find the hex-zero chars on my side with "netcat -u -l 6000" command

In the second issue, the number 2 and 3 means ready state each.
The number 2 state of the onoff is the step before turning off the power while the number 3 is the step before turning on the power.

We were mistaken. The SP3 is a UDP client. We will fix it to prevent confusion next revision.

Thank you.

rpgfun
Posts: 7
Joined: Wed Dec 29, 2021 4:10 am
languages_spoken: english
ODROIDs: HC1, HC2, C1, SP3
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: UDP logging adds 0x00 chars

Post by rpgfun »

Hello,
Could you share the output.csv without removing the CR?
I can't find the hex-zero chars on my side with "netcat -u -l 6000" command
I attached a file sp3-idle.csv. My parameters: 100ms interval, no load on output-channels. The command to create the file was:

Code: Select all

netcat -u -l 6000 > /tmp/sp3-idle.csv
I can see the 0x00 chars with hexdump, e.g.

Code: Select all

hexdump -C /tmp/sp3-idle.csv | head -n 15
00000000  30 30 30 30 30 35 38 37  38 38 2c 31 35 30 37 37  |0000058788,15077|
00000010  2c 30 30 37 37 2c 30 31  31 36 30 2c 30 2c 00 30  |,0077,01160,0,.0|
00000020  30 30 30 30 2c 30 30 30  30 2c 30 30 30 30 30 2c  |0000,0000,00000,|
00000030  30 2c 30 30 2c 00 30 30  30 30 30 2c 30 30 30 30  |0,00,.00000,0000|
00000040  2c 30 30 30 30 30 2c 30  2c 30 30 2c 00 30 65 2c  |,00000,0,00,.0e,|
00000050  31 34 0a 0d 00 30 30 30  30 30 35 38 38 38 38 2c  |14...0000058888,|
00000060  31 35 30 37 39 2c 30 30  37 34 2c 30 31 31 31 35  |15079,0074,01115|
00000070  2c 30 2c 00 30 30 30 30  30 2c 30 30 30 30 2c 30  |,0,.00000,0000,0|
00000080  30 30 30 30 2c 30 2c 30  30 2c 00 30 30 30 30 30  |0000,0,00,.00000|
00000090  2c 30 30 30 30 2c 30 30  30 30 30 2c 30 2c 30 30  |,0000,00000,0,00|
000000a0  2c 00 30 65 2c 31 34 0a  0d 00 30 30 30 30 30 35  |,.0e,14...000005|
000000b0  38 39 38 39 2c 31 35 30  38 30 2c 30 30 37 32 2c  |8989,15080,0072,|
000000c0  30 31 30 38 35 2c 30 2c  00 30 30 30 30 30 2c 30  |01085,0,.00000,0|
000000d0  30 30 30 2c 30 30 30 30  30 2c 30 2c 30 30 2c 00  |000,00000,0,00,.|
000000e0  30 30 30 30 30 2c 30 30  30 30 2c 30 30 30 30 30  |00000,0000,00000|
You can also open the file with Visual Studio Code (it will complain about a binary file because of the 0x00), then you can see the zeros are in the specified colums.

Hope this helps. I will also have a look at the firmware source, maybe I can spot something.
Attachments
sp3-idle.csv
(6.56 KiB) Downloaded 9 times

rpgfun
Posts: 7
Joined: Wed Dec 29, 2021 4:10 am
languages_spoken: english
ODROIDs: HC1, HC2, C1, SP3
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: UDP logging adds 0x00 chars

Post by rpgfun »

I have identified the problem:

Code: Select all

void Screen::runWiFiLogging(const char *buf0, const char *buf1, const char *buf2, const char *buf3)
{
	udp.beginPacket(wifiManager->ipaddr_udp, wifiManager->port_udp);
	udp.write((uint8_t *)buf0, SIZE_LOG_BUFFER0);
	udp.write((uint8_t *)buf1, SIZE_LOG_BUFFER1);
	udp.write((uint8_t *)buf2, SIZE_LOG_BUFFER2);
	udp.write((uint8_t *)buf3, SIZE_CHECKSUM_BUFFER);
	udp.endPacket();
}
You send the complete buffers, but each (char) buffer has a 0x00 at the end (end of string). In contrast, the logging over serial uses Serial.printf(...), which will print up to this 0x00 end-of-string char.

BTW: why do you use LFCR as line endings? This is extremely uncommon, I assume what you really want is CRLF.
These users thanked the author rpgfun for the post:
lsc1117 (Thu Dec 30, 2021 4:16 pm)

lsc1117
Posts: 280
Joined: Thu Aug 22, 2013 12:46 am
languages_spoken: english
Location: South Korea
Has thanked: 9 times
Been thanked: 37 times
Contact:

Re: UDP logging adds 0x00 chars

Post by lsc1117 »

@rpgfun,

Thank you for your very clear investigations.

The "\0" and CRLF exist for serial logging. Without them, you can't see the logs properly in a terminal.
I have fixed the issue by getting rid of "\0" and CRLF when SP3 sends the UDP packets.
https://github.com/leeseungcheol/smartp ... e56800215c

If there is no other side effect, it will be included in the next firmware release.

rpgfun
Posts: 7
Joined: Wed Dec 29, 2021 4:10 am
languages_spoken: english
ODROIDs: HC1, HC2, C1, SP3
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: UDP logging adds 0x00 chars

Post by rpgfun »

This sound good. But just a note regarding the CR: I don't object to the CR in the UDP-stream, it is important for Windows users and I can remove it myself on Linux with the tr-command. I would suggest not to remove it from UDP.

But I object to the order of newline and carriage-return. In your code you use `\n\r` instead of `\r\n` (i.e. LFCR instead of CRLF). I think this was just a sort of typo from the beginning.

Historically, DOS/Windows is using CRLF, Linux is using LF and (old) macOS is using CR. Only RiscOS is using LFCR, but who in the world is still using that OS?

lsc1117
Posts: 280
Joined: Thu Aug 22, 2013 12:46 am
languages_spoken: english
Location: South Korea
Has thanked: 9 times
Been thanked: 37 times
Contact:

Re: UDP logging adds 0x00 chars

Post by lsc1117 »

Oh, I misunderstood your comment.

I didn't know the difference between CRLF and LFCR. Thank you for letting me know that.
But, could you explain the difference more?

User avatar
mctom
Posts: 980
Joined: Wed Nov 11, 2020 4:44 am
languages_spoken: english, polish
ODROIDs: N2+, Game Advance, a few XU4
Location: Gdansk, Poland
Has thanked: 102 times
Been thanked: 123 times
Contact:

Re: UDP logging adds 0x00 chars

Post by mctom »

The difference between two is just a CR and LF byte order. Windows expects CRLF, so if it gets LFCR instead, it might be incompatible with some COM terminals.
Historically CR caused the printer head to go back to a beginning of a line, and LF advanced paper one line forward. Some systems assumed that both these operations always happen together, so skipped one of these two bytes. The CR+LF order didn't matter to a printer, but as a convention CRLF was adopted as a newline in Windows.

I also think this is a beneficial change, and thanks for pointing it out.
These users thanked the author mctom for the post (total 2):
rpgfun (Sat Jan 01, 2022 5:10 pm) • lsc1117 (Mon Jan 03, 2022 2:08 pm)
Punk ain't no religious cult, punk means thinking for yourself!

Maintainer of PiStackMon

rpgfun
Posts: 7
Joined: Wed Dec 29, 2021 4:10 am
languages_spoken: english
ODROIDs: HC1, HC2, C1, SP3
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: UDP logging adds 0x00 chars

Post by rpgfun »

mctom wrote:
Fri Dec 31, 2021 9:42 pm
The difference between two is just a CR and LF byte order. Windows expects CRLF, so if it gets LFCR instead, it might be incompatible with some COM terminals.
Historically CR caused the printer head to go back to a beginning of a line, and LF advanced paper one line forward. Some systems assumed that both these operations always happen together, so skipped one of these two bytes. The CR+LF order didn't matter to a printer, but as a convention CRLF was adopted as a newline in Windows.

I also think this is a beneficial change, and thanks for pointing it out.
Good explanation, just one thing to add: Linux programs also might not work as expected when they detect LFCR instead of CRLF. The latter is often silently and transparently converted to LF.
These users thanked the author rpgfun for the post:
lsc1117 (Mon Jan 03, 2022 2:08 pm)

lsc1117
Posts: 280
Joined: Thu Aug 22, 2013 12:46 am
languages_spoken: english
Location: South Korea
Has thanked: 9 times
Been thanked: 37 times
Contact:

Re: UDP logging adds 0x00 chars

Post by lsc1117 »

@mctom, @rpgfun,

Thank you for your great explanation. I have understood the difference.

I have updated the fixing of the wrong byte order.

fix the wrong byte order to CRLF from LFCR
https://github.com/leeseungcheol/smartp ... 302d101cb6

remove '\0' and carriage return in UDP packet buffers
https://github.com/leeseungcheol/smartp ... e56800215c

rpgfun
Posts: 7
Joined: Wed Dec 29, 2021 4:10 am
languages_spoken: english
ODROIDs: HC1, HC2, C1, SP3
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: UDP logging adds 0x00 chars

Post by rpgfun »

I have updated the fixing of the wrong byte order.

fix the wrong byte order to CRLF from LFCR
https://github.com/leeseungcheol/smartp ... 302d101cb6

remove '\0' and carriage return in UDP packet buffers
https://github.com/leeseungcheol/smartp ... e56800215c
The last patch now (after swapping LFCR to CRLF) removes the LF instead of the CR. I would just use "SIZE_xxx-1" for all udp.writes, keeping the CRLF. This would keep data transmitted by Serial.printf() and udp.write() identical. And Windows-users will also be happy (if they dare to run netcat after all :-D)

User avatar
mctom
Posts: 980
Joined: Wed Nov 11, 2020 4:44 am
languages_spoken: english, polish
ODROIDs: N2+, Game Advance, a few XU4
Location: Gdansk, Poland
Has thanked: 102 times
Been thanked: 123 times
Contact:

Re: UDP logging adds 0x00 chars

Post by mctom »

@rpgfun you know you may fork and pull request any changes on github, right? I highly recommend doing that, will be quicker. ;)
Punk ain't no religious cult, punk means thinking for yourself!

Maintainer of PiStackMon

rpgfun
Posts: 7
Joined: Wed Dec 29, 2021 4:10 am
languages_spoken: english
ODROIDs: HC1, HC2, C1, SP3
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: UDP logging adds 0x00 chars

Post by rpgfun »

mctom wrote:
Tue Jan 04, 2022 5:17 pm
@rpgfun you know you may fork and pull request any changes on github, right? I highly recommend doing that, will be quicker. ;)
perfectly valid, but I have not tested PIO with the smart-power-3, and I don't want to submit a pull request for untested code. But that is on my todo-list.

(off-topic, but: I had a look at your pistackmon, very nice, I will try to build on)
These users thanked the author rpgfun for the post:
mctom (Fri Jan 07, 2022 6:10 am)

User avatar
mctom
Posts: 980
Joined: Wed Nov 11, 2020 4:44 am
languages_spoken: english, polish
ODROIDs: N2+, Game Advance, a few XU4
Location: Gdansk, Poland
Has thanked: 102 times
Been thanked: 123 times
Contact:

Re: UDP logging adds 0x00 chars

Post by mctom »

You may still submit it and ask someone else for testing. It's still helping even if @lsc1117 himself would have to test it.
I just thought this may be a better idea than describing proposed changes in code here. :)
Punk ain't no religious cult, punk means thinking for yourself!

Maintainer of PiStackMon

Post Reply

Return to “Smart Power”

Who is online

Users browsing this forum: No registered users and 1 guest