13:51 <fbytez> It seems that `(char)` and `(peek)` skip single NULL bytes and return NIL on consecutive ones;

13:51 <fbytez> `(line)` seems to read them as any other character;

13:52 <fbytez> `(from)` seems to pay them no attention;

13:52 <fbytez> `(till)` will always stop at a NULL.

13:54 <abu[m]> This may well be. As I said, a null byte is not expected in an input stream. Thus it is used as "uninitialized" for the internal $Chr global variable.

13:56 <fbytez> I would've thought that `(line)` and `(from)` were implemented in terms of `(char)` and `(peek)`, yet the behaviour is different.

13:57 <abu[m]> They all use $Chr, but do not really call (char) or (peek)

13:58 <fbytez> Why do they treat the (= $Chr 0) differently?

13:58 <abu[m]> Basically, $Chr is checked before every read operation, and if zero, a prefetch triggered to fill it first

13:58 <abu[m]> Stress here on *before*

13:59 <abu[m]> i.e. to handle a freshly opened stream

13:59 <abu[m]> after that, $Chr is assumed to hold a non-zero byte

14:00 <fbytez> Yeah, I'm not understanding why `(line)` doesn't skip but `(char)` and `(peek)` do.

14:00 <fbytez> Well, and `(line)` doesn't get "interrupted".

14:01 <abu[m]> It is all a consequence of the above

14:01 <abu[m]> They do not "skip"

14:01 <abu[m]> They just fill on the first call

14:03 <abu[m]> (vi 'line)

14:03 <abu[m]> ... (unless C (setq C (call $Get))) ...

14:04 <abu[m]> Similar in all reading functions

14:04 <abu[m]> (de read1 ((i32 . End)) (unless (val $Chr) (call $Get) )

14:05 <fbytez> I mean in the sense that `(line)` "provides" the NULLs whereas the others are "interrupted"; can't determine EOF.

14:05 <abu[m]> As (char) and (peek) handle only one single char, this test happens upon each call

14:05 <abu[m]> I don't remember the details of (line) atm

14:08 <abu[m]> (out "a" (wr 65) (wr 0) (wr 66) (wr 10))

14:08 <abu[m]> (in "a" (line)) -> ("A" NIL "B")

14:09 <abu[m]> Anyway, this is undefined behavior

14:09 <fbytez> It's not something you'd ever change?

14:10 <abu[m]> Never ever ☺

14:11 <fbytez> That's unfortunate.

14:11 <abu[m]> You can produce tons of illegal input

14:11 <abu[m]> It is not only the null bytes

14:11 <abu[m]> Why care? Garbage in, garbage out

14:12 <fbytez> Security experts could give you better answers than me.

14:13 <fbytez> That aside, why waste time processing garbage as though it's not?

14:13 <abu[m]> It is not a security issue

14:13 <fbytez> "Fail fast".

14:14 <abu[m]> No! Why waste time checking for thins which should be checked at better places

14:15 <abu[m]> In 99% percent you are *sure* there is no null byte. Why check each time?

14:15 <abu[m]> Do you also check that after adding 3 and 4 it is *really* 7 ?

14:16 <fbytez> Depends on how it has supposed been determined.

14:17 <abu[m]> If you generated the data, they are OK. Period. If they might be damaged during transmission, you need general integrity checks

14:17 <fbytez> "IF"

14:17 <abu[m]> not at hoc stupid isolated checks

14:17 <abu[m]> Come on!

14:18 <fbytez> "Come on" indeed. A server doesn't mostly receive data that it generated itself.

14:19 <abu[m]> Then they must be checked in an intelligent way. Not at the low level byte reading.

14:19 <fbytez> Byte is what all the data is.

14:19 <abu[m]> But not only null bytes

14:20 <fbytez> Indeed.

14:20 <abu[m]> If a client sends garbage, it may be wrong data

14:20 <abu[m]> Or non-UTF

14:20 <abu[m]> doesn't matter

14:20 <fbytez> Yep. It has be parsed / validated / whatever.

14:20 <abu[m]> Pil does not crash, just the user's data are garbage

14:21 <abu[m]> You are free to modify pil input Have fun!

14:22 <abu[m]> I think this is not a security problem. It is all about the semantics of user data.

14:23 <abu[m]> It may be all legal digits, but still the numbers might be wrong

14:24 <abu[m]> There is no general check to be hardcoded, and which then eats CPU time every time

14:25 <abu[m]> Then, the next problem is that if you abort at such a low level, it is very hard to handle at the higher application levels

14:26 <abu[m]> Have you ever written a real practical application?

14:27 <abu[m]> And do you know what "premature optimations" are?

14:28 <fbytez> Do you know how sensitive, defensive and rude you are?

14:28 <abu[m]> Oops, no, I don't know. Sorry!

14:29 <abu[m]> I just asked you

14:30 <abu[m]> Your views are typical for academic text books, from authors who never wrote a real program ;)

14:30 <abu[m]> So I assumed you are a student

14:32 <abu[m]> Perhaps I'm rude because all this is totally opposite the PicoLisp philosopy