<razzy>
can Vip hilight some letters? or otherwise mark for user?
<Regenaxer>
There is the "mark" concept from Vim/Vi, which marks positions
<Regenaxer>
And then also a :m command (vip only) which writes the position to a file if the buffer is closed
<Regenaxer>
I use it for reading e-books
<razzy>
nice :D
<Regenaxer>
But there is no visual highlighting
<Regenaxer>
The Vim/Vi marking works with 'm' in command mode
<Regenaxer>
m plus any char
<Regenaxer>
ma marks the current position as "a"
<razzy>
Regenaxer: "mark" is not visual?
<Regenaxer>
then all commands work with it
<Regenaxer>
nope
<Regenaxer>
`a jumps to "a"
<Regenaxer>
d`a deletes till "a" etc.
<razzy>
can it be done?
<Regenaxer>
yes, but I won't do it :)
<Regenaxer>
It would be useless unless also all editing operations work on it
<Regenaxer>
So this means a complete new machinery
<Regenaxer>
If you need it, you can implement it
theruran has quit [Ping timeout: 268 seconds]
theruran_ has joined #picolisp
<razzy>
I want some visual marker. change collour on some letter, light up field under letter, change font. anything will do. is something from that easy?
<Regenaxer>
Something similar is there already for transient symbols and comments
<Regenaxer>
see the 'markup' function
<Regenaxer>
I think Erik Gustavson did some such things too. See mailing list
<razzy>
Regenaxer: perfect, it did not occur to me :D, not woken up. Thank you.
Hunar has joined #picolisp
Hunar has quit [Quit: Client closed]
Hunar has joined #picolisp
<Hunar>
Hello :) I initially thought the chat log libera.irclog.whitequark.org/picolisp can be found with google search, but the results I found were from freenode libera.irclog.whitequark.org/picolisp .. is this intentional? can we add it ourselves? or the domain owner has to do it
<Hunar>
Second link was wrong
<Hunar>
freenode.irclog.whitequark.org/picolisp
<beneroth>
that is the old archive
<Hunar>
Is the new archive google-search-able?
<beneroth>
we used freenode IRC (like many/most FOSS groups) before it was taken over by a rich maniac and libera was created by the former freenode staff
<beneroth>
I believe so, it's the same website, same guy, just a different subdomain
<beneroth>
Hunar, add "site:libera.irclog.whitequark.org/picolisp" to your search term
<beneroth>
thought older results from freenode might be useful too, depending on the topic :)
<Hunar>
I prefer the built-in search .. but I almost never found results from the new archive with google
<v_m_v>
I mean... it is working ..it is going and running jsonResponse function..but I am not getting any results on my Curl. It looks like cons is not returning anything ...right ?
<Regenaxer>
The conses are ok
<Regenaxer>
(cons "errorMsg" "Cannot parse...) is constant though
<Regenaxer>
And @Resp should bette be saved with (use @Resp ) but just to be on the safe side
<v_m_v>
I am wondering why my cond even if it is working it is not returning my jsonResponse. I am using jsonResponse in my other endpoints so I know that it is fine.
<v_m_v>
The only thing that I am using here and not in the other functions is ...cond.
<Regenaxer>
The cons results are passed to 'jsonResponse'
<v_m_v>
I've printed what I am getting to jsonResponse and it was fine .
<Regenaxer>
maybe, but just to see the flow
<v_m_v>
It is just...like ...because there is cond it would be not returned as a return statement of my function
<Regenaxer>
what happens
<Regenaxer>
I recommend to trace and/or single-step to see what happens
<v_m_v>
Oh...you know what ...double deffinition of the httpHeader was the issue
<v_m_v>
silent issue
<v_m_v>
it was the same deffinition
<Regenaxer>
oh, what does that mean?
<Regenaxer>
double definition is ok
<Regenaxer>
is the same, or says "redefined"
<v_m_v>
(httpHead "application/Json" 0) defined two times
<v_m_v>
when I have removed one of them ...everything was fine
<Regenaxer>
you mean called two times?
<Regenaxer>
ok, great!
<v_m_v>
yes ..sorry...called
<v_m_v>
You have mentioned something about (use... )
<Regenaxer>
Yes, just to be safe
<v_m_v>
why safe?
<Regenaxer>
bind the @ symbols locally
<Regenaxer>
Oherwise they are global
<v_m_v>
oh
<v_m_v>
ooohhh!!
<Regenaxer>
'use' is like 'let' without value
<v_m_v>
I see.
<Regenaxer>
It is only a problem if that var is used also by a calling function
<v_m_v>
You know what? Each day my PIL codebase is getting smaller and smaller :D It is amazing how small your codebase can be in PIL and how fast you can get the right results. You only need to know ..how.
<Regenaxer>
good :)
<v_m_v>
So this is like a 5th iteration over the same code ... and each time I am removing more then adding :D
<Regenaxer>
:)
<beneroth>
:)
<beneroth>
v_m_v, wait until you get the good and bad feeling, at the same time, after many hours of successfully reducing a 10 liner into 4 lines of picolisp xD
<beneroth>
it's like: "wonderful. so elegant. so simple. awesome!" and "so many hours for ..this few lines"
<Regenaxer>
true :)
<Regenaxer>
So better not charge customers by LOC
<beneroth>
then again, as Djikstra used to say (parphrasing): lines of codes should be considered not as "created" but as "spent"
<beneroth>
Regenaxer, yeah thats stupid. worked well for MS though :)
<v_m_v>
Hmm PIL database is quite fast when we are talking about getting data but it is not that fast when we are talking about adding data
<Regenaxer>
yes, depends on the index structures
<v_m_v>
Hmm can I make it faster?
<Regenaxer>
and you may need to optimize the data import
<v_m_v>
I mean ...it is really not that fast.
<v_m_v>
maybe bult inserts?
<v_m_v>
*bulk
<Regenaxer>
The OSM import with 400 million objects took me about one day iirc
<Regenaxer>
How many data do you have
<v_m_v>
almost nothing. I've just tested my API using "wrk" tool. For quering it is really fast..but for addint ..I am getting timeouts quite fast
<Regenaxer>
oh, this should not be
<Regenaxer>
I'm talking about index sizes which exceed RAM size
<v_m_v>
I have 8 gigs of RAM and under 200 elements in my DB ...
<Regenaxer>
and how many objects to add?
<v_m_v>
So...4 threads 50 connections od 30 seconds of constant sending request ..is getting me ...27 timeouts.
<Regenaxer>
maybe the DB locking
<v_m_v>
5257 requests in 30 seconds
<Regenaxer>
you insert with 'put!>' ?
<v_m_v>
with request!
<Regenaxer>
so there are many child processes each locking the db
<v_m_v>
oh...
<v_m_v>
should I change it to put!> ?
<Regenaxer>
it is a lot faster if a single process imports, and does request instead of request! and cimmits every few hundred thousand inserts :)
<Regenaxer>
no, put!> is the same
<Regenaxer>
it locks the db, does a single transaction, and commits
<Regenaxer>
very expensive
<Regenaxer>
but necessary if not a single process accessing the db at the same time
<v_m_v>
hmm so it would be better to ... have a one PIL app for putting request of adding into message brocker...and then the other one for getting a bunch of those items and putting them into db ...many at the same time ?
<v_m_v>
Can two PIL apps have access to the same DB ?
<Regenaxer>
only as long as they have the same parent process
<v_m_v>
if yes then I can use message brocker and put many items in the same commit
<Regenaxer>
the parent controls it
<Regenaxer>
I once used a non-forking server for such a purpose
<Regenaxer>
so a single process does it all
<v_m_v>
hmm so I can not have two PIL apps run on separate system threads ?
<Regenaxer>
and commits only after many transactions
<Regenaxer>
two different applications?
<Regenaxer>
or processes?
<v_m_v>
applications
<Regenaxer>
with the same data model?
<Regenaxer>
anyway, they must share a common parent
<v_m_v>
one application is getting events from message brocker and putting them into db ..the other one is just a server with a simple db queries
<Regenaxer>
the parent does all synchronization
<v_m_v>
what do you mean as parent?
<Regenaxer>
Parent process
<Regenaxer>
children are forked
<Regenaxer>
the children can be completely different applications
<v_m_v>
Hmm even if that would be two different apps ?
<Regenaxer>
yes
<v_m_v>
hmm ok so at the end the parent would be still an issue
<beneroth>
v_m_v, sounds like you did something wrong
razzy has joined #picolisp
<Regenaxer>
each child (load)s its code after (fork)
<v_m_v>
hmm can parent be only responsible for adding events into db?
<beneroth>
parent should not be busy, better use a dedicated child for that
<Regenaxer>
yes
<Regenaxer>
but if one child locks the db a long time
<Regenaxer>
other children are blocked
<v_m_v>
ok...so decicated child ..when I am using function "request" and then I am making "commit" from time to time?
<Regenaxer>
Whe locking is not so bad
<Regenaxer>
it is only during commit
<beneroth>
v_m_v, also, you are aware of the ca. 512 connection/child processes limit? comes from: every child has 2 pipes with parent (each direction), and standard config of Linux a single process may only have 1024 file descriptors open at same tim
<v_m_v>
Can I somehow prepare the query without locking the db?
<Regenaxer>
the other time other children can read the db
<Regenaxer>
only during the final commit the db is also read locked
<v_m_v>
so...having a one child only responsible for writing is not that bad. That child can ask about event the message brocker. So there would be back pressure. Other childs would only read from DB.
<Regenaxer>
after (dtSync) it is write-locked
<v_m_v>
I see.
<beneroth>
concurrent writing requires locking. get better performance by having (dbSync)...(commit 'upd) as short as possible, and by using multiple database files, see (dbs)
<beneroth>
v_m_v, are you using multiple files already?
<v_m_v>
how I can check it ?
<beneroth>
you would know, if you would do it ;-)
<Regenaxer>
multiple files in db/ ?
<v_m_v>
In DB folder I have @ A B C D E F G H I J K ...but it means nothing right ?
<Regenaxer>
or sources per app?
<beneroth>
you would give (pool) a directory instead of a single database file, and you would assign index and objects to files using (dbs)
<Regenaxer>
yes, so all right
<beneroth>
so you have multiple files
<Regenaxer>
the children would share those
<Regenaxer>
cause the parent opens it
<Regenaxer>
So this concept should be all right
<Regenaxer>
two processes
<Regenaxer>
the parent just has the er.l and opens the db
<Regenaxer>
one child loads reading code and waits for requests
<Regenaxer>
the other child loads the updating code
<Regenaxer>
it does the writing whenever import data arrive
<Regenaxer>
it should collect a few thousand data, then commit
<Regenaxer>
the commit takes a fraction of a second, during that time the reading child block
<Regenaxer>
perhaps only collect a few hundred objects
<Regenaxer>
then import is slower but the blocking shorter
<Regenaxer>
hmm, sorry, must go
<Regenaxer>
afp, back later
<v_m_v>
I will do it as you have said
<beneroth>
it is true that pilDB is not so well suited for high number of concurrent writes
<beneroth>
the default write locking mechanism is simple but not very advanced. more complex locking mechanics could be made, but the approach with a more coordination / specific child process to collect and batch writes is certainly easier
<beneroth>
in my productive applications I've never exhausted the performance of the pilDB, so far it was always well enough. but I run mostly business applications or websites (which have much more data reading than writing)
<beneroth>
pilDB is quite different to the major database architectures used, because changes are in place, not using a WAL. This makes the whole database usage and housekeeping very much simpler, but comes with some trade-offs.
<v_m_v>
I am collecting events from sorounding apps
<v_m_v>
Just put it into levelDB and then take it and process it later.
<beneroth>
yeah, this is a niche where you might want to scale horizontally (multiple separated pilDB instances, maybe for each type of event), or go to a NoSQL-database which were mainly development for this kind of data (but mostly without good consistency guarantees)
<v_m_v>
Still I see hope here. I just need to change my idea of using PIL
<beneroth>
you could also make a kind of your own LevelDB on top of the lower-level key-value system of pilDB and with doing batching as discussed before
<beneroth>
yeah
<beneroth>
Regenaxer (and also me) use it primarily for business applications with up to a few 100 users who seldom write concurrently the exact same entities
<beneroth>
for this type of applications pilDb is best, I'm sure, I can compare.
rob_w has quit [Ping timeout: 245 seconds]
<v_m_v>
Hmm I've saw some binding into message brockers for PIL...I will try one or two.
<v_m_v>
It is on so early stage...
<beneroth>
the most beautiful thing about pilDB is, that most part is implemented in picolisp (see @lib/btree.l and @lib/db.l) and fully transparent and debugable
<Regenaxer>
I'm still out in the rain, just a short note: In your (go) function do (unless (fork) (loop (dbSync) (do 1000 (when (rd) (request ...))) (commit 'upd)))
<beneroth>
v_m_v, yeah aw- did some work on that front, check out his github, he writes very good code and documentation
<v_m_v>
at the end I need to validate each event (gpg keys cert)...it will take some time.
<Regenaxer>
(rd) assumes PLIO input, which is the most efficient way
<beneroth>
ok, yeah if you require good data integrity than pilDB is better than NoSQL databases which are optimized for high concurrent writes but sacrified data integrity for it and can lose data
<Regenaxer>
I would use a separate process piping data per PLIO to that child
<Regenaxer>
bbl
<beneroth>
Regenaxer, sounds good. I see what you mean and agree! But I suspect v_m_v will need a bit more in-depth explanation before he can pull this off ;-)
<v_m_v>
:D
<v_m_v>
true
<v_m_v>
for now I would just keep it as it is ...it is simple. Then I will move it to other app
<v_m_v>
90% of the code would be the same
razzy has quit [Quit: leaving]
<aw->
Regenaxer: hi, I can't seem to end a process with ctrl+c anymore
<aw->
i have a pil process that spawns a child, and then I ctrl+c in the terminal but the parent or child just seem to ignore it
rob_w has joined #picolisp
<aw->
: (version)
<aw->
21.11.22
<beneroth>
Regenaxer is currently out in the rain, so might be a bit delayed until he can respond
<beneroth>
aw-, what about Ctrl+D ?
<aw->
same
<aw->
hi beneroth
<beneroth>
aw-, maybe try with different terminal? I had such problems, but then it was the terminal not pil. But I use older pil64, so maybe there is really a bug in current pil21
<aw->
actually the child process does respond to ctrl+c, but the parent doesnt
<beneroth>
hi :)
<aw->
worked fine in pil64
<beneroth>
hmmm.. no idea, sorry
<aw->
previously, ctrl+c in the terminal would kill the parent process (cleanly)
<aw->
well i separate bug too
<beneroth>
yeah that should work
<beneroth>
or is the process stuck in a infinite busy loop, doing never (wait) and so might ignore SIGTERM sent to it?
<beneroth>
you can diagnose this with cpu load or 'strace'
<beneroth>
I need to do an errand, will be back soonish, bbl
<aw->
beneroth: wow!
<aw->
beneroth: did you test with pil21?
<beneroth>
haha
<beneroth>
damn no, old pil64 xD
<beneroth>
sorry
<aw->
:P
<aw->
yeah i know it worked fine on pil64
<aw->
no problem
<aw->
ttyl
<beneroth>
rofl.. sorry for my useless help
<beneroth>
btw. interesting project, looks like something I need to check out
<beneroth>
bbl
<aw->
hmmm... ok I managed to break out of it with ctrl+z then ctrl+c which dropped me to a ! prompt
<Regenaxer>
ret
<Regenaxer>
hmm, aw-, is the process in a tight loop perhaps?
<Regenaxer>
signals are not handled in such cases
<Regenaxer>
but if ^Z works, then this is not a tight loop
<Regenaxer>
^C is initially the default, i.e. terminate the process, and then redirected to a debug break once a repl was open
<Regenaxer>
So ^C shoul never be ignored
rob_w has quit [Remote host closed the connection]
razzy has joined #picolisp
aw- has quit [Ping timeout: 240 seconds]
<v_m_v>
does PIL have bindings to Postgres?
<Regenaxer>
I do not know of any
<v_m_v>
BTW I have noticed that after some time my PIL server is consuming 99-100% of my CPU....without any load....
<v_m_v>
This is not the first time
<v_m_v>
This is the newest PIL
<Regenaxer>
Should not be
<Regenaxer>
Got into some infinite loop
<v_m_v>
it is ... each day after couple of hours
<Regenaxer>
Perhaps strace -p <pid> helps?
<v_m_v>
I will check it next time. Now I've killed it
<v_m_v>
But there is nothing which could make such loops
<Regenaxer>
Is the process size growing?
<v_m_v>
nope. Only the CPU.
<Regenaxer>
Interesting
<v_m_v>
I've it a couple a days in row
<Regenaxer>
Is CPU usage increasing slowly, or does it go up all of a sudden to 100% ?
<v_m_v>
I don't know. My lap was warmer then usual. I've checked processed by htop and there was one with 100%
<v_m_v>
PIL
<v_m_v>
I thought that ...maybe this is one of my PIL repl in emacs...but then I have noticed that it is the server. Each time
<Regenaxer>
I see
<v_m_v>
Now after restarting it is 0.1%
<Regenaxer>
Could it depend on the data?
<v_m_v>
what do you mean?
<Regenaxer>
Is it all UTF-8 ?
<v_m_v>
hmmm maybe. I don't know
<Regenaxer>
Should not really matter
<Regenaxer>
But legal is only UTF-8 ;)
<v_m_v>
on stress test it is 20%
<v_m_v>
and ..after stress test ..0.1%
<Regenaxer>
ok
<v_m_v>
I will write to you next time when I will get 100%
<Regenaxer>
Does it make sense to 'trace' some parts?
<Regenaxer>
To have a hint where it happens
<v_m_v>
Hmmm there could be one more thing. My linux here is virtualised
<v_m_v>
It is chromebook
<v_m_v>
I mean...I don't have such problems with anyothers apps
<Regenaxer>
Should be ok
<Regenaxer>
I would 'trace' some functions, and/or put 'msg' to find *where* it hangs
<Regenaxer>
Problem is that there are many processes
<Regenaxer>
So 'msg' is better, with *Pid
<Regenaxer>
(msg *Pid " on point 1")
<Regenaxer>
(msg *Pid " on point 2")
<Regenaxer>
...
<v_m_v>
Can I get access to PIL process from outside ?
<v_m_v>
like in Erlang?
<v_m_v>
debugging VM remotely?
<Regenaxer>
yes, if it listens on http, you can use psh
<Regenaxer>
gives a repl
<Regenaxer>
if it is a normal GUI program, you can even connect to an existing session with psh
<Regenaxer>
you need the port and session id for that
<Regenaxer>
I showed in PilCon some time ago how to do that
<beneroth>
v_m_v, use linux command 'strace' to check what the 100% CPU process is doing. usually infinite loop attempting to read from an invalid/broken file descriptor (usually pipe)
<beneroth>
strace -p <pid of process>
<Regenaxer>
yes, or it simply shows nothing, then it is in a tight code loop :)