<phantomics>
I've also built web servers with CL, mostly serving React apps
<Guest82>
oh, that's what I'm trying to do phantomics :)
<Guest82>
did you use ngnix at all or just lisp?
<Guest82>
what libraries do you use?
<Guest82>
qlot and utopian or caveman2?
<Guest82>
or just hunchenhoot?
zacts has quit [Ping timeout: 246 seconds]
zacts40 has quit [Ping timeout: 246 seconds]
lotuseater has quit [Ping timeout: 245 seconds]
<Guest82>
phantomics do you use mito or your own classes?
<phantomics>
Just built my own classes, haven't used with a relational DB
<Guest82>
ok, thanks... I'm trying now. I hope I will learn lisp but also be successful making an app haha
<phantomics>
What's your goal app?
aidenw has quit [Quit: Leaving]
waleee has quit [Ping timeout: 276 seconds]
engblom has quit [Quit: Lost terminal]
Guest82 has quit [Quit: Client closed]
<pjb>
phoe: note that deftype is like defmacro. If you allow closures in satisfies, each time the type is expanded you get a different closure!
<pjb>
phoe: AFAICS, this is the reason why we're restricted to functions (that have to be defined in the compilation environment too, most of the time, so eval-when).
<pjb>
phoe: but granted, it makes it very difficult to define list-of (deftype list-of (element-type) `(satisfies ,(once-or-twice (eval `(defun ,(gensym) (object) (is-proper-list-of ,',element-type object)))))) etc…
<pjb>
in the current spec, you need to define a toplevel function for each type of list-of lists.
<pjb>
Yes, that works, with eg. (defun is-proper-list-of (object element-type) (and (proper-list-p object) (every (lambda (element) (typep element element-type)) object)))
zacts has joined #commonlisp
<pjb>
But as I said, this generates a new closure and a new toplevel function for each occurences of (list-of foo) (list-of foo) (list-of foo) -> 3 equal closures, 3 toplevel functions. So you want to manage a cache, which has the misfortune of having to live across the compilation-environment and the runtime environment. ie. you need to rebuild the cache at load-time or execute…
<pjb>
satisfies would have to do the same without necessarily knowing what keys to use for the cache, if it had to deal with closures itself. at least, when the user does it, he knows what he's doing…
pranavats has left #commonlisp [#commonlisp]
zacts has quit [Quit: Client closed]
zacts has joined #commonlisp
zacts has quit [Write error: Broken pipe]
lisp123_ has joined #commonlisp
zacts has joined #commonlisp
lisp123_ has quit [Ping timeout: 276 seconds]
prxq_ has joined #commonlisp
prxq has quit [Ping timeout: 265 seconds]
zacts has quit [Ping timeout: 246 seconds]
akoana has left #commonlisp [Leaving]
zacts has joined #commonlisp
frgo has joined #commonlisp
<beach>
Good morning everyone!
frgo has quit [Ping timeout: 272 seconds]
<char>
Hello beach
pranavats has joined #commonlisp
<phantomics>
Morning beach
zacts has quit [Ping timeout: 246 seconds]
pillton has joined #commonlisp
makomo has quit [Quit: WeeChat 3.0.1]
Bike has quit [Quit: sleep]
srhm has quit [Quit: Konversation terminated!]
Inline has quit [Quit: Leaving]
waaron has quit [Quit: WeeChat 3.0.1]
lisp123_ has joined #commonlisp
cjb has quit [Read error: Connection reset by peer]
selwyn has joined #commonlisp
<char>
I just want to confirm that there is no common lisp reverse proxy?
<hayley>
I can't think of one, but nginx has worked fine for me. Is there anything specific you need from it?
gaqwas has joined #commonlisp
<edgar-rft>
(reverse "proxy") => "yxorp" if that counts
<recordgroovy>
that would make a good name for a reverse proxy
lisp123_ has quit [Ping timeout: 272 seconds]
shka has joined #commonlisp
silasfox has quit [Ping timeout: 276 seconds]
silasfox has joined #commonlisp
Lord_of_Life_ has joined #commonlisp
Lord_of_Life has quit [Ping timeout: 258 seconds]
Lord_of_Life_ is now known as Lord_of_Life
<susam>
Good morning, beach!
<pjb>
Good morning, susam!
<susam>
Good morning, pjb!
<pjb>
susam: beach is on food supply chore right now. He'll be back soon.
amb007 has quit [Ping timeout: 256 seconds]
amb007 has joined #commonlisp
dickbar__ has quit []
rain3 has joined #commonlisp
pillton has quit [Quit: ERC (IRC client for Emacs 27.2)]
amb007 has quit [Ping timeout: 258 seconds]
amb007 has joined #commonlisp
recordgroovy has quit [Ping timeout: 265 seconds]
gaqwas has quit [Ping timeout: 258 seconds]
tfb has quit [Quit: died]
amb007 has quit [Read error: Connection reset by peer]
heisig has joined #commonlisp
amb007 has joined #commonlisp
peterhil has joined #commonlisp
lisp123_ has joined #commonlisp
kakuhen has quit [Ping timeout: 250 seconds]
lisp123_ has quit [Ping timeout: 240 seconds]
attila_lendvai has joined #commonlisp
attila_lendvai has quit [Remote host closed the connection]
attila_lendvai has joined #commonlisp
frgo has joined #commonlisp
frgo has quit [Ping timeout: 256 seconds]
recordgroovy has joined #commonlisp
pve has joined #commonlisp
amb007 has quit [Read error: Connection reset by peer]
amb007 has quit [Read error: Connection reset by peer]
amb007 has joined #commonlisp
amb007 has quit [Read error: Connection reset by peer]
dsk has quit [Ping timeout: 256 seconds]
amb007 has joined #commonlisp
lisp123_ has joined #commonlisp
lotuseater has joined #commonlisp
lisp123__ has joined #commonlisp
lisp123_ has quit [Read error: Connection reset by peer]
lisp123__ has quit [Remote host closed the connection]
amb007 has quit [Read error: Connection reset by peer]
amb007 has joined #commonlisp
amb007 has quit [Read error: Connection reset by peer]
amb007 has joined #commonlisp
lisp123__ has joined #commonlisp
lisp123__ has quit [Ping timeout: 240 seconds]
lisp123_ has joined #commonlisp
lisp123_ has quit [Ping timeout: 240 seconds]
kakuhen has joined #commonlisp
cage has joined #commonlisp
lisp123_ has joined #commonlisp
lisp123_ has quit [Ping timeout: 240 seconds]
lisp123_ has joined #commonlisp
selwyn has quit [Read error: Connection reset by peer]
<lisp123_>
would it be possible to implement a reverse proxy through hunchentoot?
<flip214>
lisp123_: of course.
<flip214>
you can use drakma to make outgoing HTTP connections.
<lisp123_>
flip214: then that would be the answer to the earlier query on how to reverse proxy in CL
<lisp123_>
thanks for confirming
<flip214>
unless there is some specific rewriting to do, I'd go for a standard product, though (nginx, apache, haproxy, etc.). even casual text changes (via regexes) can be done in apache, for example.
<lisp123_>
yeah nginx is a very easy to use reverse proxy imo (well since I know how to use it, hence its easy to use :D)
<flip214>
lisp123_: depending on the answer time of the http services "behind" the proxy you might have quite a lot of simultaneous active http connections.
<flip214>
HT by default creates threads; quux-ht uses a thread pool; perhaps you should take a look at tpd2 or so, to have a bounded number of threads for an (more or less) unbounded number of connections.
frgo has joined #commonlisp
frgo has quit [Remote host closed the connection]
frgo has joined #commonlisp
<lisp123_>
flip214: Good to know :) I was just answering the earlier question on the chat, I've personally moved away from web dev for the foreseeable future
<lisp123_>
perhaps I will come back in the future, after all everything commercial is a SaaS these days
zacts has joined #commonlisp
kevingal has joined #commonlisp
lisp123_ has quit [Remote host closed the connection]
lisp123_ has joined #commonlisp
lisp123_ has quit [Ping timeout: 240 seconds]
selwyn has joined #commonlisp
selwyn has quit [Read error: Connection reset by peer]
lisp123_ has joined #commonlisp
amb007 has quit [Read error: Connection reset by peer]
amb007 has joined #commonlisp
zacts has quit [Ping timeout: 246 seconds]
random-nick has joined #commonlisp
lisp123_ has quit [Ping timeout: 265 seconds]
amb007 has quit [Read error: Connection reset by peer]
amb007 has joined #commonlisp
amb007 has quit [Read error: Connection reset by peer]
amb007 has joined #commonlisp
kakuhen has quit [Quit: Leaving...]
tfeb has joined #commonlisp
tfeb has quit [Client Quit]
frgo has quit [Remote host closed the connection]
frgo has joined #commonlisp
lisp123_ has joined #commonlisp
lisp123_ has quit [Read error: Connection reset by peer]
lisp123_ has joined #commonlisp
tyson2 has joined #commonlisp
igemnace has quit [Remote host closed the connection]
amb007 has quit [Read error: Connection reset by peer]
amb007 has joined #commonlisp
amb007 has quit [Ping timeout: 276 seconds]
yitzi has joined #commonlisp
lisp123_ has quit [Remote host closed the connection]
lisp123_ has joined #commonlisp
lisp123_ has quit [Ping timeout: 258 seconds]
amb007 has joined #commonlisp
Lycurgus has joined #commonlisp
cage has quit [Remote host closed the connection]
cage has joined #commonlisp
cage has quit [Remote host closed the connection]
cage has joined #commonlisp
cage has quit [Remote host closed the connection]
<Josh_2>
Afternoon
cage has joined #commonlisp
Bike has joined #commonlisp
lisp123_ has joined #commonlisp
lisp123_ has quit [Ping timeout: 272 seconds]
rain3 has quit [Quit: Leaving]
selwyn has joined #commonlisp
Inline has joined #commonlisp
mister_m has joined #commonlisp
Inline has quit [Ping timeout: 276 seconds]
heisig has quit [Quit: Leaving]
Inline has joined #commonlisp
waleee has joined #commonlisp
Noisytoot_ has joined #commonlisp
mrSpec has joined #commonlisp
jdz_ has joined #commonlisp
phadthai_ has joined #commonlisp
tyson2 has quit [Quit: ERC (IRC client for Emacs 27.2)]
Noisytoot has quit [*.net *.split]
spec has quit [*.net *.split]
phadthai has quit [*.net *.split]
jdz has quit [*.net *.split]
mrSpec is now known as spec
lisp123_ has joined #commonlisp
selwyn has quit [Read error: Connection reset by peer]
jdz_ has quit [Quit: I'm done]
jdz has joined #commonlisp
lisp123_ has quit [Client Quit]
lisp123_ has joined #commonlisp
hendursa1 has quit [Quit: hendursa1]
hendursaga has joined #commonlisp
zacts has joined #commonlisp
NeoCron has joined #commonlisp
pve_ has joined #commonlisp
pve_ has quit [Client Quit]
pve has quit [Ping timeout: 240 seconds]
NeoCron has quit [Quit: Leaving]
attila_lendvai_ has joined #commonlisp
attila_lendvai has quit [Ping timeout: 256 seconds]
pve has joined #commonlisp
zacts has quit [Ping timeout: 246 seconds]
eta has quit [Ping timeout: 245 seconds]
Lycurgus has quit [Quit: Exeunt]
zacts has joined #commonlisp
attila_lendvai_ has quit [Ping timeout: 276 seconds]
attila_lendvai_ has joined #commonlisp
amb007 has quit [Read error: Connection reset by peer]
amb007 has joined #commonlisp
eta has joined #commonlisp
amb007 has quit [Read error: Connection reset by peer]
amb007 has joined #commonlisp
manicennui has left #commonlisp [#commonlisp]
zacts has quit [Quit: Client closed]
selwyn has joined #commonlisp
Noisytoot_ is now known as Noisytoot
nij- has joined #commonlisp
<nij->
Can someone recommand a noob friendly CL implementation that comes with an editor, works with windows and macos, and that allows the user to start playing with the repl after a quick download?
<beach>
I would think LispWorks would do all that.
<nij->
Oh no I forgot to mentin "free" in price.
<beach>
I think they have free trial editions.
<nij->
I'm not precise enough - free in price forever xD
aeth_ has joined #commonlisp
aeth has quit [Killed (NickServ (GHOST command used by aeth_))]
<nij->
I remember there's one that satisfies all of these requirements.. but I couldn't find its name..
aeth_ is now known as aeth
dj_ has quit [Quit: Connection closed for inactivity]
attila_lendvai_ has quit [Ping timeout: 276 seconds]
<etimmons>
nij-: People like Portacle, but it's not an implementation. Just a bundling of an implementation and other things.
<Shinmera>
Unfortunately macOS' release of Portacle is pretty busted, and I don't have the time... or, rather, the patience and energy to play catch-up with Apples BS.
<Shinmera>
If there's any other poor souls willing to sacrifice themselves to do the work though, by all means, PRs are open.
tyson2 has joined #commonlisp
<beach>
I just don't understand why people put up with that "BS" at all.
<Josh_2>
Because it has an apple logo on it
<Josh_2>
don't you like the logo beach, its all about the brand
silasfox has quit [Quit: WeeChat 3.2]
silasfox has joined #commonlisp
<Shinmera>
beach: In my case it's because other people that do use Apple devices heckle me about it
<nij->
Oh yeas, it's called Portacle! Thanks etimmons!
<shka>
well, it is XXI century, essentially a cyberpunk dystopian future, but boring
<shka>
and corporations can do as they please
luna-is-here has quit [Ping timeout: 245 seconds]
luna-is-here has joined #commonlisp
silasfox has quit [Ping timeout: 252 seconds]
silasfox has joined #commonlisp
<nij->
M1 seems promising though.
<nij->
Wonder how long it will take for it to be ported to nonapple machines.
srhm has joined #commonlisp
yitzi has quit [Quit: Leaving]
dsk has joined #commonlisp
tyson2 has quit [Quit: ERC (IRC client for Emacs 27.2)]
amb007 has quit [Ping timeout: 250 seconds]
amb007 has joined #commonlisp
<jeosol>
* debugging an application that works correctly in serial mode but gives inconsistent results in parallel mode (race problems, etc) is a different kind of hell
aeth_ has joined #commonlisp
aeth has quit [Killed (NickServ (GHOST command used by aeth_))]
aeth_ is now known as aeth
<nij->
Shinmera: if Portacle is just a bunch of implementations, why wouldn't it do well on macos?
<Shinmera>
First fo all, that's being very dismissive of all the work that had to be done, and second the reason is because apple wants you to distribute stuff solely through the app store and makes everything else a huge pain in the ass to force your hand.
<jackdaniel>
I have to confirm, even windows isn't such a pita as macos for maintanance
<shka>
OSX used to be pretty nice platform for the software distribution, shame that it ends this way
<Shinmera>
Turns out when companies are doing well they start fucking everyone over. Who knew???
<nij->
Sign.
<shka>
i guess they want they share from the app store
<nij->
Sigh* Do you think doing that favors the those who will never learn any bit of programming?
<Josh_2>
Shinmera: hopefully you dont start doing well then
<Josh_2>
ofc that is a jk
<shka>
to be fair apple always was trying to fuck people over
<shka>
ever since the OG mac
<Shinmera>
Don't worry, I'm currently bleeding through all of my savings and probably will end up bankrupt within the next year, so me not doing well is virtually guaranteed :)
<Josh_2>
well guess you wont be able to fuck anyone over!
<shka>
they had a single pin incompatible keyboard cable... that cost premium
<shka>
50 years later they are doing more or less the same :P
amb007 has quit [Ping timeout: 256 seconds]
amb007 has joined #commonlisp
<jackdaniel>
I think that it is a matter of incentives; when you respond before a board assigned by shareholders, then you aim for growth and revenue; when you respond to your family business then you seek stability; when you respond to a non-profit then you have "its" goal in mind
<jeosol>
Shinmera: hopefully not, you won't go bankrupt ...
<Shinmera>
I absolutely will if I don't get any grants or other funding.
pjb` has joined #commonlisp
pjb has quit [Ping timeout: 240 seconds]
<shka>
Shinmera: can i buy something from you?
<jeosol>
I am sure we'd be able to rally round. when my lot changes soon, i should be able to make some contributions to your efforts
<shka>
like that platformer game you are working on
<jeosol>
calling out threading experts, my code exhibiting weird behavior and I don't throwing locks here and there is the way to do it. I am using bordeaux threads for simd-like tasks. I have 3 tasks A, B, C that I would like to run in parallel: A writes files required for step B, B does a system call using the file from step A and 3rd party exe, and C
<jeosol>
reads the output of step B.
<jeosol>
For step A, I occassionally, get the inputs jumbled when I look at the files for different cases, e.g., a file for a different task ends up being included for a different task.
<Bike>
i don't understand that last clause
<jeosol>
I would like to chain all three steps A, B, C, but perhaps doing A serially, B parallel, C serially is more tractable
<jeosol>
Bike: apologies
<Josh_2>
Instead of a lock you could use semaphores to signal when B when A is done, and C when B is done
<Josh_2>
signal B when A *
yitzi has joined #commonlisp
<jeosol>
Bike: It is a population-based optimization I am doing and I need to evaluate F(X) for several iterations and populations. Step B is the most expensive and can run for hours, I have created a small case that finishes in 2 seconds for my testing. steps A and C are very fast and not the bottle neck
<jeosol>
issue is when writing the files, some of the indices are jumbled. For instance in a given iteration, say iteration 1, I can have 20 solutions. so I append the iteration index and population index to the ends of files in each folder. e.g., input_iter1_sol2.dat
<shka>
Shinmera: i grabbed two for 15 bucks each, it also looks like something i could appreciate on aesthetically
<jeosol>
All relevant files in the same folder should have same iteration and solution indices: e.g., <prefix file>_iter1_sol2.dat
<jeosol>
Josh_2: thanks, I will take a look
<jeosol>
So far, I have used locks, but it stinks and works in some cases and other it doesn't. However in serial mode, everything is kosher
<Bike>
i'm sorry, you're giving a very high level description of your system but it's hard to understand the particular problem. is it something like, a file produced by an A task is being processed by the wrong kind of B task?
<Josh_2>
does sound to me like you need semaphores, you can use them to pause and synchronize the execution of the tasks. if B depends on the completion of A, then B can sit on its semaphore until A signals it, then B will execute, same for B and C
<jeosol>
Bike: I didn't want to get into too much details. I wanted to understand any CL aspects in case I am missing something. In the above only step A is critical, as if the wrong files ended up in the wrong folder, the application will file as it can't get the folder so I say objective function is some high value 1e10 (for minimization task)
<Bike>
i would guess that there's nothing CL-specific to worry about
<jeosol>
I want to be able to do a call, e.g., like (call-with-thread-lock (function-A) ...) or something similar so the file writing steps is correct
<shka>
jeosol: well, the obvious solution would be to use locks
<jeosol>
I am trying to do something like the that
<Bike>
i think you are confused about what with-lock-held does?
<Bike>
when you exit the with-lock-held form, the lock is released. your release-lock call is redundant and probably an error or something.
<jeosol>
Bike: ah I see you mean the release part
Guest82 has joined #commonlisp
<Bike>
your cleanup form thing is also very strangely written
PinealGlandOptic has quit [Ping timeout: 245 seconds]
<Bike>
it will evaluate the cleanup form once unconditionally, and then again if it returns true
<Bike>
you probably meant to check whether there's a cleanup form at macroexpansion time rather than runtime
<jeosol>
Bike: yeah, mixed up compile- and runtime here .. you are right
<Guest82>
phantomics a simple social network like app with a reactjs front end (or angular) and managing docs, videos, commenting, and the like
<Bike>
,@(when cleanup-form (list cleanup-form)) is what you want i think?
<Bike>
just to be very clear about with-lock-held - the lock will be released if the form is exited in any way. it essentially does an unwind-protect to ensure it's released.
<jeosol>
so my unwind-protect on top is reduntant since the with-lock-held would have released the lock on exit
<jeosol>
?
<jeosol>
I guess I can simplify things, I rarely have a cleanup form, or just chain all tasks together
<phantomics>
Guest82: better to choose React, from what I heard Angular has been abandoned
<Bike>
you could dispense with this macro, and instead just do (unwind-protect (with-lock-held ...) cleanup) when you need to
<yitzi>
Does anybody know if it is a hard requirement that logical pathname words get upcased? CLHS 19.3.1.1.7 seems to imply that to me, but CCL appears to preserve case words.
<jeosol>
Bike: Thanks for helping brain stop this. But in your view, if I do something like this, it should help with the race issues?
<shka>
(the non-obvious solution would be to use lparallel future and chain)
<Bike>
jeosol: like i said, i don't understand your race issues. i just know that you're doubly releasing locks and that can't be good
<jeosol>
ok
svillemot has quit [Quit: WeeChat 3.0]
<Bike>
and Josh_2 is right that from your description you might want a semaphore structure rather than locks per se
tfeb has joined #commonlisp
<_death>
it's not clear to me what you expect to gain by parallelization, if processes A and C are much faster than B
<Bike>
that's also a little mysterious
<Guest82>
phantomics oh, didn't know that... have seen many people using react, yet also see many people and companies using angular...
<Guest82>
so I wanted to learn react
<jeosol>
Josh_2: thanks for that link on semaphores, I'll need to read it to refresh.
<jeosol>
have a crazy deadline so tryign to sort this out.
<phantomics>
React is worth learning for sure, you can also use the Panic library to generate React in tandem with CL-WHO
amb007 has quit [Read error: Connection reset by peer]
<jeosol>
_death: I apologize since I it's not very clear. I am doing optimization and for each F(X) evaluation, I run three steps A, B, and C. B is a numerical simulation step (fluid modeling on a 3d grid) using a 3rd party application. Writing the input files and reading the output are easy and fast they are both textfiles
<phantomics>
It's a lot more elegant than using React's JSX
dsk has quit [Ping timeout: 240 seconds]
<jeosol>
For task B, the size of the grid affects the run time, so I created a smaller grid 40x40x1 for my test case. Realistic ones can have 500K cells and takes longer to run
amb007 has joined #commonlisp
<phantomics>
Using CL-WHO and Panic, you can meld React components, JS and HTML together in a seamless way
<_death>
jeosol: so are you running a single B process, or multiple ones doing different inputs?
thmprover has joined #commonlisp
<jeosol>
death: so in a population-based optimizer, in each iteration you get 20 solutions, there are 20 F(X)'s to compute, if each F(X) takes an hour, if I can do all F(X) in parallel, they would take 1 hour ceteris paribus, instead of 20 hours - that is where the parallelization comes in
<jeosol>
death: yes, the inputs are different, I use threads to run them. bordeaux-threads
tfeb has quit [Quit: died]
<jeosol>
one process, but I create as many threads as need and call the function that does steps A, B, and C. But I think steps A where I write the input files, results get jumbled. Onces the files are written correctly, Step B is deterministic and Step C is just read a file in the correct folder
<shka>
jeosol: it fits your use case really, really, really well
<jeosol>
shka: thanks for the link
<shka>
you are welcome
<shka>
so when you call lparallel:pmapcar, you will get list of results, just like with mapcar
<shka>
and it will block until everything is completed
<shka>
but it will complete in parallel
<shka>
which is what you want
<shka>
(i think, at least)
<jeosol>
yeah its similar to what I am doing, but I can't say I am using lparallel. My issue is the shear size of parameters - unfortunately, I have functions all over the place. So I was wondering if I could somehow tie them together and use the lock, everything will be kosher
<jeosol>
but I will look into other options, like you suggested.
<jeosol>
it seems it will do that I want especially with the blocking part but still running in parallel
<shka>
jeosol: also check lparallel:plet
<Josh_2>
Best not use locks if you are going to be blocking for a long time
<Josh_2>
because they spin (I believe) and so they take up CPU
<jeosol>
I was only using it for those Task A writing files that don't take much time, but for B, no locking, until I get to C again to read out. The task B - more expensive part, has no locking
<jeosol>
But a cleaner solution that allows me to just chain everything and not worry, if at all, is better.
<shka>
jeosol: well, you can use lparallel:future and lparallel:chain
<Bike>
Josh_2: only a really dumb lock implementation actually just spins. they try to yield the processor.
<jeosol>
several months ago, some I chatted with on linked advised to look at lparallel but since bt-threads was working, I never saw the need. I guess I have to look into it now
<shka>
most spin loccks usually have a maximum spin count
<Bike>
lparallel is a library on top of bordeaux. higher level, if you like
<Bike>
bordeaux mostly just provides primitives like threads and locks and stuff.
<jeosol>
@Bike
<_death>
jeosol: maybe you can arrange it so that you have a function (defun my-task (number) (read-outputs (magic (write-inputs number)))) .. then pmapcar it as shka suggested
<shka>
i use lparallel almost everyday, it saved me countless hours of programming and debugging
<jeosol>
ok, I was low level, writing my joins, etc. May be I need to stay higher-level since there are aspects I may not understand very well
<jeosol>
_death: I see. That's would be much cleaner - so at the level of calling my-task function, the locking (?) and blocking is taken care of?
srhm has quit [Quit: Konversation terminated!]
<Bike>
well if each indexed task can be handled independently you shouldn't really need to lock anything
<jeosol>
shka: I think I should do same, move to lparallel. I debugged yesterday and was able to isolate the issue to the parallelization as serial model gives the correct results and I noticed the jumbled input files
<jeosol>
Bike: There are some shared resources (CLOS objects) - this may be where the issues are
<shka>
jeosol: not if those are read-only
<_death>
jeosol: there's no need for locking if the inputs and outputs are written in separate files, and magic only depends on the inputs.. a naive approach would be to have start a thread for each task and just join them all.. lparallel allows you to limit the number of tasks running at once
<jeosol>
shka: good point, unfortunately, not all the way through, for instance, I am taking some parameters from a higher level object and saving them in a lower object (iteration and solution indices) - i noticed an issue there
<jeosol>
_death: ok I see, I am doing something similar but have a loop with bt-threads functionality to track the join, specify a batch-job parameter and all. But I should probably just be using a library if it will save me all the headache
<_death>
jeosol: right, often threads are too low level an abstraction
<jeosol>
ultimately, my goal is to do distributed computing, but I have only used swank-crew to run on another box, for one of computation and it's ok so far
<jeosol>
_death: I agree with you completel
<_death>
there is also an lparallel for multiple machines, called lfarm.. I've no experience with it though
<jeosol>
oh really. that'd be interesting - I checked aws but they were expensive. Someone here referred me to some European option. The plan will be to have some machines with the executable for task B installed, and SBCL running, route the jobs, and get the results back
<jeosol>
I am far from that point, but I will look into lfarm
<jeosol>
_death: thanks, I looked at the link, it should serve that distributed option
<jeosol>
I suppose
<jeosol>
_death, Bike, shka, Josh_2, others: Thanks guys, I appreciate the help. I don't want to take over the channel, but I have gotten good pointers to follow up on - should probably stay way from managing the threads creation joining etc and just use lparallel API's
Inline_ has joined #commonlisp
Inline_ has quit [Client Quit]
Inline_ has joined #commonlisp
Guest82 has quit [Quit: Client closed]
<shka>
jeosol: sometimes you have to get your hands dirty with BT directly, plus at least sometimes you need to use locks, but yeah, most frequently problems can be solved in lparallel in a few lines of code
Inline has quit [Killed (NickServ (GHOST command used by Inline_))]
Inline_ has quit [Client Quit]
<jeosol>
shka: haha, I think they are dirty enough, lol, I used to use pthreads with C++ code - not an easy experience
<shka>
C++ has support for threads in the standard since C++11
peterhil has quit [Ping timeout: 252 seconds]
<jeosol>
skha: I handle the thread batching via loops and joining, but I should probably just offload that task and not have to worry much. Yeah, the locks have helped but I should probably redesign to avoid setting variables in thread calls
<shka>
well, good luck
<jeosol>
shka: My C/C++ days was back in graduate schools, I have not coded C/C++ since leaving, like 2010
<jeosol>
shka: thanks. I have a deadline for a conference, and need to have this parallelization part to work to be able to submit the runs
edgar-rft has quit [Ping timeout: 240 seconds]
<shka>
jeosol: right
luna-is-here has quit [Ping timeout: 265 seconds]
<shka>
my tip: try to rewrite your main part so it is run in a function passed to the mapcar
edgar-rft has joined #commonlisp
<shka>
and try removing synchronization points if possible
<shka>
and then move to lparallel
<shka>
the fun part with lparallel is that you can write (apply (if parallelp #'lparallel:pmap #'cl:map) nil function input-sequence)
<shka>
well, s/apply/funcall
<shka>
which allows you to use the exact same source code for both parallel and serial execution
luna-is-here has joined #commonlisp
amb007 has quit [Read error: Connection reset by peer]
Inline has joined #commonlisp
amb007 has joined #commonlisp
<jeosol>
skha: good point: The ability to use the same code for serial and parallel is definitely import. For now, I have a use-parallel-p variable that I use to test both parts. Serial part is trival because I have a loop that calls a function withich chains all the steps I need
<jeosol>
shka: I definitely need to make my life easier when i comes to executing parallel jobs, so I should consider alternatives
gaqwas has joined #commonlisp
<jeosol>
shka: by "removing synchronization points ..." you mean avoid writing to shared resource?
<shka>
yes
<shka>
use the return value instead
<jeosol>
shka: I agree. I think I will spend some time to redesign so as not to worry about this again
<shka>
much less potential for bugs
<shka>
yeah, try to refactor code into sort of map reduce code
<jeosol>
I have a part were are save some variables in an object they get jumbled some pointing to previous indices.
<jeosol>
shka: I agree - options less prone to bugs
<shka>
and if that does not work, you can use lparallel:future and chain which allows you to link execution steps
<shka>
but what you are doing sounds more like a map reduce type of problem
PinealGlandOptic has joined #commonlisp
<jeosol>
shka: the call to F(X) is from an upstream higher level algorithm (optimizer) with F(X_i) doing F(A(X_i)), F(B(X_i)), and F(C(X_i)) - so having the main F(X_i) execute without the issues is better. The higher-level algorithm doesn't know about the CLOS object used to compute F(X_i)
<jeosol>
noticed compiler type instructures "the" and other declare statements -- what is the speed up with and without, I know it probably depends
<shka>
anyway, gotta call it a day
<shka>
jeosol: it is ONLY useful for numeric types
<jeosol>
thanks shka:
<shka>
and even then, not really that much
<jeosol>
my neural network library on mnist is very slow (type hints or anything) haha
<shka>
well, i guess perhaps for arrays as well, since maybe compiler will be able to inline memory access
<jeosol>
my matrix-vector operations are not optimized and used loops, etc. it was just to check my understanding
<shka>
sbcl is not really that great target for neural networks
<jeosol>
oh really?
<jeosol>
I thought for number crunching it's supposed to be best of the free ones
<shka>
it is!
<jeosol>
anyway, I won't take your time, we can pick up later
<shka>
just let me show you one main problem
<jeosol>
I normally do computational tasks but for my task B (numerical modeling part) I just use a 3rd party executable. It will take lots of efforts to write a 3D numerical solver and then worry about matrix conditions, optimize computations, etc. The 3rd party exe was written in Fortran
<shka>
jeosol: check the following (disassemble (lambda (a b) (declare (type double-float a b) (optimize (speed 3) (safety 0))) (* a b)))
<Inline>
so what does it get you to write the last one as (* (the double-float a) (the double-float b)) ?
<Inline>
you spare the type checking ?
<shka>
sorry, i meant
<shka>
jeosol: check the following (disassemble (lambda (a b) (declare (type single-float a b) (optimize (speed 3) (safety 0))) (* a b)))
<shka>
Inline: in this case those type assertions are redundant
<shka>
well, not assertions
<shka>
type hints
<Inline>
ah yes
<Inline>
so useful if you didn't declare beforehand but only hinted at
<jeosol>
shka: yeah, i ran in, I am supposed to notice something ...
<shka>
uh
<shka>
it used to be slightly more wonky then it is right now
<jeosol>
Oh I see, I didn't get what I should have noticed
<shka>
uh, am i crazy...
<jeosol>
Some neural network training on my desk top didn't finish after several days, other guy said it ran cuda arrays on gpu, in 24 hours - that's a massive speed up
<jeosol>
If i get some spare change next, I will look to invest in one of those machines
<jeosol>
btw, this is running SBCL
<shka>
jeosol: disregard what i said, i confused single-float < with multiplication
<jeosol>
shka: not crazy, I am not compiler guy ...
<Bike>
that multiplication is going to have to cons if in an independent function like that
<Bike>
since double floats are usually boxed
<shka>
Bike: yeah, but double-floats are rarely used in deep learning
<Bike>
single floats... shouldn't need to cons?
<shka>
i think i wanted to show how sbcl compares single-floats
<jeosol>
I occasionally work with python for the DL libraries, but it can be pain in notebooks having to go up and down to reevaluate some cells come back down, only to realize one other computation is stale, etc
<Inline>
err whaaa
notzmv has quit [Ping timeout: 256 seconds]
<Inline>
i see no threading for single-floats but when using double floats there is some threading going on
<Bike>
that just indicates it has to allocate memory
<Inline>
and CONS->RNN
<Bike>
the memory allocation is a per thread thing most of the time, probably
<shka>
you know what, i am spewing nonsense
<Inline>
so not even consing in single floats
<Inline>
but in double floats
<shka>
comparison also looks fine
<Bike>
a double float needs 64 bits to represent, so there's no room for a type tag. so they're boxed if they need to be used in a type sensitive context. ergo, consing.
<Inline>
aha
<Bike>
a single float is only 32 bits so it fits with a type tag into a 64 bit word just fine.
<shka>
yet i vividly remember when i stumbled in something float related in SBCL that worked faster with double-floats then a single floats
<shka>
because single float were getting converted to double floats anyway
<shka>
but i can't remember or google it
<Inline>
welp, the fastness i also skimmed from books recommending to use bigger types, the machine architectures are then faster because of bus design or so....
<shka>
so maybe my memory is glitching
<jeosol>
Bike: I was recently following a talk on Julia why the guy was saying it faster than python and many other languages (don't recall he mentioned C). But one thing he kept saying is that if you have to box and unbox, he said you are "dead"
<Bike>
yes, memory allocation really screws things up.
<Inline>
not sbcl specific, even python and fortran books recommend the same
<jeosol>
this was for a course on performance optimization
<Bike>
suddenly your arithmetic might involve the garbage collector and/or synchronization.
<shka>
well, you may work fine with double-floats in sbcl, but you will gonna need type declarations for arrays and use inline quite a bit
<Bike>
right, it can avoid boxing intermediate results
<shka>
it is a little bit annoying, but it can be done
<shka>
heck, statistical learning uses double-floats for floating point types
<shka>
and i did that BECAUSE OF THAT ISSUE THAT I CAN'T FIND OR REMEMBER
* shka
is frustrated
<shka>
maybe it got fixed?
<jeosol>
Btw, on my threading issues mention earlier, is passing data from upstream to downstream object a bad design or code smell - this is where I tell the lower object what their index is that is used to create a directory later on
<shka>
jeosol: reserve index beforehand
<jeosol>
hmm, ok, that's interesting
<jeosol>
as long as I can map the X_i to some folder and just pick the results, I probably don't need to pass an object (and I do a setf on a slot) this is where things are getting messed up
<shka>
nothing in the changelog
<jeosol>
I haven't been displined enough
<shka>
well, maybe i was wasting my time
<shka>
wouldn't be the first time...
<shka>
jeosol: as for index, if that's just iota, well, you can simply do something like (lparallel:pmap nil (lambda (argument index) ...) arguments (alexandria:iota (length arguments)))
<Shinmera>
shka: Thanks a lot! My finances will stay troublesome as long as I have to fund a team with no income, heh.
<shka>
Shinmera: btw, did you painted this calendar using your drawning app that name i can't remember right now?
<Shinmera>
haha, no
<shka>
*did you paint
<Shinmera>
I use Krita
<shka>
heh, that's a shame!
<shka>
but it looks nice
<jeosol>
skha: technically, the index corresponds to rows of a larger matrix (num_solutions, num_dimensions), so I slice of each row (X_i) to a separate thread
<shka>
i will grab one for myself and one as a gift
<shka>
jeosol: then, technically, you are fine
peterhil has joined #commonlisp
<shka>
jeosol: you don't need a lock
<shka>
which is great
<shka>
you may prefer to return vector which gets stacked into matrix as an after step but at this point this is really just stylistic choice
<shka>
or you may pass displaced array into the function
<shka>
all options are possible
<shka>
anyway, have a good night
<shka>
and once again, good luck with your deadline!
<jeosol>
shka: thanks buddy. I really appreciate the pointers and suggestions. I will try them
lisp123_ has quit [Read error: Connection reset by peer]
yitzi has quit [Quit: Leaving]
pjb` has quit [Quit: renaming]
pjb has joined #commonlisp
aeth has quit [Ping timeout: 272 seconds]
aeth has joined #commonlisp
attila_lendvai_ has joined #commonlisp
<jmercouris>
Shinmera: how to plump:text and avoid the text of script tags?
<jmercouris>
Shinmera: let's say I invoke it on the document root, and I want to avoid getting script tags
<Shinmera>
you do it yourself.
<jmercouris>
will I have to manually traverse the tree or something
<jmercouris>
damn, I was just about to say it
<jmercouris>
OK
<jmercouris>
that's pricy
<Shinmera>
"pricy?"
lisp123_ has joined #commonlisp
phadthai_ is now known as phadthai
shka has quit [Ping timeout: 256 seconds]
thmprover has quit [Quit: [Exit, pursued by bear]]
Guest82 has joined #commonlisp
<Guest82>
does anyone know how to print to the repl while using a controller (read: web.lisp) in caveman2? phantomics
<Guest82>
I've tried format T, print, don't know what else
<jmercouris>
Guest82: it should work
<phantomics>
Haven't used caveman2, do you know what output stream is being used?
<jmercouris>
Guest82: if you aren't seeing anything look in your inferior lisp
<Guest82>
hm, not sure
<jmercouris>
check the inferior lisp, printing works just fine in caveman
<Guest82>
I am too much of a noob to even understand what inferior lisp means... I'm using sublime text with a package called sly
<Shinmera>
Why would it be any more expensive than what the plain text does
<Guest82>
jmercouris I'm getting errors, package doesn't exist, tried doing ql:quickload "swank" but then still didn't work
<Guest82>
for (SWANK::*CURRENT-STANDARD-OUTPUT*)
<Guest82>
in the repl
dsk has joined #commonlisp
<pve>
Guest82: I meant try printing to that stream
<Guest82>
oh, meaning doing something like this (format SWANK::*CURRENT-STANDARD-OUTPUT* "hello")
<Guest82>
?
<Guest82>
@pve
<pve>
Guest82: yes, and see where the output goes
<Guest82>
pve got an error saying the variable swank is unbound
<Guest82>
thanks for the noob debug help...
<pve>
Guest82: oh, ok you're running it separately
<pve>
Guest82: how exactly are you running it?
<Guest82>
pve not sure what that means..... I started the server in the repl... but the repl is probably running as a bash process and being called from the editor sublime text
<pve>
Guest82: ok, I understand now
<Guest82>
phantomics nope, don't know...
tfeb has joined #commonlisp
<Guest82>
@pve Is there a way to see all the variables in the current script? I tried (inspect *readtable*) but didn't understand the output so much
<Guest82>
pve any ideas about the printing? or how to debug a program without printing haha
<Guest82>
I feel like there's a bit of a jump between starting to program in lisp and understanding how to deal with all these things...
<pve>
Guest82: Sorry I'm not familiar with caveman2, but you could at least print to a file if you can't see standard-output
<Guest82>
@pve oh, I hear, thanks, that's a good idea
<pve>
and turn that into a utility function like (defun log (message) ... ) or something
mister_m has quit [Remote host closed the connection]
<lisp123_>
Guest82: "I feel like there's a bit of a jump between starting to program in lisp and understanding how to deal with all these things..." --> I won't lie, it will take a bit of time, but the benefits will be great down the track
<Guest82>
pve, thanks a lot!
<pve>
Guest82: it may be that running the thing from within sublime is complicating things too much if you are unable to see any output
<Guest82>
lisp123_ I find macros are the solution to my problems fighting languages to be able to abstract things, but I feel like there should be a smaller barrier to entry than learning lisp and emacs at the same time
tfeb has quit [Quit: died]
<pve>
Guest82: You could try starting it manually from the shell instead. Then you should see the debugging output at least.
<Guest82>
pve I hear, I understand everything is easier with emacs, but I don't know how to use emacs and I feel it's too much to learn both in one shot
<pve>
Guest82: didn't mean emacs, just shell
<Guest82>
@pve I will try now, that's a good idea!
<pve>
Guest82: remember to familiarize yourself with the various command line parameters like --eval and --load to get maximum convenience
<lisp123_>
Guest82: Try Portacle. I don't disagree with you, the combo of learning lisp and emacs makes it a bit harder, but both tools have a lot of benefits too - so you will get a lot of success down the track :)
<Guest82>
pve, thanks will read the man page
<lisp123_>
Guest82: Sorry to beat a daed horse, but I _highly_ recommend emacs for any sort of lisp programming
<lisp123_>
You can start with the basics and dip your toes in
<Guest82>
lisp123_ yeah, from what've I've read it seems it's a great combination. I tried portacle but they didn't support latest mac os
<Guest82>
let me check if they updated it
attila_lendvai_ has quit [Ping timeout: 265 seconds]
<lisp123_>
For MacOs - you can install basic emacs
<pve>
Guest82: and if working from the shell, do define convenience functions or symbol-macros to reload your project quickly
<pve>
Guest82: silly example: (define-symbol-macro rr (asdf:load-system "myapp")) will make "rr" reload your stuff after you've edited it in sublime
<Guest82>
pve thanks for the idea of the convenience functions
<Guest82>
lisp123_ thanks for the link
<Guest82>
will look at it
<Guest82>
oh, that's just emacs
<lisp123_>
lol
tyson2 has joined #commonlisp
<lisp123_>
It will take 1 min to install
<lisp123_>
Have you installed SBCL already?
cage has quit [Quit: rcirc on GNU Emacs 27.1]
kakuhen has joined #commonlisp
<Guest82>
lisp123_ yeah I have emacs 2.7 in the terminal, sbcl,just added cl
<Guest82>
ccl
<lisp123_>
Cool
<lisp123_>
I'm gonna sleep but good luck and enjoy
lisp123_ has quit [Quit: Leaving...]
<Guest82>
lisp123_ what's the easiest tutorial about dealing with lisp on emacs? I feel like the tutrial here is just text editting, and then to do lisp modes and other things is another huge jump
<Guest82>
ok, thanks, good night!
<pjb>
Guest82: well, there's the emacs-lisp-intor.
<Guest82>
@pve great, thanks, I have plenty of homework now ;)
<pve>
Guest82: I fondly remember watching that video when I started lisping
<Guest82>
my emacs fails from bad init file settings
<Guest82>
hahaha
<Guest82>
old problems
<Guest82>
Warning (initialization): An error occurred while loading ‘/Users/danielnussenbaum/.emacs.d/init.el’:
<Guest82>
File is missing: Opening input file, No such file or directory, /Users/danielnussenbaum/zsh:1: command not found: rosconfig
<Guest82>
now i've commented everything out and it works!
<Guest82>
haha
notzmv has joined #commonlisp
<pve>
Guest82: I'm rewatching the slime video now, and it's incredible how useful it still is, despite being a little old
akoana has joined #commonlisp
<Guest82>
it's two years old hahaha when I see something from 20 years ago I think it's old
<pve>
nah, it's from 2005
<Guest82>
thanks
<Guest82>
ah repost..
<pve>
Guest82: he uses asdf-install in the video, but there's quicklisp these days
<Guest82>
yeah
Qwnavery has joined #commonlisp
pve has quit [Quit: bedtime]
Guest82 has quit [Quit: Client closed]
kevingal has quit [Remote host closed the connection]
<kakuhen>
fyi you almost always want to quicklisp nowadays
<kakuhen>
the nice part about quicklisp is not only that it's widely used nowadays, but also lets you quickly test your projects on other cl implementations, assuming you added it to your implementation's init file (i.e. your .sbclrc, .eclrc, and so on)
Qwnavery has quit [Quit: WeeChat 3.2]
jeosol has quit [Quit: Ping timeout (120 seconds)]
silasfox has quit [Ping timeout: 240 seconds]
sterni has joined #commonlisp
silasfox has joined #commonlisp
<char>
flip214: I'm not sure drakma will work because I need it to work for websocket too.
selwyn has quit [Read error: Connection reset by peer]
frgo has quit [Remote host closed the connection]
frgo has joined #commonlisp
<char>
flip214: i need the proxy to be configurable in common lisp
random-nick has quit [Ping timeout: 258 seconds]
subst has quit [Ping timeout: 252 seconds]
pjb has quit [Remote host closed the connection]
IPmonger has joined #commonlisp
IPmonger has quit [Remote host closed the connection]