<nij->
My lisp is consuming a lot of memories. Is there any way to trouble shoot?
jmdaemon has joined #commonlisp
<Bike>
keep fewer objects around. you can use room to get a basic idea of what's around
<aeth>
i.e. ROOM
<aeth>
beyond that there's actually a lot of benchmarking you can do but it's usually implementation-specific
<aeth>
although TIME might also tell you if something is consing
<aeth>
the #1 way to save memory is to move from a linked list to a specialized vector (e.g. string, octet vector, etc.) if you can, though
<nij->
vectors are more efficient than linked list?
<aeth>
they're more compact in memory. Even T vectors.
<nij->
I guess there must be some cache building up...
<aeth>
In general, anyway. If the implementation lets you make one for (complex double-float)s that might be larger than a cons cell.
<nij->
How do I list all dynamically bound variables?
<Bike>
a cons cell has a car and a cdr, so each list element takes two objects
<nij->
(across all packages)
<aeth>
(although storing the complex double float would still take that much space, anyway, so nevermind)
<Bike>
you could iterate over all symbolsl and check what's bound
<Bike>
(do-all-symbols (s) (when (boundp s) ...))
<aeth>
nij-: a cons cell in memory is probably [ pointer-or-tagged-object | pointer ] so it's definitely come up before where someone's problem fits in memory with a vector, but not with a list
<aeth>
because the list is probably taking twice the RAM. Same big-O memory usage, ofc, since 2n = n in big-O
<nij->
How to list all hash tables?
jmdaemon has quit [Ping timeout: 240 seconds]
<nij->
Or is it not possible?
<aeth>
list all of the hash tables in memory?
<nij->
yep
<aeth>
it's not easy, but it's probably not impossible with the debugger
<aeth>
in some implementation
<nij->
hmmm..
<aeth>
probably harder to tell if it's garbage waiting to be collected or not
NicknameJohn has quit [Ping timeout: 250 seconds]
<contrapunctus>
I thought the answer would be something similar to what Bike wrote - `(do-all-symbols (s) (when (and (boundp s) (hash-table-p (symbol-value s))) (print s)))` ...is that incorrect?
<nij->
Oh, at least I can get all symbols that point to some hash table.
<nij->
However, there could also be hash tables that are not collected because it is used by something else.
* contrapunctus
re-reads aeth 's message and realizes the issue
<nij->
Does hash-table size = 70000 mean 70MB?
<aeth>
contrapunctus: that gets all global variable/constant hash tables, but not hash tables stored in slots, active local variables (e.g. closures), etc. Or just garbage
<contrapunctus>
aeth: ah, that too
<nij->
It's handy to know the size of a hash table. Can I see the same thing for all other objects and structs?
Catie` has joined #commonlisp
<aeth>
nij-: there are two relevant sizes, though
<nij->
?
<aeth>
the size of the hash table in memory (I'm guessing it's roughly number of items times the bit size of the implementation, e.g. 64*123456)
<aeth>
and then the size of the contents of that hash table
<aeth>
imagine e.g. using pngload to load PNGs into hash tables... most of the memory used will be the PNGs not the hash tables themselves
<aeth>
there are a lot of ways to run out of memory
<Bike>
the implementation almost certainly doe snot keep a record of all hash table
<Bike>
so it would have to walk through the entirety of memory
<aeth>
Bike: right, if there's a debugger that can go through memory, you can get a count of hash tables, but without keeping a record, you don't know if it's garbage or not afaik? Without running the GC
Catie` has quit [Client Quit]
<aeth>
assuming it knows the type tags and stuff
Catie` has joined #commonlisp
Catie has quit [Ping timeout: 264 seconds]
<Bike>
depends on how the interface works
<Bike>
in order to get all non garbage it would probably need to stop the world? i dunno, i'm not a gc guy
<nij->
hmmm sounds like gc issue is very hard to resolve :(
Catie has joined #commonlisp
Catie` has joined #commonlisp
<hayley>
It would indeed; else someone else can come along and create garbage hash tables. I'd just live with it, unless you're making a lot of hash tables very quickly.
NicknameJohn has joined #commonlisp
Catie has quit [Ping timeout: 268 seconds]
shka has joined #commonlisp
akoana has joined #commonlisp
NicknameJohn has quit [Ping timeout: 268 seconds]
waleee has quit [Ping timeout: 240 seconds]
tyson2 has quit [Remote host closed the connection]
nij- has quit [Ping timeout: 240 seconds]
bilegeek has joined #commonlisp
<edwlan[m]>
sbcl has a way to map over all the objects on the heap, but I’m not sure what guarantees you’ve had
<ixelp>
GitHub - brown/swank-client: Common Lisp implementation of a Slime / Swank client.
<knusbaum>
Yeah, looked at swank-client already.
<knusbaum>
Seems great, but I'd have to build a repl on top of it.
parjanya has joined #commonlisp
torbo has left #commonlisp [ERC 5.4 (IRC client for GNU Emacs 28.2)]
jon_atack has joined #commonlisp
jonatack has quit [Ping timeout: 256 seconds]
ym has joined #commonlisp
<hefner>
SBCL should have a deluxe terminal UI out of the box, I'm certain of this
<hefner>
editing, completion, a blackjack game, the works
<hayley>
Should it be able to send email?
<hefner>
possibly.
<contrapunctus>
hefner: terminal interfaces should die, I'm certain of this 😏️
<contrapunctus>
"This machine [MacBook Pro] is cosplaying as a PDF-11!"
<contrapunctus>
* PDP
<hefner>
you will never drag the 1970 unix death cult out of their self imposed dark age, I'm equally certain of this. You just have to make the best of it.
<contrapunctus>
I hope to do so, at least for myself and a few others :)
<hefner>
there's room for more than one thing
<contrapunctus>
Sure.
<beach>
If we just had unbounded resources, that would not be a problem.
<hefner>
from each according to his ability, to all according to his whim
hineios9 has joined #commonlisp
hineios has quit [Ping timeout: 265 seconds]
hineios9 is now known as hineios
<hayley>
I would simply just make more resources.
akoana has quit [Quit: leaving]
ym has quit [Ping timeout: 264 seconds]
pranavats has left #commonlisp [Error from remote client]
pranavats has joined #commonlisp
dcb has quit [Quit: MSN Messenger 3.8]
bilegeek has quit [Quit: Leaving]
pve has joined #commonlisp
rgherdt has joined #commonlisp
msavoritias has joined #commonlisp
lottaquestions has joined #commonlisp
lottaquestions_ has quit [Ping timeout: 256 seconds]
<Nilby>
how about a terminal protocol that can both work in unix death cult mode and pure lisp to whatever backend x11/windows/etc. unfortunately to fully escape unix death cult mode you have either forgo running tty apps or split your address space with a unix process.
vassenn has joined #commonlisp
Noisytoot has joined #commonlisp
<Nilby>
but it's easier to forgo tty crap when your repl has "ls" and tetris built in.
nij- has joined #commonlisp
* White_Flame
looks forward to reusing the term "1970s unix death cult" heavily now
vassenn has quit [Quit: Leaving]
vassenn has joined #commonlisp
rgherdt has quit [Ping timeout: 256 seconds]
vassenn has left #commonlisp [#commonlisp]
azimut has quit [Ping timeout: 240 seconds]
rgherdt has joined #commonlisp
nij- has quit [Ping timeout: 256 seconds]
Inline has quit [Quit: Leaving]
NicknameJohn has joined #commonlisp
_cymew_ has joined #commonlisp
lucasta has quit [Remote host closed the connection]
rogersm has joined #commonlisp
Lord_of_Life_ has joined #commonlisp
Lord_of_Life has quit [Ping timeout: 240 seconds]
Lord_of_Life_ is now known as Lord_of_Life
Guest34 has joined #commonlisp
festerdam has joined #commonlisp
Guest34 has quit [Quit: Client closed]
euandreh has quit [Remote host closed the connection]
euandreh has quit [Remote host closed the connection]
nij- has quit [Ping timeout: 240 seconds]
<Demosthenex>
hrm. i'd argue the terminal is superior to 90% of the gui interfaces ever built
<Demosthenex>
imagine, a universal application interface where you control the font, size, colors, style, window sizze/position, have complete isolation between applications, you get your own custom hotkeys, it's also making data available in a universal format (text) without requiring the vendor to write an export. high speed, low overhead, fewer dependencies, no web browser or javascript.
<hayley>
I don't want complete isolation between applications. And text is a rather miserable format.
* hayley
wonders how they managed before Web browsers were invented.
<Demosthenex>
hayley: text is a universal format.
kevingal has joined #commonlisp
<Demosthenex>
yeah, lets burn the whole web stack. it's a cesspool
<hayley>
And a Turing machine is a universal computer. Doesn't mean I want to use one.
<hayley>
In practice you have more structure than "text", which does require parsing/exporting, or shotgun parsing. Unix utilities tend toward the latter.
<Demosthenex>
for serious stuff, sure. same with a webapp. but i'd argue my copy/paste of a putty session is readable, where a copy/paste of a web browser page is crap.
<hayley>
Copy-pasting perf stat usually doesn't go well, but maybe that's cheating as it makes a mock-GUI in a terminal.
<jackdaniel>
your copy paste from a terminal would be equal to screencap of a web page
<Demosthenex>
jackdaniel: yes, but you can't extract text from screenshot images.
<Demosthenex>
plus the computing world learned ages with the earliest viruses not to trust code from untrusted sources. here's looking at you javascript.
<hayley>
But no one trusts JS code, hence the sandbox.
<hayley>
AWS engineers should be equally frightened of the code I put on their servers. I mean, they should be anyway, because it's crap, but also because it's untrusted to them.
<Demosthenex>
you do know that sandboxes were introduced afterward? and it's a constant arms race, like spam
<jackdaniel>
well, that's a strawman, you can't extract images from the text either
<hayley>
Do I? Did JS let you delete system32 in the 90s?
<Demosthenex>
hayley: plenty of vendors were and still are launching local executables from javascript, so deleting system32 is still a thing :P
<Demosthenex>
and likely better hygeine
<jdz>
Back in the day people had to propagate viruses using floppy disks.
<jdz>
And they did.
<jackdaniel>
this argument is so dumb I'll better turn a blind eye and get back to (/me looks at another window) improving ecl compiler
mgl has joined #commonlisp
<jackdaniel>
(that terminals are superior to other uis)
lucasta has joined #commonlisp
<Demosthenex>
jackdaniel: you da man!
<Nilby>
So does anyone actually use a CLIM (or similar) listener as their main REPL? My opinon is that Emacs is still mostly a terminal.
<jdz>
Also, conflating "text based" with "command/REPL based" is not productive (back in the day, on DOS, the GUIs were text based, with mouse clicking/keyboard navigation, but no pipes or command invocations in sight).
<jdz>
Try midnight commander for the experience.
<Demosthenex>
yep. emacs + slime is totally a gui app
<beach>
Nilby: I don't, mainly because SLIME is still superior to what we can propose as CLIM-based software. But when we get the superior parsing framework for Common Lisp going, I will likely abandon SLIME and Emacs for editing Common Lisp code.
<jackdaniel>
I'm using simple clim repl when I'm working with graphical output
<hayley>
One incremental improvement I'd like to SLIME is to be able to provide hooks for looking up things in other namespaces. I've now had to hack the code which retrieves argument lists twice, and I've been unsatisfied with the result both times.
<hayley>
Initially I had annotated argument lists with types, which I still use, and today I added support for looking up function names through a FFI library.
kevingal has quit [Ping timeout: 256 seconds]
<pve>
hayley: is your slime hack public?
<hayley>
No, but perhaps it should be. I'll remember where it's saved tomorrow.
<Nilby>
beach: Thanks for answering. My feeling as a long time Emacs developer is that it's a plateau that's very hard to move on from.
<pve>
hayley: nice, thanks
euandreh has joined #commonlisp
<beach>
I tend to agree. As long as the Common Lisp code is analyzed the way it is, I think it is hard to improve the situation.
euandreh has quit [Ping timeout: 240 seconds]
tyson2 has quit [Remote host closed the connection]
<hayley>
I'm not sure what the right way to add my analyses would be, though. From memory, I had to change the way SLIME parsed lambda lists in order to intersperse type annotations with parameter names.
rogersm has quit [Ping timeout: 264 seconds]
euandreh has joined #commonlisp
piethesailor has joined #commonlisp
euandreh has quit [Remote host closed the connection]
euandreh has joined #commonlisp
msavoritias has quit [Remote host closed the connection]
slyrus has quit [Read error: Connection reset by peer]
rgherdt_ has joined #commonlisp
rgherdt has quit [Ping timeout: 264 seconds]
waleee has quit [Ping timeout: 250 seconds]
phantomics has quit [Ping timeout: 250 seconds]
nij- has joined #commonlisp
nij- has quit [Client Quit]
nij- has joined #commonlisp
<nij->
Why is my lisp taking so much VIRT but not much RES as shown in htop?
<nij->
VIRT ~10GB, RES~ 4GB.
<knusbaum>
Don't know about *your* lisp, but lots of applications do that.
<knusbaum>
Linux, at least, allocates pages lazily through page faults. I would guess others do too.
<knusbaum>
It also might have files mapped, etc.
<hayley>
SBCL will allocate virtual memory for the entire heap, but may only use a small amount of that virtual memory.
lucasta has quit [Remote host closed the connection]
lucasta has joined #commonlisp
kevingal has joined #commonlisp
phantomics has joined #commonlisp
rgherdt_ is now known as rgherdt
lucasta has quit [Remote host closed the connection]
euandreh has quit [Ping timeout: 264 seconds]
zxcvz has joined #commonlisp
karlosz has quit [Quit: karlosz]
Guest62 has joined #commonlisp
morganw has joined #commonlisp
slyrus has joined #commonlisp
green_ has joined #commonlisp
mgl has quit [Quit: Client closed]
<nij->
I see. THanks
avocadoist has quit [Ping timeout: 268 seconds]
euandreh has joined #commonlisp
<beach>
Oh, wow, I was just looking up the current price of SSDs, and 1TB is around 50USD. That means that 40MB costs 0.2 cents?
edgar-rft has joined #commonlisp
<beach>
So if you deliver your SBCL executable to (say) 1000 users, you are inflicting a total cost of 2USD. If you work at minimum wage, that's around 10 minutes of work. That's a rough estimate of the amount of time it is worth spending to think about how to minimize the size of executables.
sjl has quit [Quit: WeeChat 3.6]
sjl has joined #commonlisp
<beach>
I should modify the section on checkpointing in the Closos document to take into account that disk memory is out. SSDs probably simplify the algorithms.
sjl has quit [Client Quit]
Guest62 has quit [Quit: Client closed]
<green_>
Hey there. I've been hacking on a library management system, like quicklisp, but it leverages modern tools borrowed from the world of containers... https://github.com/ocicl/ocicl . To beach's point, above, I chose to make all package project local by default. Disk space is cheap, and it makes it easier to lock to specific library versions.
<ixelp>
GitHub - ocicl/ocicl: An ASDF system distribution and management tool for Common Lisp
msavoritias has joined #commonlisp
ym has joined #commonlisp
<nij->
green_ How to you specify library version? Using git rev hash?
kevingal has quit [Ping timeout: 240 seconds]
<green_>
I build versions from releases or git HEAD if releases aren't avaiable. For straight-from-git, the version includes the git commit hash. The version is used a verion label for the OCI artifact
<green_>
(OCI images are basically docker container images, but generalized to support non-docker things)
<green_>
and like container images, :latest always points at the most recent.
waleee has joined #commonlisp
<beach>
green_: So this is software in the form of OS executables, rather than in the form of Common Lisp applications to be run in your own Common Lisp image?
<green_>
There are two parts.. something you run in your image, and an external executable
<beach>
Hmm.
<beach>
How do they communicate?
<nij->
Wow, using org-mode!
<green_>
The embedded library just invokes the executable when it has to
<green_>
for downloading OCI blobs from the registry
<nij->
green_ I've been waiting things like this.. keep it up.
<nij->
Btw have you looked into qlot?
<green_>
Yes, I looked at all of them, but felt I could make a better integrated solution.
<beach>
green_: I am confused I guess. How do you handle specific library versions? If two applications are loaded into the same image, and they need different versions of the same library, how is this situation handled?
<green_>
However you would normally do it with ASDF.
<green_>
It doesn't change anything there. It's about managing local copies of the software, and how we get and secure them.
karlosz has joined #commonlisp
festerdam has quit [Ping timeout: 240 seconds]
<beach>
green_: So what does "it makes it easier to lock to specific library versions" mean?
<green_>
If ASDF is looking for a system that it can't find, it will pass it to the ocicl-runtime, which then looks in a file called "systems.csv", which maps system names to specific versions and where to find them on disk.
<beach>
I see. Thanks.
<green_>
The ocicl cli is for managing the csv file, and the tarballs associated with them.
<green_>
The README tries to explain it.
tyson2 has joined #commonlisp
McParen has joined #commonlisp
Catie` has quit [Changing host]
Catie` has joined #commonlisp
Catie` is now known as Catie
nij- has quit [Ping timeout: 248 seconds]
nij- has joined #commonlisp
nij- has quit [Ping timeout: 246 seconds]
tyson2 has quit [Remote host closed the connection]
azimut has joined #commonlisp
McParen has quit [Ping timeout: 240 seconds]
dcb has joined #commonlisp
ryanbw has quit [Ping timeout: 240 seconds]
tyson2 has joined #commonlisp
xristos has quit [Ping timeout: 268 seconds]
jeosol has joined #commonlisp
cage has joined #commonlisp
ym has quit [Ping timeout: 240 seconds]
tibfulv has quit [Remote host closed the connection]