<czy>
strange, sbcl is outputting information that i did not provide. in this tiny CL project that I am doing (https://github.com/chenzhawyang/metaphono-korean), when I call the function `metaphono-load' defined in `build.lisp', sbcl prints ";;; Computing Hangul syllable names" in the slime repl; this is relevant to my project (i am doing computational phonology on korean), but nowhere in my source code or system defini
<czy>
line, how on earth does CL (sbcl) know?
Lord_of_Life has quit [Ping timeout: 240 seconds]
Lord_of_Life_ has joined #commonlisp
Lord_of_Life_ is now known as Lord_of_Life
lucasta has joined #commonlisp
<kagev>
czy: maybe sbcl "knows" about unicode ... they have a dedicated #sbcl channel, also maybe read their manual's unicode section: https://www.sbcl.org/manual/#Unicode-Support
<ixelp>
SBCL 2.3.3 User Manual
NotThatRPG has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<czy>
kagev: i know that sbcl ships with unicode support, but i have not included a line of lisp that processes unicode though...
<czy>
maybe sbcl has achived singularity
<Bike>
perhaps it lazily computes character names when you use literal hangul characters.
<czy>
Bike: indeed i did that, but except in variable nouns i didn't even mention the language `korean', let alone `hangul'
<Bike>
hm, actually this message is from cl-unicode, not sbcl
<czy>
hmmm, what happened then?
johnjaye has joined #commonlisp
<Bike>
dunno. if this happens every time you could trace some functions to see what's up. i would guess that cl-unicode is building one of the many unicode databases, perhaps when it's first loaded
NicknameJohn has quit [Ping timeout: 240 seconds]
<johnjaye>
how good is the common lisp cookbook?
<johnjaye>
i know that one, the on lisp one, and the apress one
mantlepro has quit [Quit: cheerio]
wheeler has joined #commonlisp
<czy>
Bike: sure, i will take a look at the trace
jeosol has quit [Quit: Client closed]
eddof13 has joined #commonlisp
theesm has quit [Ping timeout: 248 seconds]
theesm has joined #commonlisp
kevingal_ has quit [Ping timeout: 248 seconds]
kevingal has quit [Ping timeout: 248 seconds]
NicknameJohn has joined #commonlisp
Oladon has joined #commonlisp
epony has quit [Remote host closed the connection]
azimut has quit [Ping timeout: 255 seconds]
occ has joined #commonlisp
NicknameJohn has quit [Ping timeout: 276 seconds]
lucasta has quit [Ping timeout: 250 seconds]
kevingal has joined #commonlisp
kevingal_ has joined #commonlisp
keinbock has quit [Ping timeout: 240 seconds]
occ has quit [Ping timeout: 276 seconds]
occ has joined #commonlisp
keinbock has joined #commonlisp
occ has quit [Read error: Connection reset by peer]
eddof13 has quit [Quit: My MacBook Air has gone to sleep. ZZZzzz…]
occ has joined #commonlisp
occ has quit [Ping timeout: 276 seconds]
occ has joined #commonlisp
soundmodel has joined #commonlisp
<soundmodel>
A finnish Wikipedia page writes that "Common Lisp is not so much a programming language, but a definition for the implementation of the language, in the same way as, for example, C and C++.". Input?
<soundmodel>
"Realtime synthesis with sample-by-sample or block-by-block processing (without a realtime garbage collector but it's simple to define cons-free DSP)"
<soundmodel>
I wonder if it's running in some sort of constrained scope
<soundmodel>
where one can guarantee that the CL garbage collector won't act
<soundmodel>
I am starting to think that maybe CL excludes apps that require latency guarantees
<soundmodel>
or at least I cannot understand yet how Incudine works
<edwlan[m]>
You can profile and avoid allocation
<edwlan[m]>
And you’d never run into gc pauses
<soundmodel>
it writes "In practice the init-time and performance-time functions of UGEN or DSP are arbitrary lisp functions regulated by the context (i.e. realtime audio)."
<soundmodel>
but what does that mean in C pseudocode or something
<soundmodel>
edwlan[m] what's that?
<soundmodel>
you mean like running the program and monitoring memory usage, then adjusting the program?
<edwlan[m]>
gc only runs if your code is allocating memory
<soundmodel>
so is this different from Java's or something?
<edwlan[m]>
No, people do this in Java too
<edwlan[m]>
It’s harder to write latency-sensitive code when there’s a gc around, but it’s not impossible to do it
<soundmodel>
you mean it releases stuff while it allocates new?
<soundmodel>
I thought the times for object release with ref count where largely undeterministic
<edwlan[m]>
You also don’t release memory
<soundmodel>
so is this like Java's Epsilon GC then?
<soundmodel>
that it will run out of memory eventually
<edwlan[m]>
I think a typical strategy is to allocate the memory you need and then make sure your code doesn’t either allocate any more or release any memory
<edwlan[m]>
It won’t run out of memory because the memory usage doesn’t change
<soundmodel>
yes in a sense this might be possible in DSP
<edwlan[m]>
Some lisps have garbage collectors written in lisp, for example
<soundmodel>
you only need to set up the filters etc. once and after that you have no need to dispatch them again
pieguy128 has quit [Ping timeout: 261 seconds]
pieguy128_ has joined #commonlisp
<soundmodel>
might not be possible in some kind of modular synth or live coding thing where you're supposed to also change blocks
<edwlan[m]>
And a garbage collector cannot allocate memory because there might not be any memory to be allocated when it is running
<edwlan[m]>
The other strategy is to manually trigger gc when you know it’s safe
<soundmodel>
but it has to be somehow substantially better than Java
<soundmodel>
because even if this was technically possible in Java, I've not seen any app that does it
<edwlan[m]>
But you can use sb-sprof and other tools to see hoe your code actually behaves and figure out how to make gc pauses not a problem
rgherdt has joined #commonlisp
<edwlan[m]>
Lots of high performance Java code is written for near-real time guarantees
<soundmodel>
it sounds very much like Epsilon GC
<edwlan[m]>
HFT algorithms and such
<soundmodel>
but it's not exactly meant for all sorts of real-time apps
<soundmodel>
but those that can work within the limitation of an upper limit for memory
<edwlan[m]>
They still have very tight deadlines
<edwlan[m]>
And unexpected latency can cost a lot of money
<soundmodel>
if it's the same as this one discussed in this Lisp, then there's the risk that the program will halt once it runs out of memory
<edwlan[m]>
Anyways, the gc doesn’t really cause “random pauses”
beach has joined #commonlisp
<edwlan[m]>
It causes pauses when your system is running out of memory
<edwlan[m]>
If you write your code to allocate as little memory as possible and use profilers to figure out how it behaves, you can make it work
<soundmodel>
I just don't know how CL GC could be much different from those of Java's
<soundmodel>
in principle
<soundmodel>
so it just makes me think that one ultimately wants the GC off
GreaseMonkey has quit [Quit: No Ping reply in 180 seconds.]
<edwlan[m]>
It’s not that different in principle
<edwlan[m]>
Java’s Shenandoah and ZGC gcs are better because they’re designed to minimize latency
greaser|q has joined #commonlisp
<edwlan[m]>
There’s a reason why people usually avoid languages with gc for real time applications
<ixelp>
GitHub - carp-lang/Carp: A statically typed lisp, without a GC, for real-time applications.
<beach>
soundmodel: Why would you want the GC off? Manual memory management is much more costly, and there are good GC techniques with no pauses now
greaser|q has quit [Changing host]
greaser|q has joined #commonlisp
greaser|q is now known as GreaseMonkey
<soundmodel>
beach Because the general consensus is that there are no garbage collectors that work within the limitations of < 1ms latencies without clicks
<soundmodel>
in real-time audio
<beach>
I am sure there are such garbage collectors. There are incremental collectors, and there are collectors that work in a separate thread so in parallel with the mutator threads.
Inline has quit [Quit: Leaving]
justache is now known as reddit-bot
reddit-bot is now known as justache
Oladon has quit [Quit: Leaving.]
<beach>
soundmodel: Also, the other day, you argued that Lisp should be faster because it is easier to parse. I don't know whether you got an answer to that, but parsing does not influence runtime behavior at all. It is all compile-time stuff, or rather read-time in Common Lisp.
<soundmodel>
yes because there should be a correlation with the length of machine code right?
<soundmodel>
e.g. how much it references stuff that it doesn't really need to
<beach>
I don't know what that means, but it sounds wrong.
<beach>
There is certainly no correlation between easy of parsing and size of machine code.
<beach>
And there is also no correlation between size of machine code and speed of machine code.
<soundmodel>
well possibly you can technically devise a program that's easy to parse but produces a lot of instructions compared to the length of the program
<beach>
If most of the machine code is very rarely used (which can very well be the case for Common Lisp), then there can be a lot of machine code and it can still be fast.
hayley has joined #commonlisp
<beach>
soundmodel: Absolutely. The more expressive the language, the more code will be produced for each source character.
<ixelp>
[PDF] A comparison of c++ concepts and haskell type classes | Semantic Scholar
<soundmodel>
it's similar, just syntactically slightly more verbose
<beach>
Are you kidding?
<hayley>
I wouldn't hold my breath.
<beach>
They are totally different.
<hayley>
The concepts of a "concept" and "typeclass" may be similar, but everything else in C++ and Haskell may well be different.
<beach>
soundmodel: C++ is statically typed, distinguishes between pointers and other values, which is totally fundamental.
<hayley>
Once someone on r/lisp told me that it was just my opinion that Lisp semantics involves dynamic typing and garbage collection, so you never know.
<Alfr>
Introduce lazy evaluation to C++ just to make it more similar to Haskell?
<soundmodel>
but I mean, the general topic is about compromises
<soundmodel>
and then I was thinking that maybe for real-time apps it'd be smarter to write Lisp-like C++
<soundmodel>
than seek a real-time Lisp
<mariari>
there is a strict haskell compiler
<hayley>
Well, this is most odd.
<mariari>
gotta just remove the spine
<soundmodel>
in Haskell's case the compromise is to leverage the C++ ecosystem
<soundmodel>
rather than Haskell's purity without the ecosystem
<hayley>
I'm preparing a slide deck on -- oh, that sounds painful.
<soundmodel>
so you get a bit of both
<mariari>
huh, the purity of haskell does not get in the way of IO
<mariari>
and you can reduce annoying MTL stacks down to a single monad with newer approaches, if you have issuees with overhead on that front
<mariari>
the laziness is the bigger killer
<hayley>
Well, um. I'm preparing a slide deck on a parallel garbage collector for SBCL, and the idea is to make it concurrent later, after it works well in a stop-the-world setting.
<soundmodel>
no but I mean that one exchanges the purity of Haskell to the lack of ecosystem
<soundmodel>
while one exchanges more ugliness in C++ to the ecosystem
<mariari>
Haskell has quite a few libraries these days(?)
<soundmodel>
or the nice features of CL to the lack of proper non-GC implementations
<hayley>
You assume that real-time computation cannot be done with a garbage collector. That's just false.
<soundmodel>
well that was the general consensus when I researched this a lot a while ago
<soundmodel>
at least for java
<beach>
soundmodel: Also, I did a calculation. The speed of sound is 343m/s and I don't think the ear can distinguish two sounds that are 10m apart. So that gives you roughly 30ms to buffer samples, enough for most modern GCs.
<hayley>
beach: As a rough estimate, I think I begin to lose my sense of timing while playing electric instruments with about 50ms latency.
<soundmodel>
? Pro musicians cannot play if there's more than 1.5 ms of latency or so
<soundmodel>
or they can, but they must consciously compensate for the lag
<hayley>
Well, I'm hardly a professional musician.
beach` has joined #commonlisp
<soundmodel>
the conclusion for Java was (there's a sequence of presentations around "real-time Java") that it's not possible to achieve real-time Java without removing the GC
<soundmodel>
and the Epsilon GC is the "best outcome" of that
<beach`>
soundmodel: So you don't believe my calculation?
beach has quit [Ping timeout: 240 seconds]
<mariari>
if you really are concerned about gc pause, you can do what jane street has done and make 0 allocation runs, and write with the memory you have in your hot loop
<hayley>
I haven't heard of the Real-Time Specification for Java being used. My memory of the papers on the Metronome GC was that the authors considered RTSJ removing the GC to be a mistake.
beach` is now known as beach
<Alfr>
soundmodel, that's not sound. So for a group of them to play something together, each of them must not be farther apart to any other one than ,(* 343 1.5e-3) meters.
<ixelp>
(* 343 1.5e-3) => 0.5145
<Alfr>
How may musicians can we pack in a 0.52m diameter circle?
<hayley>
Do they need to be humans?
<Alfr>
hayley, don't know.
<mariari>
do they need to have instruments?
<beach>
Alfr: That's pretty much what I was saying, no?
<Alfr>
beach, yeah, just elaborated on your numbers a little bit.
<beach>
Thanks!
<soundmodel>
but tbh Carp seems a pretty nice compromise
<soundmodel>
possibly even more contemporary than other CLs
<ixelp>
Carp/LanguageGuide.md at master · carp-lang/Carp · GitHub
<beach>
I think soundmodel was assuming a loop where each sample was generated without any buffering.
<hayley>
"The Metronome: A Simpler Approach to Garbage Collection in Real-time Systems" argues that it is very difficult to write applications where real-time and non-real-time threads must interact.
<beach>
soundmodel: It seems to me that you are conveniently ignoring statements that are contrary to your your current thinking.
<mariari>
I don't see how carp is more contemporary, what are other CLs, compilers?
<soundmodel>
contemporary to the world where Rust and Clojure are hot
<soundmodel>
and it suggest that it combines parts of those
<hayley>
Well, I don't like that world, so see if I care.
<mariari>
I don't see how you don't have them in CL itself, the types laid out in there are something CL does better
shka has joined #commonlisp
<soundmodel>
@beach Okay sorry, but I am not into doing research, but I was interested if someone knows immediately which Lisp to use for writing real-time audio programs
<beach>
soundmodel: You are not listening. It is not a question of programming language, but of the implementation of a programming language. Contrary to what the Finnish Wikipedia page says, Common Lisp is a programming language. And it has several implementations. Whether one has a GC that works for you is a different story.
<hayley>
Alfr: How far is the head of a pianist from the body of their piano?
<soundmodel>
yes so is Carp the only thing out there?
<hayley>
The only thing for solving non-existent problems, sure.
<soundmodel>
but it says it's for RT apps
<hayley>
And I say I have a well paying job, but my bank account disagrees with me on that one.
<soundmodel>
also plenty of sources say that GC is inherent for a Lisp
beach` has joined #commonlisp
<mariari>
ehhh?
<hayley>
I do say that too.
<soundmodel>
so maybe it's not the right tool for the task
<hayley>
I still don't see why you cannot make real-time applications using a garbage collector.
<beach>
soundmodel: This channel is for Common Lisp. Not for Java or for language comparisons.
<soundmodel>
no but it gives a reference for GC w.r.t. to real-time audio
<hayley>
I read "The IBM JVM [9] on the other hand typically took about 8 milliseconds per garbage collection with our application".
<beach>
So you reject our calculations. Too bad.
<soundmodel>
they conclude with hypothesis that is seen to lead to RTSJ etc.
<soundmodel>
but later on one discovers that they cannot offer the required latencies either
<hayley>
So computers did not get faster since 2007 (which I found is the year the paper was published), and low-latency GC (in general, or implementations for Java) did not become more accessible since then?
<beach>
soundmodel: I am sorry to hear that they are unable to do that. But it is off topic for this channel.
<soundmodel>
and then I just wonder how some other GC could be substantially better than the Oracle multibillion or whatever one
<hayley>
You are reading results for the IBM JVM from 2007 configured in an unknown manner.
<beach>
soundmodel: It is fine with me if you want to believe that you can't use a language with automatic memory management for your applications, but then this channels is probably inappropriate for discussions about your applications.
<soundmodel>
so are there framework for this already?
<hayley>
Nowadays IBM offers different collectors; I don't know if they did in 2007. But those collectors have differing effects on latency and throughput, and the authors don't state which was used. Which is rather sloppy writing in my opinion.
<ixelp>
Performance of Modern Java on Data-Heavy Workloads: The Low-Latency Rematch · Hazelcast Jet
<hayley>
Is your application data-heavy?
<beach>
soundmodel: Java is off topic for this channel.
<hayley>
And the page states "Keep especially in mind that latency does not equal a GC pause" - we don't know if the tail is the application, OS, some other runtime component, or the garbage collector.
<soundmodel>
again it's just for reference purposes on GCs
<soundmodel>
since I think we should assume that Java should have the best GCs
<hayley>
A reference for what though? Are you doing sliding window aggregation on three million items per second in your sound synthesis application?
<soundmodel>
or that there exists an useful GC for such
<beach>
soundmodel: Again, it is fine if you don't believe the calculations by me and Alfr, and if you have decided that your application requires a language without automatic memory management, but then your application is off topic for this channel.
<hayley>
You do not have any evidence that those results would be a good model of how the garbage collectors perform for a sound synthesis application.
<soundmodel>
"Regardless of GC model used on the JVM, all current GC’s have stop-the-world time. While these systems have developed a great deal over time, in general, they do not seem to target the kinds of latency requirements necessary for realtime audio systems, though they can work out fine if object allocation rate is managed."
<soundmodel>
which leaves open the question of whether there still exists such GC on some other platform
<beach>
soundmodel: OK, I think that's enough. You are not listening, and you are repeating the same arguments, without acknowledging arguments.
agm has joined #commonlisp
<soundmodel>
the things I read suggest that there's no such GC
<beach>
With a 30ms delay?
<soundmodel>
no 1.5
<beach>
So you are not listening. Fine!
<beach>
soundmodel: If you are not listening and not acknowledging arguments, I think it is time to stop.
<beach>
soundmodel: I think that, at this point, we should agree that you are right. There is no GC for your application, and you can't use a language with automatic memory management, so your application is off topic here. Now let's get back to Common Lisp!
<soundmodel>
I asked for a Common Lisp that meets such requirements
<soundmodel>
then I found that maybe it's Carp
<mariari>
Carp is not Common Lisp
<beach>
Again, you are not listening. Carp is not an implementation of Common Lisp.
<beach>
So Carp is off topic for this channel.
<soundmodel>
okay so is there a CL variant of it then?
<mariari>
is there a CL variant of C
<agm>
soundmodel: I'm writing a CL implementation that doesn't use strict GC, but an enhanced reference counting, so there are no random pauses. it's at https://savannah.nongnu.org/projects/alisp. it's still incomplete, but it shows that it can be done
<ixelp>
A lisp implementation - Summary [Savannah]
<beach>
soundmodel: I just told you no. There is no language with automatic memory management that has an implementation that will work for you.
<beach>
soundmodel: So now let's get back to Common Lisp.
<ixelp>
GitHub - ska80/thinlisp: ThinLisp is an open source Lisp to C translator.
<beach>
soundmodel: I doubt that thinlisp is an implementation of Common Lisp, so it's off topic here.
<beach>
soundmodel: ECL can translate Common Lisp to C, but that does not affect the garbage collector.
<beach>
soundmodel: Now, can we get back to the topic of this channel, i.e., Common Lisp?
<soundmodel>
but ThinLisp should be on-topic because it talks about a subset of CL minus the parts that are conveniences
<soundmodel>
over performance
<beach>
A subset of Common Lisp is not Common Lisp.
_cymew_ has joined #commonlisp
<beach>
soundmodel: I also don't understand why you want to submit all those off-topic links to this channel. Go ahead and read them if you think they are relevant to your application.
<beach>
soundmodel: Now, can we get back to the topic of this channel, i.e., Common Lisp?
<soundmodel>
because someone was arguing that CL can do real-time audio
<soundmodel>
and based on the multibillion Java ecosystem it's highly doubtable
<beach>
Yes, but you showed us all we were wrong.
<beach>
soundmodel: Now, can we get back to the topic of this channel, i.e., Common Lisp?
random-jellyfish has joined #commonlisp
<Alfr>
beach, not that CL prescribes a GC, we only have to invent that infinite tape.
beach` has joined #commonlisp
<random-jellyfish>
does abcl optimize (expt 2 n) to (ash 1 n)?
<random-jellyfish>
sbcl**
beach has quit [Killed (NickServ (GHOST command used by beach`))]
<random-jellyfish>
or should I just stick to ash to be sure?
beach` is now known as beach
kevingal_ has quit [Ping timeout: 255 seconds]
kevingal has quit [Ping timeout: 255 seconds]
<beach>
soundmodel: I think we are now all thoroughly convinced that no language implementation that uses a tracing garbage collector can be used for real-time sound generation. So you can not use such a language implementation for your application.
soundmodel has quit [Ping timeout: 260 seconds]
<beach>
random-jellyfish: Check the disassembly.
azimut has joined #commonlisp
<random-jellyfish>
also, when you use time to measure performance, is the number of cpu cycles that's printed out accurate or just some estimation?
random-jellyfish has quit [Quit: Client closed]
random-jellyfish has joined #commonlisp
beach` has joined #commonlisp
beach has quit [Ping timeout: 255 seconds]
random-jellyfish has quit [Quit: random-jellyfish]
beach` is now known as beach
random-jellyfish has joined #commonlisp
occ has quit [Ping timeout: 276 seconds]
beach` has joined #commonlisp
beach has quit [Ping timeout: 246 seconds]
beach` is now known as beach
heisig has joined #commonlisp
random-jellyfish has quit [Ping timeout: 255 seconds]
heisig has quit [Remote host closed the connection]
heisig has joined #commonlisp
NicknameJohn has joined #commonlisp
beach` has joined #commonlisp
beach has quit [Ping timeout: 255 seconds]
beach` is now known as beach
heisig has quit [Ping timeout: 255 seconds]
mrcom has quit [Quit: Leaving]
heisig has joined #commonlisp
agm has quit [Ping timeout: 264 seconds]
beach` has joined #commonlisp
beach has quit [Killed (NickServ (GHOST command used by beach`))]
beach` is now known as beach
soundmodel has joined #commonlisp
beach` has joined #commonlisp
beach has quit [Ping timeout: 255 seconds]
<soundmodel>
is CL emulated in C++ on-topic?
<hayley>
There is Clasp which implements CL atop a C++ runtime, but it uses a garbage collector.
<hayley>
So trying to change the topic like that won't work.
<soundmodel>
No I mean whether one can implement CL using templates
<soundmodel>
so whether CL is emulatable
<hayley>
Apparently C++ templates are Turing-complete, so the answer is yes.
<hayley>
But that would be a very odd implementation strategy. (Though I do routinely ponder if integers could ostensibly be Church-encoded in a simple language, so maybe not so odd.)
<soundmodel>
well I've been hitting the "it's not practical" -wall with this
<hayley>
And any implementation like that probably won't help your fear of garbage collection.
beach` is now known as beach
<soundmodel>
it's not a fear, it's just that it doesn't seem to solve the problem I have
thonkpod has quit [Ping timeout: 248 seconds]
thonkpod has joined #commonlisp
<hayley>
Right, statistics from 2007 from a paper on Java (which is not Common Lisp) using a JVM by IBM (which I have not seen used in the wild) which hardly suggest how to make reproducible results, and statistics from a recent blog post series on processing big data are relevant to doing sound synthesis in Common Lisp in 2023.
<hayley>
*hardly suggests
<soundmodel>
well yes as long as you don't want it to run 100 tracks with 2 ms latency
beach` has joined #commonlisp
beach has quit [Killed (NickServ (GHOST command used by beach`))]
beach` is now known as beach
<hayley>
Well, in that case, I must agree with beach, and should add that you should also beware that unicorn farts may clog up your CPU in the worst-case.
<beach>
soundmodel: One can implement Common Lisp in pretty much any language with varying difficulty.
<beach>
soundmodel: But the implementation language typically does not affect the memory management strategy. It is usually a tracing garbage collector.
<beach>
soundmodel: Since Common Lisp is good for almost any programming task, it is also the best language to use for writing a Common Lisp implementation.
<soundmodel>
it says that one can set the latency to 20 ms
<beach>
Shinmera certainly does.
<soundmodel>
so is that an idea about real-world latency of CL in audio apps?
<hayley>
One way to find out.
<beach>
Latency is a characteristic of the application and the Common Lisp implementation used. You can't generalize that to the Common Lisp language.
<soundmodel>
no but I assumed that the developer of such library would aim for the best performance
<hayley>
The latency is customisable so that the developer of the application can choose an appropriate latency.
<beach>
The best performance might involve buying a commercial Common Lisp implementation, and I am pretty sure Shinmera was not interested in that alternative
<hayley>
If an application does more than just generating sound, a larger buffer might be needed to be safe. I can change the buffer size on the DAW I use; are the developers of the DAW not aiming for low latency there?
<soundmodel>
okay so what kind of GC would I find in LispWorks or something?
<Shinmera>
the GC is not a problem.
<Shinmera>
And that's all I'll say.
<hayley>
Franz has a parallel GC, I dunno LispWorks.
<beach>
soundmodel: You need to ask them. This is Libera and is dedicated to free software.
<hayley>
Again, I'm catching up here.
<soundmodel>
okay well what's the best free CL implementation then?
<soundmodel>
best in performance
<Shinmera>
Actually: soundmodel, consider this an official mod warning. Your presumptuous behaviour so far in this channel has not impressed me. I urge you to read and understand what people here are telling you, rather than repeating some nebulous arguments you've found online.
<beach>
soundmodel: Probably SBCL overall, but the particular application may change that.
<soundmodel>
what else is there than GC though?
<soundmodel>
because that's what the other papers hold as the crucial bit
<hayley>
Even if we're talking about garbage collectors, there is huge variance in how the objects you allocate and hold onto will make the collector perform.
<beach>
soundmodel: Having done research in the analysis and synthesis of sound, I can tell you that the way the samples are produced from multiple sound sources is very important to performance.
<beach>
soundmodel: Also oscillator drift must be addressed for long-running applications.
<beach>
soundmodel: And if you do FFT or inverse FFT, the algorithm you choose can be crucial.
<soundmodel>
but that's not related to recording is it?
mrcom has joined #commonlisp
<beach>
soundmodel: I have seen no reference to recording. Only to producing sound in real time. If you just want to record sound, there is not much real time involved.
<soundmodel>
or playthrough for that matter: MIDI in -> produce a sound
<hayley>
I've done that, the latency was just PulseAudio being annoying. Note to self: don't use PulseAudio for that.
<beach>
That's sound synthesis in real time, and everything I said applies.
<beach>
hayley: Yeah, sound on Linux is a disaster it seems.
<Equill>
It is? JACK has worked nicely for me, for years.
<Equill>
It *was* pretty bad, back in the OSS days, but those are long past.
<beach>
Equill: Good to know. Last time I tried it, it required detailed knowledge about connecting what source to what output.
<beach>
Equill: And aren't there applications that use PulseAudio by default like Firefox?
<beach>
Anyway, off topic I guess. I'll talk to you about it in private some other time.
<soundmodel>
is it possible to infer the possible GC activity based on source code only or "in theory"?
<Equill>
It is indeed offtopic. I'm happy to talk about it outside this channel, any time.
<soundmodel>
or does it need to be tested in practice
<beach>
Equill: Thanks.
<beach>
soundmodel: Not possible. It depends on the particular GC implementation, and not just on your code.
<Equill>
soundmodel: what problem are you actually trying to solve, here? I get the feeling you're spinning your wheels.
<hayley>
When GC occurs is fundamentally a global property.
<soundmodel>
I wanted to understand how to use Common Lips in the context of hard real time
<soundmodel>
Lisp*
<beach>
Equill: soundmodel is convinced, despite solid calculations, that a pause of 1.5ms is the maximum allowed. But then soundmodel also knows that no GC algorithm can do that, so soundmodel knows that no language with a GC can be used to write real-time sound applications.
Bocaneri has joined #commonlisp
Bocaneri is now known as Guest4705
<beach>
soundmodel: Again, it is not the language that determines that, but the implementation.
<soundmodel>
I wonder if there are follow-ups somewhere
<Equill>
Hmmm. "Hard real-time" is pretty challenging. I'm not an expert, but IIUC the OS is as crucial as the language, and it's a rare problem-domain that genuinely needs it.
Sauvin has quit [Ping timeout: 276 seconds]
<Equill>
1.5ms is pretty tight. I'm working with 32ms at the moment, according to qjackctl's calculations: 3 periods of 512 frames per buffer, at 48KHz. That's good enough for a mixture of synthesized and recorded tracks, plus whatever I'm recording live.
<Equill>
Neither is written in CL, but it could be worth looking through the source-code for Ardour (DAW) and Surge XT (synth) to see how they approach memory-management. Both are mature, and are known to perform very well indeed. That's the pre-research side.
<beach>
Equill: Are you accusing soundmodel of being wrong that 1.5ms is required? How dare you?
<Equill>
Presumptuous git, that's me.
<beach>
:)
<Equill>
I'm an infrastructure engineer: I've made a career out of explaining inconvenient practicalities to people. See also RFC 1925. Actually, *everybody* should read that one, because it applies to a lot more than just networking.
<Equill>
soundmodel: my best advice is to try it, and see how it works out. Best-case, you'll learn a lot, and performance will be better than you expect. More likely, you'll run into performance issues, and the smart approach there is to instrument the code and find out where they're actually coming from - it usually isn't where you thought.
<soundmodel>
I'm going to learn some CL anyways, because I also want to understand how I can write that style in C++
<Equill>
This is a good reason.
<Equill>
Probably better than trying to write C++ in CL, too :)
<soundmodel>
and yes the latency can be higher than that, but so far I've not found any pointers to what kind of latencies to expect
<soundmodel>
the takeuchi paper seemed hard to read
<Equill>
If you wait until you get an a priori prediction you like, you'll never get started. Just try it, then measure the results, then adjust as necessary and repeat.
<soundmodel>
the numbers on p. 13 also seem oddly small
agm has joined #commonlisp
<soundmodel>
if the worst delays are ~100 musec
<Equill>
I say this because I used to get paralysed in the same way. Eventually I accepted that theory is only ever an approximation, and the real world is more complicated and unpredictable than you expect.
<soundmodel>
those would be like 100 times smaller than the worst cases they give for Java's
<Equill>
soundmodel: I've just looked at the Takeuchi paper. It's based on hardware made *30 years ago*. If you want to try doing it in CL in the hope that it'll be fast enough, try it and see. If you want to make audio software with non-GC memory management, Rust is probably your better bet.
agm has quit [Ping timeout: 255 seconds]
<soundmodel>
TBH that paper is very weird
<soundmodel>
they suggest later on as if they've made some sort of rare discovery
<soundmodel>
"Quick survey revealed that wake-up delay or pause time has long b een told in term of millisecond. We wonder
<soundmodel>
why this simple idea, incremental up date with write-barrier has not b een realized so long time. Nothing very
<soundmodel>
sp ecial is involved in our basic algorithm and implementation. We susp ect this is a sort of \Columbus's egg";
<soundmodel>
that is, something simple or p ossible but which no one has ever tried actually or seriously"
<soundmodel>
when it sounds a bit like they may have set up things wrong if the results are 100-1000 times better
<soundmodel>
or not even "up to" but more like "on average". The worst cases seem to go further than that
karlosz has joined #commonlisp
McParen has joined #commonlisp
<Equill>
Clearly, it's a doomed proposition, and you should abandon this idea before you waste any more time on it.
<Bike>
this paper is over thirty years old, man.
<McParen>
does anyone know when the next quicklisp release is scheduled? (is it scheduled at all?)
<mzan>
soundmodel: take also in consideration that in CL you can annotate some variables with "dynamic-extent" and if the compiler supports it, they will be allocated on the stack http://clhs.lisp.se/Body/d_dynami.htm
<ixelp>
CLHS: Declaration DYNAMIC-EXTENT
<mzan>
As other noted: a sound processing algorithm is often a pipeline, so you can pre allocate the buffers with the data, and if you reuse them, probably the GC will never work
cage has joined #commonlisp
<mzan>
So in an ideal situation you have preallocated buffers for processing data, and local stack-allocated variables when you call procedures, i.e. nearly no GC at all
<soundmodel>
Bike Oops yep
<soundmodel>
but it's also very difficult to find much papers on CL
<hayley>
Then you might have to invent some.
<Equill>
I was going to suggest that: do some new research, and publish the results.
<soundmodel>
"As a result, scavenging took 2.9 milliseconds
<soundmodel>
on average, tracing took 1.9 milliseconds, and sweeping took 1.6
<soundmodel>
milliseconds."
beach has quit [Ping timeout: 260 seconds]
<hayley>
Never heard of it.
<hayley>
But check the context, and the "future work" section.
random-nick has joined #commonlisp
<soundmodel>
p. 7 has benchmarks
<soundmodel>
it's running the game Kandria
<soundmodel>
that seems to suggest worst case around 20 ms
<soundmodel>
with possible optimization down to 5-7 ms
beach`` has joined #commonlisp
<hayley>
A bit less, if you're using the normal SBCL collector and not that presented in the paper. Not that I would know about that collector.
beach`` is now known as beach
beach` has quit [Ping timeout: 255 seconds]
agm has quit [Ping timeout: 255 seconds]
<soundmodel>
but what about Figure 4
<hayley>
Is your application constantly allocating messages in a ring buffer?
<soundmodel>
it suggests far worse performance with ring buffers
<soundmodel>
I thought they're quite central in DSP applications
<hayley>
"The benchmark involves a ring buffer of small unboxed arrays, each array containing a one kilobyte “message”, with each new message being a new allocation from the heap."
<soundmodel>
and?
<hayley>
That benchmark was derived from a real application (though certainly a weird one), but I don't think you'd put those in a ring buffer for DSP.
<hayley>
A ring buffer of samples seems reasonable, and samples tend to be small integers or floats, which don't put any pressure on GC.
<soundmodel>
I haven't done much DSP, but IIRC a possibly use case is the output from a spectral process involving FFT
beach` has joined #commonlisp
<soundmodel>
possible*
beach has quit [Ping timeout: 255 seconds]
<Bike>
if you haven't done much DSP why on earth are you getting into these weeds about latency?
McParen has left #commonlisp [#commonlisp]
<soundmodel>
well it didn't seem to work in the Java land
<Bike>
why do you care? what is your purpose?
<soundmodel>
avoid doing experiments on something that's already known in theory
<soundmodel>
on a side note some other sources treat CL as an AI language
<soundmodel>
and not as a general-purpose one
<Equill>
That's probably because AI research is where Lisp (and thus CL) comes from.
<Bike>
So you found Harmony, a sound library used in a commercial product, and now on the basis of half understood 1980s "theory" papers doing experimental benchmarks, you think doing sound in Lisp is impossible?
<Bike>
You already got a mod warning. You don't have to write in Lisp if you don't want to. Emulating Lisp in C++ is not going to end well. If you want to write in C++ you should just write in C++ rather than try to fit a square peg into a round hole.
tibfulv has quit [Read error: Connection reset by peer]
<msavoritias>
Also you dont cant avoid expermenting. Programming is above all experimentation
agm has joined #commonlisp
agm has quit [Remote host closed the connection]
tibfulv has joined #commonlisp
agm has joined #commonlisp
<Bike>
you're certainly not going to find a mathematical proof or disproof that you can do DSP. the question is inherently based on what kind of processing you're doing and what computer you're running it on.
tyson2 has joined #commonlisp
soundmodel has quit [Ping timeout: 260 seconds]
jfrent has joined #commonlisp
beach`` has joined #commonlisp
beach` has quit [Ping timeout: 255 seconds]
soundmodel has joined #commonlisp
dcb has quit [Read error: Connection reset by peer]
_dcb_ has joined #commonlisp
makomo has joined #commonlisp
makomo has quit [Client Quit]
jmdaemon has quit [Ping timeout: 250 seconds]
Guest4705 is now known as Sauvin
soundmodel has quit [Quit: Client closed]
beach``` has joined #commonlisp
beach`` has quit [Ping timeout: 276 seconds]
beach```` has joined #commonlisp
soundmodel has joined #commonlisp
beach``` has quit [Ping timeout: 240 seconds]
occ has joined #commonlisp
pfd has joined #commonlisp
<Shinmera>
beach````: FWIW cl-mixed has native backends for ALSA, PulseAudio, JACK, and OSS (and hopefully soon also PipeWire).
occ has quit [Ping timeout: 240 seconds]
<Shinmera>
Though, with the exception of PulseAudio, only for playback. Nobody's done the input equivalents.
<Shinmera>
(which would be a very easy implementation task if anyone's bored)
beach```` has quit [Ping timeout: 240 seconds]
agm has quit [Ping timeout: 240 seconds]
<Shinmera>
Oh, it also just recently gained AAudio support, too, so it runs on Android.
beach has joined #commonlisp
<beach>
Shinmera: I see. Thank you!
McParen has joined #commonlisp
heisig has quit [Ping timeout: 255 seconds]
igemnace has quit [Read error: Connection reset by peer]
jfrent has left #commonlisp [#commonlisp]
beach has quit [Ping timeout: 260 seconds]
beach`` has joined #commonlisp
occ has joined #commonlisp
cage has quit [Remote host closed the connection]
cage has joined #commonlisp
beach`` has quit [Ping timeout: 240 seconds]
beach```` has joined #commonlisp
pfd has quit [Ping timeout: 260 seconds]
soundmodel has quit [Ping timeout: 260 seconds]
epony has joined #commonlisp
beach```` has quit [Ping timeout: 240 seconds]
beach` has joined #commonlisp
skin has joined #commonlisp
beach` has quit [Ping timeout: 276 seconds]
Krystof has quit [Ping timeout: 252 seconds]
karlosz has quit [Quit: karlosz]
skin has quit [Quit: Leaving.]
skin has joined #commonlisp
skin has quit [Quit: Leaving.]
skin has joined #commonlisp
ebrasca has joined #commonlisp
skin has quit [Quit: Leaving.]
skin has joined #commonlisp
skin has quit [Client Quit]
beach has joined #commonlisp
<contrapunctus>
beach: I had a question regarding CLOSOS - how is the object store to be implemented? i.e. does it replace existing filesystems (such as ext4), or is it a database on a typical filesystem, containing metadata for CLOSOS objects and external files?
Equill has quit [Remote host closed the connection]
<beach>
contrapunctus: It has nothing to do with external memory. It is just a data structure in the unified memory.
skin has joined #commonlisp
Equill has joined #commonlisp
skin has quit [Client Quit]
skin has joined #commonlisp
<beach>
contrapunctus: Does that make sense?
_dcb_ has quit [Remote host closed the connection]
_dcb_ has joined #commonlisp
fourier has joined #commonlisp
<contrapunctus>
beach: somewhat. I understand that the unified memory (and therefore the object store too) is persisted to disk in some form. And I guess the simplest option is to use existing filesystems for that...
<beach>
Well, Closos is intended to run on a machine without any other OS, so it is using whatever secondary memory is available.
<beach>
But if it runs on top of something like Linux, yes, then perhaps a single file could be used as the entire Closos secondary memory.
<beach>
I haven't thought much about that case.
skin has quit [Quit: Leaving.]
skin has joined #commonlisp
<beach>
contrapunctus: There would be no file system on a computer without an operating system.
occ has quit [Ping timeout: 255 seconds]
heisig has joined #commonlisp
heisig has quit [Client Quit]
<beach>
contrapunctus: Maybe I am not expressing myself very well, so if you have more questions, let me know.
<nytpu>
beach: uhhh... filesystems exist on the disk and don't need a hosted operating system to be used. have you ever used an embedded device that supports SD cards or anything like that? unless they're treating it like arbitrary formatless flash then they're using a filesystem so it can actually be accessed and understood by other devices
<beach>
That's strange to me.
<contrapunctus>
Ah, nytpu brought up what I was struggling to for the last ten minutes :D
<beach>
What file system would there be? I mean ext4 is Linux specific, and I am pretty sure Windows uses something else.
<edwlan[m]>
Isn’t a filesystem a datastructure for organizing the layout of information on a disk?
<edwlan[m]>
That’s more or less how I’ve always understood the term
<beach>
But all the file systems I know are OS-specific.
<beach>
Are you saying that something like ext4 runs on top of something that is independent of the OS?
<Bike>
exFAT is used for external drives for example
<edwlan[m]>
There are implementations of the code necessary to read disks formatted as ext4 on other OSes
<Bike>
the OS does the actual operations, but the data on the disk is formatted in the ext4/exFAT/etc fashion so the OS can deal with it
<beach>
Bike: Sure, but it is still used to create files, right?
<beach>
And since Closos doesn't have files like that, why would there be a file system on the disk?
<Bike>
The OS is used to create files? Yes
<beach>
So if you have an OS that doesn't create files, why would you have a file system on disk?
<Bike>
I suppose for CLOSOS the "file system" would be the serialization format, which is identical to the memory layout
<drakonis>
you need a filesystem on disk so data can be moved out of the OS?
<beach>
drakonis: Only of you have files on disk, no?
<drakonis>
basically, the gist of the whole thing is that filesystems exist so that data can be persisted across reboots and can be moved to other devices
<beach>
Yes, Closos would use consecutive disk blocks directly, just like a file system would.
<drakonis>
they also exist as a protocol of sorts
<drakonis>
so that if you move the device, it can be understood and the files in it can be read by another OS
<beach>
drakonis: But only in operating systems that use files.
NotThatRPG has joined #commonlisp
<drakonis>
yes but you're inevitably going to have to do this to share data with other devices
<beach>
I don't understand.
<beach>
Let me put it this way: What is the file system that is used by ext4, given that you seem to say that there is a file system on disk before ext4 is installed?
<drakonis>
ext4 is the file system
<beach>
It can't be according to what was said before, i.e. that the file system is independent of the OS.
skin has quit [Quit: Leaving.]
<drakonis>
which can be installed on a block device as a way to organize the data there, the OS needs to understand how the data is organized to actually read and write to it
<beach>
Of course. But in this discussion, the OS is Closos, and it uses disk blocks and not files.
<drakonis>
a file is an abstraction of persistent data, i suppose?
<beach>
At some point, some software must access the disk blocks (sectors or whatever) directly, no?
<beach>
drakonis: Yes, but not in Closos.
<drakonis>
it typically accesses through a filesystem abstraction
<beach>
In an OS that has files, yes.
<drakonis>
because it is simpler to do so
skin has joined #commonlisp
NotThatRPG has quit [Ping timeout: 240 seconds]
<beach>
drakonis: Simpler than what?
<drakonis>
simpler than having to directly think about how the data is stored in the block device and how you can accidentally clobber existing data by writing too much
<drakonis>
or even reading too much data and accidentally reading more than you wanted
<beach>
drakonis: But the file system has to do that thinking, and I did the analogous thinking for Closos, which doesn't have files.
<drakonis>
so, congratulations, you have a file system in a way?
<beach>
I do not.
<beach>
Because I don't have the concept of files on disk.
<drakonis>
bear with me for i will go abstract now
<drakonis>
a filesystem is normally a tree data structure persisted onto some form of storage media
<beach>
Seems correct.
<drakonis>
typically
<drakonis>
closos has a object storage that roughly translates to this, yes?
<beach>
Nope.
<beach>
The object store is just a mapping from (nontrivial) names to objects.
<beach>
Nothing to do with trees, external memory, or anything like that.
<drakonis>
so its like smalltalk then?
<beach>
I don't know Smalltalk.
<beach>
Closos is not much more than a persistent Common Lisp implementation. The object store is just another data structure in memory.
<drakonis>
yes but can the object store be translated to a tree structure?
<beach>
Nope.
<beach>
Are you saying that a tree of CONSes in Common Lisp is a file system?
<drakonis>
yes you could treat it that way
<drakonis>
the file metadata and its contents
<beach>
OK, then all of Closos is a file system with that definition. Not just the object store.
<drakonis>
there's more complexities to a filesystem than that though
<beach>
So is every Common Lisp implementation.
<beach>
Oh, there must be more? Then Closos object store is not a file system, because that's all there is to it.
<mzan>
'In modern usage, the term usually refers to the organization of a computing system in which there are no files, only persistent objects (sometimes called segments), which are mapped into processes' address spaces (which consist entirely of a collection of mapped objects). The entire storage of the computer is thought of as a single two-dimensional plane of addresses (segment, and address within segment). '
_dcb_ has quit [Remote host closed the connection]
_dcb_ has joined #commonlisp
<beach>
mzan: Yes, but Closos doesn't have segments (like Multics did/does) either.
<beach>
mzan: It has only Common Lisp objects.
X-Scale has joined #commonlisp
<mzan>
yes, clear
jathd has quit [Ping timeout: 246 seconds]
<beach>
And it doesn't have processes either, the way Multics did.
<mzan>
You have only one way for accessing data in the system, and then transparently under the hood you map them to RAM or disk
<beach>
Right.
<mzan>
The 'filesystem' if exists, is not exposed to the user, but used internally from your system
<beach>
I don't see a file system. You can read the relevant chapter in the Closos specification and see what resembles a file system. I don't see anything.
<beach>
I should take this discussion as evidence that the brainwashing of generations of computer users has been so successful that the presence of a file system seems like a must, and not only a feature of "modern" operating systems.
<drakonis>
filesystem is the wrong term
<drakonis>
the right one is object persistence
<beach>
Oh, so now we have had a very long discussion with the wrong term.
<beach>
Which I said right in the beginning.
<beach>
And object persistence is also not right. Because its abstraction is the saving and restoring of objects to and from secondary memory.
<beach>
Closos doesn't propose that abstraction to the user.
eddof13 has joined #commonlisp
<beach>
Closos just saves memory pages that have been modified.
<beach>
I am very puzzled by this discussion, actually.
<edwlan[m]>
My experience is that the terms used in discussing what closos does have are unfamiliar to me
<edwlan[m]>
Probably because the model is different from the oses I’m used to
<beach>
edwlan[m]: It is fine for things to be unfamiliar, and that's a fixable problem.
<beach>
edwlan[m]: But what I see is a bunch of arguments that are just wrong and no willingness to understand why.
<edwlan[m]>
I think what’s sometimes helpful in this situation is to say things like “it’s like a filesystem in this way and …”
<beach>
But I would then by lying.
<edwlan[m]>
“It’s like a filesystem because it’s an organization of data on disk, but there are no files, just a mapping of names to objects”
<beach>
So a Common Lisp vector (or a hash table, or a list, or a standard object) is like a file system?
<beach>
That's a very strange way of seeing those objects.
<edwlan[m]>
The part that took me a long time to understand was that closos hides the distinction between memory and disk more than typical systems do.
<drakonis>
not every common lisp data type is a file system
<beach>
Yes, and that's because you and most other people have been brainwashed for generations to believe that there has to be a distinction (to the user) between primary and secondary memory. This is despite the fact that we knew how to do that before Unix was invented.
<johnjaye>
i mean it's a pretty clear hardware distinciton. some memory stays intact when you turn of the power, others don't
<edwlan[m]>
The only current system I’m aware of that even claims to do something similar is the IBM i
<johnjaye>
are you saying the hardware is badly designed?
<mzan>
The confusion is that "filesystem" can mean two things: the physical dataformat on disk for storing data; the API or logical paradigm for accessing data on disk.
<beach>
edwlan[m]: That previous thing was for you.
<edwlan[m]>
And I think even that doesn’t succeed 100%
<drakonis>
mzan: that, yes.
<beach>
johnjaye: I guess you are not familiar with the concept of building high-level abstractions on top of low-level things.
<beach>
Wow, this is getting more and more weird.
<pjb>
If you put all your lisp objects in hash-tables, with uniform keys, yes. But this is not what a file system is, it's just what a hierarchical directory subsystem in a file system is.
<johnjaye>
of course not, i'm a programmer. i just connect things with perl scripts.
<beach>
contrapunctus: Now do you have a clearer picture of Closos and the object store?
<johnjaye>
anyway you mentioned primary and secondary memory. maybe we are brainwashed sure. but the physical presence of hardware differences sure makes it convincing
<contrapunctus>
beach: I suspect I will, after I go through the discussion a few more times. Thanks :)
<beach>
johnjaye: Absolutely, and that's why the creators of Unix took the easy way out, and preserved that distinction to the user, despite the fact that Multics (which Unix is modeled after) did not have that distinction.
<beach>
johnjaye: So as a result, we have been living with that distinction for half a century or so, with fewer and fewer people questioning the wisdom of it.
<beach>
And I can't even begin to estimate the millions of programmer hours wasted because of it.
<edwlan[m]>
I like this idea
<beach>
edwlan[m]: Which one?
<edwlan[m]>
The idea of hiding the difference between primary and secondary memory
<beach>
Oh, thank you! It is not new though. Half a century or so like I said.
<edwlan[m]>
The main concern I have is that I’ve worked with various systems that claim to do this and you almost always end up in abstraction leaks.
NotThatRPG has joined #commonlisp
<edwlan[m]>
(I’m aware, I think I’ve discussed this with you before under the name fiddlerwoaroof)
<johnjaye>
is that Strandh pdf the specification of clos? or more of an informal description?
<beach>
edwlan[m]: Probably so, yes.
<beach>
johnjaye: It is as much as I have had the energy to produce.
<johnjaye>
fair enough. i will keep this bookmarked!
<beach>
Great!
monoidog has joined #commonlisp
<edwlan[m]>
On a different topic, does clobber provide any way to compact or repair the transaction log?
<beach>
I don't think so. I wrote Clobber mostly as a protest against "object prevalence".
<beach>
I never meant it to be used. Though I am sure it is usable, perhaps with a few modifications.
X-Scale has quit [Quit: Client closed]
<edwlan[m]>
I like the concept, it’s more or less how I think a data store should work
<beach>
If you are forced to have the primary/secondary memory distinction, then sure.
<edwlan[m]>
And industry streaming systems have sort of reinvented some of its ideas. (See the talk “turning the database inside out” for an example)
<beach>
Interesting!
<edwlan[m]>
In a distributed system, it’s pretty common to have a protocol for publishing messages and then the systems that need to use the messages construct a data store that’s optimized for their access patterns from reading a stream of messages
lucasta has joined #commonlisp
<beach>
I see.
d0g has joined #commonlisp
monoidog has quit [Ping timeout: 265 seconds]
X-Scale has joined #commonlisp
<beach>
Time for me to go back to my room where the network is flaky.
beach has quit [Quit: ERC (IRC client for Emacs 27.1)]
skin has quit [Quit: Leaving.]
skin has joined #commonlisp
Oladon has quit [Quit: Leaving.]
tyson2 has quit [Remote host closed the connection]
Gleefre has joined #commonlisp
<pjb>
https://termbin.com/8gx88 ; but I'm not sure of the point of restricting access to lisp object with paths…
<d0g>
any recommendation on a markdown parser library? or should I rather build one on my own?
_dcb_ has quit [Remote host closed the connection]
<ixelp>
me/nrdl: Nestable Readable Document Language - nrdl - git.djha.skin
__dcb__ has quit [Remote host closed the connection]
__dcb__ has joined #commonlisp
<monoidog>
I only have minimal experience with esrap, used it for a scribble subset for a project, it was good enough for my needs, but parsnip looks promising
<edwlan[m]>
Parser combinators often have a bunch of sharp edges
<skin>
It's very much mirrors how I wrote my parser but from scratch. I probably would have used this if I knew about it before I started
<skin>
edwlan[m]: tell me more
__dcb__ is now known as dcb
<edwlan[m]>
Getting good error messages is a bit tricky because of how they backtrack.
<skin>
Makes sense
<edwlan[m]>
And because they work by calling functions, you can run into issues with large documents blowing the stack
<skin>
I avoided backtracking on purpose. Makes things messy doesn't it.
<skin>
That's a good point too. I wonder if I took that into account. That's good feedback
<edwlan[m]>
I used a parser-combinator library for a parser for EDN that works pretty well, but it ran into significant performance issues with documents in the 100s of MB
<edwlan[m]>
that the Clojure implementation doesn't have
<skin>
That's pretty big
<skin>
I think I'd be happy to use your parser for my needs
<skin>
What were you using EDN for
<edwlan[m]>
It's one of my favorite data formats
<skin>
Fair
<skin>
I like that it supports keywords but I also think that a lot of stuff in it is molded to the JVM
<edwlan[m]>
Not really
<skin>
It has big number literals where JSON just has numbers
<skin>
It has big float literals where JSON just has numbers
<edwlan[m]>
That's pretty important
<edwlan[m]>
JSON runs into a bunch of issues with cross-language compatibility because of this
<edwlan[m]>
JSON doesn't specify a range or precision for its numeric types and you'll run into issues where different languages interpret the same numeric value differently
<edwlan[m]>
Notably, since JavaScript doesn't have integers, big enough integer values will lose precision when parsed in Javascript
<skin>
That's true but Java doesn't have unsigned integers. So you still sort of running to the same problem going from Java to C with edn
<edwlan[m]>
I also like EDN's tagged literals which let you extend the types it can represent
<skin>
Frankly I love EDN. If it had embeddable documents like yaml I probably wouldn't have written my own language
<skin>
I would have just used EDN
<edwlan[m]>
I find YAML's string representations pretty bad
<skin>
Those tags only make sense in untyped languages though
<skin>
Me too
<edwlan[m]>
the multi-line strings are just ok, but the syntax is really confusing
<skin>
Yup
<edwlan[m]>
the single-line strings are annoying because of ambiguity between types
<edwlan[m]>
The tags make sense in any language, you just have to make sure the parser knows how to interpret them
<edwlan[m]>
YAML and XML have equivalent concepts
<skin>
True
<edwlan[m]>
The other thing EDN gets right is namespaced identifiers
<edwlan[m]>
I think YAML has something like this
<edwlan[m]>
And its one of the things that makes me prefer XML to a lot of other simpler formats
<skin>
Yeah but that's another thing that's molded to closure. It works well for common lisp but it doesn't take into account languages that have nestable namespaces
<skin>
Like Java
<edwlan[m]>
I don't think that's particularly relevant
<skin>
What are an interesting opinion that you prefer XML
<skin>
That is really intriguing
<skin>
Yeah it doesn't really matter much
<edwlan[m]>
For one thing, the namespaces don't have to map to language namespaces
<skin>
I was just thinking about this stuff when I wrote mine
<skin>
True
<edwlan[m]>
Anyways, the xmlns system in XML is great
<skin>
XML gives you a lot of rope. Is there a particular style of XML serialization that you prefer
<edwlan[m]>
And I've never really understood why people don't like XML, it has some verbosity downsides, but its also well-designed in a lot of ways
<edwlan[m]>
And tools like XPath/XSLT are pretty nice
<skin>
That's what I don't like about it. There's eight different ways to write the same thing
<skin>
That's true
<skin>
That's why I didn't like Perl and why I switched away from it
<edwlan[m]>
To Common Lisp :)
<edwlan[m]>
No one could accuse common lisp of only having one way to do the same thing
<skin>
I guess what I mean by that is: how would you serialize a hash table for example
<skin>
Would you make keys and values attributes of a tag or would you make a key its own tag and a value its own tag etc
<edwlan[m]>
It depends on what the intention behind the hash-table is
<edwlan[m]>
attributes as metadata and tags as data makes a lot of sense to me
<skin>
Maybe there's a library that you use that you prefer that makes decisions like this?
<edwlan[m]>
I don't actually use XML all that much, it's more that it has a bunch of properties that simpler data formats lost
<skin>
The metadata data thing makes sense I guess
<edwlan[m]>
Like, unambiguous tag and attribute names
<skin>
But when writing nrdl I realized I didn't want something that was too busy. I omitted the yaml tagging for example
<edwlan[m]>
Because, if you're using xmlns, every tag has a url that it's namespaced to
tyson2 has joined #commonlisp
karlosz has joined #commonlisp
Gleefre has quit [Remote host closed the connection]
<splittist>
One of the things I like about xml is being able to treat it as text and throw regexes at it (:
skin has quit [Ping timeout: 264 seconds]
skin has joined #commonlisp
karlosz has quit [Quit: karlosz]
karlosz has joined #commonlisp
Gleefre has joined #commonlisp
eddof13 has quit [Quit: My MacBook Air has gone to sleep. ZZZzzz…]
karlosz has quit [Quit: karlosz]
karlosz has joined #commonlisp
<mister_m>
Is a hash table really a “document” though? Is that the kind of thing xml was really supposed to be used for? My impression is that a lot of the hate comes from people using a markup format for documents as a something for gobs of data points or what have you where they actually need virtually none of the XML strengths and features
tyson2 has quit [Remote host closed the connection]
azimut has quit [Ping timeout: 255 seconds]
skin has quit [Ping timeout: 260 seconds]
skin has joined #commonlisp
X-Scale has quit [Quit: Client closed]
occ has joined #commonlisp
Oladon has joined #commonlisp
skin1 has joined #commonlisp
skin1 has quit [Client Quit]
kevingal has joined #commonlisp
kevingal_ has joined #commonlisp
dcb has quit [Remote host closed the connection]
_dcb_ has joined #commonlisp
_dcb_ has quit [Client Quit]
euandreh has joined #commonlisp
NicknameJohn has quit [Ping timeout: 240 seconds]
lucasta has quit [Remote host closed the connection]
epony has quit [Remote host closed the connection]
epony has joined #commonlisp
d0g has quit [Quit: Leaving]
cage has quit [Quit: rcirc on GNU Emacs 28.2]
skin has quit [Ping timeout: 255 seconds]
skin has joined #commonlisp
kevingal_ has quit [Ping timeout: 240 seconds]
kevingal has quit [Ping timeout: 240 seconds]
boogsbunny has joined #commonlisp
boogsbunny has quit [Client Quit]
skin has quit [Ping timeout: 255 seconds]
skin has joined #commonlisp
McParen has left #commonlisp [#commonlisp]
skin has quit [Ping timeout: 260 seconds]
skin has joined #commonlisp
_cymew_ has quit [Ping timeout: 265 seconds]
attila_lendvai has joined #commonlisp
waleee has joined #commonlisp
skin has quit [Read error: Connection reset by peer]
skin has joined #commonlisp
lucasta has joined #commonlisp
skin has quit [Ping timeout: 240 seconds]
skin has joined #commonlisp
epony has quit [Ping timeout: 240 seconds]
epony has joined #commonlisp
skin has quit [Ping timeout: 248 seconds]
skin has joined #commonlisp
eddof13 has joined #commonlisp
skin1 has joined #commonlisp
eddof13 has quit [Client Quit]
Oladon has quit [Quit: Leaving.]
Oladon has joined #commonlisp
akoana has joined #commonlisp
tyson2 has joined #commonlisp
modula2 has joined #commonlisp
edgar-rft has quit [Quit: Leaving]
edgar-rft has joined #commonlisp
dcb has joined #commonlisp
skin has quit [Ping timeout: 255 seconds]
skin has joined #commonlisp
jmdaemon has joined #commonlisp
shka has quit [Ping timeout: 252 seconds]
attila_lendvai has quit [Remote host closed the connection]
attila_lendvai has joined #commonlisp
lucasta has quit [Remote host closed the connection]
skin has quit [Ping timeout: 250 seconds]
attila_lendvai has quit [Ping timeout: 276 seconds]
skin has joined #commonlisp
tankrim has joined #commonlisp
occ has quit [Ping timeout: 255 seconds]
ober has quit [Ping timeout: 240 seconds]
lucasta has joined #commonlisp
samedi has joined #commonlisp
tankrim has quit [Ping timeout: 255 seconds]
ober has joined #commonlisp
johnjaye has quit [Ping timeout: 248 seconds]
johnjaye has joined #commonlisp
rgherdt has quit [Remote host closed the connection]
X-Scale has joined #commonlisp
johnjaye has quit [Ping timeout: 240 seconds]
NicknameJohn has joined #commonlisp
random-nick has quit [Ping timeout: 240 seconds]
monoidog has quit [Remote host closed the connection]