<aw->
but (split) has nother uses, so these kind of utils wouldn't be valuable in the main distribution unless we augment 'sc' to accept additional arguments to (split).. at that point 'sc' will still be too inflexible for all use-cases. So, to answer your question: no, I don't think it should be in the standard distro.
hrberg has quit [Quit: No Ping reply in 180 seconds.]
hrberg has joined #picolisp
_whitelogger has joined #picolisp
<abu[7]>
Yeah, we should not clobber the distro with too many trivial functions
pablo_escoberg has joined #picolisp
<pablo_escoberg>
Cool. In the utils file it goes :)
msavoritias has joined #picolisp
pablo_escoberg has quit [Quit: Client closed]
pablo_escoberg has joined #picolisp
<pablo_escoberg>
BTW, I put a few questions in the Matrix chat. Has anyone seen them?
aw- has quit [Quit: Leaving.]
felter3 has quit [Remote host closed the connection]
pablo_escoberg has quit [Ping timeout: 246 seconds]
msavoritias has quit [Remote host closed the connection]
<abu[7]>
Hmm, I see no question in #picolisp:7fach.de
aw- has joined #picolisp
Nistur has quit [Ping timeout: 246 seconds]
Nistur has joined #picolisp
avocadoist has quit [Remote host closed the connection]
pablo_escoberg has joined #picolisp
<pablo_escoberg>
Well, that explains why nobody answered me :). Very strange; it shows up in my client. IAC, aay as well continue this discussion here:
<pablo_escoberg>
GM. Please ping me when you get the time to continue this discussion (not just
<pablo_escoberg>
abu
<pablo_escoberg>
; anyone who wants to chime in). My concern here is that I need to maintain different files with different block sizes for different classes of objects, combining classes of similar sizes into appropriately sized files. Hence, when the size of a class changes, I may want/need to change the file it's in and may need to rebuild it. Seems like a
<pablo_escoberg>
nightmare, but obviously you guys don't find it so.
<pablo_escoberg>
Also, there's the issue with short field names. I tend to make iterate over field names and translate them into human readable labels ('first_name -> "First Name"). And whereas I'm not that concerned about disk space.... yikes! I had thought of writing a layer that keeps a hash of field names to integers and just stores the integers, looking up the
<pablo_escoberg>
field names as necessary, but it seems a non-trivial undertaking requiring deep understanding of how the thing works.
<pablo_escoberg>
If I can address both of those, I will probably end up using the pilDB.
<abu[7]>
Ha pablo_escoberg, strange, the message is still not visible here
<abu[7]>
Concerning block sizes: You misunderstand something
<abu[7]>
You can ignore them for now
<abu[7]>
Just stay with a single file with the default block size of 254 bytes
<pablo_escoberg>
OK, that I can do.
<abu[7]>
Only later when you have a really big production DB you define your sizes. It is an optimization
<abu[7]>
And for lelation names, you can make them as long as you like. The take up space though
<abu[7]>
The article is correct, but it is just a matter of performance differences
<pablo_escoberg>
ok, so it's kind of one of the last things you do. That makes sense.
<abu[7]>
exactly
<pablo_escoberg>
But then the change management thing comes in. Say I'm already in production, and I need to make changes; add several relations to a class.
<pablo_escoberg>
Now it's in the wrong DB file.
<abu[7]>
It gets important if you have millions of objects
<abu[7]>
It is never in the wrong file
<pablo_escoberg>
ah, ok. But is it as least reasonably easy to move it to a different file.
<pablo_escoberg>
It's not in the "Wrong" file. It's in a suboptimal file.
<abu[7]>
If you change the 'dbs' later, only *new* objecbns go to the new file
<pablo_escoberg>
But I still have to rebuild the old file to reflect changes in class structure or I have several versions of the same class sitting around.
<abu[7]>
If you really want to move objects later, there is (move!> . +Entity) in @lib/too.l
<abu[7]>
You don't have to rebuild a file
<abu[7]>
only if an index changed, you rebuilt *it*
<abu[7]>
'rebuild' function in @lib/too.l
<pablo_escoberg>
But if I add a relation, say, it will be available in all existing classes?
<abu[7]>
Again, you are worrying too early!!
<abu[7]>
First do some practical experiments
<pablo_escoberg>
if it's just optimization, yes, way too early.
<abu[7]>
and try to understand it
<pablo_escoberg>
will do.
<abu[7]>
Just (pool "mydb")
<pablo_escoberg>
ITMT, work on the sqlite driver is coming along very nicely, so I'll be able to do side by side comparisons as well.
<abu[7]>
it uses a single file with 256-byte blocks
<abu[7]>
cool
<abu[7]>
Here is a big PicoLisp DB, which did an many optims as possible: https://picolisp.com/osm
<abu[7]>
It has 340,261,042 node objects and 55,528,532 way objects
<abu[7]>
Plus several indexes
<abu[7]>
The DB takes up 79 GiB
aw- has quit [Ping timeout: 264 seconds]
<pablo_escoberg>
Impressive, but doesn't really address my concern about change management. What happens when your schema changes dramatically in the next iteration? What I'm used to is having a bunch of `alter table` statements in a file, which I can run on an existing database to make it like another one. I'm sure this is doable in pilDB as well, but my
<pablo_escoberg>
concern is it might be substantially harder.
<abu[7]>
You can change anything any time :)
<pablo_escoberg>
of course! It's turing complete; everything is possible. The question is how hard is it compared to having a sql DB.
<abu[7]>
I cannot help you here
<abu[7]>
Just stop worrying!!!
<pablo_escoberg>
I may just need to reconceptualize change management.
<abu[7]>
You start a new field of worrying each day, before tou really investigated the worries of yesterday ;)
<pablo_escoberg>
Nah, I investigate all worries simulatneously :).
<abu[7]>
haha, yes. Good :)
<pablo_escoberg>
You live entirely in the picolisp world so you may not appreciate just how different it is from everything else.
<abu[7]>
yes, so please ask beneroth
<pablo_escoberg>
ask them about what?
<abu[7]>
"The question is how hard is it compared to having a sql DB."
<pablo_escoberg>
ah, yes, ok. I'll figure that out as well.
<abu[7]>
You are too deep in the wrong thinking
<pablo_escoberg>
yeah, that's almost definitely it.
<abu[7]>
A Pil DB is deeply integrated with the application logig
<abu[7]>
"alter table" is no concept here
<pablo_escoberg>
Yeah, that I get.
<abu[7]>
there are no tables
<pablo_escoberg>
right. it's a network db.
<pablo_escoberg>
VERY cool.
<pablo_escoberg>
but again, requires a different way of thinking about change management.
<abu[7]>
I have one database running since 22 years, changed an infinite number of times over the years
<abu[7]>
survived pil32 -> pil64 -> pil21
<pablo_escoberg>
wow! I didn't realize pilDB was that old.
<abu[7]>
was changed from single-file to multi-file to distributed
<abu[7]>
All that is not of concern
<abu[7]>
it is the application logic
<abu[7]>
So your question is in he wrong directions
<abu[7]>
makes no sense
<pablo_escoberg>
yeah, I get that. but a few things: First, I'm not going to be using the picolisp application logic. It's great for what it does, but it doesn't do what I need, which is a stateless API.
<pablo_escoberg>
so overall, I'm loving picolisp the language. But picolisp the ecosystem doesn't exactly do what I want.
aw- has joined #picolisp
<abu[7]>
I think you misunderstood "application logic" here. It is not the GUI, and not HTTP
<pablo_escoberg>
So what I'm doing is creating another ecosystem around the language.
<abu[7]>
It is the *whole* model
<pablo_escoberg>
the `app` thing, right?
<abu[7]>
no
<abu[7]>
The "logic" means the *program* in general
<abu[7]>
Yuu cannot use a DB withoubn a model
<pablo_escoberg>
you mean the DB schema or a model of the schema?
<abu[7]>
It is "er.l" plus all operations on these entity/relations
<pablo_escoberg>
Yeah, it's like an ORM, but there's no R, so no need for translation.
<abu[7]>
I think "DB schema" is not apbropriate
<abu[7]>
at least I don't use it
<pablo_escoberg>
yeah, may not apply to this kind of DB.
<abu[7]>
It is the Domain Model
<abu[7]>
You also do not need to use entities and relations
<abu[7]>
or OO
<pablo_escoberg>
right, which woks nicely with the whole code is data concept.
<abu[7]>
You must make a model
<pablo_escoberg>
Right, you can just extern symbols. that's also very cool.
<abu[7]>
then see hopn to implement it
<abu[7]>
It may be that b-trees are enough
<abu[7]>
yes, or even that
<abu[7]>
For complicated models I find E/R easier
<abu[7]>
especially easier to extend
<abu[7]>
i.e. inheritance and polymorphism
<abu[7]>
but that's a matter of taste
<pablo_escoberg>
well, I guess the upshot is I'll definitely learn the pilDB and probably use it for some things. But it will be nice to have the sqlite thing as well, if only to compare.
<pablo_escoberg>
Does the object system have a mechanism like Ruby's `method_missing`, specifically a message that gets called when a class/object is sent a message that is not predefined?
<abu[7]>
You mean something for *all* classes?
<pablo_escoberg>
no, something for each individual class and the classes that inherit from it.
<abu[7]>
Well, then you can simply define it in the top superclass, no?
<pablo_escoberg>
so in Ruby, you can define a `method_missing` method for each class. When an object is called with an undefined method, `method_missing` is called with the name of the method called and all it's arguments.
<pablo_escoberg>
If it doesn't exist, I'd have to add it to the definition `+Class` I guess...
<abu[7]>
There is no such mechanism in Pil. You could catch the error though
<abu[7]>
"Bad message"
<pablo_escoberg>
that's probably the way.
<abu[7]>
BTW, "an object is called with an undefined method", in Pil the terminology is the other way round
<abu[7]>
"a message is sent to an object"
<abu[7]>
i.e. an object is never "called"
<abu[7]>
only a function or method is called
<pablo_escoberg>
it's the same thing in the end. But yes, technically it's better to think of it that way because it maps nicely to the syntax.
<abu[7]>
Yes, syntax. But also semantically, an object is not executable and thus cannot be called
<abu[7]>
(as I see it)
<abu[7]>
Only code is called
<abu[7]>
Though perhap "call" is not right either. Code is "evaluated"
<pablo_escoberg>
yup, I'll make an effort to use the right terminology. It really does help.
<abu[7]>
Some terms are not well defined yet
<pablo_escoberg>
indeed. So now, of course, I'm having a hard time with error handling. I need to catch the "Bad message" error along with the message itself and run some kind of handler. I can't figure out how to do that with `catch`.
<pablo_escoberg>
it seems to just print the error I'm catching and doesn't let me specify a handler.
<abu[7]>
Either NIL or the message is returned from 'catch'
<pablo_escoberg>
hmmm... that was my first instinct. glad to know it was right. Guess I just typed it in wrong.
<abu[7]>
I think redefining 'send' does not help
<abu[7]>
it is not called explicitly
<abu[7]>
(let send '((...) ...) ... ?
<abu[7]>
This redefines it, but 'url>' is not influenced by that
<abu[7]>
Why do you send illegal messages?
<abu[7]>
If this is possibly, look at 'try'
<abu[7]>
(try 'url> 'Obj)
<pablo_escoberg>
It's the reverse of the dispatch pattern. So, for example, if I have an object representing a row in a table, I want to be able to send it a column name and have it look in the table for the value.
<pablo_escoberg>
not quite working yet, but that's the gist.
<abu[7]>
well, as I said, 'send' is not called
<abu[7]>
usually
<abu[7]>
and if, then you can directly call 'try', no?
<pablo_escoberg>
how else are class/object methods executed apart from `send`?
<abu[7]>
tankf33der: (symbols 'pico 'simul) is not rigxt
<abu[7]>
The lib should override pico symbols
<abu[7]>
In fact, I would do (symbols 'ap 'simul 'pico)
<abu[7]>
si the simul namespace is not clobbered
<abu[7]>
pablo_escoberg, 'send' is used only in *very* special cases
<abu[7]>
I think I have no case anywhere ;)
<pablo_escoberg>
wow. how do you execute methods, then? I thought that was the only way...
<abu[7]>
It is there for completeness, to allow possible dynamic manipulations
<pablo_escoberg>
yes, very useful for dynamic dispatch.
<abu[7]>
haha, no.
<abu[7]>
(dm foo> (...) ...)
<abu[7]>
(foo> Obj)
<abu[7]>
Did you not look at OOP examples for Pil?
<pablo_escoberg>
I did. I guess not closely enough. I guess my brain just put the `send` there because I was expecting it. But that is definitely a more succinct syntax.
<abu[7]>
tankf33der, besides the symbols call it looks good
<pablo_escoberg>
So I guess I'm back to the catch thing....
<abu[7]>
In fact, the very first versions of Pil did only have 'send', so you are on the historically right way
<abu[7]>
Catch is the safest, but ugliest
<abu[7]>
Why do you send illegal messages?
<pablo_escoberg>
To build the methods dynamically, primarily from within the database.
<abu[7]>
I would try to avoid errors *before* they happen
<pablo_escoberg>
well, this isn't an error
<pablo_escoberg>
it's intended behavior
<abu[7]>
ok
<pablo_escoberg>
I think it's called the active record pattern.
<abu[7]>
But after you catch the error, what to do? Go into another error?
alexshendi has quit [Read error: Connection reset by peer]
alexshendi has joined #picolisp
<pablo_escoberg>
No, send the missing method name and it's arguments to a single method that knows what to do with random method calls.
<pablo_escoberg>
So it lets me build the class dynamically, at runtime.
<abu[7]>
Sounds weird
<abu[7]>
You can always build
<abu[7]>
but it should be correct
<abu[7]>
e.g. check
<pablo_escoberg>
between `method_missing` and `send` Ruby can do some real magic.
<abu[7]>
check a whitelist
<pablo_escoberg>
well, the column list is the white list.
<abu[7]>
I do not see this is useful
<pablo_escoberg>
ok, let me describe
<pablo_escoberg>
i am defining a class to access a table. instances of the class represent rows.
<abu[7]>
good
<pablo_escoberg>
If the column definitions change, I don't want to have to reflect that in the picolisp code.
<pablo_escoberg>
So any time a method tries to access a value from an object with a column name, the method will check if such a column name exists and if it does, return the value (if not, throw an error).
<pablo_escoberg>
It's mindbogglingly useful once you start using it.
<pablo_escoberg>
but then again, you don't use `send` so I guess you work differently.
<abu[7]>
I'm sure it is useless in Pil
<pablo_escoberg>
Interesting. How would you do something like this instead?
<abu[7]>
There are many ways a column definition changes
<pablo_escoberg>
exactly. and I want to change it in the DB without changing the picolisp code.
<abu[7]>
E9. you change a +Ref to an +IdxFold
<pablo_escoberg>
and i want the same code to handle it.
<abu[7]>
There is no method involved
<abu[7]>
or you add an attribute
<abu[7]>
or remove one
<abu[7]>
There is no method that changs
<pablo_escoberg>
I guess I can build the methods dynamically at read time.
<pablo_escoberg>
but then if I change the DB structure elsewhere, the class no longer as the correct structrue.
<pablo_escoberg>
structure*
<abu[7]>
(with Obj (: foo bar blub))
<abu[7]>
You are over-abstracting before you even grokked the use cases
<abu[7]>
But your worry about model changes in tfne wrong place
<pablo_escoberg>
IOW the code needs to handle stuff that may change without knowing about it to begin with.
<abu[7]>
In pil you dont insert data this way
<abu[7]>
You manipulate Lisp symbols
<pablo_escoberg>
right, the piDB is very tightly integrated
<abu[7]>
Never as a whole
<pablo_escoberg>
so you would never need this kind of thing.
<abu[7]>
And where do you need it?
<abu[7]>
Show me a simple case
<pablo_escoberg>
but for external structures (DB or otherwise) over which the code has little direct control, it's a different thing.
<pablo_escoberg>
simple case:
<pablo_escoberg>
Table currently has 3 columns: id, name, birthday
<abu[7]>
ok
<pablo_escoberg>
They are defined ONLY in the DB
<pablo_escoberg>
I now add a comment field, and I don't want to change the class definition.
<abu[7]>
In Pil, there is nothing defined *in* the DB
<pablo_escoberg>
right, this is for external db's. not the pilDB
<abu[7]>
ah, ok
<pablo_escoberg>
could be useful for parsing JSON as well.
<pablo_escoberg>
you can build an object dynamically that reflects the structure.
<abu[7]>
But for a Pil class nobnhing is needed
<abu[7]>
You just do (put!> Obj 'comment "Comment)
<pablo_escoberg>
yeah, this is not useful for pilDB access. Only for dealing with structure external to picolisp (JSON, external DB, etc).
<abu[7]>
after adding (rel comment (+String))
<abu[7]>
No methods change
<abu[7]>
So you want to model these external structures
<abu[7]>
I seee
<pablo_escoberg>
ok, great.
<pablo_escoberg>
Just trying to dynamically create classes to reflect external structures. method_missing makes things much easier for that.
<abu[7]>
OK, but as I said, I would check *before* the error
<abu[7]>
You can do that with 'method'
<pablo_escoberg>
oops, have to run. I'll play with the `catch` thing probably tomorrow and hopefully figure it out. If not, I'll figure something out, possibly without objects. talk soon.
<abu[7]>
(if (method 'foo> Ob:) ... or so
<abu[7]>
OK, good night for now :)
<beneroth>
oh my
<abu[7]>
Hi beneroth :)
<beneroth>
hey abu[7]
<beneroth>
pablo_escoberg, every database has blocksizes. the sqlite-blocksize is 4KB.
<beneroth>
don't worry about it until you would worry about it while using SQLite or any other database.
<abu[7]>
Anyway, I must leave :)
<abu[7]>
Good night to all!
<beneroth>
good night abu[7] :)
<beneroth>
pablo_escoberg, about catching errors... in picolisp we usually don't catch errors but fix them.
<beneroth>
I cannot really answer any of your DB schema/modeling questions because you didn't write any meaningful questions really.
<beneroth>
you should learn first about the basic usage before going into advanced topics which only make sense after working with databases for a while.
<beneroth>
stop thinking magic and focus, try to understand what actually happens.
<beneroth>
bbl
pablo_escoberg has quit [Ping timeout: 246 seconds]
pablo_escoberg has joined #picolisp
<pablo_escoberg>
beneroth I appreciate the advice, but I came to picolisp for the magic. I already have tons in Ruby, and I came here for still more. So far, I have not been disappointed, but it's a lot to take in at once and it's never really clear where to go next. Also, my intuition about what's hard and what isn't is failing me here. Several things I
<pablo_escoberg>
thought would be hard turn out to be quite easy and vice versa, this being a prime example. I assumed that what I was looking for was already there and as it turns out, not only isn't it there, it's pretty hard to implement. So it goes...
<pablo_escoberg>
Another way of putting it: You're right and I'd love to have the basics down before going into advanced topics, but it's not clear to me what is and what is not advanced.
<pablo_escoberg>
Oh, and I'm totally over the whole block thing, TY :).