<Regenaxer>
beneroth, maybe interesting for you: @lib/too.l now has a 'move!>' method for '+Entity'
<Regenaxer>
Lets you move an object from one db-file to another
<Regenaxer>
Takes care of fixing all references (I hope ;)
<beneroth>
wooooot
<beneroth>
that I need to check out
<Regenaxer>
:)
<beneroth>
how do you find all references, scanning all the files?
<Regenaxer>
yes
<beneroth>
not to expensive on huge databases?
<Regenaxer>
takes a few seconds per object
<beneroth>
okay
<Regenaxer>
depending in size of course
<Regenaxer>
It knows about standar
<Regenaxer>
d relations
<beneroth>
I hope I could ease this with more metadata bookkeeping, but haven't implemented it really so far. I consider this the hardest of all schema changes in pilDb.
<Regenaxer>
Perhaps you need to adjust fer your exrensions
<Regenaxer>
exactly
<beneroth>
I'll look into it, thanks.
<Regenaxer>
I think other relatiin types are not critical
<Regenaxer>
it basically knows about blobs and swaps
<beneroth>
my teammate and I think we found a small issue with +Swap
<Regenaxer>
Nothing special with links and joinys
<Regenaxer>
oh, really?
<beneroth>
T. just annoying to find them.
<Regenaxer>
well, +Swap is a bit limited
<beneroth>
yeah it's natural how it came to be, but we consider it leaking storage memory
<Regenaxer>
+Joint in a +Swap won't work iirc
<beneroth>
(put> 'obj 'rel NIL) on a +Swap is changing the value of the +Swap external symbol to NIL, it's not setting the property to NIL
<Regenaxer>
ah
<Regenaxer>
ok
<Regenaxer>
the ext obj remains
<beneroth>
thereby you "leak" / keep external symbols around when they're not needed.
<beneroth>
yeah
<beneroth>
you agree?
<Regenaxer>
I know, but this is intended
<Regenaxer>
you could zap it
<Regenaxer>
But usually it may be needed again
<beneroth>
ah I see
<beneroth>
but...
<Regenaxer>
In upd> a zap
<beneroth>
it's also not calling (upd>) I believe?
<Regenaxer>
there is no other ref
<beneroth>
not sure, haven't tested
<beneroth>
I have logging / data history which is based on upd>
<Regenaxer>
upd> is called for every change
<Regenaxer>
yes, me too
<Regenaxer>
+Hist prefix
<beneroth>
ok, I haven' tested what the issue is there. Maybe my upd> just doesn't recognize it correctly, or it is just that it logs NILling when we prefer to log that as deletion... nevermind, I need to check first
<Regenaxer>
But if you zap it, it will be dbgc'd
<beneroth>
yeah same concept
<beneroth>
does zap anything besides setting T property (on external symbol) ?
<Regenaxer>
if the new value is NIL and the value is an ext?, it can be zapped
<Regenaxer>
no, zap completely removes
<Regenaxer>
no properties
<Regenaxer>
Normally only dbgc calls zap
<Regenaxer>
so for a multi-user system, you need delayed zap
<beneroth>
ah ok. so A) lose> removes the object from index trees, but does not change any properties besides setting T property. B) zap is removing all properties and setting the T property. correct?
<Regenaxer>
'zap_' I meant
<beneroth>
ah
<Regenaxer>
yes
<beneroth>
another new thing, zap_
<beneroth>
I see, nice
<beneroth>
I see you want to lure me into switching more quickly :D
<beneroth>
good!
<Regenaxer>
:)
<beneroth>
I guess all this is only in pil21? :P
<Regenaxer>
But as I said, I think it is not worth to zap
<beneroth>
though I would guess the move> would be compatible without any changes
<Regenaxer>
yes, pil21
<Regenaxer>
could be used
<beneroth>
yeah I grok that argument
<Regenaxer>
copied
<beneroth>
kk
<Regenaxer>
creating objects is expensive
<Regenaxer>
free list etc
<beneroth>
it's allocation
<Regenaxer>
yep
<beneroth>
allocation is always the most expensive thing in pil
<Regenaxer>
and even more freeing
<Regenaxer>
cells too
<beneroth>
currently we found developing dump scripts the most exhausting...
<Regenaxer>
'dump' from too.l ?
<beneroth>
primarily intended for move> action, not big changes
<Regenaxer>
ah, I see
<beneroth>
you know, moving from single database file to multiple once you have good data
<Regenaxer>
yes
<Regenaxer>
I use it to start a new db
<beneroth>
we have many entities, that makes the task a bit involved. I guess you usually work with not so many entities (but in the current case they come from the actual use case, my metalayer entities stuff doens't affect dump really)
<Regenaxer>
But now I also use it in PilBox to save a DB
<Regenaxer>
I once dumped a big DB
<Regenaxer>
ERP system when porting to pil64
<Regenaxer>
so a different block format
<beneroth>
yeah it's something like that.. less ERP more datawarehouse to be specific, but nevertheless. many entangled entities.
<Regenaxer>
right, gets tough
<beneroth>
I will check out move>, thank you! I'm sure I will love it :)
<Regenaxer>
great :)
<Regenaxer>
I needed it for BTG, because production use started long before the program was ready ;)
<beneroth>
maybe I can optimize the scanning part with my metadata bookkeeping, but of course scanning is the safest option when you are not absolutely sure :)
<beneroth>
yeah, always the case...
<beneroth>
well was the right strategy in our case, there were many changes after they started using it
<Regenaxer>
There are related functions 'refObj' and 'disqlaced' in pil21
<beneroth>
because they were not sure of the process before, and changed it some more times
<beneroth>
ok
<Regenaxer>
'displaced' I mean
<beneroth>
thought so.. finding objects which are not in the file which is assigned to the entity?
<Regenaxer>
'refObj' is in @lib/db.l, uses the same algo
<Regenaxer>
yes
<beneroth>
ok, I will get the newest version and have a look.
<beneroth>
not right now though
<Regenaxer>
ah, it is also in too.l
<Regenaxer>
refObj
<Regenaxer>
refObj was the first
<Regenaxer>
I needed for the distributed DBs
<beneroth>
yes, obviously
<Regenaxer>
all these functions look also in extenal dBs
<beneroth>
powerful
<Regenaxer>
T
<Regenaxer>
there is a global *ExtDBs
<Regenaxer>
Normally NIL, so a no-op
<beneroth>
I see
<beneroth>
yeah that approach to distribute database is more involved than the more common "lets make multiple servers look like one single database", but it's more powerful. The common approach kicks ACID transactions and persistence guarantees over board. Which is okay for data where a single record doesn't matter much, but devastating for anything else.
<Regenaxer>
Right, but ext DBs are just read-only here
<Regenaxer>
The comuncation stuff is separate
<Regenaxer>
eg to/from PilBox
<beneroth>
yeah, that is why you can tailor it to the actual use case instead of using an approach with often works but not with 100% reliability
<Regenaxer>
yes
<beneroth>
anyway, thanks for the hint! I take a break now, I'm tired of having to deal with stupid a lot today...
<Regenaxer>
good ;)
<Regenaxer>
I'm listening to a podcast
<razzy>
I find joy in reading chat about something that works.
<Regenaxer>
Good! Let's hope all really works :)
<Regenaxer>
Did you read chats about things that don't work?
<razzy>
I am ussually stressed with problems :). not in pil chat tho :)
<Regenaxer>
yeah
<razzy>
move> seems powerfull. i would just do one move per second in cron and let it run over weeks :]
<Regenaxer>
yes, for example
<Regenaxer>
It could run as a 'task' all day long
<Regenaxer>
one per minute
<Regenaxer>
for a small DB
<Regenaxer>
In my case there are only 74 displaced objects
<razzy>
how long move> take?
<razzy>
i guess under 200ms
<Regenaxer>
In this small DB perhaps 4 secs
<Regenaxer>
:)
<Regenaxer>
It scans all DB files
<razzy>
hmm, expensive :]
<Regenaxer>
and then iterates all properties of each object
<Regenaxer>
I see no faster way
<Regenaxer>
Perhaps by analyzing the model
<Regenaxer>
and take care of non-indexed +Link's
<Regenaxer>
But thats difficult
<Regenaxer>
error-prone
<Regenaxer>
A +Link may be in a list in a +Bag of +Bag's :)
<Regenaxer>
Even links inside a list in a +Swap are handled correctly
<Regenaxer>
(I hope ;)
<razzy>
bigger DB, exponentially longer time :]
<Regenaxer>
right
<Regenaxer>
Unusable for big DBs
<Regenaxer>
Normally displaced objects are only in the early stage of development
<razzy>
when searching, it is locked for read only. :]
<Regenaxer>
You mean dbSync?
<razzy>
i have only my mental model. so i do not know.
<Regenaxer>
yes, it is correct
<razzy>
i guess, you lock database for read only if you check for reations
<razzy>
*relations
<Regenaxer>
Though I'm not sure if it is a good idea to use it in multi-user
<Regenaxer>
It is normal dbSync+commit
<Regenaxer>
Like every transaction
<razzy>
i would still use it for big database :]. i would do a pipe of changes, run it at night, maybe multiple searchs per one read-lock. help it manually only when pipeline overflow year or so :]
<beneroth>
definitely not good idea unless you have some other way to ensure that nobody else might change the symbol you handle between your reading and your writing
<beneroth>
then you would need to dbSync before reading until writing to prevent dirty reads
<Regenaxer>
As razzy said, it is locked
<Regenaxer>
so it stops other users
<beneroth>
yeah
<Regenaxer>
better at night
<Regenaxer>
a pipe is not useful
<beneroth>
T
<beneroth>
afk
<Regenaxer>
Just (mapc 'move!> (displaced))
<Regenaxer>
cu
<Regenaxer>
Or (mapc 'move!> (filter '((X) (isa '+Entity X)) (displaced)))
<Regenaxer>
if there are e.g. +Swap's
<razzy>
pilDB does not allow multiple users to read data at once?
<Regenaxer>
Read is allowed for an unlimited number
<Regenaxer>
also after (dbSync)
<Regenaxer>
only (commit) locks writing too
<razzy>
multiple searchs per one read-lock is wrong. i see.
<Regenaxer>
Which searches?
<Regenaxer>
Parallel calls to move!> ?
<razzy>
search, i mean check for relations
<razzy>
at least not trivial
<Regenaxer>
As move!> does a dbSync, other processes will do so too and thus must wait
<Regenaxer>
As I said, I would not run it when other users are active