pablo_escoberg has quit [Quit: Ping timeout (120 seconds)]
beneroth has quit [Read error: Connection reset by peer]
beneroth has joined #picolisp
rob_w has quit [Remote host closed the connection]
rob_w has joined #picolisp
rob_w_ has joined #picolisp
<beneroth>
abu[7], I just greatly improved performance on an import by first building the list with 'push instead of using (put> . +Lst) with single elements, as (put> . +Entity) kinda checks for uniqueness with (has> . +Lst) which becomes slower the larger the list is
<beneroth>
now I do (put> '+Entity 'lst (flip Lst)). I figure one 'flip at the end should be cheaper than 'queue (requiring finding the end of the list on every call)
<beneroth>
I wonder if (make (made Lst) (link Element)) might be better
<abu[7]>
Yes, this may well be. But I don't think the performance depends much on the actual list operations in the gereral case, but on the relation operations triggered by them
<abu[7]>
(indexes etc.)
<abu[7]>
The bottleneck of imports are the indexes
<abu[7]>
That's why we have 'create'
<beneroth>
my case is about (+List +Ref +Link), so your point should apply
<abu[7]>
Many indexes, yes
<beneroth>
but I found quite big performance impact when using (put> . +Entity) to add the elements one-by-one versus first building the list and then adding it (which then calls the indexing)
<beneroth>
lists of a few thousand elements. performance difference of minutes vs. milliseconds
<abu[7]>
Perhaps the problem is long lists in DB objects in general
<abu[7]>
Oh
<abu[7]>
yes, that's not a good idea
<beneroth>
all elements are unique. fresh empty database, though running the same import multiple times does no difference in import duration (I create most elements with (request))
<beneroth>
only a single commit at the end
<beneroth>
<abu[7]> Perhaps the problem is long lists in DB objects in general
<beneroth>
yeah, usually it should be split up so its made into index trees instead of lists
<abu[7]>
I would rather use many small objects
<beneroth>
I do that, but I have one main +Entity representing one import in which I have this lists to track in which imports which objects are used.
<abu[7]>
Is only the import slow?
<abu[7]>
or also runtime?
<beneroth>
what do you mean? so far I only do importing and not much else xD
<abu[7]>
Read accesses
<beneroth>
I have not tested it yet. You fear that read access (separate from global db locking with dbSync, which I don't do here) might get slower from index tree operations caused by (put>) big +Ref +List ?
<beneroth>
as I do only one commit per import, I don't see how that should influence it
<abu[7]>
If only the import (once) is slow, it is not a big issue
<abu[7]>
If there is only one commit, you must take care that you have enough RAM
<beneroth>
the import is planned to run multiple times a day. most times they will not make big changes, most of the imported data gets handled with (request ...), but I would like to keep the Import object to gather all stuff it imported
<beneroth>
I want the big import object instead of having references to import in the value objects because I want them unchanged in all imports unless import really brings new values
<beneroth>
yes, I have enough RAM :)
<abu[7]>
:)
<beneroth>
<beneroth> but I found quite big performance impact when using (put> . +Entity) to add the elements one-by-one versus first building the list and then adding it (which then calls the indexing)
<beneroth>
<beneroth> lists of a few thousand elements. performance difference of minutes vs. milliseconds
<beneroth>
it suprised me to have such a huge impact
<beneroth>
but it makes sense, because (put> . +Entity) calls (has> . +List) which does a 'find, so it runs slower and slower the bigger the list gets
<beneroth>
must be that, I think
<abu[7]>
In fact, the GUI also always puts full lists (from +Chart)
<beneroth>
oh! that would explain why it's not a more prominent issue then.
<abu[7]>
In the GUI it is not time critical
<beneroth>
yeah
<beneroth>
I'm right in assuming that 'push should be the quickest compared to 'queue and (make (made ..)) ?
<beneroth>
of course (make) without initialising is the fastest, but I need to handle multiple lists at the same time
<abu[7]>
All truc
<abu[7]>
e
<beneroth>
thanks
<abu[7]>
queue is inefficient
<beneroth>
yeah, so I do now 'push and 'flip to get back the original order
<abu[7]>
Good
<abu[7]>
Perhaps 'made' plus 'link' is also so fast
<beneroth>
if multiple links are to be done, then yes.. but made still has to find the end of the list once same as queue, no?
<abu[7]>
Hmm, but 'made' also traverses I think
<abu[7]>
T
<beneroth>
I'd think 'made plus 'link is better than 'queue when doing multiple inserts, but 'push is more universal, maybe even better when doing multiple inserts
<abu[7]>
T, and flip is fast
<beneroth>
yeah it's really fast
<beneroth>
microseconds
<abu[7]>
(but in the same order as queue
<abu[7]>
)
<beneroth>
yeah, but only one flip at the end is faster than many calls to queue :)
<abu[7]>
Yep
bjorkint0sh has quit [Quit: "Every day, computers are making people easier to use." David Temkin]
<beneroth>
haha, nice quit message by bjorkint0sh
<abu[7]>
'flip' is destructive, so one should knowhat one is doing ;)
<beneroth>
T ;)
<abu[7]>
I had to look up in the log, my client does not show the message
<beneroth>
yeah, sometimes its better to filter out all the leaving and rejoining
bjorkintosh has joined #picolisp
bjorkintosh has joined #picolisp
rob_w has quit [Remote host closed the connection]