<freemangordon>
but there is no such id when I select from the shell
<Wizzup>
maybe it's some id that is itself temporaraily adds
<Wizzup>
I wonder if you can just check if there is a result, and see if the first result is "None"
<freemangordon>
or sqlite caches data
<freemangordon>
no, first result is urn\
<Wizzup>
I wouldn't blame sqlite3 here, but maybe how tracker uses sqlite3
<Wizzup>
aha
<Wizzup>
got the full query for me?
xmn has quit [Quit: ZZZzzz…]
<freemangordon>
'SELECT ?u { GRAPH <urn:uuid:472ed0cc-40ff-4e37-9c0c-062d78656540> { ?u a nfo:FileDataObject ; tracker:available true ; a ?class . FILTER (?class IN (nfo:Media,nfo:Icon,nfo:Document,nmm:Playlist,nmm:MusicPiece,nfo:Audio,nfo:PlainTextDocument,nmm:Video,nfo:SourceCode,nfo:HtmlDocument,nfo:Video,nfo:SoftwareApplication,nfo:EBook,nmm:Photo,nfo:VectorImage,nfo:TextDocument,nfo:PaginatedTextDocument,nfo:FilesystemImage,nfo:Image,nfo:M
<freemangordon>
do you want me to pastebin it?
<Wizzup>
I can work with this I think
<Wizzup>
so you say from cli this returns nothing?
<Wizzup>
oh actually it is cut off
<freemangordon>
this returns no result in the shell and uuid in tracker-miner-fs
<freemangordon>
yes
<freemangordon>
and there is no such uuid in the database
<Wizzup>
is there anything other than uuid in the row?
<freemangordon>
no, see the query
<freemangordon>
?u a nfo:FileDataObject
<Wizzup>
so one possibility is that this is some tmp file
<Wizzup>
I've seen tracker-miner-fs find various tmp files in /var/tmp
<Wizzup>
no, hang on, that is not right
<Wizzup>
I've seen it on dbus-monitor as picked up by some inotify daemon, but not tracker-miner-fs
<Wizzup>
but maybe it is some tmp file that it picks up when the program starts up
<freemangordon>
Wizzup: i debugged to the pont where sqlite returns result
<freemangordon>
*point
<freemangordon>
this is *not* tracker caching something or picking up wrong file or whatever
<freemangordon>
it id DB returning stalled data
<freemangordon>
tracker sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject. FILTER (?u = urn:uuid:bb8cbccd-320e-456c-9383-9a23124e4328)}' returns nothing in the console
<bencoh>
sqlite showing stalled data might mean that two processes opened the db
<Wizzup>
freemangordon: you mean that you stopped gdb?
<Wizzup>
stopped it from gdb?
<freemangordon>
bencoh: yes, looks like that
<Wizzup>
sqlite3 has a notion of transations, journals, write ahead log, and so oh, so this doesn't have to be a sqlite3 problem
<freemangordon>
Wizzup: stooped? no, stepped :)
<freemangordon>
and yes, I started tracker-miner-fs from gdb
<Wizzup>
like if tracker-miner-fs starts a transaction, crawls fs, adds something, then runs that query before commit, only it will see it
<Wizzup>
I mean you know all of these things, but just saying
<freemangordon>
look at the select
<Wizzup>
right
<freemangordon>
it has a condition
<freemangordon>
and (i suspect) that gets set by tracker-extract
<Wizzup>
tracker-extract doesn't run at the time of this query though
<freemangordon>
yes
<freemangordon>
and DB as such is correct
<freemangordon>
but it seems there is cached data in the miner
<freemangordon>
as bencoh said, because we have 2 processes opening the DB
<freemangordon>
I see the same issue with mafw-tracker-source
<Wizzup>
but only one active process :)
<freemangordon>
where sparql queries return some obsolete data
<freemangordon>
does not matter
<Wizzup>
ok, well, this still does not look like a sqlite3 problem to me at all, but you clearly spent a lot more time on it
<Wizzup>
maybe in how gnome folks use it
<freemangordon>
so it is either sqlite or how tracker uses it
<freemangordon>
right
<freemangordon>
but I see nothing special in how they open the db
<Wizzup>
still it would be good to know where this row comes from
<freemangordon>
result = sqlite3_open_v2 (db_interface->filename, &db_interface->db, mode | SQLITE_OPEN_NOMUTEX, NULL);
<freemangordon>
that's look fine to me
<Wizzup>
the pragma's they run afterwards matter more
<freemangordon>
*that looks
<freemangordon>
yeah
<freemangordon>
lemme chech what they do
<freemangordon>
*check
<Wizzup>
is there any way to run tracker-miner-fs in a way where it *prints* what files it finds / wants analysed?
<Wizzup>
well, I guess it does write that to the log (Creating new item)
<freemangordon>
'PRAGMA synchronous = OFF;'
<Wizzup>
I've seen tracker-extract run a lot of pragma's, but tracker-miner-fs not so much, but I maybe did not see it restart as often
<Wizzup>
freemangordon: yes, because they use journal mode and wal, I think that is fine
<freemangordon>
you have to G_MESSAGES_DEBUG=all
<Wizzup>
do you see tracker-miner-fs add any file before running that select, or not?
<Wizzup>
I'm not sure yet how you know the problem isn't just within the tracker-miner-fs transaction/session
<freemangordon>
yes, I see adding many files
<freemangordon>
or rather it checks db for consistency on startup
<freemangordon>
looks fine to me
<Wizzup>
also in the odd behaviour where it keeps starting extract?
slep has joined #maemo-leste
<freemangordon>
odd behaviour is because it thinks there is something to be extracted
<freemangordon>
because the query returns result
<Wizzup>
yes, but something has to be written to be db for that to return results
<Wizzup>
and I doubt that tracker-extract adds new rows by itself, it should just update them with data, no?
<freemangordon>
hmm, maybe you are right
<Wizzup>
those are the same pragma's as with tracker-extract at least
<freemangordon>
I have a full startup log, lemme see where this uuid comes form
<freemangordon>
hmm, maybe I shall delete most of the music, to speed-up the process
<freemangordon>
Wizzup: ok, what I think happens is:
<freemangordon>
miner crawls all directories and inserts files that are extract subject
<freemangordon>
the tracker-extract (or tracker-store) deletes those records once processed
<freemangordon>
somehow a stale data for a deleted record remains in the miner sqlite connection
<rafael2k_>
hi all! I'm flying to conference today, but on Sunday I'm back and can leave the PPP on and accessible over the VPN for anyone wanting to help with h-d and everything else!
<freemangordon>
rafael2k_: thanks, I guess Wizzup will unpack his device finally :)
<rafael2k_>
eheheh
<rafael2k_>
cool!
<freemangordon>
I am kind of reluctant to do development on a device noone has physical access to
<rafael2k_>
I'll use openvpn, as I already settled it up, but running to the airport now
<freemangordon>
in theory I can have a runaway process that will melt the cores
<rafael2k_>
ehehehe
<freemangordon>
no, really
<rafael2k_>
I always do this when compiling the kernel
<rafael2k_>
:P
<freemangordon>
hehe
<rafael2k_>
and now with 6 cores...
<freemangordon>
but you know it will finish at some points
<freemangordon>
runaway process will most-probably destroy the device if left running for sevral days
<Wizzup>
back
<rafael2k_>
btw, libcamera properly list the cameras, and Pinchart will add to libcamera some tuning files for the rk1isp
<rafael2k_>
for photo / video, PPP will work well
<Wizzup>
freemangordon: I don't know if tracker-extract deletes or update records but this sounds correct
<freemangordon>
deletes them
<Wizzup>
ok
<freemangordon>
tracker sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject}'
<Wizzup>
well, tracker-extract doesn't find the record, do they even use the same kind of queries?
<freemangordon>
who cares? the same query gives no results from the cli
<freemangordon>
execute ^^^ query
<Wizzup>
right, it's not in the same transaction
<Wizzup>
in any case it sounds like what you said above seems correct, but we still might want to know what this stale row is, where it comes from
<Wizzup>
it doesn't seem like tracker-extract is doing anything wrong, is what I mean, it just starts, sees nothing, stops
<freemangordon>
it comes from the initial inserts in the miner
<freemangordon>
exactly
<sunshavi>
But when you open from the cli. Yo do not use sqlite3_open_v2 which is multi-thread. so conn is different
<freemangordon>
and miner restarts it because it thinks extract has something to do :)
<freemangordon>
sunshavi: so? it is still the same db
<Wizzup>
freemangordon: but it seems to be wrong in this belief, so I'm wondering where this stale record comes from
<rafael2k_>
* Kieran Bingham will help with the ISP side
<freemangordon>
Wizzup: exactly
<Wizzup>
rafael2k_: great @ libcamera (sorry caught up in this tracker issue chat :) )
<Wizzup>
freemangordon: the uuid doesn't show which one it is in the logs I guess?
<freemangordon>
miner does not print uuds
<Wizzup>
I'm kind of surprised that the table with the record wouldn't contain the *path* of the file, like how does tracker-extract what to do
<rafael2k_>
Wizzup, : )) I wish good luck with it!
<freemangordon>
but it does not matter eitehr
<freemangordon>
Wizzup: it does
<Wizzup>
like, how does tracker extract know what to do with it*
<Wizzup>
freemangordon: ok
<freemangordon>
it does know, while the record is still there
<Wizzup>
can we get the path from tracker-miner-fs when it runs this query?
<freemangordon>
sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject; nie:url ?url. FILTER (?u = urn:uuid:14ec2b01-d9ad-432a-a4ce-12474c0f1f97)}'
<freemangordon>
was giving me file:///home/user/Xorg.0.log, urn:uuid:14ec2b01-d9ad-432a-a4ce-12474c0f1f97
<freemangordon>
before extract did his job
<freemangordon>
and none after it did
<freemangordon>
that's why I said extract (or store) deletes the records that were put there by the miner
<Wizzup>
ok, so this record is there saved in the db before tracker runs, regardless of whether you look from miner-fs or not?
<Wizzup>
or is this just -a- record, not the stale one ? (sorry I'm bad at matching uuids :D )
<freemangordon>
what is 'tracker' in this context?
<Wizzup>
tracker-extract, my bad
<freemangordon>
yes, just a record
<Wizzup>
ok
<freemangordon>
a valid one
<Wizzup>
btw, just to double check, when I run:
<Wizzup>
$ tracker sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject}'
<Wizzup>
Results: None
<freemangordon>
mhm
<Wizzup>
I get the above, and that is not a 1 row result, yeah?
<freemangordon>
everything is deleted
<Wizzup>
ok
<freemangordon>
no, this is empty result
<Wizzup>
ok
<freemangordon>
in the same time miner gets results for teh same (lets assume) query
<freemangordon>
so, to me it is sqlite cached something
<freemangordon>
give me 2 minutes to debug something
<Wizzup>
k, it might be that miner-fs doesn't finish/flush it's work
<freemangordon>
oh, still extracting ::(
<Wizzup>
before it does a search itself, but even that should work
<freemangordon>
hmm, wait
<freemangordon>
maybe it is on the opposite
<freemangordon>
it has it in its cache, but it is not flushed to the db
<freemangordon>
is that possible?
<sunshavi>
Yes that could be possible. before doing commit or rollback
<freemangordon>
does it read uncommited?
rafael2k_ has quit [Ping timeout: 246 seconds]
<Wizzup>
freemangordon: sure it is possible, but then it would flush a stale record to the db, that is also weird
<Wizzup>
at least I am still under the assumption that tracker-miner-fs doesn't actually have any work that needs done
<freemangordon>
Wizzup: seems sqlite does "read uncommited" if a query is in the same transaction
<Wizzup>
yes, that is how it should be
<Wizzup>
I don't know if tracker-miner-fs does these things though
<freemangordon>
so it is possible that miner does not commit the last transaction
<Wizzup>
(just didn't look at the src)
<Wizzup>
yup, that would be hella stupid, but it's possible
<freemangordon>
so it will see the data, while noone else will
<freemangordon>
well, I don;t see any other plausible option
<Wizzup>
yes, that would make sense if we can't see it, even it tracker-extract does not run
<freemangordon>
either that or sqlite3 is FUBAR :)
<Wizzup>
freemangordon: can we change tracker-miner-fs and print the rows it finds?
<Wizzup>
before it runs tracker-miner-fs ?
<Wizzup>
er
<Wizzup>
tracker-extract
<freemangordon>
as I said, I already did
<freemangordon>
what do you mean "rows"?
<freemangordon>
url?
<Wizzup>
to get the filename
<Wizzup>
yeah
<freemangordon>
what data do you need?
<freemangordon>
ok, sec
<Wizzup>
btw, this is super unrelated, but probably also want to tweak the actual files that tracker looks at, there is the gsettings ignore rules and rules in /usr/share/tracker-miners/extract-rules
<Wizzup>
but we probably*
<freemangordon>
right
<sunshavi>
what tracker has to do with an X log file?
<Wizzup>
nevermind the log file, that is a red herring
<Wizzup>
sunshavi: GNOME tracker reads all kinds of files in various places, including the Xorg.0.log file in /home/user
<freemangordon>
this is a text file
<freemangordon>
there is no reason for it to not be indexed
<Wizzup>
freemangordon: tracker indexes text files for full text search
<Wizzup>
which actually is quite awesome
<Wizzup>
so there is a reason for it, but not for OMP
<Wizzup>
it also extracts text from PDFs
<sunshavi>
but that log file should have nothing interesting. It could be important If You are searching for an ERR perhaps
<Wizzup>
sunshavi: again, red herring, just nevermind
<Wizzup>
:)
<freemangordon>
Wizzup: I didn;t say it is bad
<freemangordon>
"there is no reason for it to not be indexed"
<Wizzup>
misread :)
<sunshavi>
excluding that file does not help?
<Wizzup>
sunshavi: *red herring*, it's not related to the problem :D
<freemangordon>
sunshavi: this file has nothing to do
<freemangordon>
period :)
<sunshavi>
perhaps on the miner's codepath a commit or rollback have been forgotten?
<Wizzup>
perhaps, we'll find out
<freemangordon>
Wizzup: runnung the new miner that will dump all the 'stale' rows
<freemangordon>
the fuck
<freemangordon>
I deleted most of the music and now it works ok
<freemangordon>
(tracker-miner-fs:17937): Tracker-DEBUG: 14:01:49.455: Not starting extractor. Nothing to do.
<freemangordon>
ok, lemme add data back
<Wizzup>
lol: 06 Dec 2023, 13:03:51: Tracker: Processing file 'file:///usr/share/applications'...
<Wizzup>
06 Dec 2023, 13:04:15: Tracker: Creating new item 'file:///usr/share/applications/hildon-status-menu/ham-notifier.desktop'\
<Wizzup>
not sure why but ok
<freemangordon>
it indexes applications as well
<freemangordon>
I think they disabled that in 3.0
<Wizzup>
$ gsettings get org.freedesktop.Tracker.Miner.Files index-recursive-directories
<freemangordon>
"Hardware recoveries work now,..."
<freemangordon>
ok, we I hit the bug again
<freemangordon>
Result [0]: urn:uuid:9b88e7a9-30de-4153-8b60-e8b6ea7ed1a3, file:///home/user/MyDocs/.sounds/Hammerfall/Renegade/06%20-%20The%20Way%20Of%20The%20Warrior.mp3
<freemangordon>
seems tracker does not like Hammerfall :)
<freemangordon>
have to join a mtg, bbl
<Wizzup>
freemangordon: looks like my phone is in this state again
<Wizzup>
the tracker one
<freemangordon>
mine as well
<Wizzup>
when I then run tracker daemon -k, it loses -all- the data
<freemangordon>
but will continue debuggin when have time
<Wizzup>
sigh
xmn has joined #maemo-leste
<freemangordon>
Wizzup: seems tracker misuses sqlite API
<freemangordon>
they use sqlite3_prepare_v2() and deprecated sqlite3_expired()
<freemangordon>
also, no check is made for the result of sqlite3_reset()
<sunshavi>
perhaps it is old code never updated
<freemangordon>
no, there is a note there
<sunshavi>
mmm
akossh has quit [Quit: Leaving.]
<sunshavi>
what about checking the res of sqlite3_reset(). Does that helps?
<freemangordon>
hehe, that's what I am trying to do now
<freemangordon>
waiting for it to be hit
<sunshavi>
ok.
fab_ has quit [Quit: fab_]
<freemangordon>
hmm, now sqlite3_step does not return an error
<freemangordon>
it returns SQLITE_ROW
<sunshavi>
according to documentation two alternatives sqlite3_reset | sqlite3_step
<freemangordon>
sunshavi: seems to be uncommitted data
<sunshavi>
then commit first?
<freemangordon>
hmm?
<sunshavi>
or rollback (more context would be needed to reach the right alternative)
<freemangordon>
where?
<sunshavi>
that uncommited data should be flushed? then commit otherwise rollback
<freemangordon>
where?
<freemangordon>
I know what has to be done
xmn has quit [Ping timeout: 255 seconds]
<sunshavi>
at which point in the source code commit is going to be hit?. this data never flushes cos commit never happens. is this a kind of daemon?. Yes it is cos it reuses the connection. could I do a clone of this code to see the specific line we are talking about?
<Wizzup>
sunshavi: he's telling you that he knows what you know, but he needs to find the place in the tracker code and find where it is not flushed
<freemangordon>
sunshavi: believe me, I know the theory
<sunshavi>
ok
xmn has joined #maemo-leste
<freemangordon>
the code in question is tracker
<freemangordon>
no idea where in it the bug is
<freemangordon>
Wizzup: ok, at least one issue - tracker reset -r deletes sqlite db, but mafw never knows about that
<freemangordon>
so it keeps the deleted files openend :)
<Wizzup>
yes that is a problem that I also experience
<Wizzup>
but it's not the cause for our troubles
<freemangordon>
one of them being some file called $something-shm
<freemangordon>
could be
<freemangordon>
but I will try to fix that first
<Wizzup>
I had the trouble occur when mafw source tracker was stopped