hexnewbie_ has joined #maemo-leste
hexnewbie has quit [Ping timeout: 264 seconds]
hexnewbie_ is now known as hexnewbie
akossh has quit [Quit: Leaving.]
lamontc has joined #maemo-leste
parazyd has quit [Ping timeout: 255 seconds]
parazyd has joined #maemo-leste
Juest has quit [Read error: Connection reset by peer]
Juest has joined #maemo-leste
lamontc has quit [Quit: Leaving]
joerg has quit [Ping timeout: 255 seconds]
joerg has joined #maemo-leste
fab_ has joined #maemo-leste
<freemangordon> Wizzup: that query results in valid id when called from within miner
<freemangordon> starting to look like race condition
fab_ has quit [Quit: fab_]
ceene has joined #maemo-leste
fab_ has joined #maemo-leste
sch has joined #maemo-leste
ceene has quit [Ping timeout: 264 seconds]
rafael2k_ has joined #maemo-leste
attah_ has joined #maemo-leste
attah has quit [Ping timeout: 255 seconds]
attah_ is now known as attah
<sunshavi> a count before the select?
<Wizzup> freemangordon: what is the id?
<freemangordon> urn:uuid:bb8cbccd-320e-456c-9383-9a23124e4328
<freemangordon> but there is no such id when I select from the shell
<Wizzup> maybe it's some id that is itself temporaraily adds
<Wizzup> I wonder if you can just check if there is a result, and see if the first result is "None"
<freemangordon> or sqlite caches data
<freemangordon> no, first result is urn\
<Wizzup> I wouldn't blame sqlite3 here, but maybe how tracker uses sqlite3
<Wizzup> aha
<Wizzup> got the full query for me?
xmn has quit [Quit: ZZZzzz…]
<freemangordon> 'SELECT ?u { GRAPH <urn:uuid:472ed0cc-40ff-4e37-9c0c-062d78656540> { ?u a nfo:FileDataObject ; tracker:available true ; a ?class . FILTER (?class IN (nfo:Media,nfo:Icon,nfo:Document,nmm:Playlist,nmm:MusicPiece,nfo:Audio,nfo:PlainTextDocument,nmm:Video,nfo:SourceCode,nfo:HtmlDocument,nfo:Video,nfo:SoftwareApplication,nfo:EBook,nmm:Photo,nfo:VectorImage,nfo:TextDocument,nfo:PaginatedTextDocument,nfo:FilesystemImage,nfo:Image,nfo:M
<freemangordon> do you want me to pastebin it?
<Wizzup> I can work with this I think
<Wizzup> so you say from cli this returns nothing?
<Wizzup> oh actually it is cut off
<freemangordon> this returns no result in the shell and uuid in tracker-miner-fs
<freemangordon> yes
<freemangordon> and there is no such uuid in the database
<Wizzup> is there anything other than uuid in the row?
<freemangordon> no, see the query
<freemangordon> ?u a nfo:FileDataObject
<Wizzup> so one possibility is that this is some tmp file
<Wizzup> I've seen tracker-miner-fs find various tmp files in /var/tmp
<Wizzup> no, hang on, that is not right
<Wizzup> I've seen it on dbus-monitor as picked up by some inotify daemon, but not tracker-miner-fs
<Wizzup> but maybe it is some tmp file that it picks up when the program starts up
<freemangordon> Wizzup: i debugged to the pont where sqlite returns result
<freemangordon> *point
<freemangordon> this is *not* tracker caching something or picking up wrong file or whatever
<freemangordon> it id DB returning stalled data
<freemangordon> tracker sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject. FILTER (?u = urn:uuid:bb8cbccd-320e-456c-9383-9a23124e4328)}' returns nothing in the console
<bencoh> sqlite showing stalled data might mean that two processes opened the db
<Wizzup> freemangordon: you mean that you stopped gdb?
<Wizzup> stopped it from gdb?
<freemangordon> bencoh: yes, looks like that
<Wizzup> sqlite3 has a notion of transations, journals, write ahead log, and so oh, so this doesn't have to be a sqlite3 problem
<freemangordon> Wizzup: stooped? no, stepped :)
<freemangordon> and yes, I started tracker-miner-fs from gdb
<Wizzup> like if tracker-miner-fs starts a transaction, crawls fs, adds something, then runs that query before commit, only it will see it
<Wizzup> I mean you know all of these things, but just saying
<freemangordon> look at the select
<Wizzup> right
<freemangordon> it has a condition
<freemangordon> and (i suspect) that gets set by tracker-extract
<Wizzup> tracker-extract doesn't run at the time of this query though
<freemangordon> yes
<freemangordon> and DB as such is correct
<freemangordon> but it seems there is cached data in the miner
<freemangordon> as bencoh said, because we have 2 processes opening the DB
<freemangordon> I see the same issue with mafw-tracker-source
<Wizzup> but only one active process :)
<freemangordon> where sparql queries return some obsolete data
<freemangordon> does not matter
<Wizzup> ok, well, this still does not look like a sqlite3 problem to me at all, but you clearly spent a lot more time on it
<Wizzup> maybe in how gnome folks use it
<freemangordon> so it is either sqlite or how tracker uses it
<freemangordon> right
<freemangordon> but I see nothing special in how they open the db
<Wizzup> still it would be good to know where this row comes from
<freemangordon> result = sqlite3_open_v2 (db_interface->filename, &db_interface->db, mode | SQLITE_OPEN_NOMUTEX, NULL);
<freemangordon> that's look fine to me
<Wizzup> the pragma's they run afterwards matter more
<freemangordon> *that looks
<freemangordon> yeah
<freemangordon> lemme chech what they do
<freemangordon> *check
<Wizzup> is there any way to run tracker-miner-fs in a way where it *prints* what files it finds / wants analysed?
<Wizzup> well, I guess it does write that to the log (Creating new item)
<freemangordon> 'PRAGMA synchronous = OFF;'
<Wizzup> I've seen tracker-extract run a lot of pragma's, but tracker-miner-fs not so much, but I maybe did not see it restart as often
<Wizzup> freemangordon: yes, because they use journal mode and wal, I think that is fine
<freemangordon> you have to G_MESSAGES_DEBUG=all
<Wizzup> do you see tracker-miner-fs add any file before running that select, or not?
<Wizzup> I'm not sure yet how you know the problem isn't just within the tracker-miner-fs transaction/session
<freemangordon> yes, I see adding many files
<freemangordon> or rather it checks db for consistency on startup
<freemangordon> looks fine to me
<Wizzup> also in the odd behaviour where it keeps starting extract?
slep has joined #maemo-leste
<freemangordon> odd behaviour is because it thinks there is something to be extracted
<freemangordon> because the query returns result
<Wizzup> yes, but something has to be written to be db for that to return results
<freemangordon> pragmas
<Wizzup> and I doubt that tracker-extract adds new rows by itself, it should just update them with data, no?
<freemangordon> hmm, maybe you are right
<Wizzup> those are the same pragma's as with tracker-extract at least
<freemangordon> I have a full startup log, lemme see where this uuid comes form
<freemangordon> hmm, maybe I shall delete most of the music, to speed-up the process
<freemangordon> Wizzup: ok, what I think happens is:
<freemangordon> miner crawls all directories and inserts files that are extract subject
<freemangordon> the tracker-extract (or tracker-store) deletes those records once processed
<freemangordon> somehow a stale data for a deleted record remains in the miner sqlite connection
<rafael2k_> hi all! I'm flying to conference today, but on Sunday I'm back and can leave the PPP on and accessible over the VPN for anyone wanting to help with h-d and everything else!
<freemangordon> rafael2k_: thanks, I guess Wizzup will unpack his device finally :)
<rafael2k_> eheheh
<rafael2k_> cool!
<freemangordon> I am kind of reluctant to do development on a device noone has physical access to
<rafael2k_> I'll use openvpn, as I already settled it up, but running to the airport now
<freemangordon> in theory I can have a runaway process that will melt the cores
<rafael2k_> ehehehe
<freemangordon> no, really
<rafael2k_> I always do this when compiling the kernel
<rafael2k_> :P
<freemangordon> hehe
<rafael2k_> and now with 6 cores...
<freemangordon> but you know it will finish at some points
<freemangordon> runaway process will most-probably destroy the device if left running for sevral days
<Wizzup> back
<rafael2k_> btw, libcamera properly list the cameras, and Pinchart will add to libcamera some tuning files for the rk1isp
<rafael2k_> for photo / video, PPP will work well
<Wizzup> freemangordon: I don't know if tracker-extract deletes or update records but this sounds correct
<freemangordon> deletes them
<Wizzup> ok
<freemangordon> tracker sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject}'
<Wizzup> well, tracker-extract doesn't find the record, do they even use the same kind of queries?
<freemangordon> who cares? the same query gives no results from the cli
<freemangordon> execute ^^^ query
<Wizzup> right, it's not in the same transaction
<Wizzup> in any case it sounds like what you said above seems correct, but we still might want to know what this stale row is, where it comes from
<Wizzup> it doesn't seem like tracker-extract is doing anything wrong, is what I mean, it just starts, sees nothing, stops
<freemangordon> it comes from the initial inserts in the miner
<freemangordon> exactly
<sunshavi> But when you open from the cli. Yo do not use sqlite3_open_v2 which is multi-thread. so conn is different
<freemangordon> and miner restarts it because it thinks extract has something to do :)
<freemangordon> sunshavi: so? it is still the same db
<Wizzup> freemangordon: but it seems to be wrong in this belief, so I'm wondering where this stale record comes from
<rafael2k_> * Kieran Bingham will help with the ISP side
<freemangordon> Wizzup: exactly
<Wizzup> rafael2k_: great @ libcamera (sorry caught up in this tracker issue chat :) )
<Wizzup> freemangordon: the uuid doesn't show which one it is in the logs I guess?
<freemangordon> miner does not print uuds
<Wizzup> I'm kind of surprised that the table with the record wouldn't contain the *path* of the file, like how does tracker-extract what to do
<rafael2k_> Wizzup, : )) I wish good luck with it!
<freemangordon> but it does not matter eitehr
<freemangordon> Wizzup: it does
<Wizzup> like, how does tracker extract know what to do with it*
<Wizzup> freemangordon: ok
<freemangordon> it does know, while the record is still there
<Wizzup> can we get the path from tracker-miner-fs when it runs this query?
<freemangordon> sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject; nie:url ?url. FILTER (?u = urn:uuid:14ec2b01-d9ad-432a-a4ce-12474c0f1f97)}'
<freemangordon> was giving me file:///home/user/Xorg.0.log, urn:uuid:14ec2b01-d9ad-432a-a4ce-12474c0f1f97
<freemangordon> before extract did his job
<freemangordon> and none after it did
<freemangordon> that's why I said extract (or store) deletes the records that were put there by the miner
<Wizzup> ok, so this record is there saved in the db before tracker runs, regardless of whether you look from miner-fs or not?
<Wizzup> or is this just -a- record, not the stale one ? (sorry I'm bad at matching uuids :D )
<freemangordon> what is 'tracker' in this context?
<Wizzup> tracker-extract, my bad
<freemangordon> yes, just a record
<Wizzup> ok
<freemangordon> a valid one
<Wizzup> btw, just to double check, when I run:
<Wizzup> $ tracker sparql -q 'SELECT * WHERE { ?u a nfo:FileDataObject}'
<Wizzup> Results: None
<freemangordon> mhm
<Wizzup> I get the above, and that is not a 1 row result, yeah?
<freemangordon> everything is deleted
<Wizzup> ok
<freemangordon> no, this is empty result
<Wizzup> ok
<freemangordon> in the same time miner gets results for teh same (lets assume) query
<freemangordon> so, to me it is sqlite cached something
<freemangordon> give me 2 minutes to debug something
<Wizzup> k, it might be that miner-fs doesn't finish/flush it's work
<freemangordon> oh, still extracting ::(
<Wizzup> before it does a search itself, but even that should work
<freemangordon> hmm, wait
<freemangordon> maybe it is on the opposite
<freemangordon> it has it in its cache, but it is not flushed to the db
<freemangordon> is that possible?
<sunshavi> Yes that could be possible. before doing commit or rollback
<freemangordon> does it read uncommited?
rafael2k_ has quit [Ping timeout: 246 seconds]
<Wizzup> freemangordon: sure it is possible, but then it would flush a stale record to the db, that is also weird
<Wizzup> at least I am still under the assumption that tracker-miner-fs doesn't actually have any work that needs done
<freemangordon> Wizzup: seems sqlite does "read uncommited" if a query is in the same transaction
<Wizzup> yes, that is how it should be
<Wizzup> I don't know if tracker-miner-fs does these things though
<freemangordon> so it is possible that miner does not commit the last transaction
<Wizzup> (just didn't look at the src)
<Wizzup> yup, that would be hella stupid, but it's possible
<freemangordon> so it will see the data, while noone else will
<freemangordon> well, I don;t see any other plausible option
<Wizzup> yes, that would make sense if we can't see it, even it tracker-extract does not run
<freemangordon> either that or sqlite3 is FUBAR :)
<Wizzup> freemangordon: can we change tracker-miner-fs and print the rows it finds?
<Wizzup> before it runs tracker-miner-fs ?
<Wizzup> er
<Wizzup> tracker-extract
<freemangordon> as I said, I already did
<freemangordon> what do you mean "rows"?
<freemangordon> url?
<Wizzup> to get the filename
<Wizzup> yeah
<freemangordon> what data do you need?
<freemangordon> ok, sec
<Wizzup> btw, this is super unrelated, but probably also want to tweak the actual files that tracker looks at, there is the gsettings ignore rules and rules in /usr/share/tracker-miners/extract-rules
<Wizzup> but we probably*
<freemangordon> right
<sunshavi> what tracker has to do with an X log file?
<Wizzup> nevermind the log file, that is a red herring
<Wizzup> sunshavi: GNOME tracker reads all kinds of files in various places, including the Xorg.0.log file in /home/user
<freemangordon> this is a text file
<freemangordon> there is no reason for it to not be indexed
<Wizzup> freemangordon: tracker indexes text files for full text search
<Wizzup> which actually is quite awesome
<Wizzup> so there is a reason for it, but not for OMP
<Wizzup> it also extracts text from PDFs
<sunshavi> but that log file should have nothing interesting. It could be important If You are searching for an ERR perhaps
<Wizzup> sunshavi: again, red herring, just nevermind
<Wizzup> :)
<freemangordon> Wizzup: I didn;t say it is bad
<freemangordon> "there is no reason for it to not be indexed"
<Wizzup> misread :)
<sunshavi> excluding that file does not help?
<Wizzup> sunshavi: *red herring*, it's not related to the problem :D
<freemangordon> sunshavi: this file has nothing to do
<freemangordon> period :)
<sunshavi> perhaps on the miner's codepath a commit or rollback have been forgotten?
<Wizzup> perhaps, we'll find out
<freemangordon> Wizzup: runnung the new miner that will dump all the 'stale' rows
<freemangordon> the fuck
<freemangordon> I deleted most of the music and now it works ok
<freemangordon> (tracker-miner-fs:17937): Tracker-DEBUG: 14:01:49.455: Not starting extractor. Nothing to do.
<freemangordon> ok, lemme add data back
<Wizzup> lol: 06 Dec 2023, 13:03:51: Tracker: Processing file 'file:///usr/share/applications'...
<Wizzup> 06 Dec 2023, 13:04:15: Tracker: Creating new item 'file:///usr/share/applications/hildon-status-menu/ham-notifier.desktop'\
<Wizzup> not sure why but ok
<freemangordon> it indexes applications as well
<freemangordon> I think they disabled that in 3.0
<Wizzup> $ gsettings get org.freedesktop.Tracker.Miner.Files index-recursive-directories
<Wizzup> ['&DESKTOP', '&DOCUMENTS', '&MUSIC', '&PICTURES', '&VIDEOS', '/mnt/sd/']
<Wizzup> it shouldn't I think, but ok
<freemangordon> hmm, looks like I am not hitting the bug with the modified query :(
<freemangordon> or I still don;t have enough music
<freemangordon> Wizzup: Result [0]: urn:uuid:d8630ae1-784b-4112-8e9b-47edd6060d8c, file:///home/user/MyDocs/.sounds/Blind%20Guardian/(1986-1987)%20-%20Lucifer's%20Heritage%20E.P.'s/(1987)%20-%20Battalions%20Of%20Fear/05%20-%20Run%20For%20The%20Night.mp3
<freemangordon> hmm, wait
<freemangordon> this is the first time after restart so it is normal
<Wizzup> mhm
<freemangordon> the second time it still says "(tracker-miner-fs:18865): Tracker-DEBUG: 14:21:25.565: Not starting extractor. Nothing to do."
<freemangordon> lemme add more music
<freemangordon> maybe the changed query made it run properly, dunno
<freemangordon> I added OPTIONAL {?u nie:url ?url} in the select clause
<Wizzup> it might take a bit of time to get into this problem
<freemangordon> looks like, yeah
<Wizzup> I'm also reindexing my files, even though I did the same yesterday but some how it got lost I guess
<freemangordon> will take some time until music gets copied over ssh
akossh has joined #maemo-leste
<Wizzup> freemangordon: hm pvr on my d4 is unhappy, any debug I can get for you?
<freemangordon> elaborate on 'unhappy'
<Wizzup> screen is stuck, nothing happens, dmesg is filled with repeating messages
<freemangordon> I don;t think I can do anything about it
<Wizzup> interestingly it starts with:
<Wizzup> Dec 6 13:07:54 localhost kernel: [130585.629302] PVR_K: User requested SGX debug info
<freemangordon> in the newer blobs they say reset works properly
<freemangordon> no idea what that means
<Wizzup> xorg log says:
<Wizzup> [130585.607] (EE) OMAP(0): ERROR: waitForBlitsCompleteOnDeviceMem: PVR2DQueryBlitsComplete failed with error code: -8 (Blit not complete)
<Wizzup> [130588.704] (EE) OMAP(0): ERROR: sgxCopyNextBatch: PVR2DBlt failed with error code: -2 (Device unavailable)
<Wizzup> [130589.204] (EE) OMAP(0): ERROR: waitForBlitsCompleteOnDeviceMem: PVR2DQueryBlitsComplete failed with error code: -8 (Blit not complete)
<freemangordon> sgx crashed
<freemangordon> this shall be handled in the driver/blobs
<freemangordon> but it is not
<freemangordon> in our version of the blobs that is
<Wizzup> ok
<Wizzup> I'll restart the device when tracker finishes
<Wizzup> brb
<freemangordon> see the commit message https://git.ti.com/cgit/graphics/omap5-sgx-ddk-um-linux/commit/?h=1.17.4948957/mesa/glibc-2.35&id=337832df9662f0842ce4704342c0e42acab22209
<freemangordon> "Hardware recoveries work now,..."
<freemangordon> ok, we I hit the bug again
<freemangordon> Result [0]: urn:uuid:9b88e7a9-30de-4153-8b60-e8b6ea7ed1a3, file:///home/user/MyDocs/.sounds/Hammerfall/Renegade/06%20-%20The%20Way%20Of%20The%20Warrior.mp3
<freemangordon> seems tracker does not like Hammerfall :)
<freemangordon> have to join a mtg, bbl
<Wizzup> freemangordon: looks like my phone is in this state again
<Wizzup> the tracker one
<freemangordon> mine as well
<Wizzup> when I then run tracker daemon -k, it loses -all- the data
<freemangordon> but will continue debuggin when have time
<Wizzup> sigh
xmn has joined #maemo-leste
<freemangordon> Wizzup: seems tracker misuses sqlite API
<freemangordon> they use sqlite3_prepare_v2() and deprecated sqlite3_expired()
<freemangordon> also, no check is made for the result of sqlite3_reset()
<sunshavi> perhaps it is old code never updated
<freemangordon> no, there is a note there
<sunshavi> mmm
akossh has quit [Quit: Leaving.]
<sunshavi> what about checking the res of sqlite3_reset(). Does that helps?
<freemangordon> hehe, that's what I am trying to do now
<freemangordon> waiting for it to be hit
<sunshavi> ok.
fab_ has quit [Quit: fab_]
<freemangordon> hmm, now sqlite3_step does not return an error
<freemangordon> it returns SQLITE_ROW
<sunshavi> according to documentation two alternatives sqlite3_reset | sqlite3_step
<freemangordon> sunshavi: seems to be uncommitted data
<sunshavi> then commit first?
<freemangordon> hmm?
<sunshavi> or rollback (more context would be needed to reach the right alternative)
<freemangordon> where?
<sunshavi> that uncommited data should be flushed? then commit otherwise rollback
<freemangordon> where?
<freemangordon> I know what has to be done
xmn has quit [Ping timeout: 255 seconds]
<sunshavi> at which point in the source code commit is going to be hit?. this data never flushes cos commit never happens. is this a kind of daemon?. Yes it is cos it reuses the connection. could I do a clone of this code to see the specific line we are talking about?
<Wizzup> sunshavi: he's telling you that he knows what you know, but he needs to find the place in the tracker code and find where it is not flushed
<freemangordon> sunshavi: believe me, I know the theory
<sunshavi> ok
xmn has joined #maemo-leste
<freemangordon> the code in question is tracker
<freemangordon> no idea where in it the bug is
<freemangordon> Wizzup: ok, at least one issue - tracker reset -r deletes sqlite db, but mafw never knows about that
<freemangordon> so it keeps the deleted files openend :)
<Wizzup> yes that is a problem that I also experience
<Wizzup> but it's not the cause for our troubles
<freemangordon> one of them being some file called $something-shm
<freemangordon> could be
<freemangordon> but I will try to fix that first
<Wizzup> I had the trouble occur when mafw source tracker was stopped
<freemangordon> yes, I know
<freemangordon> I had it stopped here as well
<freemangordon> but we have multiple issues
<freemangordon> anyway, have to run now, bbl
Danct12 has quit [Quit: ZNC 1.8.2 - https://znc.in]
asriel has quit [Quit: Don't drink the water. They put something in it to make you forget.]
Danct12 has joined #maemo-leste
asriel has joined #maemo-leste
xmn has quit [Ping timeout: 256 seconds]
xmn has joined #maemo-leste
elastic_dog has quit [Ping timeout: 256 seconds]
elastic_dog has joined #maemo-leste
Langoor has quit [Remote host closed the connection]
Langoor has joined #maemo-leste