07:30 <frojnd> Hi there.

07:39 <frojnd> Anyone used lexbor parser before? I'm trying to parse this site: https://www.biblija.net/biblija.cgi?Bible=Bible&m=mr+13%3A24-32&id33=1&pos=0&set=3&l=en and get only text inside p class. Problem is that I also get all the text I _don't want_ like `.: a Isa .; Jol .,; .; Rev .; b Isa .; Ezk ..` and `u .: the time is near, ready to begin; or he is near, ready to come.` and so one. This is my

07:39 <frojnd> code so far: bible_parser.cr https://carc.in/#/r/ca8x and get_verse.cr https://carc.in/#/r/ca8y

07:56 <FromGitter> <paulocoghi> Hi, Fronjd ⏎ ⏎ Giving a fast look in your case, it seems that the "text" example may be helpful. https://github.com/kostya/lexbor/blob/master/examples/texts.cr ⏎ It extracts only the text, removing inner tags like links, bold and other text formatting tags. [https://gitter.im/crystal-lang/crystal?at=6194b5c45c91883da790ada1]

07:57 <frojnd> Let me take a look

07:59 <FromGitter> <paulocoghi> Sorry, now I understand your problem better

08:00 <FromGitter> <paulocoghi> You are correctly extracting the text under the desired <table> tag

08:01 <FromGitter> <paulocoghi> I'm looking into it

08:04 <FromGitter> <paulocoghi> I will continue in a separated thread, here

08:04 <FromGitter> <paulocoghi> Now I understand your problem better

08:05 <FromGitter> <paulocoghi> Since the text you want have undesired extra texts inside it, using extra "span" tags

08:06 <frojnd> Yeah problem is those extra span tags and don't know how to ignore them

08:06 <FromGitter> <paulocoghi> One approach would be to separately select the undesired texts, with an extra "span" selector

08:06 <FromGitter> <paulocoghi> and later search and eliminate their occurrences on the main text

08:07 <FromGitter> <paulocoghi> cleaning it

08:08 <FromGitter> <paulocoghi> There is not a CSS selector for the "pure text" inside an element

08:08 <FromGitter> <paulocoghi> As stated here: ⏎ https://stackoverflow.com/questions/7787520/css-class-selector-to-select-text-inside-a-div

08:09 <FromGitter> <paulocoghi> So one possible approach would be to separately select the undesired texts, search and remove them from the main text

08:13 <FromGitter> <paulocoghi> Another approach would be to download the original Bible document used by BibliJa.net, which seems to be provided by The Digital Bible Library

08:13 <FromGitter> <paulocoghi> also from United Bible Societies

08:13 <FromGitter> <paulocoghi> https://thedigitalbiblelibrary.org/

08:14 <FromGitter> <paulocoghi> Or, if this is a non-commercial project, you can use the free API on https://scripture.api.bible/

08:15 <frojnd> Problem is that Slovenian language doesn't have any public API so I'm stuck with biblija.net

08:17 <frojnd> And while I'm at doing it for one language I though I might just support all that are listed (en,si,fr,es,ca,eu)

08:17 <frojnd> Since content is similar if not the same with those spans

08:19 <FromGitter> <paulocoghi> I found the Slovenian version used on Biblija.net on The Digital Bible Library, as well as the other languages

08:19 <FromGitter> <paulocoghi> https://thedigitalbiblelibrary.org/whats-inside/

08:19 <FromGitter> <paulocoghi> I understand this is not your desired approach,

08:19 <FromGitter> <paulocoghi> but it my be easier and more durable

08:20 <frojnd> Hm can't access it

08:20 <frojnd> Ah jsut slow

08:22 <frojnd> Haha searching for that "Download" button ;D

08:24 <FromGitter> <paulocoghi> Haha, it's fine :)

08:24 <FromGitter> <paulocoghi> If my previous suggestions don't help

08:25 <FromGitter> <paulocoghi> I found this list of other Slovenian bibles online

08:25 <frojnd> It's not open lol

08:25 <frojnd> So there is no download

08:26 <FromGitter> <paulocoghi> Maybe one of the alternatives provides a better HTML structure, allowing an easier extraction

08:26 <FromGitter> <paulocoghi> http://worldbibles.org/language_detail/eng/slv/Slovene

08:26 <FromGitter> <paulocoghi> Please inform me if one of them helps you

08:30 <frojnd> No, most of them points to biblija.net so...

08:30 <frojnd> Format is also not acceptable. I'm stuck with biblija.net

11:53 <frojnd> @paulocoghi I've done as you suggested: https://git.sr.ht/~frojnd/bible-parser-cli/commit/master however is there a better way (as in faster way to do it?)

11:54 <frojnd> I'm looping over redundat text and then removing it from main text

11:54 <frojnd> s/redundat/redundant

12:00 <FromGitter> <paulocoghi> Considering the limitations on CSS selectors, by now I believe your approach is the appropriate one (maybe not the fastest, but it works pretty well).

12:01 <straight-shoota> You could also take a look at https://shardbox.org/shards/sanitize

12:02 <straight-shoota> It allows defining a custom transform policy which could take care of your special sanitization need

12:03 <frojnd> straight-shoota: interesting

21:40 <riza> in ruby I can do "string".hex to get a numeric which equates to the string, provided that it's a hexidecimal -- https://ruby-doc.org/core-3.0.2/String.html#method-i-hex

21:40 <riza> is there an equivalent in crystal? looking at .hexbytes, but that returns a Bytes not a numeric of any sort

21:44 <FromGitter> <Blacksmoke16> `pp "0x0a".to_i prefix: true`

21:46 <riza> I was just looking at https://crystal-lang.org/api/1.2.2/String.html#to_big_i%28base%3D10%29%3ABigInt-instance-method

21:48 <riza> though it looks like that might be a new method in 1.2.2 and carc.in isn't updated

21:48 <FromGitter> <Blacksmoke16> pretty sure its been around for a while

21:49 <FromGitter> <Blacksmoke16> is it not working for you?

21:49 <riza> https://carc.in/#/r/cagh

21:49 <riza> maybe I need to take an eye break and get a snack

21:50 <FromGitter> <Blacksmoke16> ah, need to `require "big"`

21:50 <riza> derp

21:51 <riza> thanks