<_whitenotifier-a>
[scopehal] azonenberg labeled issue #494: CSV Import issue with floats (same comma both used as separator and decimal point). - https://git.io/JGrDx
<_whitenotifier-a>
[scopehal] azonenberg labeled issue #494: CSV Import issue with floats (same comma both used as separator and decimal point). - https://git.io/JGrDx
<azonenberg>
Hmmmm
<azonenberg>
Soooo if we upgrade to C++ 17 we get std::from_chars and std::to_chars
<azonenberg>
which lets us do nice clean locale insensitive conversion from strings to floats and such
<azonenberg>
And doing this any other way is going to be *really* ugly
<azonenberg>
So I guess the question now is, what target platforms do not currently support C++ 17? How many current/potential users do we alienate by doing this?
<azonenberg>
right now the only one I know of is debian oldstable which is in LTS mode now
<azonenberg>
GyrosGeier, xzcvczx: thoughts?
Degi_ has joined #scopehal
Degi has quit [Ping timeout: 272 seconds]
Degi_ is now known as Degi
<azonenberg>
We're now sitting on like three different local related tickets that i want to squish all at once
<azonenberg>
with an overhaul of all the file io code
<azonenberg>
we can use printf-style functions that use the current locale for anything screen facing
<azonenberg>
but file facing stuff needs to be well defined fixed format
<_whitenotifier-a>
[scopehal] azonenberg pushed 1 commit to master [+0/-0/±2] https://git.io/JGiZV
<_whitenotifier-a>
[scopehal] azonenberg 3d74746 - Unit class now uses the configured locale for printing numbers for display, keeping the thread's locale at "C" the rest of the time so we always use '.' as the decimal separator for file I/O. Fixes #494.
<_whitenotifier-a>
[scopehal] azonenberg closed issue #494: CSV Import issue with floats (same comma both used as separator and decimal point). - https://git.io/JGrDx
<_whitenotifier-a>
[scopehal-apps] azonenberg pushed 1 commit to master [+0/-0/±5] https://git.io/JGint
<_whitenotifier-a>
[scopehal-apps] azonenberg 6e7f5e1 - LC_NUMERIC is now set to "C" globally, and only changed to the user's requested locale when printing numbers for display. Fixes #301.
<_whitenotifier-a>
[scopehal-apps] azonenberg closed issue #301: Preference file loading fails on locales that use commas as decimal separator - https://git.io/JLtVO
<_whitenotifier-a>
[scopehal] azonenberg pushed 1 commit to master [+0/-0/±1] https://git.io/JGic3
<_whitenotifier-a>
[scopehal] azonenberg e5b5a91 - Unit: default locale should still use user's requested locale for all settings other than numbers
<azonenberg>
So I fixed one thing and broke another, great. This might take me a bit...
<_whitenotifier-a>
[scopehal] azonenberg pushed 1 commit to master [+0/-0/±1] https://git.io/JGilm
<_whitenotifier-a>
[scopehal] azonenberg 5a4d09b - Seems like LC_NUMERIC doesn't work properly and we have to use LC_ALL?
<_whitenotifier-a>
[scopehal-apps] azonenberg pushed 1 commit to master [+0/-0/±2] https://git.io/JGilC
<_whitenotifier-a>
[scopehal-apps] azonenberg d392054 - Updated scopehal. Fixed warning message in PreferenceTree for malformed preferences
<_florent_>
Hi
<_florent_>
thanks azonenberg for the quick look at #494
<_florent_>
I can confirm it fixes the import of the sample values as floats, but it now seems that time values are not imported correctly
<azonenberg>
time should always be in seconds
<azonenberg>
If your input data doesn't have time in seconds, that's your problem
<azonenberg>
_florent_: I'm not sure if this is documented anywhere yet. Going over docs of everything to ensure completeness is on the v0.1 todo list
<azonenberg>
If you have time in seconds and it's still failing to import, that's a separate bug
<_florent_>
azonenberg: The .csv is generated from a python script, so can give the signification I want to the time column, but I just see that by multiplying time by 1e12 (to be sure to fall back >1 integer part), the import works
<azonenberg>
Hmm
<azonenberg>
Can you pastebin a few lines of the file and let me know what locale you're using?
<_florent_>
yes sure
<_florent_>
Here for this one, the import does not work:
<azonenberg>
Yeah that means the sample interval was converted to 0. If you set LANG="C" or en_US.utf-8 do you get the same problem?
<GyrosGeier>
azonenberg, ideally locale is stream specific
<azonenberg>
also can you send a test case file for that issue? or did it happen with one of your existing test cases you sent me?
<azonenberg>
GyrosGeier: Right now I have locale as "C" globally except when explicitly formatting numbers for display
<azonenberg>
Which seems to work fine for the moment
<GyrosGeier>
but that means switching the global locale, no?
<GyrosGeier>
we have that in KiCad, and it sucks
<azonenberg>
We use per thread locale. And yes, I'd like to clean it up more thoroughly in the future but right now, string formatting is not a critical path or even close
<azonenberg>
so the overhead isn't a big deal
<GyrosGeier>
the C++ locale system uses multiple locale objects, and the global locale is used only as the default initializer
<GyrosGeier>
so reading from a stream uses the locale that was in effect when the stream was created
<azonenberg>
Yeah but we use c sscanf/snprintf for most numeric I/O right now
<GyrosGeier>
ow
<azonenberg>
Retooling that might happen eventually, not for v0.1
<_florent_>
azonenberg: it also happens with the first non working CSV I provided
<_florent_>
for now I'm testing with the import functions of GLScopeClient directly as a first step, but that would be nice to have drivers in the long term yes
<azonenberg>
anyway, I cannot reproduce with either french or german locales
<azonenberg>
both of your test CSVs seem to import OK
<_florent_>
for the SDS1104X-E, it's not directly using LiteScope, it's directly storing the samples in DRAM and uploading over UDP
<azonenberg>
We also have support for some Siglent scopes, but the 1000X-E is not on the supported list yet as it uses a different protocol from the 2xxx and 5xxx series
<_florent_>
Now that we figured it out, setting LC_NUMERIC=en_US.utf-8 before using GLScopeClient is fine for me
<azonenberg>
I believe GenTooMan and xzcvczx are working on 1000X-E
<azonenberg>
Well I want to get to the bottom of it
<azonenberg>
But i've gone about as far as i can without being able to reproduce on my end
<azonenberg>
there's probably some other variable i'm missing
<azonenberg>
BTW also, the CSV import is analog only right now. Digital signals will be interpreted as analog signals with values of 0V or 1V at all times
<azonenberg>
Both are blockers for v0.1 so should get done soon
<_florent_>
OK thanks, I was going to use Math.Threshold to convert the analog import to Digital
<azonenberg>
Correct, that's how you'd do it if the input was truly analog
<azonenberg>
But you'll be able to skip that once the CSV importer handles digital values natively
<azonenberg>
VCDs are imported as digital already since VCD is a purely digital format
<_florent_>
OK I see, thanks
dang`r`us has quit [Quit: WeeChat 3.1]
<azonenberg>
And obviously if you were talking directly to the instrument it wouldn't be an issue either
<_florent_>
I was trying to use the UART decoder yesterday, by doing: .CSV import --> Math.Threshold --> Bus.UART decoder but haven't been able to get it working
<_florent_>
I haven't tried to put traces in the code to understand but was just wondering if the decoders were supposed to work on imports (or were just working on continuous waveforms?)
<azonenberg>
Most decodes should work fine with data sampled at arbitrary frequencies
<azonenberg>
the general scopehal architecture is built around that
<azonenberg>
that said, UART is one of the oldest decodes in the library and it's possible it may not handle that well
<azonenberg>
I'm not sure if i've ever tested it on sparsely sampled data
<azonenberg>
If you have a test case I'll definitely look into it
<azonenberg>
Every scopehal waveform is stored in a sparse format with a timestamp and duration attached to each sample
<azonenberg>
Samples must be in chronological order and cannot overlap, but can have gaps between them and irregular lengths
<azonenberg>
There is also a "dense packed" flag that indicates the waveform is uniformly sampled which allows certain optimizations in decoding and rendering
<azonenberg>
but all processing blocks should work correctly on sparse waveforms
<azonenberg>
Anything that fails to do so is a bug
<_florent_>
Thanks, that's also maybe me doing things incorrectly. I'll first do more tests, add traces to understand and will fill an issue if I really think there's something wrong.
<azonenberg>
I'm sure there are things that are not obvious to a new user. The manual is far from complete
<azonenberg>
it probably has 50+ blank pages with just headings on them
<azonenberg>
and a LOT of recently added features have no documentation whatsoever
<azonenberg>
But so far nobody's been willing to put in the time to polish it up
<azonenberg>
I'd say upwards of 80% of the community contributions have been on instrument drivers
<azonenberg>
with a handful on protocol decodes and one person who built built the preferences system then dropped off the map
<azonenberg>
Some of the driver maintainers have documented their drivers
<azonenberg>
but other than that all of the documentation, the GUI, and the application core is me
<azonenberg>
So I have my hands pretty full
<azonenberg>
Not that driver development isn't a valuable contribution, but it'd be nice if we got some people working on other parts of the project too :p
<_florent_>
Sure I can understand, I have the same with LiteX :) You already did quite some work and GlScopeClient is already really pleasant to use
<azonenberg>
And we've had several instrument vendors throwing free hardware at us
<azonenberg>
Which is nice, but they're not providing engineering time to go with it :p
<_florent_>
For now I'll just try to provide feedback as I use it, but could try to help on some features in the future when I'll be more familiar with it
<azonenberg>
Great
<azonenberg>
Down the road, I want support for on-chip logic analyzers with cross triggering to off chip scopes
<azonenberg>
so you can do full system debug
<azonenberg>
We already have multi-scope support however the code that de-skews the trigger so that they sample coherently does cross-correlations on analog values
<azonenberg>
So we're going to have to figure out another sync path for deskewing a digital-only instrument relative to others
<_florent_>
in fact your demos are very useful and provide additional information that would be difficult to understand even if in the doc, that's with one of them for example that I understood how to create a digital signal with Math.Threshold
<azonenberg>
Yeah, I plan to be doing more comprehensive tutorials down the road
<azonenberg>
and probably reshooting a lot of the older ones as I have a better sound setup and the software has greatly evolved
<_whitenotifier-a>
[scopehal] azonenberg pushed 1 commit to master [+0/-0/±2] https://git.io/JGXte
<_whitenotifier-a>
[scopehal] azonenberg 7b07d88 - MockOscilloscope: added import support for lots of I/Q file formats, and some heuristics for guessing the correct format of an unknown file. Fixes #487.