<mohammad1722>
While you might have seen me around before, I'm re-introducing myself as a GSoC applicant for OpenSCAD. teepee suggested it would be a good idea to get to know the community before the final selection.
<mohammad1722>
My proposal is about the OpenSCAD package manager and I'd love to hear your thoughts and answer any questions you might have. You can find all the details about my proposal, including implementation ideas, here: https://drive.google.com/drive/folders/120mTMrRDbM0TAXjP2Zsp3SPYVyL-HYyr
<mohammad1722>
Looking forward to chatting with you all!
<teepee>
hi mohammad1722
<teepee>
as the package manager topic is a bit different from classic "change some code in openscad" it might be good to toss around some ideas how to make this works if the project would be selected
<teepee>
like the normal use case is to just check-in the code into a PR and everyone can just fetch that and try it out
<InPhase>
mohammad1722: I'd have some questions about the dev timeline and how this would work with early testing feedback on MVPs (tying into teepee's comments). But a bigger picture question that I think preceeds this, is how version management will be handled. There are some complicated and nuanced problems with version management. For example, different scad files might require different versions of
<InPhase>
libraries. There's currently no way to handle this with a centralized library file. Dependencies will almost certainly also be version dependent, which you hit upon in part of the proposal with the manifest file, but this ties into notions of install, update, and remove in complicated ways. And there are a wide variety of ways to try to handle that, none of which are addressed in the proposal.
<InPhase>
I worry that a library system that doesn't address version management up front in its design risks grandfathering in a system which is unusable for people in the sorts of practical scenarios that routinely arise out there in every other language's package management system.
<teepee>
Agreed, but I would see even a limited (e.g. Arduino style) management as a step forward
<teepee>
what I've wondered is if it is possible to actually have multiple version in one project
<InPhase>
From the name of the wiki article, it is obvious nobody likes this: https://en.wikipedia.org/wiki/Dependency_hell But a review of the solutions indicates the usability risks of failing to plan for it. :)
<teepee>
we don't have a linker step that would make things collide
<InPhase>
teepee: Well as long as the contexts are kept separate this should be fine.
<InPhase>
Although if a library management system confined it to one for simplicity, that wouldn't be too troubling.
<teepee>
that's what I'm thinking too, so if we manage to get object bases imports (or some other means of namespacing) there is no huge problem with multiple versions
<InPhase>
The bigger problem is that nothing in scad code specifies a version at import (and maybe it shouldn't), but nothing in the program's operation currently specifies a manifest file either at run time.
<InPhase>
Although maybe a convention could be built around identical filename manifest files.
<teepee>
yeah, that's some separate topic on the openscad side, providing that
<InPhase>
It would be okay if a scad file required a manifest file to run right with library management.
<InPhase>
It wouldn't even break backwards compatibility, because you could just manually provide libraries without the manifest library file, or without a version that could process it.
<teepee>
that's pretty much the concensus right now as far as I've seen various languges using package manager
<teepee>
just lets try to find a reasonable name :)
<InPhase>
The proposal however only proposed those for libraries. But it could go for the top-level scripts as well, I think.
<teepee>
but most annoing names is with the cmake stuff
<InPhase>
Yeah, we should avoid directory-based naming like cmake does. This is not the scad way.
<InPhase>
Well...
<InPhase>
Maybe there could also be a special name to specify for larger directory based packages, so the manager knows what the top level is. for installation.
<InPhase>
It's tricky because some libraries are traditionally files, and some are directories of many files.
<teepee>
yeah, that's the problem with adding that so late in the game
<teepee>
but I would hope if we provide maybe 2 most used cases people will add this at some point
<InPhase>
And if you go to dependency walk, for installation, it's unclea what you need to search for.
<InPhase>
s/unclea/unclear/
hyperair has quit [Ping timeout: 260 seconds]
<InPhase>
Like first pass I imagine every open, use, or input might check for a directory level and then a file matched manifest file, silently skipping if these don't exist, and use those to set the active dependency versions in a file-specific context manner?
<InPhase>
But if we supported only 2 paths for dependency walking at installation you could have people installing libraries with directory based specification that have file-based specification embedded that ends up not processed at installation. Which is not a deal breaker. People can learn to not do that.
<InPhase>
mohammad1722: So those are my preliminary thoughts about how to avoid shooting ourselves in the foot. So for some code, you can do everything iteratively and it's all good. But there are certain systems, of which this is one, where what you put out there results in people building up infrastructure around it, and then half-baked parts that are not fully handling requirements can trap you into a
<InPhase>
scenario where things can no longer be extended to meet actual usage needs. We'd need to be explicit about some of these issues to make sure that we don't create a trap where a library system is inadequate but cannot be made better without breaking things.
<InPhase>
mohammad1722: Thinking through those issues is probably best handled with a lot of discussion. No one person is ever going to think of all the diverse ways such a thing could go wrong in the future.
mohammad38 has joined #openscad
mohammad38 has quit [Client Quit]
mohammad1722 has quit [Ping timeout: 250 seconds]
mohammad1722 has joined #openscad
hyperair has joined #openscad
teepee_ has joined #openscad
teepee has quit [Ping timeout: 260 seconds]
teepee_ is now known as teepee
<mohammad1722>
InPhase: "There are some complicated and nuanced problems with version management."
<mohammad1722>
Thanks for raising this important point about version management. It's definitely a critical aspect to consider. You're right, the initial proposal didn't go into enough detail on versioning. I've broken down the explanation into points to make it easier to discuss each aspect individually.
<mohammad1722>
1. Library developers should (or must?) use semantic versioning for their libraries
<mohammad1722>
2. Dependencies in a library's manifest should include a specific version or a version range.
<mohammad1722>
3. Only one version of a library can be installed on the user machine
<mohammad1722>
4. When a developer tries to create a library, the dependency graph of it is examined for loops (Circular dependencies) and conflicts (Depending on different versions of the same library whether directly or indirectly). If any problem exists then this library can't be created)
<mohammad1722>
5. When installing a library, the same conflict issue may arise, one could be trying to install a library that depends on a different version of an already installed library. Think the following scenario: installed libraries are {libA/1.0.0, libB/1.0.0} where libA is a dependency of libB, and the user wishes to install libC/x.x.x which depends on
<mohammad1722>
libA/2.0.0. In this case a helpful error message can be displayed showing that libC conflicts with libB and one of them should be removed to use the other. (IIRC this is how pip handles this type of conflicts)
<mohammad1722>
However, a more interesting point to me that you have mentioned is "different scad files might require different versions of libraries". Is this use-case really present with OpenSCAD users (files within the same library having different versions of dependencies)? If so, how could people keep the different library versions separated in their
<mohammad1722>
projects?
<teepee>
right now that is difficult to do, mostly due to the global namespace
<teepee>
there's some tricks that could make it work for some cases I think, but not just in general
<mohammad1722>
I think if a file has its own set of dependencies, then it should be considered a package
<mohammad1722>
And the same rules would apply to it.
<teepee>
what's the difference between library and package?
<linext_>
why does there need to be a package manager?
<linext_>
aren't people able to find the library and import it
<linext_>
it's a dependency and crutch
<teepee>
a good number of people are not even aware libraries exist and how useful they are
RoyK has quit [Ping timeout: 264 seconds]
<linext_>
if you put the .scad file in a zip with the library and use a relative file path, won't it find the library?
<mohammad1722>
teepee: It is the same thing, I use the terms interchangeably to refer to a (one or more .scad files)
<teepee>
mohammad1722: so files have (local) dependencies + dependencies that are coming in from libraries/packages
<teepee>
linext_: if you package everything yes, but if I upload stuff to some platform like yours or the bambulab thingiy I don't want to include all the libraries
<mohammad1722>
Yes, exactly
<linext_>
i'd say better to package everything
<teepee>
mohammad1722: ok, it sounded like you said every dependency would need to go into a separate package
<teepee>
mohammad1722: while it's pretty common to have one big scad file, specifically as Thingiverse did never support multi-file scripts, it's not the only case
<teepee>
BOSL2 is 13MB, I really don't want to package (e.g. when uploading to github) that with every small script I do that might use it
<linext_>
what if you put the URL as an annotation next to the import? Then do a pop-up to ask whether to download the lib?
<linext_>
possibly using git to check for new versions
<teepee>
why do hacks if we can get a reasonable solution?
<linext_>
yea, it's not for me. maybe for the beginners
<teepee>
also just a URL is not enough as that still has the library in undefined state
<linext_>
i kind of liked arduino better before the package manager
<teepee>
I don't think it will be required, it's still possible to just do everything manually
<mohammad1722>
linext_: While packaging everything (current project + dependencies) in a monolithic package is straightforward and keeps each project self-contained with its own dependencies, doing that manually can quickly become a nightmare for complex projects with many nested dependencies. It also consumes more space for every project that uses the same
<mohammad1722>
dependency.
RoyK has joined #openscad
<mohammad1722>
InPhase: "The proposal however only proposed those for libraries. But it could go for the top-level scripts as well, I think."
<mohammad1722>
What are top-level scripts?
mohammad1722 has quit [Quit: Client closed]
mmu_man has quit [Ping timeout: 264 seconds]
<InPhase>
linext_: When I use one of my libraries, I'm always having to do something like manually type out: use <threads.scad> // https://github.com/rcolyer/threads-scad
<linext_>
yea
<linext_>
that's not that bad
<InPhase>
linext_: And you know what happens? Half the people I share a file with don't want to even try it because it requires obtaining a library, and that seems like too much work to them or they're not sure how to do it.
<linext_>
maybe they just want the STL or 3MF
<InPhase>
That's about half when I'm actually speaking to a person directly. I think the ratios get much worse if it's a file posted online.
<linext_>
i think a better project would be to embed openscad .scad file and libraries into .3MF files
<InPhase>
linext_: Which means people are actively avoiding projects that use libraries, do the the lack of built-in support for this.
<linext_>
yea
<InPhase>
z/do the the/due to the/
<linext_>
in PHP they get around this by allowing librarie includes to be URLs
<linext_>
although it's not used much
<InPhase>
Whereas with OpenSCAD getting so much faster, I think the need and potential of libraries has radically increased.
mmu_man has joined #openscad
<InPhase>
Also with all the new language features around functions, modules, and the object proposal.
<InPhase>
linext_: Yes, urls were considered by me as a proposal. Although it's a bit of a hack. :)
<InPhase>
And even if you do urls, you still need library management infrastructure to manage these.
<InPhase>
Because you want to cache locally, and then you want to manage those caches...
<linext_>
i think a better use of effort would be to build these features into a web app version of openscad
<linext_>
that way it works on mobile phones
<teepee>
argh, no ;-)
<InPhase>
I like the notion of a web app version working, but it's never going to be my primary method, because of it being more terrible. :)
<linext_>
many people won't install openscad
<linext_>
let alone get far enough to install libraries
<teepee>
that's fine in addition
<InPhase>
But I would like web app versions to retain compatibility, for broader usage.
<teepee>
but I want things to work if I'm not online
<linext_>
it's not like you can use openscad if you have a chromebook or android
<InPhase>
But power users should not be shoved onto the web. That's not how power users use. And power users are more likely to be content producers rather than content users.
<teepee>
easy: if anyone cares enough about those systems, help making it work
<linext_>
i think this dependency package manager could be done with very little interaction
<InPhase>
linext_: For a web version, you could imagine it caching for hours instead of months/years, and thus not requiring any management. But most people want stable offline copies of dependencies for permanently reproducible content.
<linext_>
who retains the list of packages?
<linext_>
how does it get updated?
mmu_man has quit [Ping timeout: 264 seconds]
<linext_>
who is allowed to update it?
<linext_>
what if two libraries are have conflicting namespaces?
<InPhase>
That won't happen, because we don't have namespaces!
<InPhase>
Which is a separate problem.
<InPhase>
And the proposal does mention an index registry. However nothing about the proposal actually requires it, since it proposed putting the e.g. github url into the manifest file. So if this were added also to the dependency section, the registry need disappears.
<InPhase>
It's a matter of choice though. A registry can promote some discoverability.
<teepee>
yes, allowing a decentralized use case is nice
<InPhase>
And naming standardization.
<teepee>
search does need some sort of centalization of at least the search index
mohammad1722 has joined #openscad
<linext_>
so the goal is to make openscad easier to use for beginners?
<linext_>
or to make it easier to use for all users?
<InPhase>
Right, the search functionality is the one part that would actually require the registry. Although that doesn't necessarily need to be a thing. Like if you want to find a python library, you hit up google typically.
<InPhase>
and maybe you put pypi or conda into the search bar if you are targetting a particular type of distribution. :)
<linext_>
node.js has a package manager which mostly works but can cause weird errors from over-simplification
<InPhase>
mohammad1722: By "top-level scripts" I just meant the one you actually open in OpenSCAD and try to preview/render.
<InPhase>
mohammad1722: Also, note the channel log in the topic, if you want to catch up on what you missed when you dropped.
mmu_man has joined #openscad
erectus has quit [Remote host closed the connection]
erectus has joined #openscad
<linext_>
maybe write a demo that takes a .SCAD file as input, then downloads the dependencies into the proper folder, then launches openscad
<linext_>
easy to start as a completely separate project
<linext_>
make the .SCAD associate temporarily with the package manager
<linext_>
when i eventually get around to fixing up 3dcustomizer.net maybe i'll add that auto-download libraries as a feature
mohammad1722 has quit [Ping timeout: 250 seconds]
JordanBrown has quit [Read error: Connection reset by peer]
JordanBrown has joined #openscad
ccox has joined #openscad
guso78k has quit [Quit: Client closed]
erectus has quit [Remote host closed the connection]