<greyltc>
That thing seemingly has nothing to do with its "source" repo since 5 years and 5 releases ago
<greyltc>
25M downloads
jenrzzz has quit [Ping timeout: 260 seconds]
<konsolebox>
greyltc: Well they added two-factor
Doc_X has joined #ruby
<konsolebox>
greyltc: I doubt they can do anything about authors uploading malicious code whether it's intentional or not. Reviewing each upload would have to be manual.
<[0x1eef]>
I think it is a problem without a good solution, and it's best to minimize the number of dependencies you install to a small number. Generally I would treat rubygems.org as untrusted.
<greyltc>
Anybody can upload anything there and say it's anything, right?
<[0x1eef]>
AFAIK it's not that different to distributing a tarball with any contents you want.
<havenwood>
greyltc: In 2022 RubyGems started mandating MFA for popular packages and this year Giddens is funded to dedicate his work to RubyGems security for the first time.
<greyltc>
I've been trying to only use github/gitlab generated tarballs for the software I build
<greyltc>
Or better, clone the repo and build from that
<havenwood>
A "popular" package is more than 180 million downloads.
otisolsen70 has quit [Quit: Leaving]
<havenwood>
greyltc: Often more folk have keys to the repo than to pushing gems.
<havenwood>
greyltc: You can run your own gem server, if you'd like, and manually review any changes for gems you add to it.
<havenwood>
I'd not recommend pointing to an arbitrary GitHub or GitLab repo in the name of security.
<[0x1eef]>
IMO it is a good argument for minimalism :) when the dependency tree is small it's not impossible to audit and keep track of. At a certain point, like in a lot of Rails applications, it is almost impossible and the best you can do is rely on bundle-audit.
<greyltc>
When I end up needing to package these things (for Arch) I try to skip rubygems.org and get the source from wherever it's being developed
<havenwood>
Getting known CVEs is certainly better than nothing!
<greyltc>
But sometimes I find a package like gemojione
<greyltc>
I don't know what to thing
<greyltc>
think*
<havenwood>
greyltc: Just note that MFA may be required for RubyGems deployment and not for the GitHub repo and most the projects I'm on more people have the commit bit than have push credentials to RubYGems.
<greyltc>
like it's being developed in secret and published to rubygems.org or what?
<havenwood>
Some folk (thinking normalperson, but there are plenty) won't use commercial hosting for their FOSS.
<greyltc>
mfa doesn't really address a malicious developer who has control of the gem page
<havenwood>
My point is there are fewer people controlling pushing gems, so it's a tighter circle of trust.
jenrzzz has joined #ruby
<havenwood>
Let's say 10 folk have the commit bit, 2 can push to RubyGems. Is the repo or the published gem a larger attack surface?
<havenwood>
That's all I mean.
<greyltc>
is there closed source software hosted by rubygems.org?
<havenwood>
greyltc: Hosting on RubyGems means you're uploading your source, but projects are free to choose closed source license or even no license at all.
<havenwood>
If by "closed source" you mean without disclosing your source code, "no." If by "closed source" you mean proprietary license or unlicensed, "yes."
<greyltc>
Seems like auditing would become easier if rubygems didn't take source tarballs, but rather a git url and a commit spec
<greyltc>
Then an auditor could actually inspect the source repo the gem's code must have come from
<havenwood>
greyltc: Most gems are open source and there are already tools to show the diff to review and changelog with the existing system.
jenrzzz has quit [Ping timeout: 272 seconds]
<havenwood>
greyltc: Relying on nearly 180,000 URLs is super brittle and doesn't work at the scale we're talking.
<havenwood>
greyltc: There have been over 164 trillion gem downloads from RubyGems.
osc4rpt has quit [Ping timeout: 264 seconds]
<greyltc>
you mean "super brittle" like it would be too hard to expect gem authors to provide working URLs to their gem's source repos?
osc4rpt has joined #ruby
<havenwood>
greyltc: Not only would URLs go down (GitHub, GitLab, etc would throttle and ban for overuse even) but we'd lose critical things like compression, caching, CDNs, etc.
<greyltc>
Rubygems would still cache the gems
<greyltc>
we'd just know where that cached gem actually came from and have a chance of actually being able to audit the source
<havenwood>
greyltc: It already works like that, just isn't mandatorily tied to git.
<havenwood>
greyltc: I used to review every commit of gems we updated. In practice, there's convention to tag releases and have working RubyGems source links.
<greyltc>
where did the gemojione code come from?
<caleb>
I just wish enough popular gems were signed that we could use "-P HighSecurity" and have it mostly work
<havenwood>
greyltc: Where did what about it come from?
<havenwood>
Sorry, I don't quite follow.
<greyltc>
the source. who wrote it? where is it?
<greyltc>
it's not coming from the git repo that the metadata on rubygems.org says its coming from
<greyltc>
seemingly, i mean the homepage link points to a repo that is not what's feeding the gem (seems to have diverged some years ago)
<greyltc>
But if rubygems.org needs to be told where to fetch the source from, then my question is where is it fetching these releases from?
<[0x1eef]>
I think cargo, npmjs, etc will have the same issues. npmjs might be best prepared because they've had real world situations to deal with in the past.
<konsolebox>
greyltc: Authors push gems to rubygems.org
<greyltc>
ah. that's what i originally thought
<KOTP>
Does auditing a gem in production require someone to look at the source of the gem, or just unzip the gem and audit what is literally delivered?
jenrzzz has joined #ruby
<greyltc>
Auditing becomes a lot easier (read actually possible) when you have a coherent revision control repo to look over
<konsolebox>
greyltc: You implying gems versioning isn't coherent?
<KOTP>
Yes, you can match SHA's for example, but what you have in hand is literally what is being used. So the trust level has to be given to "this is the source, and this is what is delivered".
<greyltc>
KOTP I don't want to have to match SHAs. I want rubygems to tell me what commit hash they pulled from repo X and published on rubygems.org as gem release x.y.z
<havenwood>
greyltc: I just took a look, and it seems like that GitHub repo is very much where the code is coming from.
<greyltc>
is it?
<greyltc>
maybe I missed something
<havenwood>
greyltc: Whoever is pushing the gem might not have tagged a few recent versions? But yeah.
jenrzzz has quit [Ping timeout: 255 seconds]
<havenwood>
greyltc: Just cloning the repo, and fetching the gem source, I can see they seem to differ by the last commit.
<havenwood>
> Files gemojione/lib/gemojione/index.rb and gemojione-4.3.3/lib/gemojione/index.rb differ
<havenwood>
The last commit is to that file. I didn't inspect further.
<[0x1eef]>
KOTP: You could take it so seriously that audit meant reviewing the diff of every gem update in your Gemfile, you could verify the contents don't have binary blobs, etc. A malicious rubygem could do a lot on install, and at runtime.
<havenwood>
It's open source, so you could add the missing tags if it's bothering you via PR.
<greyltc>
rubygems.org seems to host releases 4.2.0, 4.3.0, 4.3.1, 4.3.2, 4.3.3 that are not from the gem's source repo
<greyltc>
(on github)
<havenwood>
greyltc: They simply didn't tag the releases.
<havenwood>
greyltc: Like `git tag`.
<havenwood>
It's a housekeeping thing.
<havenwood>
Folk either use a release tool that automatically tags or do it manually as part of the release process.
<havenwood>
If someone does't use a tool and forgets to tag, there's no tag.
<havenwood>
It doesn't mean the gem didn't come from the source.
<havenwood>
You can add the tags now.
<greyltc>
mustache-sinatra is another
<havenwood>
greyltc: This kind of thing happens when someone is manually going through a process and just starts forgetting to tag, or the person pushing changes and they just never bothered to tag.
jenrzzz has joined #ruby
<havenwood>
greyltc: Most people diff the code. I personally like to always tag releases and appreciate when others do.
<havenwood>
greyltc: Even if someone doesn't git tag properly, you can still compare the code.
<greyltc>
diffing between releases is all we can do to audit gems at the moment
<havenwood>
Especially if they were doing tags and stopped, I'd do a PR to add tags I cared about.
<havenwood>
greyltc: Tags can lie so it's always about comparing the source. I still like tags as a nicety. Popular gems often tag uniformly and correctly and they're handy for quickly finding the commit of a version.
<greyltc>
i'm just suggesting auditing would become easier if rubygems.org enforced a link to a revision control repo instead of accepting any source tarball a person uploads to it
<havenwood>
greyltc: That hardcodes to a single way of managing source code, which isn't acceptable.
<greyltc>
who writes code without revision control?
<KOTP>
Yep, an audit is an audit, unzip the gem and audit what is delivered. Looking at source repository is an audit at that location, but if something is not intact, you can not just rely on the audit at the source.
<havenwood>
greyltc: There are many types of revision control. Folk using `git` can tag properly and then those sources look nice on GitHub or GitLab.
<greyltc>
If we could trust that rubygems.org pulled their release code from some place they tell us they did, then an audit of where they told us they pulled it from is a good audit. That would be a nice improvement.
jenrzzz has quit [Ping timeout: 268 seconds]
<havenwood>
greyltc: The whole point is not having to trust these arbitrary federated places.
<havenwood>
greyltc: As someone who has audited gems a ton, it currently works great.
<greyltc>
Nah, i think that's impossible
<havenwood>
greyltc: Can you diff between one gem version and another? Like super simply see the diff?
<havenwood>
Even if you don't want to write a line of code, there are tons of tools that will show you the diff in the PR or where you can find the diff.
<greyltc>
If I decide that I trust github.com and rubygems.org for example, then proper auditing of gems might move from something impossible to something that might enter the realm of possibility
jenrzzz has joined #ruby
<havenwood>
greyltc: It's very possible to audit gems. Companies run their own gem servers and review diffs before merging even. You're not trusting anyone. You're getting one version, another version, and comparing *exactly* what changed between them.
<havenwood>
Which part isn't possible? I'm not following.
<greyltc>
idk. i guess my point is auditing is too hard. and i think there are things that places like rubygems.org and pypi.org could do to make it easier (maybe even at the expense of some dev's workflows)
<greyltc>
security is important and maybe a little bit of the audit burden should be moved upstream
<havenwood>
greyltc: The RubyGems website shows you the diff between gem versions. ¯\_(ツ)_/¯ I guess they could do more.
<havenwood>
"But at what cost?!" - Aaron Patterson
<greyltc>
i dunno. imagine the xz backdoor had made it into ubunutu LTS
<greyltc>
what would the cost of that have been?
<havenwood>
Ruby stopped shipping `.xz` extension in their releases a few years ago.
<havenwood>
I think that's not a RubyGems problem.
<greyltc>
that was an example of a supply chain attack that could have been absolutely massive
<greyltc>
i'm not saying it had anything to do with ruby
<havenwood>
Sure, and RubyGems is dedicating a grant for an expert to work on security for RubyGems this year.
<havenwood>
Ability to diff gem versions isn't the problem. That's easy.
<havenwood>
We could use a team to work on security.
<havenwood>
Nobody is going to read all the gem code. RubyGems already rapidly responds to reports of abuse.
tarel2 has quit [Quit: Client closed]
<greyltc>
Right. It would be cool for a gem auditor to be aided by the added metadata and change granularity that comes with linking gem code with a proper revision control repo
graywolf has joined #ruby
<greyltc>
be that git or svn or whatever rev control the gem author is using
<havenwood>
Just tag your releases?
<havenwood>
PR if someone else forgets.
<havenwood>
You could suggest a git tag be required by Bundler gem Rake workflow, but unsure if that would be accepted.
<havenwood>
I like nice git tagging as much as anybody, but don't really consider it related to security.
<greyltc>
the mustache-sinatra gem for example is not a release tagging issue
<greyltc>
the upstream github repo has been dead for 9 years
<havenwood>
Ok.
<greyltc>
yet the gem got a new release last year
<havenwood>
When things are nominally maintained, folk tend to not necessarily bother with all the steps to fully polish.
<greyltc>
this is a symptom of someone being able to upload whatever source tarball they pleased and call it gem version 2.0.0
<havenwood>
Also of someone just kindly keeping things working while not necessarily caring about nonessentials.
<greyltc>
Yeah. totally
<greyltc>
I'm sure that's the case here
<havenwood>
If you don't like maintenance mode, pick the new, shiny things.
<havenwood>
Bitrot is sad, but happens.
<greyltc>
But this is the sort of disconnect that makes gem security auditing really hard
<havenwood>
Hanami, Bridgetown, Roda and friends are getting more love.
<johnjaye>
what about the case of the man that put a hard drive in a drawer and 10 years later it was fine
<johnjaye>
i heard from someone who heard from someone that happened
<havenwood>
greyltc: That's where I disagree. The security auditing isn't made harder here. Tool show you exactly how the code changed. There's no blocker.
<havenwood>
It's a nicety thing where it doesn't look right on GitHub. As a security auditor, no impact.
<greyltc>
it's easy in this case because the change is minor and obvious. but what if that rubygems.org provided release diff contained 42000 lines of code changed (made over 10k commits)?
<havenwood>
Then you have a lot of code to review?
<havenwood>
How does that change?
<greyltc>
Auditing that via one single diff is much more difficult than having access to the actual 10k commits that the revision control system has
<greyltc>
fo example, from the revision control metadata, we might see that the malicious XZ dev made a commit to the gem
<havenwood>
greyltc: So in your scenario this code is on GitHub, they've just not tagged it, and you are worried about the time it will take you to manually add the version tags before you begin reviewing the 10K commits?
bambanxx has quit [Quit: My MacBook Air has gone to sleep. ZZZzzz…]
<greyltc>
i'm not sure what git tags have to do with this example
<havenwood>
greyltc: A git tag specifies the commit that corresponds to a specific gem version release.
<greyltc>
but, yes. the revision control metadata would make that big audit much easier than one single raw diff
<havenwood>
greyltc: In your scenario, is this code not on GitHub?
<greyltc>
i don't care where it's hosted
<havenwood>
greyltc: If you want to review 42K lines of proprietary code, ask your vendor for the commit history.
<havenwood>
greyltc: If it's on GitHub, properly tagged, what's the problem?
<greyltc>
rubygems.org will tell me the source control URL where they pulled the release from
<greyltc>
that might be github.com
<havenwood>
Gem authors specify where the source can be found, if it can be. What's the problem?
bambanxx has joined #ruby
<havenwood>
If you're merging 42K lines of close source code, what does that have at all to do with RubyGems? That's your problem.
<havenwood>
I just literally can't figure out what you're saying the problem with RubyGems is here.
<[0x1eef]>
I think greyltc's brings a valid point that doesn't have a robust solution. I don't think rubygems.org is well prepared for this type of thing, I think it has neglected security for quite a while and I wouldn't be surprised if something happens in the future.
<havenwood>
[0x1eef]: I'm just not following what type of thing we're talking about. Maybe I need more coffee.
<greyltc>
rubygems could force the gem author to provide the gem's source code release as pointer to a commit in a revision control system, they could then publish the revision control URL and the commit that they used to craft gem ABC version x.y.z
<greyltc>
this would make the life of a security auditor less difficult than just having diffs of release code tarballs to work with
<havenwood>
It's nice when folk use git and tag so that works. RubyGems doesn't enforce it since we'd lose out on a lot of great code.
<greyltc>
the revision control system doesn't have to be git
<havenwood>
greyltc: With the tools I've used auditing code it hasn't been a problem, FWIW. When commits or changelogs are available they're detected and shown.
<havenwood>
The status quo works well from that angle. There are a ton of other RubyGems security issues that need addressing.
<[0x1eef]>
The conversation has become a bit hard to follow, and greyltc appears to have moved onto a potential solution for their original question, which I think is the real problem; we usually trust rubygems.org blindly, we don't know if the source code on github and rubygems.org is a match, we don't know if the gem distributes malicious binary blob, and so on.
<konsolebox>
This is just honestly desperate. Authors can neglect revision data if they want to.
<greyltc>
kinda my point is that I think that since XZ (my opinion is that) the status quo is no longer good enough
<greyltc>
konsolebox: yeah, they could. but maybe that would be a red flag
<greyltc>
[0x1eef]: right, right
<konsolebox>
I mean if they don't upload their code anywhere but in Rubygems they'll find it pointless.
<greyltc>
They will be forced to make a public repo for their gem if they wish it to be graciously hosted and distributed for free by rubygems.org
elcuervo has quit [Ping timeout: 240 seconds]
elcuervo has joined #ruby
TomyWork has quit [Remote host closed the connection]
rhe has joined #ruby
<havenwood>
greyltc: xz was a fairly sophisticated attack, of an insidious kind where trust is legitimately gained and the code is cleverly hidden as plausible test material.
<greyltc>
yeah, it was nuts
<greyltc>
it highlights that the project maintainers themselves might even be malicious
<greyltc>
meaning that 3rd party audits are all the more important
<havenwood>
greyltc: On the other hand, no auditing caught this even though the code was carefully reviewed. It took edge case use with someone profiling background noise to be discovered.
<havenwood>
So not a great win for code auditing.
<greyltc>
i know!
<greyltc>
We need better/more auditing
<greyltc>
what else can we possibly do, besides give up?
<havenwood>
That would suggest maybe we need more pen testers rather than code auditors.
<havenwood>
More folk benchmarking. Silence all the noise!
<greyltc>
yeah i mean it being caught by a benchmark was probably random luck
<havenwood>
greyltc: I think an auditing takeaway could be "random bytes are inherently suspicious" even if there are plausibly good reasons for them to be introduced so they deserve extra scrutiny.
<greyltc>
yeah
<greyltc>
I'm not sure how often binary blobs appear in ruby gem sources
<greyltc>
but there was human readable code that enabled the blobs in the XZ case that could have been caught by careful audit
<greyltc>
it's just such an insanely huge job
<greyltc>
that *anything* we can possibly do to make auditing more effective/easier is worth doing IMOP
jenrzzz has quit [Ping timeout: 264 seconds]
<[0x1eef]>
There's definitely room to add constraints. Last I checked 'bundle install' could replace 'rake' with the 'rake' of a different gem, there's things like that that are unique to Ruby and largely ignored.
bambanxx has quit [Quit: My MacBook Air has gone to sleep. ZZZzzz…]
<[0x1eef]>
I feel like no one takes those issues seriously until they're used in some sort of attack, and then there's a patch. Until then no one seems to care. That's my admittely negative impression of rubygems.
<[0x1eef]>
Or to say it another way, the approach is reactive rather than proactive. That would sum it up.
<greyltc>
yeah, it's not easy to find the resources for anything except reactive security
<havenwood>
[0x1eef]: At least that reports a conflict, with the default being not to overwrite the exectuable.
<havenwood>
Overwrite the executable? [yN] ERROR: Error installing ... "rake" from whatever conflicts with installed executable from rake
<[0x1eef]>
That's fixed then :) Before only the 'gem' command did it.
jenrzzz has joined #ruby
<havenwood>
[0x1eef]: Though rake is a bundled gem and can be uninstalled, so you can still possibly hijack the executable. Makes more sense for hijacking something that's not installed by default but likely might be.
osc4rpt has quit [Ping timeout: 268 seconds]
<havenwood>
[0x1eef]: Like do a `rails`, which would fail if Rails was installed but would be a placeholder attack for if it was used before being installed.
<havenwood>
RubyGems could forbid executable collision with popular gem executables.
osc4rpt has joined #ruby
<havenwood>
I don't know if there are existing collisions amongst the 180K+ downloads crowd.
<konsolebox>
As an aside I wish Rubygems add namespaces already just like in npm. Give me that @konsolebox/xyz.
<konsolebox>
I have a Ruby script for example named as xn that I can't convert to a gem because it already exists.
jenrzzz has quit [Ping timeout: 255 seconds]
<[0x1eef]>
havenwood: I see something a bit different with bundle install writing to .gems/. 'bundle exec rake' works but prints this warning:
<[0x1eef]>
The `rake` executable in the `foogem` gem is being loaded, but it's also present in other gems (rake).
<[0x1eef]>
If you meant to run the executable for another gem, make sure you use a project specific binstub (`bundle binstub <gem_name>`).
<[0x1eef]>
If you plan to use multiple conflicting executables, generate binstubs for them and disambiguate their names.
<[0x1eef]>
Then proceeds to execute the duplicate rake as expected.
<[0x1eef]>
konsolebox: Yeah me too. I'd like namespaced gems as well.