Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vernier + profiling enabled causing silent crashes #2575

Open
adam12 opened this issue Mar 6, 2025 · 2 comments
Open

Vernier + profiling enabled causing silent crashes #2575

adam12 opened this issue Mar 6, 2025 · 2 comments

Comments

@adam12
Copy link

adam12 commented Mar 6, 2025

Issue Description

With Vernier installed and configured as the profiler, there is a random segmentation fault encoutered inside Sentry, causing a 500 error user-facing, but no exception ever reported to Sentry. This can cause silent failures where there are no visible signs that the website is flakey other than failed requests.

I had this manifest on gemdocs.org.

From Gemfile.lock

    concurrent-ruby (1.3.5)
    sentry-rails (5.22.4)
      railties (>= 5.0)
      sentry-ruby (~> 5.22.4)
    sentry-ruby (5.22.4)
      bigdecimal
      concurrent-ruby (~> 1.0, >= 1.0.2)
    vernier (1.5.0)
Stacktrace

[9403b-a7c0-4727-b91c-e3518d748965] Started GET "/gems/datadog_api_client/2.23.0/DatadogAPIClient/V2/RUMApplicationCreateRequest.html" for 37.19.207.34 at 2025-02-23 04:27:54 +0000
/app/vendor/bundle/ruby/3.4.0/gems/vernier-1.5.0/lib/vernier/collector.rb:77: [BUG] Segmentation fault at 0x0000000000000000
ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +YJIT +PRISM [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0036 p:---- s:0243 e:000242 CFUNC :finish
c:0035 p:0003 s:0239 e:000238 METHOD /app/vendor/bundle/ruby/3.4.0/gems/vernier-1.5.0/lib/vernier/collector.rb:77
c:0034 p:0016 s:0231 e:000230 METHOD /app/vendor/bundle/ruby/3.4.0/gems/vernier-1.5.0/lib/vernier.rb:56
c:0033 p:0018 s:0226 e:000225 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/vernier/profiler.rb:76
c:0032 p:0043 s:0221 e:000220 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/transaction.rb:260
c:0031 p:0017 s:0211 e:000210 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/rack/capture_exceptions.rb:83
c:0030 p:0102 s:0205 e:000204 BLOCK /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/rack/capture_exceptions.rb:43
c:0029 p:0020 s:0198 e:000197 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/hub.rb:269
c:0028 p:0017 s:0193 e:000192 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry-ruby.rb:420
c:0027 p:0006 s:0188 e:000187 BLOCK /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/rack/capture_exceptions.rb:21
c:0026 p:0009 s:0184 e:000183 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/hub.rb:59
c:0025 p:0017 s:0179 e:000178 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry-ruby.rb:400
c:0024 p:0024 s:0174 e:000173 METHOD /app/vendor/bundle/ruby/3.4.0/gems/sentry-ruby-5.22.4/lib/sentry/rack/capture_exceptions.rb:20
c:0023 p:0007 s:0169 e:000168 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/show_exceptions.rb:32
c:0022 p:0047 s:0160 E:001a38 METHOD /app/vendor/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/rack/logger.rb:41
c:0021 p:0048 s:0147 e:000146 METHOD /app/vendor/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/rack/logger.rb:29
c:0020 p:0028 s:0141 e:000140 METHOD /app/vendor/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/rack/silence_request.rb:28
c:0019 p:0034 s:0136 e:000135 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/remote_ip.rb:96
c:0018 p:0030 s:0130 e:000129 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/request_id.rb:34
c:0017 p:0057 s:0124 e:000123 METHOD /app/vendor/bundle/ruby/3.4.0/gems/rack-3.1.9/lib/rack/method_override.rb:28
c:0016 p:0013 s:0118 e:000117 METHOD /app/vendor/bundle/ruby/3.4.0/gems/rack-3.1.9/lib/rack/runtime.rb:24
c:0015 p:0019 s:0108 e:000107 METHOD /app/vendor/bundle/ruby/3.4.0/gems/activesupport-8.0.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29
c:0014 p:0016 s:0101 e:000100 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/executor.rb:16
c:0013 p:0018 s:0092 e:000091 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/static.rb:27
c:0012 p:0007 s:0087 e:000086 METHOD /app/vendor/bundle/ruby/3.4.0/gems/rack-3.1.9/lib/rack/sendfile.rb:114
c:0011 p:0021 s:0074 e:000073 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/ssl.rb:92
c:0010 p:0043 s:0068 e:000067 METHOD /app/vendor/bundle/ruby/3.4.0/gems/actionpack-8.0.1/lib/action_dispatch/middleware/assume_ssl.rb:24
c:0009 p:0016 s:0063 e:000062 METHOD /app/vendor/bundle/ruby/3.4.0/gems/railties-8.0.1/lib/rails/engine.rb:535
c:0008 p:0017 s:0057 e:000056 METHOD /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/configuration.rb:279
c:0007 p:0007 s:0052 e:000051 BLOCK /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/request.rb:99
c:0006 p:0015 s:0049 e:000048 METHOD /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/thread_pool.rb:390
c:0005 p:0353 s:0044 e:000043 METHOD /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/request.rb:98
c:0004 p:0093 s:0029 e:000028 METHOD /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/server.rb:472
c:0003 p:0005 s:0018 e:000017 BLOCK /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/server.rb:254
c:0002 p:0071 s:0014 e:000013 BLOCK /app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/thread_pool.rb:167 [FINISH]
c:0001 p:---- s:0003 e:000002 DUMMY [FINISH]

-- Ruby level backtrace information ----------------------------------------
/app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/thread_pool.rb:167:in 'block in spawn_thread'
/app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/server.rb:254:in 'block in run'
/app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/server.rb:472:in 'process_client'
/app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/request.rb:98:in 'handle_request'
/app/vendor/bundle/ruby/3.4.0/gems/puma-6.6.0/lib/puma/thread_pool.rb:390:in 'with_force_shutdown'
/pre>

Reproduction Steps

Unknown.

Expected Behavior

Not to crash.

Actual Behavior

Crashes silently.

Ruby Version

3.4.1

SDK Version

5.22.4

Integration and Its Version

Rails (8.0.1)

Sentry Config

Sentry.init do |config|
  config.dsn = ENV["SENTRY_DSN"]

  # get breadcrumbs from logs
  config.breadcrumbs_logger = [:active_support_logger, :http_logger]

  # enable profiling
  # this is relative to traces_sample_rate
  config.profiles_sample_rate = 1.0

  # Use Vernier for profiling
  config.profiler_class = Sentry::Vernier::Profiler

  config.traces_sampler = lambda do |sampling_context|
    # if this is the continuation of a trace, just use that decision (rate controlled by the caller)
    unless sampling_context[:parent_sampled].nil?
      next sampling_context[:parent_sampled]
    end

    0.5
  end
end
@getsantry getsantry bot moved this to Waiting for: Product Owner in GitHub Issues with 👀 3 Mar 6, 2025
@solnic solnic self-assigned this Mar 7, 2025
@solnic
Copy link
Collaborator

solnic commented Mar 7, 2025

Thanks for reporting this Adam 🙇🏻 I'll be looking into this today, but since this is coming from Vernier I'm afraid I won't be able to fix it quickly. One thing I need to check is why segfaults are not being reported for you by Sentry.

@sl0thentr0py
Copy link
Member

we are simply a library that runs within ruby, segfaults crash ruby entirely so there is no way we could catch those. That would require an agent living parallel to the ruby process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

3 participants