Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ekump/apmsp 1756 add trace exporter integration tests backup #894

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Ekump/apmsp 1756 add trace exporter integration tests backup #894

wants to merge 7 commits into from

Conversation

ekump
Copy link
Contributor

@ekump ekump commented Feb 24, 2025

DO NOT MERGE

ekump added 7 commits February 5, 2025 16:11
@ekump
APMSP-1779 add span deserialization and Payload collection constructi…
3f257c0 
…on to send data integration tests
@ekump
return the offending key when decoding v04 msgpack spans
ad0e57f 
If the decoder sees a key it doesn't recognize for a span we return a Decode error to the caller. The least we can do is tell them the problematic key. The original error message was getting swallowed when mapping errors.
@ekump
wip, broken
029fc39
@ekump
WIP: refactor send_deser_ser
9f2ca5a
@ekump
wip
bf87e55
@ekump
support query params for communicating with the test agent
530581c
@ekump
wip, broken test
3cf2678
@pr-commenter
Copy link

pr-commenter bot commented Feb 24, 2025 •
edited
Loading

Benchmarks

Comparison

Benchmark execution time: 2025-02-24 18:21:12

Comparing candidate commit 3cf2678 in PR branch ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP with baseline commit e47f5e9 in branch main.

Found 3 performance improvements and 47 performance regressions! Performance is the same for 2 metrics, 2 unstable metrics.

scenario:benching deserializing traces from msgpack to their internal representation

  • 🟥 execution_time [+3.116ms; +3.219ms] or [+5.690%; +5.879%]

scenario:benching string interning on wordpress profile

  • 🟩 execution_time [-12.045µs; -11.926µs] or [-7.990%; -7.911%]

scenario:concentrator/add_spans_to_concentrator

  • 🟥 execution_time [+314.425µs; +317.850µs] or [+5.213%; +5.270%]

scenario:credit_card/is_card_number/

  • 🟥 execution_time [+373.941ns; +374.917ns] or [+9.555%; +9.580%]
  • 🟥 throughput [-22339753.339op/s; -22281907.327op/s] or [-8.743%; -8.720%]

scenario:credit_card/is_card_number/ 3782-8224-6310-005

  • 🟥 execution_time [+7.879µs; +8.138µs] or [+9.542%; +9.856%]
  • 🟥 throughput [-1088395.272op/s; -1053589.374op/s] or [-8.987%; -8.699%]

scenario:credit_card/is_card_number/ 378282246310005

  • 🟥 execution_time [+3.955µs; +4.138µs] or [+4.975%; +5.204%]
  • 🟥 throughput [-623033.441op/s; -595926.005op/s] or [-4.953%; -4.738%]

scenario:credit_card/is_card_number/37828224631

  • 🟥 execution_time [+371.748ns; +374.191ns] or [+9.496%; +9.559%]
  • 🟥 throughput [-22296460.689op/s; -22146171.218op/s] or [-8.728%; -8.669%]

scenario:credit_card/is_card_number/378282246310005

  • 🟥 execution_time [+4.363µs; +4.502µs] or [+5.723%; +5.905%]
  • 🟥 throughput [-732710.139op/s; -709402.864op/s] or [-5.586%; -5.408%]

scenario:credit_card/is_card_number/37828224631000521389798

  • 🟥 execution_time [+7.226µs; +7.238µs] or [+14.050%; +14.073%]
  • 🟥 throughput [-2398864.366op/s; -2394934.782op/s] or [-12.338%; -12.318%]

scenario:credit_card/is_card_number/x371413321323331

  • 🟥 execution_time [+405.251ns; +407.245ns] or [+6.713%; +6.746%]
  • 🟥 throughput [-10472319.369op/s; -10419400.069op/s] or [-6.322%; -6.290%]

scenario:credit_card/is_card_number_no_luhn/

  • 🟥 execution_time [+374.423ns; +375.433ns] or [+9.567%; +9.593%]
  • 🟥 throughput [-22369439.562op/s; -22310538.731op/s] or [-8.754%; -8.731%]

scenario:credit_card/is_card_number_no_luhn/ 3782-8224-6310-005

  • 🟥 execution_time [+3.735µs; +3.844µs] or [+5.645%; +5.810%]
  • 🟥 throughput [-829724.856op/s; -807143.478op/s] or [-5.490%; -5.341%]

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

  • 🟥 execution_time [+4.880µs; +4.993µs] or [+8.172%; +8.363%]
  • 🟥 throughput [-1292043.517op/s; -1264636.568op/s] or [-7.715%; -7.551%]

scenario:credit_card/is_card_number_no_luhn/37828224631

  • 🟥 execution_time [+373.987ns; +375.068ns] or [+9.557%; +9.584%]
  • 🟥 throughput [-22351358.941op/s; -22287631.016op/s] or [-8.747%; -8.722%]

scenario:credit_card/is_card_number_no_luhn/378282246310005

  • 🟥 execution_time [+5.035µs; +5.084µs] or [+8.854%; +8.940%]
  • 🟥 throughput [-1444291.233op/s; -1429474.886op/s] or [-8.213%; -8.129%]

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

  • 🟥 execution_time [+7.228µs; +7.243µs] or [+14.054%; +14.082%]
  • 🟥 throughput [-2400106.333op/s; -2395544.978op/s] or [-12.345%; -12.321%]

scenario:credit_card/is_card_number_no_luhn/x371413321323331

  • 🟥 execution_time [+402.943ns; +405.329ns] or [+6.674%; +6.713%]
  • 🟥 throughput [-10422317.613op/s; -10358414.101op/s] or [-6.293%; -6.254%]

scenario:ip_address/quantize_peer_ip_address_benchmark

  • 🟥 execution_time [+377.975ns; +394.371ns] or [+7.444%; +7.767%]

scenario:normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...

  • 🟩 execution_time [-27.263µs; -26.846µs] or [-13.036%; -12.837%]
  • 🟩 throughput [+704584.524op/s; +717081.550op/s] or [+14.735%; +14.997%]

scenario:normalization/normalize_name/normalize_name/bad-name

  • 🟥 execution_time [+2.836µs; +2.888µs] or [+15.459%; +15.741%]
  • 🟥 throughput [-7421300.566op/s; -7291884.190op/s] or [-13.614%; -13.376%]

scenario:normalization/normalize_name/normalize_name/good

  • 🟥 execution_time [+3.532µs; +3.547µs] or [+32.868%; +33.003%]
  • 🟥 throughput [-23098134.422op/s; -23010770.713op/s] or [-24.822%; -24.728%]

scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...

  • 🟥 execution_time [+195.738µs; +196.034µs] or [+38.701%; +38.759%]
  • 🟥 throughput [-552416.736op/s; -551539.859op/s] or [-27.940%; -27.895%]

scenario:normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて

  • 🟥 execution_time [+20.256µs; +20.385µs] or [+4.466%; +4.495%]
  • 🟥 throughput [-94857.105op/s; -94266.375op/s] or [-4.302%; -4.275%]

scenario:normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters

  • 🟥 execution_time [+14.552µs; +14.644µs] or [+8.238%; +8.290%]
  • 🟥 throughput [-433507.000op/s; -430694.712op/s] or [-7.658%; -7.608%]

scenario:normalization/normalize_service/normalize_service/[empty string]

  • 🟥 execution_time [+8.807µs; +8.831µs] or [+23.400%; +23.463%]
  • 🟥 throughput [-5049980.076op/s; -5037342.457op/s] or [-19.007%; -18.959%]

scenario:normalization/normalize_trace/test_trace

  • 🟥 execution_time [+16.760ns; +21.592ns] or [+6.574%; +8.470%]

scenario:redis/obfuscate_redis_string

  • 🟥 execution_time [+6.736µs; +7.200µs] or [+20.441%; +21.850%]

scenario:sql/obfuscate_sql_string

  • 🟥 execution_time [+2.905µs; +2.987µs] or [+4.343%; +4.466%]

scenario:tags/replace_trace_tags

  • 🟥 execution_time [+279.060ns; +287.455ns] or [+11.397%; +11.739%]

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 39.356µs 39.919µs ± 0.882µs 39.515µs ± 0.054µs 39.601µs 41.781µs 41.819µs 43.111µs 9.10% 1.727 1.226 2.20% 0.062µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [39.797µs; 40.041µs] or [-0.306%; +0.306%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 263.675ns 274.107ns ± 13.706ns 267.594ns ± 2.394ns 273.990ns 304.047ns 318.248ns 320.224ns 19.67% 1.751 2.023 4.99% 0.969ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [272.208ns; 276.007ns] or [-0.693%; +0.693%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 6.328ms 6.347ms ± 0.010ms 6.346ms ± 0.005ms 6.352ms 6.358ms 6.390ms 6.426ms 1.26% 3.851 24.453 0.16% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [6.346ms; 6.349ms] or [-0.022%; +0.022%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 69.603µs 69.836µs ± 0.230µs 69.814µs ± 0.037µs 69.857µs 69.952µs 70.042µs 72.921µs 4.45% 12.146 160.461 0.33% 0.016µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [69.804µs; 69.868µs] or [-0.046%; +0.046%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
ip_address/quantize_peer_ip_address_benchmark execution_time 5.395µs 5.464µs ± 0.038µs 5.460µs ± 0.037µs 5.499µs 5.510µs 5.528µs 5.547µs 1.59% -0.015 -1.443 0.69% 0.003µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
ip_address/quantize_peer_ip_address_benchmark execution_time [5.459µs; 5.469µs] or [-0.096%; +0.096%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 138.156µs 138.768µs ± 0.297µs 138.724µs ± 0.141µs 138.893µs 139.358µs 139.834µs 139.994µs 0.92% 1.321 3.185 0.21% 0.021µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [138.727µs; 138.809µs] or [-0.030%; +0.030%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 179.500µs 182.081µs ± 1.499µs 181.930µs ± 1.090µs 183.112µs 184.598µs 185.769µs 187.712µs 3.18% 0.587 0.167 0.82% 0.106µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 5327312.760op/s 5492431.244op/s ± 45011.008op/s 5496631.322op/s ± 32742.742op/s 5527832.517op/s 5556734.085op/s 5566731.970op/s 5571022.882op/s 1.35% -0.543 0.058 0.82% 3182.759op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 20.912µs 21.207µs ± 0.138µs 21.184µs ± 0.104µs 21.315µs 21.421µs 21.488µs 21.757µs 2.70% 0.515 0.053 0.65% 0.010µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 45961954.992op/s 47156262.891op/s ± 305090.110op/s 47204920.915op/s ± 231213.297op/s 47402065.367op/s 47571619.606op/s 47655974.923op/s 47819474.652op/s 1.30% -0.480 -0.044 0.65% 21573.129op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 14.202µs 14.286µs ± 0.045µs 14.281µs ± 0.031µs 14.314µs 14.362µs 14.408µs 14.413µs 0.93% 0.523 -0.172 0.32% 0.003µs 1 200
normalization/normalize_name/normalize_name/good throughput 69381728.522op/s 69999663.118op/s ± 221839.642op/s 70025336.440op/s ± 154252.646op/s 70167004.728op/s 70328838.568op/s 70387751.004op/s 70412822.815op/s 0.55% -0.508 -0.194 0.32% 15686.432op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [181.873µs; 182.289µs] or [-0.114%; +0.114%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [5486193.152op/s; 5498669.337op/s] or [-0.114%; +0.114%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [21.188µs; 21.226µs] or [-0.090%; +0.090%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [47113980.336op/s; 47198545.446op/s] or [-0.090%; +0.090%] None None None
normalization/normalize_name/normalize_name/good execution_time [14.280µs; 14.292µs] or [-0.044%; +0.044%] None None None
normalization/normalize_name/normalize_name/good throughput [69968918.277op/s; 70030407.958op/s] or [-0.044%; +0.044%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 700.261µs 701.660µs ± 0.805µs 701.622µs ± 0.486µs 702.116µs 702.721µs 703.226µs 708.750µs 1.02% 3.505 28.249 0.11% 0.057µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1410934.591op/s 1425192.810op/s ± 1629.556op/s 1425269.658op/s ± 986.710op/s 1426229.004op/s 1427085.456op/s 1427494.398op/s 1428039.915op/s 0.19% -3.444 27.548 0.11% 115.227op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 472.980µs 473.830µs ± 0.384µs 473.817µs ± 0.260µs 474.077µs 474.483µs 474.814µs 475.013µs 0.25% 0.291 -0.034 0.08% 0.027µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2105205.775op/s 2110461.505op/s ± 1711.673op/s 2110519.228op/s ± 1155.841op/s 2111670.117op/s 2113128.292op/s 2113796.881op/s 2114255.949op/s 0.18% -0.286 -0.039 0.08% 121.034op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 190.340µs 191.247µs ± 0.199µs 191.252µs ± 0.124µs 191.368µs 191.533µs 191.724µs 191.904µs 0.34% -0.386 2.417 0.10% 0.014µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5210947.854op/s 5228846.306op/s ± 5437.753op/s 5228700.687op/s ± 3396.371op/s 5232443.943op/s 5237512.110op/s 5243553.929op/s 5253752.897op/s 0.48% 0.399 2.446 0.10% 384.507op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 46.324µs 46.457µs ± 0.074µs 46.455µs ± 0.037µs 46.490µs 46.544µs 46.599µs 47.003µs 1.18% 3.119 20.152 0.16% 0.005µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 21275327.003op/s 21525307.115op/s ± 34099.273op/s 21526172.263op/s ± 16965.747op/s 21543379.872op/s 21570304.005op/s 21582578.370op/s 21587220.122op/s 0.28% -3.060 19.643 0.16% 2411.183op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 48.970µs 49.121µs ± 0.068µs 49.119µs ± 0.047µs 49.164µs 49.243µs 49.305µs 49.332µs 0.43% 0.473 0.154 0.14% 0.005µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 20270779.906op/s 20357973.641op/s ± 28322.463op/s 20358696.561op/s ± 19554.964op/s 20379061.202op/s 20398911.628op/s 20415624.658op/s 20420743.296op/s 0.30% -0.465 0.141 0.14% 2002.701op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [701.549µs; 701.772µs] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1424966.969op/s; 1425418.651op/s] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [473.777µs; 473.884µs] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2110224.284op/s; 2110698.726op/s] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [191.219µs; 191.275µs] or [-0.014%; +0.014%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5228092.686op/s; 5229599.926op/s] or [-0.014%; +0.014%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [46.447µs; 46.467µs] or [-0.022%; +0.022%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [21520581.284op/s; 21530032.947op/s] or [-0.022%; +0.022%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [49.111µs; 49.130µs] or [-0.019%; +0.019%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [20354048.420op/s; 20361898.862op/s] or [-0.019%; +0.019%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 18.887µs 28.450µs ± 13.630µs 19.104µs ± 0.180µs 36.893µs 45.723µs 47.574µs 146.022µs 664.35% 3.815 27.760 47.79% 0.964µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [26.561µs; 30.339µs] or [-6.640%; +6.640%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 4.270µs 4.288µs ± 0.003µs 4.288µs ± 0.001µs 4.289µs 4.292µs 4.294µs 4.297µs 0.23% -1.621 14.015 0.06% 0.000µs 1 200
credit_card/is_card_number/ throughput 232694203.910op/s 233202422.061op/s ± 145870.806op/s 233220891.484op/s ± 54148.087op/s 233260654.763op/s 233333481.150op/s 233508019.463op/s 234174242.700op/s 0.41% 1.646 14.153 0.06% 10314.624op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 89.530µs 90.582µs ± 0.635µs 90.538µs ± 0.421µs 90.965µs 91.498µs 91.888µs 95.076µs 5.01% 1.882 11.050 0.70% 0.045µs 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 10517918.411op/s 11040297.655op/s ± 76424.926op/s 11045040.587op/s ± 51069.737op/s 11095349.385op/s 11145535.921op/s 11165766.624op/s 11169461.766op/s 1.13% -1.694 9.444 0.69% 5404.058op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 83.038µs 83.549µs ± 0.388µs 83.517µs ± 0.125µs 83.658µs 83.858µs 83.993µs 88.263µs 5.68% 8.974 106.798 0.46% 0.027µs 1 200
credit_card/is_card_number/ 378282246310005 throughput 11329744.763op/s 11969270.562op/s ± 53465.721op/s 11973548.648op/s ± 17987.727op/s 11989369.626op/s 12018588.710op/s 12031894.126op/s 12042626.079op/s 0.58% -8.584 100.602 0.45% 3780.597op/s 1 200
credit_card/is_card_number/37828224631 execution_time 4.269µs 4.288µs ± 0.005µs 4.287µs ± 0.001µs 4.289µs 4.291µs 4.294µs 4.343µs 1.29% 7.905 95.017 0.11% 0.000µs 1 200
credit_card/is_card_number/37828224631 throughput 230270878.699op/s 233228121.303op/s ± 252653.304op/s 233249046.746op/s ± 56035.699op/s 233293164.524op/s 233462388.270op/s 233551663.605op/s 234249906.797op/s 0.43% -7.792 93.507 0.11% 17865.286op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 79.896µs 80.673µs ± 0.245µs 80.628µs ± 0.103µs 80.735µs 81.160µs 81.597µs 81.693µs 1.32% 1.375 3.602 0.30% 0.017µs 1 200
credit_card/is_card_number/378282246310005 throughput 12240920.667op/s 12395902.516op/s ± 37552.278op/s 12402620.054op/s ± 15880.058op/s 12417065.409op/s 12437205.281op/s 12457202.449op/s 12516344.428op/s 0.92% -1.341 3.513 0.30% 2655.347op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 58.475µs 58.665µs ± 0.035µs 58.664µs ± 0.019µs 58.686µs 58.722µs 58.733µs 58.735µs 0.12% -0.924 3.758 0.06% 0.002µs 1 200
credit_card/is_card_number/37828224631000521389798 throughput 17025599.831op/s 17045989.488op/s ± 10078.052op/s 17046100.231op/s ± 5629.570op/s 17051119.131op/s 17062362.391op/s 17069927.877op/s 17101448.493op/s 0.32% 0.933 3.803 0.06% 712.626op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 6.433µs 6.443µs ± 0.003µs 6.442µs ± 0.001µs 6.444µs 6.448µs 6.452µs 6.460µs 0.28% 1.153 4.755 0.05% 0.000µs 1 200
credit_card/is_card_number/x371413321323331 throughput 154787769.213op/s 155213988.134op/s ± 82037.443op/s 155222546.075op/s ± 34906.764op/s 155253737.200op/s 155331770.307op/s 155395936.873op/s 155453139.893op/s 0.15% -1.145 4.720 0.05% 5800.923op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 4.271µs 4.288µs ± 0.003µs 4.288µs ± 0.001µs 4.290µs 4.293µs 4.296µs 4.300µs 0.29% -0.445 6.863 0.07% 0.000µs 1 200
credit_card/is_card_number_no_luhn/ throughput 232540283.217op/s 233185726.152op/s ± 161752.886op/s 233203393.109op/s ± 57987.126op/s 233247436.698op/s 233439723.925op/s 233525807.968op/s 234148512.208op/s 0.41% 0.463 6.933 0.07% 11437.656op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 69.193µs 69.957µs ± 0.364µs 69.943µs ± 0.250µs 70.197µs 70.607µs 70.861µs 70.879µs 1.34% 0.307 -0.336 0.52% 0.026µs 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 14108552.585op/s 14294813.700op/s ± 74336.034op/s 14297418.602op/s ± 51083.722op/s 14347783.140op/s 14413304.077op/s 14434616.806op/s 14452286.635op/s 1.08% -0.283 -0.356 0.52% 5256.351op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 63.852µs 64.646µs ± 0.399µs 64.603µs ± 0.295µs 64.945µs 65.347µs 65.545µs 65.602µs 1.55% 0.336 -0.658 0.62% 0.028µs 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 15243514.155op/s 15469398.507op/s ± 95350.274op/s 15479236.931op/s ± 70420.901op/s 15544099.006op/s 15600567.813op/s 15639835.836op/s 15661205.486op/s 1.18% -0.314 -0.675 0.61% 6742.283op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 4.272µs 4.288µs ± 0.003µs 4.288µs ± 0.001µs 4.289µs 4.293µs 4.295µs 4.300µs 0.29% -0.069 4.346 0.07% 0.000µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 232546479.642op/s 233213727.080op/s ± 165754.910op/s 233224388.655op/s ± 72811.779op/s 233276514.891op/s 233465939.419op/s 233576519.799op/s 234062646.321op/s 0.36% 0.082 4.370 0.07% 11720.642op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 61.764µs 61.925µs ± 0.061µs 61.930µs ± 0.032µs 61.958µs 62.017µs 62.096µs 62.171µs 0.39% 0.195 1.359 0.10% 0.004µs 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 16084728.791op/s 16148587.071op/s ± 15961.727op/s 16147347.537op/s ± 8446.993op/s 16157955.384op/s 16175242.829op/s 16187771.224op/s 16190536.631op/s 0.27% -0.185 1.341 0.10% 1128.665op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 58.546µs 58.669µs ± 0.041µs 58.667µs ± 0.020µs 58.689µs 58.735µs 58.773µs 58.929µs 0.45% 1.308 7.592 0.07% 0.003µs 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 16969587.027op/s 17044818.875op/s ± 12035.048op/s 17045462.191op/s ± 5804.171op/s 17050696.632op/s 17062726.222op/s 17069325.879op/s 17080463.366op/s 0.21% -1.291 7.493 0.07% 851.006op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 6.433µs 6.442µs ± 0.003µs 6.442µs ± 0.001µs 6.444µs 6.447µs 6.451µs 6.455µs 0.19% 0.402 2.049 0.05% 0.000µs 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 154929795.254op/s 155230846.983op/s ± 78247.697op/s 155229502.521op/s ± 35174.777op/s 155266004.509op/s 155374110.344op/s 155408979.303op/s 155447208.506op/s 0.14% -0.396 2.038 0.05% 5532.948op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [4.288µs; 4.288µs] or [-0.009%; +0.009%] None None None
credit_card/is_card_number/ throughput [233182205.770op/s; 233222638.351op/s] or [-0.009%; +0.009%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [90.494µs; 90.670µs] or [-0.097%; +0.097%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [11029705.895op/s; 11050889.414op/s] or [-0.096%; +0.096%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [83.495µs; 83.603µs] or [-0.064%; +0.064%] None None None
credit_card/is_card_number/ 378282246310005 throughput [11961860.728op/s; 11976680.397op/s] or [-0.062%; +0.062%] None None None
credit_card/is_card_number/37828224631 execution_time [4.287µs; 4.288µs] or [-0.015%; +0.015%] None None None
credit_card/is_card_number/37828224631 throughput [233193105.985op/s; 233263136.621op/s] or [-0.015%; +0.015%] None None None
credit_card/is_card_number/378282246310005 execution_time [80.639µs; 80.707µs] or [-0.042%; +0.042%] None None None
credit_card/is_card_number/378282246310005 throughput [12390698.132op/s; 12401106.901op/s] or [-0.042%; +0.042%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [58.660µs; 58.670µs] or [-0.008%; +0.008%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [17044592.767op/s; 17047386.209op/s] or [-0.008%; +0.008%] None None None
credit_card/is_card_number/x371413321323331 execution_time [6.442µs; 6.443µs] or [-0.007%; +0.007%] None None None
credit_card/is_card_number/x371413321323331 throughput [155202618.533op/s; 155225357.734op/s] or [-0.007%; +0.007%] None None None
credit_card/is_card_number_no_luhn/ execution_time [4.288µs; 4.289µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/ throughput [233163308.758op/s; 233208143.547op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [69.907µs; 70.008µs] or [-0.072%; +0.072%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [14284511.441op/s; 14305115.960op/s] or [-0.072%; +0.072%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [64.591µs; 64.702µs] or [-0.086%; +0.086%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [15456183.876op/s; 15482613.138op/s] or [-0.085%; +0.085%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [4.287µs; 4.288µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [233190755.043op/s; 233236699.116op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [61.916µs; 61.933µs] or [-0.014%; +0.014%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [16146374.929op/s; 16150799.213op/s] or [-0.014%; +0.014%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [58.663µs; 58.675µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [17043150.933op/s; 17046486.817op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [6.442µs; 6.442µs] or [-0.007%; +0.007%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [155220002.605op/s; 155241691.362op/s] or [-0.007%; +0.007%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.396µs 3.379µs ± 1.490µs 3.179µs ± 0.023µs 3.203µs 3.808µs 14.570µs 15.606µs 390.97% 7.437 56.048 43.98% 0.105µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [3.172µs; 3.585µs] or [-6.110%; +6.110%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 57.689ms 57.931ms ± 0.159ms 57.898ms ± 0.094ms 58.011ms 58.239ms 58.422ms 58.521ms 1.08% 1.189 1.672 0.27% 0.011ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [57.909ms; 57.953ms] or [-0.038%; +0.038%] None None None

Group 13

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 3cf2678 1740420328 ekump/APMSP-1756-add-trace-exporter-integration-tests-BACKUP
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.667µs 2.732µs ± 0.017µs 2.729µs ± 0.008µs 2.738µs 2.768µs 2.775µs 2.777µs 1.75% 0.156 1.900 0.62% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.730µs; 2.734µs] or [-0.087%; +0.087%] None None None

Baseline

Omitted due to size.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ekump