Skip to content

Commit 78d93d4

Browse files
Samzesethboyles
Samze
and
sethboyles
authored
Add Canary steps (#4219)
Adds Canary steps * Storing json in the DB as the overhead of a new canary_steps table seemed to have little benefit. We can always migrate in the future if required. Resolves #4172 --------- Co-authored-by: Seth Boyles <[email protected]>
1 parent b2948ee commit 78d93d4

24 files changed

+1255
-346
lines changed

‎app/actions/deployment_continue.rb

+15-5
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,21 @@ def continue(deployment:, user_audit_info:)
1212
reject_invalid_state!(deployment) unless deployment.continuable?
1313

1414
record_audit_event(deployment, user_audit_info)
15-
deployment.update(
16-
state: DeploymentModel::DEPLOYING_STATE,
17-
status_value: DeploymentModel::ACTIVE_STATUS_VALUE,
18-
status_reason: DeploymentModel::DEPLOYING_STATUS_REASON
19-
)
15+
16+
if deployment.canary_steps && deployment.canary_current_step < deployment.canary_steps.length
17+
deployment.update(
18+
state: DeploymentModel::PREPAUSED_STATE,
19+
status_value: DeploymentModel::ACTIVE_STATUS_VALUE,
20+
status_reason: DeploymentModel::DEPLOYING_STATUS_REASON,
21+
canary_current_step: deployment.canary_current_step + 1
22+
)
23+
else
24+
deployment.update(
25+
state: DeploymentModel::DEPLOYING_STATE,
26+
status_value: DeploymentModel::ACTIVE_STATUS_VALUE,
27+
status_reason: DeploymentModel::DEPLOYING_STATUS_REASON
28+
)
29+
end
2030
end
2131
end
2232

‎app/actions/deployment_create.rb

+14-9
Original file line numberDiff line numberDiff line change
@@ -54,13 +54,15 @@ def create(app:, user_audit_info:, message:)
5454
revision_guid: revision&.guid,
5555
revision_version: revision&.version,
5656
strategy: message.strategy,
57-
max_in_flight: message.max_in_flight
57+
max_in_flight: message.max_in_flight,
58+
canary_steps: message.options&.dig(:canary, :steps)
5859
)
60+
5961
MetadataUpdate.update(deployment, message)
6062

6163
supersede_deployment(previous_deployment)
6264

63-
process_instances = starting_process_instances(message, desired_instances)
65+
process_instances = starting_process_instances(deployment, desired_instances)
6466

6567
process = create_deployment_process(app, deployment.guid, revision, process_instances)
6668
# Need to transition from STOPPED to STARTED to engage the ProcessObserver to desire the LRP.
@@ -176,7 +178,8 @@ def deployment_for_stopped_app(app, message, previous_deployment, previous_dropl
176178
revision_guid: revision&.guid,
177179
revision_version: revision&.version,
178180
strategy: message.strategy,
179-
max_in_flight: message.max_in_flight
181+
max_in_flight: message.max_in_flight,
182+
canary_steps: message.options&.dig(:canary, :steps)
180183
)
181184

182185
MetadataUpdate.update(deployment, message)
@@ -217,12 +220,14 @@ def starting_state(message)
217220
end
218221
end
219222

220-
def starting_process_instances(message, desired_instances)
221-
if message.strategy == DeploymentModel::CANARY_STRATEGY
222-
1
223-
else
224-
[message.max_in_flight, desired_instances].min
225-
end
223+
def starting_process_instances(deployment, desired_instances)
224+
starting_process_count = if deployment.strategy == DeploymentModel::CANARY_STRATEGY
225+
deployment.canary_step[:canary]
226+
else
227+
desired_instances
228+
end
229+
230+
[deployment.max_in_flight, starting_process_count].min
226231
end
227232

228233
def log_rollback_event(app_guid, user_id, revision_id, strategy)

‎app/messages/deployment_create_message.rb

+74-7
Original file line numberDiff line numberDiff line change
@@ -10,17 +10,22 @@ class DeploymentCreateMessage < MetadataBaseMessage
1010
options
1111
]
1212

13+
ALLOWED_OPTION_KEYS = %i[
14+
canary
15+
max_in_flight
16+
].freeze
17+
18+
ALLOWED_STEP_KEYS = [
19+
:instance_weight
20+
].freeze
21+
1322
validates_with NoAdditionalKeysValidator
1423
validates :strategy,
1524
inclusion: { in: %w[rolling canary], message: "'%<value>s' is not a supported deployment strategy" },
1625
allow_nil: true
1726
validate :mutually_exclusive_droplet_sources
1827

19-
validates :options,
20-
allow_nil: true,
21-
hash: true
22-
23-
validate :validate_max_in_flight
28+
validate :validate_options
2429

2530
def app_guid
2631
relationships&.dig(:app, :data, :guid)
@@ -46,14 +51,76 @@ def mutually_exclusive_droplet_sources
4651
errors.add(:droplet, "Cannot set both fields 'droplet' and 'revision'")
4752
end
4853

49-
def validate_max_in_flight
50-
return unless options.present? && options.is_a?(Hash) && options[:max_in_flight]
54+
def validate_options
55+
return if options.blank?
5156

57+
unless options.is_a?(Hash)
58+
errors.add(:options, 'must be an object')
59+
return
60+
end
61+
62+
disallowed_keys = options.keys - ALLOWED_OPTION_KEYS
63+
errors.add(:options, "has unsupported key(s): #{disallowed_keys.join(', ')}") if disallowed_keys.present?
64+
65+
validate_max_in_flight if options[:max_in_flight]
66+
validate_canary if options[:canary]
67+
end
68+
69+
def validate_max_in_flight
5270
max_in_flight = options[:max_in_flight]
5371

5472
return unless !max_in_flight.is_a?(Integer) || max_in_flight < 1
5573

5674
errors.add(:max_in_flight, 'must be an integer greater than 0')
5775
end
76+
77+
def validate_canary
78+
canary_options = options[:canary]
79+
unless canary_options.is_a?(Hash)
80+
errors.add(:'options.canary', 'must be an object')
81+
return
82+
end
83+
84+
if canary_options && strategy != 'canary'
85+
errors.add(:'options.canary', 'are only valid for Canary deployments')
86+
return
87+
end
88+
89+
validate_steps if options[:canary][:steps]
90+
end
91+
92+
def validate_steps
93+
steps = options[:canary][:steps]
94+
if !steps.is_a?(Array) || steps.any? { |step| !step.is_a?(Hash) }
95+
errors.add(:'options.canary.steps', 'must be an array of objects')
96+
return
97+
end
98+
99+
steps.each do |step|
100+
disallowed_keys = step.keys - ALLOWED_STEP_KEYS
101+
102+
errors.add(:'options.canary.steps', "has unsupported key(s): #{disallowed_keys.join(', ')}") if disallowed_keys.present?
103+
end
104+
105+
validate_step_instance_weights
106+
end
107+
108+
def validate_step_instance_weights
109+
steps = options[:canary][:steps]
110+
111+
errors.add(:'options.canary.steps', 'missing key: "instance_weight"') if steps.any? { |step| !step.key?(:instance_weight) }
112+
113+
if steps.any? { |step| !step[:instance_weight].is_a?(Integer) }
114+
errors.add(:'options.canary.steps.instance_weight', 'must be an Integer between 1-100 (inclusive)')
115+
return
116+
end
117+
118+
errors.add(:'options.canary.steps.instance_weight', 'must be an Integer between 1-100 (inclusive)') if steps.any? { |step| (1..100).exclude?(step[:instance_weight]) }
119+
120+
weights = steps.pluck(:instance_weight)
121+
return unless weights.sort != weights
122+
123+
errors.add(:'options.canary.steps.instance_weight', 'must be sorted in ascending order')
124+
end
58125
end
59126
end

‎app/models/runtime/deployment_model.rb

+44
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
module VCAP::CloudController
22
class DeploymentModel < Sequel::Model(:deployments)
3+
plugin :serialization
4+
35
DEPLOYMENT_STATES = [
46
DEPLOYING_STATE = 'DEPLOYING'.freeze,
57
PREPAUSED_STATE = 'PREPAUSED'.freeze,
@@ -76,6 +78,8 @@ class DeploymentModel < Sequel::Model(:deployments)
7678
add_association_dependencies labels: :destroy
7779
add_association_dependencies annotations: :destroy
7880

81+
serialize_attributes :json, :canary_steps
82+
7983
dataset_module do
8084
def deploying_count
8185
where(state: DeploymentModel::PROGRESSING_STATES).count
@@ -87,6 +91,16 @@ def before_update
8791
set_status_updated_at
8892
end
8993

94+
def before_create
95+
self.canary_current_step = 1 if strategy == DeploymentModel::CANARY_STRATEGY
96+
97+
# unless canary_steps.nil?
98+
# # ensure that canary steps are in the correct format for serialization
99+
# self.canary_steps = canary_steps.map(&:stringify_keys)
100+
# end
101+
super
102+
end
103+
90104
def deploying?
91105
DeploymentModel::PROGRESSING_STATES.include?(state)
92106
end
@@ -99,6 +113,36 @@ def continuable?
99113
state == DeploymentModel::PAUSED_STATE
100114
end
101115

116+
def current_canary_instance_target
117+
canary_step[:canary]
118+
end
119+
120+
def canary_total_instances
121+
canary_step[:canary] + canary_step[:original]
122+
end
123+
124+
def canary_step
125+
raise 'canary_step is only valid for canary deloyments' unless strategy == CANARY_STRATEGY
126+
127+
current_step = canary_current_step || 1
128+
canary_step_plan[current_step - 1]
129+
end
130+
131+
def canary_step_plan
132+
raise 'canary_step_plan is only valid for canary deloyments' unless strategy == CANARY_STRATEGY
133+
134+
return [{ canary: 1, original: original_web_process_instance_count }] if canary_steps.nil?
135+
136+
canary_steps.map do |step|
137+
weight = step['instance_weight']
138+
target_canary = (original_web_process_instance_count * (weight.to_f / 100)).round.to_i
139+
target_canary = 1 if target_canary.zero?
140+
target_original = original_web_process_instance_count - target_canary + 1
141+
target_original = 0 if weight == 100
142+
{ canary: target_canary, original: target_original }
143+
end
144+
end
145+
102146
private
103147

104148
def set_status_updated_at

‎app/presenters/v3/deployment_presenter.rb

+38-11
Original file line numberDiff line numberDiff line change
@@ -10,18 +10,9 @@ def to_hash
1010
guid: deployment.guid,
1111
created_at: deployment.created_at,
1212
updated_at: deployment.updated_at,
13-
status: {
14-
value: deployment.status_value,
15-
reason: deployment.status_reason,
16-
details: {
17-
last_successful_healthcheck: deployment.last_healthy_at,
18-
last_status_change: deployment.status_updated_at
19-
}
20-
},
13+
status: status(deployment),
2114
strategy: deployment.strategy,
22-
options: {
23-
max_in_flight: deployment.max_in_flight
24-
},
15+
options: options(deployment),
2516
droplet: {
2617
guid: deployment.droplet_guid
2718
},
@@ -64,6 +55,42 @@ def new_processes
6455
end
6556
end
6657

58+
def options(deployment)
59+
options = {
60+
max_in_flight: deployment.max_in_flight
61+
}
62+
63+
if deployment.strategy == VCAP::CloudController::DeploymentModel::CANARY_STRATEGY && deployment.canary_steps
64+
options[:canary] = {
65+
steps: deployment.canary_steps
66+
}
67+
end
68+
69+
options
70+
end
71+
72+
def status(deployment)
73+
status = {
74+
value: deployment.status_value,
75+
reason: deployment.status_reason,
76+
details: {
77+
last_successful_healthcheck: deployment.last_healthy_at,
78+
last_status_change: deployment.status_updated_at
79+
}
80+
}
81+
82+
if deployment.strategy == VCAP::CloudController::DeploymentModel::CANARY_STRATEGY
83+
status[:canary] = {
84+
steps: {
85+
current: deployment.canary_current_step,
86+
total: deployment.canary_steps&.length || 1
87+
}
88+
}
89+
end
90+
91+
status
92+
end
93+
6794
def build_links
6895
{
6996
self: {

‎db/migrations/20250130172009_add_canary_steps_to_deployments.rb

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
Sequel.migration do
2+
up do
3+
alter_table(:deployments) do
4+
add_column :canary_steps, String, size: 4096
5+
add_column :canary_current_step, :integer
6+
end
7+
end
8+
down do
9+
alter_table(:deployments) do
10+
drop_column :canary_steps
11+
drop_column :canary_current_step
12+
end
13+
end
14+
end

‎docs/v3/source/includes/api_resources/_deployments.erb

+16
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,27 @@
88
"details": {
99
"last_successful_healthcheck": "2018-04-25T22:42:10Z",
1010
"last_status_change": "2018-04-25T22:42:10Z"
11+
},
12+
"canary": {
13+
"steps": {
14+
"current": 1,
15+
"total": 2
16+
}
1117
}
1218
},
1319
"strategy": "canary",
1420
"options" : {
1521
"max_in_flight": 3,
22+
"canary": {
23+
"steps": [
24+
{
25+
"instance_weight": 10
26+
},
27+
{
28+
"instance_weight": 20
29+
}
30+
]
31+
}
1632
},
1733
"droplet": {
1834
"guid": "44ccfa61-dbcf-4a0d-82fe-f668e9d2a962"

‎docs/v3/source/includes/resources/deployments/_create.md.erb

+1
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ Name | Type | Description | Default
7878
**revision**<sup>[1]</sup> | _object_ | The [revision](#revisions) whose droplet to deploy for the app; this will update the app's [current droplet](#get-current-droplet-association-for-an-app) to this droplet |
7979
**strategy** | _string_ | The strategy to use for the deployment | `rolling`
8080
**options.max_in_flight** | _integer_ | The maximum number of new instances to deploy simultaneously | 1
81+
**options.canary.steps** | _array of [canary step objects](#canary-steps-object)_ | An array of canary steps to use for the deployment
8182
**metadata.labels** | [_label object_](#labels) | Labels applied to the deployment
8283
**metadata.annotations** | [_annotation object_](#annotations) | Annotations applied to the deployment
8384

‎docs/v3/source/includes/resources/deployments/_object.md.erb

+9-1
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,12 @@ Name | Type | Description
1515
**status.value** | _string_ | The current status of the deployment; valid values are `ACTIVE` (meaning in progress) and `FINALIZED` (meaning finished, either successfully or not)
1616
**status.reason** | _string_ | The reason for the status of the deployment;<br>following list represents valid values:<br>1. If **status.value** is `ACTIVE`<br>- `DEPLOYING`<br>- `PAUSED` (only valid for canary deployments) <br>- `CANCELING`<br>2. If **status.value** is `FINALIZED`<br>- `DEPLOYED`<br>- `CANCELED`<br>- `SUPERSEDED` (another deployment created for app before completion)<br>
1717
**status.details.last_successful_healthcheck** | _[timestamp](#timestamps)_ | Timestamp of the last successful healthcheck
18-
**status.details.last_status_change** | _[timestamp](#timestamps)_ | Timestamp of last change to status.value or status.reason
18+
**status.details.last_status_change** | _[timestamp](#timestamps)_ | Timestamp of last change to status.value or status.reason**status.details.last_status_change** | _[timestamp](#timestamps)_ | Timestamp of last change to status.value or status.reason
19+
**status.canary.steps.current** | _integer_ | The current canary step. Only available for deployments with strategy 'canary'. (experimental)
20+
**status.canary.steps.total** | _integer_ | The total number of canary steps. Only available for deployments with strategy 'canary'. (experimental)
1921
**strategy** | _string_ | Strategy used for the deployment; supported strategies are `rolling` and `canary` (experimental)
2022
**options.max_in_flight** | _integer_ | The maximum number of new instances to deploy simultaneously
23+
**options.canary.steps** | _array of [canary step objects](#canary-steps-object)_ | Canary steps to use for the deployment. Only available for deployments with strategy 'canary'. (experimental)
2124
**droplet.guid** | _string_ | The droplet guid that the deployment is transitioning the app to
2225
**previous_droplet.guid** | _string_ | The app's [current droplet guid](#get-current-droplet-association-for-an-app) before the deployment was created
2326
**new_processes** | _array_ | List of processes created as part of the deployment
@@ -26,3 +29,8 @@ Name | Type | Description
2629
**metadata.labels** | [_label object_](#labels) | Labels applied to the deployment
2730
**metadata.annotations** | [_annotation object_](#annotations) | Annotations applied to the deployment
2831
**links** | [_links object_](#links) | Links to related resources
32+
33+
34+
#### Canary steps object
35+
36+
**instance_weight** | _integer_ | The percentage of instances to be deployed as part of the canary process in this step (experimental)

0 commit comments

Comments
 (0)