Skip to content

Add node affinity support #1166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Dec 16, 2020
Merged

Add node affinity support #1166

Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
f842529
Merge pull request #1 from zalando/master
sagor999 Oct 10, 2020
abc2cb4
Merge pull request #3 from zalando/master
sagor999 Nov 17, 2020
d412754
Merge pull request #4 from zalando/master
sagor999 Dec 6, 2020
abc48a6
Merge pull request #5 from zalando/master
sagor999 Dec 15, 2020
c7aadba
Adding nodeaffinity support alongside node_readiness_label
spohner May 13, 2020
4014442
Add nodeAffinity support to ConnectionPooler
adastleyatvi Jul 3, 2020
b6a83ca
Add tests for ConnectionPooler nodeAffinity
adastleyatvi Oct 10, 2020
9b53763
fix compareStatefulSet to take into account nodeAffinity
sagor999 Oct 11, 2020
b8751e8
add documentation for node affinity
sagor999 Oct 12, 2020
58c2142
add node affinity test
sagor999 Oct 12, 2020
300183f
fix print statement
sagor999 Oct 18, 2020
04be950
update codegen
sagor999 Oct 20, 2020
a3dea03
fix crd api for node affinity
sagor999 Nov 17, 2020
f032cd5
fix node affinity test
sagor999 Nov 17, 2020
fc9d5bf
fix e2e test for node affinity
sagor999 Nov 18, 2020
88743f8
replace hardcoded sleep with wait_for_pod_start in node affinity e2e
sagor999 Nov 18, 2020
a8c9b0e
remove extra affinity check, as it is already done
sagor999 Dec 6, 2020
8f05753
improve crd readability by moving Required field up
sagor999 Dec 6, 2020
03f91c3
improve node affinity e2e test
sagor999 Dec 6, 2020
867b294
add node affinity into various crds and add example into complete man…
sagor999 Dec 6, 2020
3bfa3af
fix missing type in crd
sagor999 Dec 6, 2020
8e8cd93
remove node affinity from connection pooler
sagor999 Dec 15, 2020
f5a871b
fix unit test for node affinity
sagor999 Dec 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions charts/postgres-operator/crds/postgresqls.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,97 @@ spec:
type: string
caSecretName:
type: string
nodeAffinity:
type: object
properties:
preferredDuringSchedulingIgnoredDuringExecution:
type: array
items:
type: object
required:
- weight
- preference
properties:
preference:
type: object
properties:
matchExpressions:
type: array
items:
type: object
required:
- key
- operator
properties:
key:
type: string
operator:
type: string
values:
type: array
items:
type: string
matchFields:
type: array
items:
type: object
required:
- key
- operator
properties:
key:
type: string
operator:
type: string
values:
type: array
items:
type: string
weight:
format: int32
type: integer
requiredDuringSchedulingIgnoredDuringExecution:
type: object
required:
- nodeSelectorTerms
properties:
nodeSelectorTerms:
type: array
items:
type: object
properties:
matchExpressions:
type: array
items:
type: object
required:
- key
- operator
properties:
key:
type: string
operator:
type: string
values:
type: array
items:
type: string
matchFields:
type: array
items:
type: object
required:
- key
- operator
properties:
key:
type: string
operator:
type: string
values:
type: array
items:
type: string
tolerations:
type: array
items:
Expand Down
24 changes: 23 additions & 1 deletion docs/user.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -517,7 +517,7 @@ manifest the operator will raise the limits to the configured minimum values.
If no resources are defined in the manifest they will be obtained from the
configured [default requests](reference/operator_parameters.md#kubernetes-resource-requests).

## Use taints and tolerations for dedicated PostgreSQL nodes
## Use taints, tolerations and node affinity for dedicated PostgreSQL nodes

To ensure Postgres pods are running on nodes without any other application pods,
you can use [taints and tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)
Expand All @@ -531,6 +531,28 @@ spec:
effect: NoSchedule
```

If you need the pods to be scheduled on specific nodes you may use [node affinity](https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/)
to specify a set of label(s), of which a prospective host node must have at least one. This could be used to
place nodes with certain hardware capabilities (e.g. SSD drives) in certain environments or network segments,
e.g. for PCI compliance.

```yaml
apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
name: acid-minimal-cluster
spec:
teamId: "ACID"
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: environment
operator: In
values:
- pci
```

## How to clone an existing PostgreSQL cluster

You can spin up a new cluster as a clone of the existing one, using a `clone`
Expand Down
106 changes: 106 additions & 0 deletions e2e/tests/test_e2e.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -929,6 +929,112 @@ def test_zzz_taint_based_eviction(self):
new_master_node = nm[0]
self.assert_distributed_pods(new_master_node, new_replica_nodes, cluster_label)

@timeout_decorator.timeout(TEST_TIMEOUT_SEC)
def test_node_affinity(self):
'''
Add label to a node and update postgres cluster spec to deploy only on a node with that label
'''
k8s = self.k8s
cluster_label = 'application=spilo,cluster-name=acid-minimal-cluster'

# verify we are in good state from potential previous tests
self.eventuallyEqual(lambda: k8s.count_running_pods(), 2, "No 2 pods running")
self.eventuallyEqual(lambda: len(k8s.get_patroni_running_members("acid-minimal-cluster-0")), 2, "Postgres status did not enter running")
self.eventuallyEqual(lambda: self.k8s.get_operator_state(), {"0": "idle"}, "Operator does not get in sync")

# get nodes of master and replica(s)
master_node, replica_nodes = k8s.get_pg_nodes(cluster_label)

self.assertNotEqual(master_node, [])
self.assertNotEqual(replica_nodes, [])

# label node with environment=postgres
node_label_body = {
"metadata": {
"labels": {
"node-affinity-test": "postgres"
}
}
}

try:
# patch current master node with the label
print('patching master node: {}'.format(master_node))
k8s.api.core_v1.patch_node(master_node, node_label_body)

# add node affinity to cluster
patch_node_affinity_config = {
"spec": {
"nodeAffinity" : {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "node-affinity-test",
"operator": "In",
"values": [
"postgres"
]
}
]
}
]
}
}
}
}

k8s.api.custom_objects_api.patch_namespaced_custom_object(
group="acid.zalan.do",
version="v1",
namespace="default",
plural="postgresqls",
name="acid-minimal-cluster",
body=patch_node_affinity_config)
self.eventuallyEqual(lambda: self.k8s.get_operator_state(), {"0": "idle"}, "Operator does not get in sync")

# node affinity change should cause replica to relocate from replica node to master node due to node affinity requirement
k8s.wait_for_pod_failover(master_node, 'spilo-role=replica,' + cluster_label)
k8s.wait_for_pod_start('spilo-role=replica,' + cluster_label)

podsList = k8s.api.core_v1.list_namespaced_pod('default', label_selector=cluster_label)
for pod in podsList.items:
if pod.metadata.labels.get('spilo-role') == 'replica':
self.assertEqual(master_node, pod.spec.node_name,
"Sanity check: expected replica to relocate to master node {}, but found on {}".format(master_node, pod.spec.node_name))

# check that pod has correct node affinity
key = pod.spec.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms[0].match_expressions[0].key
value = pod.spec.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms[0].match_expressions[0].values[0]
self.assertEqual("node-affinity-test", key,
"Sanity check: expect node selector key to be equal to 'node-affinity-test' but got {}".format(key))
self.assertEqual("postgres", value,
"Sanity check: expect node selector value to be equal to 'postgres' but got {}".format(value))

patch_node_remove_affinity_config = {
"spec": {
"nodeAffinity" : None
}
}
k8s.api.custom_objects_api.patch_namespaced_custom_object(
group="acid.zalan.do",
version="v1",
namespace="default",
plural="postgresqls",
name="acid-minimal-cluster",
body=patch_node_remove_affinity_config)
Copy link
Member

@FxKu FxKu Dec 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the affinity is another rolling update. Therefore, we should guarantee that everything is running when we leave the test.. Check other e2e tests like test_zz_node_readiness_label or test_zzz_taint_based_eviction how it is handled there.

Copy link
Contributor Author

@sagor999 sagor999 Dec 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will update!

self.eventuallyEqual(lambda: self.k8s.get_operator_state(), {"0": "idle"}, "Operator does not get in sync")

# remove node affinity to move replica away from master node
nm, new_replica_nodes = k8s.get_cluster_nodes()
new_master_node = nm[0]
self.assert_distributed_pods(new_master_node, new_replica_nodes, cluster_label)

except timeout_decorator.TimeoutError:
print('Operator log: {}'.format(k8s.get_operator_log()))
raise

@timeout_decorator.timeout(TEST_TIMEOUT_SEC)
def test_zzzz_cluster_deletion(self):
'''
Expand Down
11 changes: 11 additions & 0 deletions manifests/complete-postgres-manifest.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,3 +172,14 @@ spec:
# When TLS is enabled, also set spiloFSGroup parameter above to the relevant value.
# if unknown, set it to 103 which is the usual value in the default spilo images.
# In Openshift, there is no need to set spiloFSGroup/spilo_fsgroup.

# Add node affinity support by allowing postgres pods to schedule only on nodes that
# have label: "postgres-operator:enabled" set.
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: postgres-operator
# operator: In
# values:
# - enabled
91 changes: 91 additions & 0 deletions manifests/postgresql.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,97 @@ spec:
type: string
caSecretName:
type: string
nodeAffinity:
type: object
properties:
preferredDuringSchedulingIgnoredDuringExecution:
type: array
items:
type: object
required:
- weight
- preference
properties:
preference:
type: object
properties:
matchExpressions:
type: array
items:
type: object
required:
- key
- operator
properties:
key:
type: string
operator:
type: string
values:
type: array
items:
type: string
matchFields:
type: array
items:
type: object
required:
- key
- operator
properties:
key:
type: string
operator:
type: string
values:
type: array
items:
type: string
weight:
format: int32
type: integer
requiredDuringSchedulingIgnoredDuringExecution:
type: object
required:
- nodeSelectorTerms
properties:
nodeSelectorTerms:
type: array
items:
type: object
properties:
matchExpressions:
type: array
items:
type: object
required:
- key
- operator
properties:
key:
type: string
operator:
type: string
values:
type: array
items:
type: string
matchFields:
type: array
items:
type: object
required:
- key
- operator
properties:
key:
type: string
operator:
type: string
values:
type: array
items:
type: string
tolerations:
type: array
items:
Expand Down
Loading