The Deis Control Plane, Data Plane, and Router Mesh components all depend on an etcd cluster for service discovery and configuration.
Whether built for evaluation or to host production applications, when managing a small Deis cluster (three to five nodes), it is reasonable to accept the platform’s default behavior wherein etcd runs on every node within the cluster.
In larger Deis clusters however, running etcd on every node can have a deleterious effect on overall cluster performance since it increases the time required for nodes to reach consensus on writes and leader elections. In such cases, it is beneficial to isolate etcd to a small, fixed number of nodes. All other nodes in the Deis cluster may run an etcd proxy. Proxies will forward read and write requests to active participants in the etcd cluster (leader or followers) without affecting the time required for etcd nodes to reach consensus on writes or leader elections.
The benefit of running an etcd proxy on any node not running a full etcd
process is that any container or service depending on etcd can connect to
etcd easily via
localhost from any node in the Deis cluster.
Also see CoreOS cluster architecture documentation for further details.
The approach documented here works as of Deis 1.9. Older versions of Deis utilize an older version of etcd that did not include the proxy functionality.
To realize the topology described above, it is necessary, at the time of provisioning, to provide different cloud-config for those hosts that will run etcd and for those that will only run an etcd proxy.
For the small, fixed number of hosts running full etcd and satisfying the “central services” role (as described in the CoreOS documentation), the cloud-config provided with Deis is sufficient.
For hosts running only an etcd proxy, satisfying the “worker” role (as described
in the CoreOS documentation), cloud-config must be tweaked slightly to include
-proxy on flag. For example:
#cloud-config coreos: etcd2: discovery: <discovery URL here> proxy: on # ...
Isolating etcd as described here requires subsets of a cluster’s hosts to be
configured differently from one another (including or excluding the
-proxy on flag). Deis provisioning scripts do not currently account for
this, so managing separate cloud-config for each subset of nodes in the cluster
is left as an exercise for the advanced operator.