Storage
On-disk files in a container are ephemeral and can’t be shared with other members of a pod. For some applications, this is not an issue, but for many persistent storage is required.
Charmed Kubernetes makes it easy to add and configure different types of persistent storage for your Kubernetes cluster, as outlined below. For more detail on the concept of storage volumes in Kubernetes, please see the Kubernetes documentation.
Ceph storage
Charmed Kubernetes can make use of Ceph to provide persistent storage volumes. The following sections assume you have already deployed a Charmed Kubernetes cluster and you have internet access to the Juju Charm Store.
Deploy Ceph
Check that the current Juju model is the one where you wish to deploy Ceph
Begin by adding a minimum number of Ceph monitor nodes:
For the storage nodes we will also need to specify storage volumes for the backing cloud to add. This is done by using the --storage
option. The ceph-osd
charm defines two useful types of storage, osd-devices
for the volumes which will be formatted and used to provide storage, and osd-journals
for storage used for journalling.
The format for the --storage
option is <storage pool>,<size>,<number>
. The storage pools available are dependent on and defined by the backing cloud. However, by omitting the storage type, the default pool for that cloud will be chosen (E.g. for AWS, the default pool is EBS storage).
So, for example, to deploy three ceph-osd
storage nodes, using the default storage pool, with 2x 32G volumes of storage per node, and one 8G journal, we would use the command:
Note that actually deploying these charms with storage may take some time, but you can continue to run other Juju commands in the meantime.
The ceph-osd
and ceph-mon
deployments should then be connected:
If you wish to include CephFS support, which allows for ReadWriteMany volumes,
you can also deploy and relate ceph-fs
:
Charmed Kubernetes will then deploy the CephFS provisioner pod
and create a cephfs
storage class in the cluster.
When deploying Charmed Kubernetes on Ubuntu 18.04(Bionic), you will need
to explicitly set the install_sources
config option on the kubernetes-control-plane
charm to include cloud:bionic-ussuri
(or whatever OpenStack release you are
using).
When using OpenStack Train, ReadWriteMany (RWX) CephFS volumes on containers running as a non-root user will be mounted as owned by root instead of the container’s user, potentially leading to permissions issues. You can work around this by adding an initContainer to your pod to adjust the mounted volume’s ownership or permissions. For example:
Relate to Charmed Kubernetes
Making Charmed Kubernetes aware of your Ceph cluster requires some Juju configuration and the ceph-csi
charm.
Create storage pools
By default, the ceph-csi
charm will create the required pools defined
in the storage class. To view the default options, run:
If you’re happy with this, you can skip the section. Otherwise, if you want to change these, you can delete the pools:
Then recreate them, using the options listed from the list-actions
command ran
earlier. For example:
Verification
Now you can look at your Charmed Kubernetes cluster to verify things are working. Running:
... should return output similar to:
Note that the CephFS storage class and pods will only be present if CephFS was included above.
Scaling out
To check existing storage allocation, use the command:
If extra storage is required, it is possible to add extra ceph-osd
units as
desired:
Once again, it is necessary to attach appropriate storage volumes as before. In this case though, the storage needs to be added on a per-unit basis.
Confirm the running units of ceph-osd
Add additional storage to existing or new units with the add-storage
command. For example, to add two volumes of 32G to the unit ceph-osd/2
:
Using a separate Juju model
In some circumstances it can be useful to locate the persistent storage in a different Juju model, for example to have one set of storage used by different clusters. The only change required is in adding relations between Ceph and Charmed Kubernetes.
For more information on how to achieve this, please see the Juju documentation on cross-model relations.
NFS
It is possible to add simple storage for Kubernetes using NFS. In this case, the storage is implemented on the root disk of units running the nfs
charm.
Deploy NFS
Make use of Juju constraints to allocate an instance with the required amount of storage. For example, for 200G of storage:
Relate to Charmed Kubernetes
The NFS units can be related directly to the Kubernetes workers:
Verification
Now you can look at your Charmed Kubernetes cluster to verify things are working. Running:
... should return output similar to:
Scaling out
If extra storage is required, it is possible to add extra nfs
units as desired. For example, to add three new units, each with 100G of storage:
There is no requirement that these additional units should have the same amount of storage space as previously.
See the guide to contributing or discuss these docs in our public Mattermost channel.