Deployment Charts

All of our existing Kubernetes clusters make use of a single git repository located at operations/deployment-charts:

This repository is organized as follows:

Helm charts in /charts
Helmfile deployments in /helmfile.d
Common templates in /_scaffold
Template modules in /modules
JSON schemas of all CustomResourceDefinition objects in /jsonschema
Any custom deployments that cannot use helm for any reason, in /custom_deploy.d
Supporting test files and scripts

See the README file in the repository for more detailed information.

Helm Charts

A Helm chart is a set of files that describes a packaging together of Kubernetes resources so that they can be easily deployed as a unit and updated programmatically.

Once committed to the master branch, our helm charts are automatically published to ChartMuseum.

Common Templates

We have developed a set of common templates for our helm charts.

Enabling egress to services external to Kubernetes

Install the external-services helm template

In a new chart

The scaffolding should inject the necessary templates automatically. No need to do anything.

In an existing chart

Note: As a prerequisite, you will need to have sextant installed.

1. In order to enable egress to services living outside of Kubernetes (such as Kafka, Presto, Kerberos, etc), add "base.external-services-networkpolicy:1.0", to your chart's package.json file.

Run the following command to update the chart dependency lockfile, as well as install the base/external-services-networkpolicy template in your chart templates/vendor directory:

# Assuming you're working on charts/superset
$ cd path/to/deployment-charts
$ sextant --modulepath modules vendor charts/superset
INFO:sextant:Copied modules/base/external-services-networkpolicy_1.0.1.tpl => charts/superset/templates/vendor/base/external-services-networkpolicy_1.0.1.tpl

2. Add the following line at the end of your chart's templates/networkpolicy.yaml file

{{ include "base.networkpolicy.egress.external-services" . }}

Note: make sure this include isn't specified in a range loop, as it will lead to duplicate NetworkPolicy resources, which will cause your deployment to fail in cryptic ways.

3. Add the following lines in your chart's values.yaml file

# The set of external services to allow egress to
# Example:
# kafka:
# - main-codfw
# - main-eqiad
# presto:
# - analytics
#
# See https://wikitech.wikimedia.org/wiki/Kubernetes/Deployment_Charts#Enabling_egress_to_services_external_to_Kubernetes
# for the list of supported services
external_services: {}

4. Finally, bump your chart version in your chart's Chart.yaml file.

Select what services you want to enable egress to

Add the following structure to your chart or release values, by adapting the values associated with each service type:

# Note: only include the services you actually need, and not *all* of them.
# We mainly include them here to help you figure out what's available.
external_services:
  presto: [analytics]
  kerberos: [kdc]
  cas: [idp]
  druid: [analytics, public]
  redis-6379: [misc]
  kafka: [jumbo-eqiad]
  zookeeper: [flink-eqiad]
  cassandra: [ml-cache-a-eqiad]
  hadoop-worker: [analytics]
  hadoop-master: [analytics]
  mariadb: [analytics-meta]
  postgresql: [analytics]

The NetworkPolicy resource(s) allowing egress traffic to reach the specified services will be created next time you redeploy your application. Note: all exposed service types and associated instances are listed in Kubernetes. Connect to the deployment server, and use kubectl to list all services in the external-services namespace.

brouberol@deploy1002:~$ kubectl get services -n external-services -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n'
cas-idp
cas-idp-test
cassandra-analytics-query-service-storage-a-codfw
cassandra-analytics-query-service-storage-a-eqiad
cassandra-analytics-query-service-storage-b-codfw
cassandra-analytics-query-service-storage-b-eqiad
cassandra-cassandra-dev-a-codfw
cassandra-cassandra-dev-b-codfw
cassandra-ml-cache-a-codfw
cassandra-ml-cache-a-eqiad
cassandra-restbase-a-codfw
cassandra-restbase-a-eqiad
cassandra-restbase-b-codfw
cassandra-restbase-b-eqiad
cassandra-restbase-c-codfw
cassandra-restbase-c-eqiad
cassandra-sessions-a-codfw
cassandra-sessions-a-eqiad
druid-analytics
druid-analytics-test
druid-public
elasticsearch-cloudelastic-chi-eqiad-cloudelastic
elasticsearch-cloudelastic-omega-eqiad-cloudelastic
elasticsearch-cloudelastic-psi-eqiad-cloudelastic
elasticsearch-production-search-codfw-cirrus
elasticsearch-production-search-eqiad-cirrus
elasticsearch-production-search-omega-codfw-cirrus
elasticsearch-production-search-omega-eqiad-cirrus
elasticsearch-production-search-psi-codfw-cirrus
elasticsearch-production-search-psi-eqiad-cirrus
elasticsearch-relforge-eqiad-relforge
elasticsearch-relforge-eqiad-small-alpha-relforge
hadoop-master-analytics
hadoop-master-analytics-test
hadoop-worker-analytics
hadoop-worker-analytics-test
kafka-jumbo-eqiad
kafka-logging-codfw
kafka-logging-eqiad
kafka-main-codfw
kafka-main-eqiad
kafka-test-eqiad
kerberos-kdc
mariadb-analytics-meta
opensearch-datahubsearch
postgresql-analytics
presto-analytics
presto-analytics-test
redis-6378-misc
redis-6379-misc
redis-6380-misc
redis-6381-misc
redis-6382-misc
zookeeper-analytics-eqiad
zookeeper-druid-analytics-eqiad
zookeeper-druid-analytics-test-eqiad
zookeeper-druid-public-eqiad
zookeeper-flink-codfw
zookeeper-flink-eqiad
zookeeper-main-codfw
zookeeper-main-eqiad

Testing a Chart

Our CI will run a set of tests to validate a chart. In order to do that locally you will need:

Python 3.9 or higher
python3-yaml linux package or PyYAML (via pip)
Ruby 3.0 or higher
rake, a ruby make-like utility
helm3
docker or another container engine

To ensure the validity of our templates, we use two sets of fixtures.

fixtures under charts/mychart/.fixtures for various test cases
fixtures under .fixtures providing general data (eg mariadb sections, kafka_brokers etc) for misc enviroments (staging, eqiad, etc)
- run rake refresh_fixtures to get the latest ones from our puppet repo.

Quick tests

To quickly test against a specific fixture:

helm template \ 
  --values .fixtures/general-eqiad.yaml \
  -f charts/mychart/.fixtures/tls_enabled.yaml \
  charts/mychart \
  --debug

CI tests

To Validate against our CI:

rake run_locally["check_charts[lint/validate\,mychart]"]