How to install VerneMQ MQTT broker in GKE

IoT
How to install VerneMQ MQTT broker in GKE
Author

Rafa Sanchez

Published

August 25, 2022

Clone and builc image locally

Clone the official repo and build locally:

git clone https://github.com/vernemq/docker-vernemq
cd docker-vernemq/
docker build -t docker-vernemq .

See the local image and run locally to check for errors. Note you must accept the EULA terms by setting env variable properly:

docker image ls
docker run -p 1883:1883 -e "DOCKER_VERNEMQ_ACCEPT_EULA=yes" docker-vernemq

docker run -e DOCKER_VERNEMQ_ACCEPT_EULA=“yes” europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/vernemq/docker-vernemq:latest

Push image to cloud

Push to your container to Google Cloud Artifact Registry:

gcloud auth configure-docker europe-west4-docker.pkg.dev
docker tag docker-vernemq europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/vernemq/docker-vernemq
docker push europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/vernemq/docker-vernemq

Run the image in the cloud:

docker run -e DOCKER_VERNEMQ_ACCEPT_EULA="yes" europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/vernemq/docker-vernemq:latest

If you want to avoid accepting the EULA terms every time you run the container, you must build from source.

Create GKE cluster

Create GKE cluster and obtain credentials to work with kubectl:

gcloud container clusters create vernemq-mqtt-broker --num-nodes=1 --zone=europe-west4 \
--network=argolis-rafaelsanchez-vpc-network --tags=vernemq
gcloud container clusters get-credentials vernemq-mqtt-broker --region=europe-west4

Open firewall ports for the MQTT broker:

gcloud compute firewall-rules create vernemq-mqtt --network default --allow=tcp:1883,tcp:8883,tcp:8080,tcp:8888 \
--source-ranges 0.0.0.0/0 --target-tags vernemq

Deploy VerneMQ app on GKE. The simple way

Apply the following two yaml files: vernemq.yaml and service.yaml with these commands:

kubectl apply -f vernemg.yaml service.yaml

Content of vernemq.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: broker-verne-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      run: bkr-verne
  template:
    metadata:
      labels:
        run: bkr-verne
    spec:
      containers:
        - name: broker-verne
          image: europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/vernemq/docker-vernemq:latest
          ports:
            - containerPort: 80
            - containerPort: 1883
            - containerPort: 8883
            - containerPort: 8888
            - containerPort: 8080
          env:
            - name: DOCKER_VERNEMQ_LOG__CONSOLE
              value: 'both'
            - name: DOCKER_VERNEMQ_LOG__CONSOLE__LEVEL
              value: 'debug'
            - name: DOCKER_VERNEMQ_ACCEPT_EULA
              value: 'yes'
            - name: DOCKER_VERNEMQ_USER_student
              value: 'm2mcomm'

Content of service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: my-lb-service
spec:
  type: LoadBalancer
  selector:
    run: bkr-verne
  ports:
  - name: mqtt
    port: 1883
    targetPort: 1883
  - name: mqtts
    port: 8883
    targetPort: 8883
  - name: mqtt-ws
    port: 8080
    targetPort: 8080
  - name: http
    port: 8888
    targetPort: 8888

Deploy VerneMQ app on GKE. With namespaces and RBAC

Option 1: Following the documentation:

curl -L https://codeload.github.com/vernemq/vmq-operator/zip/master --output repo.zip; unzip -j repo.zip '*/examples/only_vernemq/*' -d only_vernemq; kubectl apply -f only_vernemq

Option 2: four yaml files will be deployed: deployment, service (Load Balancer),

# Create namesapce
kubectl apply -f  0vernemq-operator-0namespace.yaml
# Bind service account system:serviceaccount:default:default (which is the default account bound to Pod) with role cluster-admin.
kubectl apply -f fabric8-rbac.yaml
kubectl apply -f verne.yaml
kubectl apply -f vernemq-service.yaml

Manifest verne.yaml. Note you must accept the EULA terms by setting env variable properly:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: broker-verne-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      run: bkr-verne
  template:
    metadata:
      labels:
        run: bkr-verne
    spec:
      containers:
        - name: broker-verne
          image: europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/vernemq/docker-vernemq:latest
          ports:
            - containerPort: 80
            - containerPort: 1883
            - containerPort: 8883
            - containerPort: 8888
            - containerPort: 8080
          env:
            - name: DOCKER_VERNEMQ_LOG__CONSOLE
              value: 'both'
            - name: DOCKER_VERNEMQ_LOG__CONSOLE__LEVEL
              value: 'debug'
            - name: DOCKER_VERNEMQ_ACCEPT_EULA
              value: "yes"

Manifest of fabric8-rbac.yaml:

# NOTE: The service account `default:default` already exists in k8s cluster.
# You can create a new account following like this:
#---
#apiVersion: v1
#kind: ServiceAccount
#metadata:
#  name: <new-account-name>
#  namespace: <namespace>

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fabric8-rbac
subjects:
  - kind: ServiceAccount
    # Reference to upper's `metadata.name`
    name: default
    # Reference to upper's `metadata.namespace`
    namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Manifest of vernemq-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: my-lb-service
spec:
  type: LoadBalancer
  selector:
    run: bkr-verne
  ports:
  - name: mqtt
    port: 1883
    targetPort: 1883
  - name: mqtts
    port: 8883
    targetPort: 8883
  - name: mqtt-ws
    port: 8080
    targetPort: 8080
  - name: http
    port: 8888
    targetPort: 8888

Manifest of ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: neg-demo-ing
spec:
  defaultBackend:
    service:
      name: neg-svc # Name of the Service targeted by the Ingress
      port:
        number: 1883 # Should match the port used by the Service

Manifest of 0vernemq-operator-0namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: messaging

Annex: install the VerneMQ image in a VM (create-with-container)

gcloud compute instances create-with-container vernemq     --container-image europe-west4-docker.pkg.dev/argolis-rafaelsanchez-ml-dev/vernemq/docker-vernemq:latest --network=argolis-rafaelsanchez-vpc-network

File-based authentication

You can set up File Based Authentication by adding users and passwords as environment variables as follows:

DOCKER_VERNEMQ_USER_=‘password’

where is the username you want to use. This can be done as many times as necessary to create the users you want. The usernames will always be created in lowercase

References

[1] https://medium.com/geekculture/part-2-vernemq-acl-using-mysql-database-and-cluster-setup-288fb3d2eff8
[2] https://smedegaard.io/mqtt-broker-on-digital-ocean/