Installing Grafana and Prometheus on Amazon EKS

Contents

I had used ELK and GitLab as operational tools at work, but had never set them up from scratch. This gave me the chance to do so.
I have not even used Grafana or Prometheus before, so this installation feels a little different.

0. Install kubectl (link)

1. Install Helm (link)

2. Create an EKS Cluster

bash
root@gphost:~# cat cluster.yml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: gp-test
  region: ap-southeast-1
nodeGroups:
  - name: ng-1
    instanceType: t3.medium
    desiredCapacity: 2
    volumeSize: 10
root@gphost:~# eksctl create cluster -f cluster.yml
2022-12-18 10:28:12 [ℹ]  eksctl version 0.122.0
2022-12-18 10:28:12 [ℹ]  using region ap-southeast-1
2022-12-18 10:28:12 [ℹ]  setting availability zones to [ap-southeast-1b ap-southeast-1c ap-southeast-1a]
2022-12-18 10:28:12 [ℹ]  subnets for ap-southeast-1b - public:192.168.0.0/19 private:192.168.96.0/19
2022-12-18 10:28:12 [ℹ]  subnets for ap-southeast-1c - public:192.168.32.0/19 private:192.168.128.0/19
2022-12-18 10:28:12 [ℹ]  subnets for ap-southeast-1a - public:192.168.64.0/19 private:192.168.160.0/19
...
...
2022-12-18 10:46:58 [ℹ]  nodegroup "ng-1" has 0 node(s)
2022-12-18 10:46:58 [ℹ]  waiting for at least 2 node(s) to become ready in "ng-1"
2022-12-18 10:48:11 [ℹ]  nodegroup "ng-1" has 2 node(s)
2022-12-18 10:48:11 [ℹ]  node "ip-192-168-20-2.ap-southeast-1.compute.internal" is ready
2022-12-18 10:48:11 [ℹ]  node "ip-192-168-87-227.ap-southeast-1.compute.internal" is ready
2022-12-18 10:48:13 [ℹ]  kubectl command should work with "/root/.kube/config", try 'kubectl get nodes'
2022-12-18 10:48:13 [✔]  EKS cluster "gp-test" in "ap-southeast-1" region is ready

3. Create an Amazon EBS CSI Driver IAM Role for the Service Account (link)

4. Manage the Amazon EBS CSI Driver as an Amazon EKS Add-on (link)

5. Create PersistentVolumes for grafana, prometheus-server, and alertmanager

6. Add Helm Repositories

bash
root@gphost:~# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories
root@gphost:~# helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories
root@gphost:~# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
root@gphost:~# 
root@gphost:~# 
root@gphost:~# helm repo list
root@gphost:~# 
NAME                    URL
prometheus-community    https://prometheus-community.github.io/helm-charts
grafana                 https://grafana.github.io/helm-charts
root@gphost:~# 
root@gphost:~# kubectl create ns prometheus
namespace/prometheus created

6. Deploy Prometheus

bash
root@gphost:~# helm install prometheus prometheus-community/prometheus -f values-prometheus.yaml -n prometheus
NAME: prometheus
LAST DEPLOYED: Sun Dec 18 12:59:38 2022
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.prometheus.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9090


The Prometheus alertmanager can be accessed via port  on the following DNS name from within your cluster:
prometheus-%!s(<nil>).prometheus.svc.cluster.local


Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Pod Security Policy has been disabled by default since    #####
######            it deprecated after k8s 1.25+. use                        #####
######            (index .Values "prometheus-node-exporter" "rbac"          #####
###### .          "pspEnabled") with (index .Values                         #####
######            "prometheus-node-exporter" "rbac" "pspAnnotations")       #####
######            in case you still need it.                                #####
#################################################################################


The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-prometheus-pushgateway.prometheus.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus-pushgateway,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/

7. Deploy Grafana

bash
root@gphost:~# helm install grafana grafana/grafana -f values-grafana.yaml -n prometheus
W1218 13:01:12.502934    6566 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1218 13:01:13.777265    6566 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: grafana
LAST DEPLOYED: Sun Dec 18 13:01:11 2022
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace prometheus grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.prometheus.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:
     export NODE_PORT=$(kubectl get --namespace prometheus -o jsonpath="{.spec.ports[0].nodePort}" services grafana)
     export NODE_IP=$(kubectl get nodes --namespace prometheus -o jsonpath="{.items[0].status.addresses[0].address}")
     echo http://$NODE_IP:$NODE_PORT

3. Login with the password from step 1 and the username: admin

8. Test Access

Post image 1


9. Troubleshooting

Creating EBS volumes felt tedious, so I created the resources in just one Availability Zone. The cluster did not automatically handle mounting them, though. I therefore created plenty of EBS volumes in the Availability Zones where the nodes were located. Each service also required a different volume size. I think Grafana needed a 100GiB volume, but I had created 10GiB EBS volumes, which seems to have prevented it from starting. In any case, resolving the EBS volume issues was enough to get it working.

10. References

https://may9noy.tistory.com/303

Diagram

100%

Drag to pan; pinch or use the wheel to zoom. Keyboard: arrow keys pan, + − zoom, 0 fits, Esc closes. The same content is in “Read the diagram as text” below the figure.