Introduction

If you have a couple of servers in your homelab and have set up monitoring, chances are you’ve implemented alerting to notify you when something doesn’t go as expected. Two of the most popular options for managing alerts are Grafana Alerts and Prometheus Alertmanager. Unless you have very specific requirements for alert grouping or silencing—or you don’t use Grafana for visualization—Prometheus Alertmanager might not be your first choice.

For my homelab, where I monitor my Raspberry Pis, I chose to configure Grafana Alerts. Since this is just a personal setup and I don’t use integrations with popular tools like Slack or Teams, I opted for the straightforward email integration. Or so I thought! It turned out not to be as trivial as expected when I realized I didn’t have an SMTP server to send the emails 😄.

That’s when I decided to set up AWS SES to get quick access to an SMTP server. I wrote this article because I couldn’t find a clear, straightforward guide on configuring Grafana Alerts’ email integration with AWS SES.

Setup

I installed Prometheus and Grafana in my k3s cluster using the popular kube-prometheus-stack helm chart. This choice provided a quick and effective way to gain insights into my cluster nodes and pods, thanks to the default setup that includes node-exporter for node metrics and comprehensive baseline dashboards. As part of the helm values, I disabled the alertmanager installation given that I am planning to use Grafana Alerts.

The grafana.ini configuration is stored in the prom-grafana ConfigMap within the namespace where the Helm chart is installed. For testing purposes, we can modify this ConfigMap and restart the prom-grafana pod. Once the testing is complete, we can incorporate the updated grafana.ini configuration into a new Helm release by passing it as a value. Before updating the [smtp] section in grafana.ini, we first need the details of the SMTP server.

AWS SES is a reliable email platform that provides an SMTP server for sending emails using your own domain. It is straightforward to set up but requires a domain that can be verified. In my case, I used the domain I host my website on.

You need to go to the AWS SES setup page and create a new deployment. After the SMTP server is deployed, several tasks will be made available for you to complete before you can use the service to send emails. The mandatory ones are ‘Verify email address’ and ‘Verify sending domain’.

AWS SES will require you to verify any email that you use to send/receive email from while your SES deployment is still in ‘sandbox’ environment. The difference between ‘sandbox’ and ‘production’ environment is that the former has quotas limits imposed, however it should be enough for this demo or any homelab environment where you do not plan to send more than 200 emails in 24 hours. When you start the ‘Verify email address’ task, you will receive an email from AWS with a verification link. Once your click it your email will be available to use in SES.

After you verified your email address, you should verify the domain that you are sending from. SES will ask you to add 4 DNS records to your hosted zone: one TXT record containing DMARC settings and 3 CNAME records containing DKIM records. There are good articles from Cloudflare explaining DMARC records and DKIM records. Usually the DKIM records are in TXT format however AWS SES uses CNAME records pointing to other records containing the DKIM public key.

Having completed the mandatory tasks, you might now be curious about generating the actual SMTP credentials needed for our Grafana configuration. The process for creating these credentials is detailed in this guide. By clicking the “Generate SMTP credentials” button, AWS will prompt you to specify a name for an IAM user, which it will then create with access to your SMTP server. The resulting SMTP username and password will correspond to the access key and secret access key of the newly created IAM user. Make sure to save them after you create the user because you will not be able to access the secret key again (unless you create a new user).

Demo

With our SES configured and SMTP credentials created, we are in a good place to add the SMTP details in the grafana.ini file. In my case, given that Grafana is a deployment as part of the kube-prometheus-stack helm chart, I added the below to the existing prom-grafana ConfigMap:

    [smtp]
    enabled = true
    host = email-smtp.us-east-1.amazonaws.com:465
    user = SES_USER_ACCES_KEY
    password = SES_USER_SECRET_ACCESS_KEY
    startTLS_policy = MandatoryStartTLS
    from_address = grafana@alexstan.cloud
    from_name = Grafana

After restarting the pod to pick up the latest ConfigMap changes, I went to the Grafana UI and created a new contact point:

alt

When pushing a test alert, I could see an email from Grafana, having the same configuration which I setup in the grafana.ini file:

alt

Conclusion

I hope that this quick tutorial provided enough details on how to configure email alerts in Grafana Alerts using AWS SES as an SMTP server. Unless you have noisy alerts, you will probably incurr no costs for this given that you have a free quota of 3k emails sent per month. Until next time, happy learning!