> For the complete documentation index, see [llms.txt](https://utm-1.gitbook.io/utm-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://utm-1.gitbook.io/utm-docs/dokumentaciya/utm-it/resheniya/kubernetes/prilozheniya/monitoring/prometheus-+-grafana.md).

# Prometheus + Grafana

## Ссылки

Мониторинг в K8s с помощью Prometheus: [selectel.ru](https://selectel.ru/blog/tutorials/monitoring-in-k8s-with-prometheus/?ysclid=mhuck4cvuz801334568)

Prometheus - Configuration: [prometheus.io](https://prometheus.io/docs/alerting/latest/configuration/)

Мониторинг кластера Kubernetes: общий обзор и знакомство с Prometheus: [habr.com](https://habr.com/ru/companies/slurm/articles/516748/?ysclid=mhuck8hs60282121613)

Основы мониторинга (обзор Prometheus и Grafana): [habr.com](https://habr.com/ru/articles/709204/?ysclid=mhucn62y9n467442368)

Путь в GitOps или как мы перевели кластер Kubernetes под управление Argo CD: [habr.com](https://habr.com/ru/articles/842934/)

AzamatKomaev / argo-demo-habr: [github.com](https://github.com/AzamatKomaev/argo-demo-habr/tree/main/apps/monitoring)

Вас много, а я одна: обзорная система мониторинга на Prometheus и Grafana: [habr.com](https://habr.com/ru/companies/doubletapp/articles/736602/?ysclid=mhxinw1oyw13899751)

## Установка

### Установка через Argo CD

В github в репозитории argocd в каталоге `env-prod\apps` создать подкаталог `monitoring\prometheus` в котором создать файл `app.yaml` следующего содержания:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name     : k8s-prometheus
  namespace: argocd
spec:
  project: default
  destination:
    name     : in-cluster
    namespace: monitoring
  sources:
  - repoURL: git@github.com:<user-code>/argocd.git
    targetRevision: HEAD
    ref: values
  - repoURL: git@github.com:<user-code>/argocd.git
    targetRevision: HEAD
    path: "env-prod/apps/monitoring/prometheus/manifests_before"
  - chart: kube-prometheus-stack
    repoURL: https://prometheus-community.github.io/helm-charts
    targetRevision: 79.4.1
    helm:
      releaseName: prometheus-stack
      valueFiles:
      - $values/env-prod/apps/monitoring/prometheus/values.yaml
  - repoURL: git@github.com:<user-code>/argocd.git
    targetRevision: HEAD
    path: "env-prod/apps/monitoring/prometheus/manifests_after"
  syncPolicy:
    automated:
      prune   : true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
```

В каталоге `monitoring\prometheus` создать подкаталог `manifests_before` в котором создать следующие файлы:

**monitoring.yaml**

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
```

**kube-prometheus-stack-grafana.yaml**

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: kube-prometheus-stack-grafana
  namespace: monitoring
type: Opaque
data:
  admin-user: <имя пользователя в base64>
  admin-password: <пароль в base64>
  # example
  # echo -n "<data>" | base64
```

telegram-bot-token-secret.yaml

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: telegram-bot-token-secret # secret name will be referred in botToken in AlertmanagerConfig
  namespace: monitoring # same `namespace` as the `AlertmanagerConfig`.
type: Opaque
data:
  # bot-token will be used as key for botToken in AlermanagerConfig.
  bot-token: <токен в base64>
  # example
  # echo -n "<BOT_ACCESS_TOKEN>" | base64 (Token without bot prefix, exactly the token get from bot_father)
```

В каталоге `monitoring\prometheus` создать подкаталог `manifests_after` в котором создать следующие файлы:

**alert-config-telegram.yaml**

```yaml
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alert-config-telegram
  namespace: monitoring
  labels:
    release: prometheus-stack # the release label should be the exact label of the prometheus helm release label
    alertmanagerConfig: alertmanager-telegram # custom label and also set this label to alertmanager in the prometheus-stack values.yaml
spec:
  route:
    groupBy: ["alertname","job","namespace"] # it could be other labels to be grouped by
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 1h
    receiver: "telegram"
    routes:
    - receiver: "telegram"
      matchers:
        - name: severity
          matchType: "=~"
          value: "warning|critical"
  receivers:
    - name: telegram
      telegramConfigs:
        - botToken:
            name: telegram-bot-token-secret # refer to the secret name has been created before
            key: bot-token # bot-token data in the secret
          chatID: <id чата>
          parseMode: 'MarkdownV2' #'html'  #'MarkdownV2' # The template I provide is in Markdown Mode
          disableNotifications: false
          sendResolved: true
```

k8s-grafana-ingress.yaml

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: k8s-grafana-ingress
  namespace: monitoring
  annotations: {}

spec:
  ingressClassName: nginx
  rules:
  - host: <url сайта>
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: prometheus-stack-grafana
            port: 
              number: 80
```

В каталоге `monitoring\prometheus` внести изменения в файл `values.yaml` (базовый файл взять [отсюда](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack))

{% hint style="info" %}
Всё что необходимо изменить в файле `values.yaml` смотри в репозитории argocd в реальном файле. Все изменения помечены меткой `TEST!!!`
{% endhint %}

Сделать коммит в github. После этого Argo CD развернет Prometheus + Grafana в связанном кластере Kubernetes.

В Grafana заходить под логином и паролем, указанном в файле `monitoring\prometheus\manifests_before\kube-prometheus-stack-grafana.yaml`

## Информация

#### Прокси <a href="#command-line-proxy" id="command-line-proxy"></a>

**prometheus:**

```
kubectl -n monitoring port-forward svc/prometheus-stack-kube-prom-prometheus 9090:9090
```

[http://localhost:9090](http://localhost:9090/)

**grafana:**

```bash
kubectl -n monitoring port-forward svc/prometheus-stack-grafana 80:80
```

[http://localhost:80](http://localhost/)

**alertmanager:**

```bash
kubectl -n monitoring port-forward svc/prometheus-stack-kube-prom-alertmanager 9093:9093
```

[http://localhost:9093](http://localhost:9093/)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://utm-1.gitbook.io/utm-docs/dokumentaciya/utm-it/resheniya/kubernetes/prilozheniya/monitoring/prometheus-+-grafana.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.