Skip to main content

Object Storage Setup

Pre-requisite​

This guide assumes you have your own MinIO Cluster setup on a remote location.

Setup VictoraMetrics - Bandwidth Accounting​

We will setup VictoriaMetrics a time-series database which will store those metrics in the TSDB for us to use for querying purposes.

VictoriaMetrics - Github

You can look for a file named victoria-metrics-linux-amd64-vXXX.tar.gz and vmutils-linux-amd64-vXXX.tar.gz

And download the same on your linux server, for our use case at the time of writing this guide, we will go with the assumption that it's v1.126.0

wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.126.0/victoria-metrics-linux-amd64-v1.126.0.tar.gz -O /tmp/vm.tar.gz
tar -zxvf /tmp/vm.tar.gz
cp victoria-metrics-prod /usr/local/bin/victoria-metrics-prod
chmod +x /usr/local/bin/victoria-metrics-prod

Now we will prepare a systemd file for running the executable using the following content

#/etc/systemd/system/victoriametrics.service
[Unit]
Description=High-Permance, scalable time series database for prometheus
After=network.target

[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/victoria-metrics-prod -storageDataPath=/mnt/victoria_metrics -httpListenAddr=127.0.0.1:8429 -retentionPeriod=3 -search.disableAutoCacheReset
ExecStop=/bin/kill -s SIGTERM 
LimitNOFILE=65536
LimitNPROC=32000

[Install]
WantedBy=multi-user.target
info

-retentionPeriod flag is important, this will decide how many months of historical data do you want to maintain.

TSDB Authentication Layer​

Now we will download the VM Utilities, package, which will have 2 files that we will need. vmauth-prod & vmagent-prod

wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.126.0/vmutils-linux-amd64-v1.126.0.tar.gz -O /tmp/vmutils.tar.gz
tar -zxvf /tmp/vmutils.tar.gz 
cp vmauth-prod /usr/local/bin/
cp vmagent-prod /usr/local/bin/

We will now create another systemd unit file for running this service

#/etc/systemd/system/vmauth.service
[Unit]
Description=Proxy for victoria-metrics
After=network.target

[Service]
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/vmauth-prod -auth.config=/etc/vmauth.yaml -httpListenAddr=0.0.0.0:8428
ExecStop=/bin/kill -s SIGTERM $MAINPID


[Install]
WantedBy=multi-user.target

We will then create the following authentication config for the above service.

#/etc/vmauth.yaml
users:
  - username: 'grafana-write'
    password: 'somerandompassword'
    url_prefix: 'http://127.0.0.1:8429'

Setup MinIO metrics endpoint​

The following command would give you a json output with the bearer token, that's all we need for now.

mc admin prometheus generate <ALIAS> bucket --json

Setting up Scraping​

info

You can setup vmagent-prod on the same server as minio-cluster or some server that has access to the said minio-cluster directly

#/etc/systemd/system/vmagent.service
[Unit]
Description=Proxy for prometheus style scraping
After=network.target

[Service]
Restart=always
RuntimeMaxSec=1d
RestartSec=5
ExecStart=/usr/local/bin/vmagent-prod -promscrape.configCheckInterval=10s \
-httpListenAddr=127.0.0.1:8430 \
-promscrape.config=/etc/scrape.yaml \
-remoteWrite.url=http://<vmauth or victoriametrics ip>:8428/api/v1/write \
-httpAuth.username "grafana-write" \
-httpAuth.password "somerandompassword"
ExecStop=/bin/kill -s SIGTERM $MAINPID

[Install]
WantedBy=multi-user.target
#/etc/scrape.yaml
scrape_configs:
  - job_name: minio-job
    bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoiYWRtaf4iLGJleHAiOjQ5MTA4NTg3ODl9.gagR4U2KfvRj6zPUYqVIDXkh9ND6dsJem7XhcqY_WJgdmpzTuS7RL8Xj62mEvuPlDnU0cdk2u-_TihoMZWbi1Q
    metrics_path: /minio/v2/metrics/cluster
    scheme: http
    static_configs:
      - targets: ['<your-minio-cluster>:<port>']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance

Install MC client on your Management Server​

curl https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/local/bin/mc
chmod +x /usr/local/bin/mc
mc --help

Adding the Minio Server in your Management Server​

Add Server

  • Host Your Host Address for MinIO Cluster
  • Port It can be 80 or 443 if you have a reverse-proxy setup like nGINX or directly your MinIO API port which is 9000 by default
  • Use SSL If SSL certificate is present, then use this.
  • Access Key The admin username or a username with Administrator rights.
  • Secret Key The password for the above user
  • VictoriaMetrics URL This would be your victoriametrics url in our case http://<your-victoriametrics-ip>:8428/
  • VictoriaMetrics Username The username your put in vmauth-prod
  • VictoriaMetrics Password The password your put in vmauth-prod

Rest is all self-explanatory and you are good to go.