Installation

Download page: https://prometheus.io/download/

apt install prometheus  # prometheus & prometheus node exporter

• Prometheus
  • http://localhost:9090/metrics
• Prometheus Node Exporter
  • http://localhost:9100/metrics

service prometheus status
service prometheus-node-exporter status
ps -u prometheus
id prometheus
systemctl enable prometheus.service  # my guess: reboot config
ss -ntlp | grep prometheus

Access:

• http://ip:9090
• http://ip:9090/metrics
• http://ip:9100
• http://ip:9100/metrics
  • prometheus will pull this url

reverse proxy with nginx:

apt install nginx
cd /etc/nginx/sites-enabled
vim prometheus

server {
    listen 80;
    listen [::]:80;
    server_name  YOUR-DOMAIN-NAME;

    location / {
        proxy_pass           http://localhost:9090/;
    }
}

nginx -t
service nginx restart
service nginx status

Add HTTPS:

snap install core
snap refresh core
snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot
certbot --nginx  # Now we can run CertBot.

Add basic authentication:

cd /etc/nginx/
apt install apache2-utils
htpasswd -c /etc/nginx/.htpasswd admin  # password

open /etc/nginx/sites-enabled/prometheus:

server {
    ...

    #additional authentication properties
    auth_basic  "Protected Area";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass           http://localhost:9090/;
    }

    ...
}

update:

nginx -t
service nginx restart
service nginx status

Drop external port:

iptables -A INPUT -p tcp -s localhost --dport 9090 -j ACCEPT
iptables -A INPUT -p tcp --dport 9090 -j DROP
iptables -A INPUT -p tcp -s localhost --dport 9100 -j ACCEPT
iptables -A INPUT -p tcp --dport 9100 -j DROP
iptables -L

iptables settings will be lost in case of system reboot. You will need to reapply them manually,

or

install iptables-persistent

apt install iptables-persistent

This will save your settings into two files called,

/etc/iptables/rules.v4

/etc/iptables/rules.v6

Any changes you make to the iptables configuration won’t be auto saved to these persistent files, so if you want to update these files with any changes, then use the commands,

iptables-save > /etc/iptables/rules.v4
iptables-save > /etc/iptables/rules.v6

On Ubuntu, the Prometheus configuration file is typically located at: /etc/prometheus/prometheus.yml.

Other Operations

service prometheus restart

Install External Node

apt install prometheus-node-exporter
service prometheus-node-exporter status

Edit Prometheus server config, edit it, then check:

promtool check config /etc/prometheus/prometheus.yml
service prometheus restart

What Is Prometheus?

Scraping

Service discovery and relabeling give us a list of targets to be monitored. Now Prometheus needs to fetch the metrics. Prometheus does this by sending an HTTP request called a scrape. The response to the scrape is parsed and ingested into storage. Several useful metrics are also added in, such as if the scrape succeeded and how long it took. Scrapes happen regularly; usually you would configure it to happen every 10 to 60 seconds for each target.

What Prometheus Is Not

Prometheus is not suitable for storing event logs or individual events. Nor is it the best choice for high-cardinality data, such as email addresses or usernames.

Prometheus is designed for operational monitoring, where small inaccuracies and race conditions due to factors like kernel scheduling and failed scrapes are a fact of life. Thus in applications involving money or billing, Prometheus should be used with caution.

Prometheus’s Data Model

It has three main data types that are all built on each other: metrics, time series, and samples.

Metrics

Every metric has – at a minimum – a name that identifies it. This name can contain letters, digits, underscores, and/or colons. To be valid, it must match the [a-zA-Z_:][a-zA-Z0-9_:]* regex.

Additionally, metrics may also include a HELP text and a TYPE field. These are optional but highly recommended to improve usability.

A full metric that’s been exposed to Prometheus may look like this:

# HELP mastering_prometheus_readers_total Number of readers of this book
# TYPE mastering_prometheus_readers_total counter
mastering_prometheus_readers_total 123467890

The type doesn’t affect how the data is stored in Prometheus’s TSDB at all – it’s all the same under the hood. However, all of the Prometheus client libraries for instrumenting applications use the same common metric types to help organize metrics into different classifications. Additionally, it makes it easier to reason about which PromQL functions to use with a metric. At the time of writing, there are four core metric types in Prometheus: counters, gauges, histograms, and summaries.

Counters

They start at zero and can only go up during a process’s lifetime. They can easily be identified by the semantic convention of ending their metric name in a _total suffix. Possible usages include error counters or counting the number of requests received by a web server.

Gauges

Gauges are similar to counters in their simplicity, but they allow for values that can go up or down. Possible uses include tracking active connections to a web server or the memory usage of a system. Additionally, they are commonly used to track Boolean values such as in the built-in up metric for Prometheus scrape jobs, where a value of 0 indicates a scrape job is down and a value of 1 means it is up. Finally, gauges are also frequently used for information style metrics, which is a pattern of defining metrics with a info suffix, where the value is always set to 1 and the labels provide additional information about an application or server. An example of this is Prometheus’s prometheus build_info metric, which – among other things – includes labels such as version to report the running Prometheus version.

Histograms

A histogram samples observations of values and counts them in configurable buckets. In a way, a histogram is a collection of multiple counters.

Summaries

Summaries are similar to histograms in that they also are derived from observed values. However, they do some precomputation to expose special quantile labels instead of le labels.

Time series

One metric can have dozens of time series. But how? Through the usage of labels!

Labels are key/value pairs that add additional detail to a metric.

Metric	node_cpu_seconds_total
Time Series	node_cpu_seconds_total{cpu=”0″,instance=”serverA”}

A metric is just the name – nothing within the curly braces (although, as we’ll see later, the name itself is stored as a hidden label). If you’re familiar with object-oriented programming, you can think of a metric as a base class and a time series as a unique instance of that class. That uniqueness is also important.

All series that are exposed to Prometheus in a scrape job must be unique, so series with the same metric name should be grouped under a single HELP/TYPE definition and have unique labels. If they are not unique, the Prometheus scrape will fail. For example, the following would be invalid since the time series is duplicated without unique labels:

# HELP mastering_prometheus_readers_total Number of readers of this book
# TYPE mastering_prometheus_readers_total counter 
mastering_prometheus_readers_total 123467890 
mastering_prometheus_readers_total 789

This would be valid:

# HELP mastering_prometheus_readers_total Number of readers of this book
# TYPE mastering_prometheus_readers_total counter 
mastering_prometheus_readers_total{stage="published"} 123467890 
mastering_prometheus_readers_total{stage="draft"} 789

Time series are tracked over time through a series of samples, which brings us to our final data type: samples.

Samples

Samples are built on top of time series, just like how time series are built on top of metrics. A metric isn’t much use without a time series that tracks it, and a time series isn’t much use without samples that represent its values over time.

Each time that Prometheus scrapes a target, it collects samples for all of the time series returned by that scrape target. A sample is extremely simple: it’s just a tuple of a timestamp and a value. Commonly, samples are denoted using the (t0, v0), (t1, v1), . . . format, where tX is a timestamp and vX is a value.

Values are represented in a float64 type in Go (this means it has a decimal point) and timestamps are represented using the number of milliseconds since the Unix epoch (January 1, 1970).

Prometheus’s TSDB

Labels

The job label indicates the type of application.

PromQL

There are only four possible types that PromQL expressions can be evaluated as. An expression or sub-expression can evaluate:

• Instant vector – a set of time series containing a single sample for each time series, all sharing the same timestamp
• Range vector – a set of time series containing a range of data points over time for each time series
• Scalar – a simple numeric floating point value
• String – a simple string value; currently unused

Both instant and range vectors have similar basic syntax consisting of a metric name and/or label matchers.

A metric name in Prometheus is a label named name under the hood, but PromQL has syntax niceties that allow you to just specify the metric name as the start of the selector. For example, the following are equivalent and return the same data:

• Explicit label matcher on metric name

{__name__="prometheus_build_info"}

• Equivalent query without an explicit label matcher syntax

prometheus_build_info

Label matchers are defined using a label key, a label matching operator, and the label value. Here are some possible label-matching operators:

• =
• !=
• =~: The label value matches the regex in the specified string
• !~: The label value does not match the regex in the specified string

References

• Mastering Prometheus
• Prometheus Up and Running
• https://sbcode.net/prometheus/