Time: 60 minutes
Audience: Developers & DevOps Personnel
Purpose: Introduce basic monitoring using Prometheus so you may apply new skills.
Goals:
Before beginning these exercises, it is assumed that the student has a certain level of familiarity with Linux and the command-line, as well as Git commands.
For this exercise, we will use a version of Linux (CentOS) provided by Hashicorp called a Bento Box. If you have experience with other virtual environments, you can use them.
Regardless of which environment is selected, it is assumed that some Linux command line skills are possessed.
If the selected environment is a physical environment, skip ahead to download the correct version of Prometheus for your OS. If a virtual environment is selected, please follow the following steps.
A host operating system is the OS the virtual environment will run on. The guest is the virtual environment.
vagrant --versionvagrant init bento/centos-7.6 2:
vagrant up 
vagrant ssh
It is important to put vagrant in front of the ssh command so it finds the key correctly to log in. Feel free to tour around inside the guest.
~/Vagrantfile on a Mac, or C:\Users\username. Look for the line: # config.vm.network "private_network", ip: "192.168.33.10". The comment # can be removed, or the line retyped. Any valid IP address can be used. 
vagrant reload.You will need to add the git application on the virtual machine.
vagrant ssh$ sudo yum install git -yTo demonstrate the Java metrics and Grafana, we will install a Docker container with some prebaked examples, but to run them, we need to install Docker.
$ sudo yum install -y yum-utils device-mapper-persistent-data lvm2$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo$ sudo yum install docker-ce$ sudo usermod -aG docker $(whoami)$ sudo systemctl enable docker.service$ sudo systemctl start docker.serviceFor this exercise we will not need Docker Compose.
vagrant plugin install vagrant-scp Then type: vagrant scp local_file_path_in_HostOS :remote_file_path_in_GuestOS. For example: vagrant scp prometheus-2.7.2.linux-amd64.tar.gz :~/ (If your guestOS is not running you will get an error message. You can type vagrant up to start it). 
tar -zxvf prometheus-2.7.2.linux-amd64.tar.gz
$ ./prometheus which will spin up a number of messages like:
And then type <CTRL>-C to stop it, for now.
Adding Java instrumentation to your code can be managed in a couple of ways. As we discussed in the class, we have the following examples.
import io.prometheus.client.Counter;
class YourClass {
static final Counter requests = Counter.build()
.name("requests_total").help("Total requests.").register();
void processRequest() {
requests.inc();
// Your code here.
}
}class YourClass {
static final Gauge inprogressRequests = Gauge.build()
.name("inprogress_requests").help("Inprogress requests.").register();
void processRequest() {
inprogressRequests.inc();
// Your code here.
inprogressRequests.dec();
}
}import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;
import io.prometheus.client.Histogram;
import io.prometheus.client.Summary;
import io.prometheus.client.exporter.HTTPServer;
import java.io.IOException;
import java.util.Random;
public class Main {
private static double rand(double min, double max) {
return min + (Math.random() * (max - min));
}
public static void main(String[] args) {
Counter counter = Counter.build().namespace("java").name("my_counter").help("This is my counter").register();
Gauge gauge = Gauge.build().namespace("java").name("my_gauge").help("This is my gauge").register();
Histogram histogram = Histogram.build().namespace("java").name("my_histogram").help("This is my histogram").register();
Summary summary = Summary.build().namespace("java").name("my_summary").help("This is my summary").register();
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.inc(rand(0, 5));
gauge.set(rand(-5, 10));
histogram.observe(rand(0, 5));
summary.observe(rand(0, 5));
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
try {
HTTPServer server = new HTTPServer(8080);
} catch (IOException e) {
e.printStackTrace();
}
}
}You can also modify the build to include the appropriate repository. For Maven, you edit your POM.XML something like this:
<!-- The client -->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>0.6.0</version>
</dependency>
<!-- Hotspot JVM metrics-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>0.6.0</version>
</dependency>
<!-- Exposition HTTPServer-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_httpserver</artifactId>
<version>0.6.0</version>
</dependency>
<!-- Pushgateway exposition-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_pushgateway</artifactId>
<version>0.6.0</version>
</dependency>For more examples, please consult the Prometheus Java Client page.
A Docker container with the examples above can be installed.
$ git clone https://github.com/sysdiglabs/custom-metrics-examples$ sudo docker build custom-metrics-examples/prometheus/java -t prometheus-javaDepending on the state of your Docker container, it may rebuild itself while it downloads. This will take a couple of minutes.
$ sudo docker run -d --rm --name prometheus-java -p 8080:8080 -p 80:80 prometheus-java $ curl localhost:8080 which will give you something like:
These are now exporting successfully from the application. Now we need to configure Prometheus.
prometheus.yml file. This is the core configuration file for Prometheus. This file is YAML (Yet Another Markup Language) and so spaces matter. Because our Java is being exported to port 8080, we will configure Prometheus to listen on 9090.
global:
scrape_interval: 10s
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090This is a basic configuration.
global: Defines the global parameters for Prometheus operation. The scrape interval is the time, in seconds, between system scrapes. Prometheus defaults to one minute [1m] as the time between scrapes. For demonstration purposes we will use ten seconds. The actual time should be a balance between need for metrics and load on the system.
scrape_configs: Define job name, labels, and other things that define the scrape. It scrapes http by default, but can scrape other protocols as configured. There is also the ability to pass usernames and passwords if needed. The entire documented suite of variable is available in the Prometheus documents.
For this exercise, we will scrape the localhost, port 9090, every 10 seconds. The job name is prometheus. Save the YAML above as prometheus.yml, then start Prometheus with $ ./prometheus.
Open a web browser and point it at: 192.168.33.10:9090 (or the address you configured in Step 10 in setting up Vagrant). This is the main visualization page. From here you can examine a number of default metrics collected by Prometheus. You can verify the host status by clicking on Status | Targets, which will show you:
If you click on the Prometheus name, you will return to the PromQL Expression Browser main screen where you any type any number of PromQL (the Prometheus Query Language) queries. For example, by typing up will result in a returned value of 1, for up. This is useful for debugging.
If you click on Insert Metric at Cursor, you can get an idea of the default metrics that can be viewed. You will notice there are no Java variables.
To add them:
global:
scrape_interval: 10s
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- localhost:8080
There are hundreds of configuration options for Prometheus, including hosts and other library metrics. For example, if you have number of hosts, you might see a configuration like this:
global:
scrape_interval: 10s
evaluation_interval: 10s
rule_files:
- rules.yml
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- 192.168.1.181:9090
- 192.168.1.183:9090
- 192.168.1.184:9090
- 192.168.1.181:8080
- job_name: node
static_configs:
- targets:
- localhost:9100
- 192.168.1.181:9100
- 192.168.1.183:9100
- 192.168.1.184:9100This configuration comes from a network with four instances of prometheus, collecting data from the node_exporter module (which exports metrics related to the OS, network, and other node related information. You would get a report like this if we aggregate network interfaces:
The Prometheus documentation provides a number of configuration suggestions, and there is a prometheus channel in Stack Overflow with just as many suggestions. There is also a great book available from O’Reilly’s Safari subscription Prometheus Up and Running.
The Prometheus dashboard is handy for quick sanity checks, but it is not a great solution for long-term use. Enter Grafana, a popular tool for building just these sorts of dashboards.
The easiest way to plug into our existing configuration is to grab a copy of the Grafana Docker console. This will not use a volume mount so it is not a good long term solution.
$ sudo docker run -d --name=grafana --net=host grafana/grafana:5.0.0
Grafana will present itself on port 3000. Log in as admin/admin
java_my_guage and you will get something like this being pulled from your Prometheus instance. 
Now that you have an overview of the pieces, take some existing code and modify it to include counters or gauges as appropriate, build it, and then run it where Prometheus can capture the metrics. Once you have done that, validate that they are being picked up by looking in the Prometheus web page and then in Grafana.
$ /home/vagrant/custom-metrics-examples/prometheus/java/src/main/java/Main.java in the custom-metrics-examples repository you clonedrate(node_network_receive_bytes_total[1m])