Integrate Prometheus

Learn how to configure Prometheus to get metrics from the Armory Scale Agent service and display them in a Grafana dashboard.

Available metrics

If prometheus.enabled is true in the Armory Scale Agent service configuration, the Armory Scale Agent exposes metrics on port 8008 (prometheus.port) on path /metrics or /prometheus_metrics like other Armory CD services. Both paths serve the same data.

go_gc_duration_secondssummaryAmount of time spent in garbage collection.
go_goroutinesgaugeNumber of go routines.
go_memstats_alloc_bytesgaugeAmount of memory allocation used by Go.
go_memstats_alloc_bytesgaugeNumber of bytes allocated and still in use.
go_memstats_alloc_bytes_totalcounterTotal number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytesgaugeNumber of bytes used by the profiling bucket hash table.
go_memstats_frees_totalcounterTotal number of frees.
go_memstats_gc_cpu_fractiongaugeThe fraction of this program's available CPU time used by the GC since the program started.
go_memstats_gc_sys_bytesgaugeNumber of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytesgaugeNumber of heap bytes allocated and still in use.
go_memstats_heap_idle_bytesgaugeNumber of heap bytes waiting to be used.
go_memstats_heap_inuse_bytesgaugeNumber of heap bytes that are in use.
go_memstats_heap_objectsgaugeNumber of allocated objects.
go_memstats_heap_released_bytesgaugeNumber of heap bytes released to OS.
go_memstats_heap_sys_bytesgaugeNumber of heap bytes obtained from system.
go_memstats_last_gc_time_secondsgaugeNumber of seconds since 1970 of last garbage collection.
go_memstats_lookups_totalcounterTotal number of pointer lookups.
go_memstats_mallocs_totalcounterTotal number of mallocs.
go_memstats_next_gc_bytesgaugeNumber of heap bytes when next garbage collection will take place.
go_memstats_sys_bytesgaugeNumber of bytes obtained from system.
go_threadsgaugeNumber of Go threads.
kubesvc_connection_countcounterNumber of connections.
kubesvc_disconnection_totalcounterNumber of disconnections.
kubesvc_events_bytes_sent_totalcounterAmount of data sent by caching events.
kubesvc_events_sent_totalcounterNumber of caching events sent.
kubesvc_resource_agent_totalcounterNumber of watched/polled resources.
process_cpu_seconds_totalcounterTotal user and system CPU time spent in seconds.
process_max_fdsgaugeMaximum number of open file descriptors.
process_open_fdsgaugeNumber of open file descriptors.
process_resident_memory_bytesgaugeResident memory size in bytes.
process_start_time_secondsgaugeStart time of the process since unix epoch in seconds.
process_virtual_memory_bytesgaugeVirtual memory size in bytes.
process_virtual_memory_max_bytesgaugeMaximum amount of virtual memory available in bytes.

Configure Prometheus

If you are using the Prometheus operator, you can scrape metrics with:

kind: ServiceMonitor
    app: spin
  name: armory-agent-service-monitor
    - path: /prometheus_metrics
      port: metrics
#    - path: /metrics
#      port: metrics
    app: spin

Import a Grafana dashboard

You can import this Grafana dashboard definition to use with Prometheus.

Last modified March 3, 2023: (2d069084)