Configure Default Integrations

Each Monitoring Integration holds a specific job that scrapes its metrics and sends them to Sysdig Monitor. To optimize metrics scraping for building dashboards and alerts in Sysdig Monitor, Sysdig offers default jobs for these integrations. Periodically, the Sysdig agent connects with Sysdig Monitor retrieves the default jobs, and makes the Monitoring Integrations available for use. See the list of the available integrations and corresponding jobs.

You can find all the jobs in the /opt/draios/etc/promscrape.yaml file in the sysdig-agent container in your cluster.

Two types of integrations are found:

• Enabled by default: Upon installing the Sysdig Agent, the integration starts sending metrics to the Sysdig backend.
• Out of the box: An integration that requires no exporter or extra configuration to retrieve metrics from the application. The application is already orchestrated.

Supported Monitoring Integrations

Enable and Disable Integrations

Some integrations are disabled by default due to the potential high cardinality of their metrics. To enable or disable integrations, follow the below steps:

Customize a Default Job

The default jobs offered by Sysdig for integrations are optimized to scrape the metrics for building dashboards and alerts in Sysdig Monitor. Instead of processing all the metrics available, you can determine which metrics to include or exclude for your requirements. To do so, you can overwrite the default configuration in the prometheus.yaml file. The prometheus.yaml file is located in the sysdig-agent ConfigMap in the sysdig-agent namespace.

You can overwrite the default job for a specific integration by adding a new job to the prometheus.yaml file with the same name as the default job that you want to replace. For example, if you want to create a new job for the Apache integration, create a new job with the name apache-default. The jobs defined by the user has precedence over the default ones.

See Supported Monitoring Integrations for the complete list of integrations and corresponding job names.

Add Sysdig Annotations in Exporters

Sysdig provides a set of Helm charts that helps you configure the exporters for the integrations. For more information on installing Monitor Integrations, see the Monitoring Integrations option in Sysdig Monitor. Additionally, the Helm charts are publicly available in the Sysdig Helm repository.

If exporters are already installed in your cluster, you can use the standard Prometheus annotations and the Sysdig agent will automatically scrape them.

For example, if you use the annotation given below, the incoming metrics will have the information about the pod that generates the metrics.

spec:
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: '9100'
        prometheus.io/scrape: 'true'

If you use an exporter, the incoming metrics will be associated with the exporter pod, not the application pod. To change this behavior, you can use the Sysdig-provided annotations and configure the exporter on the agent.

Annotate the Exporter

Use the following annotations to configure the exporter:

spec:
  template:
    metadata:
      annotations:
        promcat.sysdig.com/port: '9187'
        promcat.sysdig.com/target_ns: my-namespace
        promcat.sysdig.com/target_workload_type: deployment
        promcat.sysdig.com/target_workload_name: my-workload
        promcat.sysdig.com/integration_type: my-integration

• port: The port to scrape for metrics on the exporter.
• target_ns: The namespace of the workload corresponding to the application (not the exporter).
• target_workload_type: The type of the workload of the application (not the exporter). The possible values are deployment, statefulset, and daemonset.
• target_workload_name: The name of the workload corresponding to the application (not the exporter).
• integration_type: The type of the integration. The job created in the Sysdig agent use this value to find the exporter.

Configure a New Job

Edit the prometheus.yaml file to configure a new job in Sysdig agent. The file is located in the sysdig-agent ConfigMap in the sysdig-agent namespace.

You can use the following example template:

- job_name: my-integration
  tls_config:
    insecure_skip_verify: true
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - action: keep
      source_labels: [__meta_kubernetes_pod_host_ip]
      regex: __HOSTIPS__
    - action: drop
      source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_omit]
      regex: true
    - source_labels: [__meta_kubernetes_pod_phase]
      action: keep
      regex: Running
    - action: keep
      source_labels:
        - __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
      regex: 'my-integration' # Use here the integration type that you defined in your annotations
    - action: replace
      source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_target_ns]
      target_label: kube_namespace_name
    - action: replace
      source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_target_workload_type]
      target_label: kube_workload_type
    - action: replace
      source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_target_workload_name]
      target_label: kube_workload_name
    - action: replace
      replacement: true
      target_label: sysdig_omit_source
    - action: replace
      source_labels: [__address__, __meta_kubernetes_pod_annotation_promcat_sysdig_com_port]
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: replace
      source_labels: [__meta_kubernetes_pod_uid]
      target_label: sysdig_k8s_pod_uid
    - action: replace
      source_labels: [__meta_kubernetes_pod_container_name]
      target_label: sysdig_k8s_pod_container_name

Exclude a Deployment from Being Scraped

If you want the agent to exclude a deployment from being scraped, use the following annotation:

spec:
  template:
    metadata:
      annotations:
        promcat.sysdig.com/omit: 'true'

Learn More

• Install Sysdig Agent
• Configure Monitoring Integrations
• Application Integrations
• Collect Prometheus Metrics
• Custom Integrations
• Troubleshoot Monitoring Integrations