Blackbox_exporter Source Reading

Blackbox_exporter is a Prometheus monitoring system used to collect http,dns,icmp related information on the agent machine, passing parameters and target via Prometheus, Mapped to the corresponding agent on the Web interface for processing. For example, the following collection configuration is from the Prometheus configuration file

Backbox exporter

Scrape_configs:
  -job_name: ' Blackbox '
    metrics_path:/probe
    params:
      module: [http_2xx]  # look For a HTTP response.
    Static_configs:
      -Targets:
        -Http://prometheus.io    # Target to probe with HTTP.
        -Https://prometheus.io   # Target to probe with HTTPS.
        -http://example.com:8080 # Target to probe with HTTP on port 8080.
    Relabel_configs:
      -source_labels: [__address__]
        Target_label: __param_target
      -source_labels: [__ Param_target]
        target_label:instance
      -target_label: __address__
        replacement:127.0.0.1:9115  # The blackbox exporter ' s real hostname:port.

Blackbox exporter modules

The path of the acquisition is defined in the configuration information, and the parameters of the HTTP pass are defined as module:[http_2xx] and a list of machines for the purpose of the collection is passed. This is consistent with the effect of sending HTTP requests directly, prometheus blackbox exporter. http://localhost:9115/probe?target=http://prometheus.io&module=http_2xx You just need to send multiple requests to get the contents of the list. configuration file Import


Blackbox exporter docker


Configuration files are mainly used to configure the supported modules and methods of acquisition, for example, we can define the DNS acquisition module DNS_UDP and DNS_TCP respectively to capture target (test target machine) for the specified domain name of the data collection

Blackbox exporter tcp example

  DNS:
    prober:dns
    timeout:5s
    DNS:
      query_type: "A"
      query_name: "example.com"
  dns_tcp:
    Prober:dns
    timeout:5s
    DNS:
      transport_protocol: "TCP"
      Preferred_ip_protocol: "IP4"
      query _name: "example.com"



We can look at how the import of a specific configuration file is implemented:

Blackbox exporter tcp

If err: = SC. Reloadconfig (*configfile); Err! = Nil {level
    . Error (Logger). Log ("msg", "Error loading config", "err", err)
    OS. Exit (1)
}



The above code executes to import the contents of the BLACKBOX.YML (default configuration), and if the import succeeds, all configuration information will be saved in SC. C "config struct body" below. The config is defined as follows. Contains a map structure that corresponds to the custom module name and the configuration information for module.



Type Config struct {
    Modules map[string]module ' yaml: "Modules" '

    //Catches all undefined fields and must is empty After parsing.
    XXX map[string]interface{} ' yaml: ', ' inline '
}



The module contains the specific configuration content, but in order to satisfy the supported components, the module is defined as follows. Any one module contains all the probe supported structures, only need to specify a component to use, stored in the Prober, the specific configuration is stored in the respective structure, such as if Probe:dns to find the configuration needs to be obtained in DNS.



Type Module struct {
    prober  string        ' Yaml: ' prober,omitempty ' '
    Timeout time. Duration ' Yaml: ' timeout,omitempty ' '
    HTTP    httpprobe     ' yaml: ' http,omitempty ' '
    TCP     Tcpprobe      ' yaml: ' tcp,omitempty ' '
    ICMP    icmpprobe     ' yaml: ' icmp,omitempty ' '
    DNS     Dnsprobe      ' yaml: "Dns,omitempty" '

    //Catches all undefined fields and must is empty after parsing.
    XXX map[string]interface{} ' yaml: ', ' inline '
}



The specific import works by overloading the Unmarshaler interface under the YAML module with the internal implementation of different components to implement YML file-to-struct import. For example, the following is the import of dnsprobe structure. In addition, Checkoverflow is used to check if an unknown configuration has been written to the configuration file.



Unmarshalyaml implements the Yaml. Unmarshaler interface.
Func (S *dnsprobe) Unmarshalyaml (unmarshal func (interface{}) error) error {
    type plain dnsprobe
    if err: = Unmarsha L ((*plain) (s)); Err! = Nil {
        return err
    }
    If err: = Checkoverflow (s.xxx, "DNS Probe"); Err! = Nil {
        return err
    }
    I F s.queryname = = "" {
        return errors. New ("Query name must is set for DNS module")
    }
    return nil
}



This allows the configured configuration config structure to contain all the custom module information, including the module name and the specific configuration. This enables acquisition of target objects by specifying the module name and target when we use interface access. Data Acquisition Interface





Blackbox_exporter by default, the/metrics path only collects information about the go run, and the core processing logic is handled in the/probe route handler function.



http. Handlefunc ("/probe", func (w http). Responsewriter, R *http. Request) {
        sc. Lock ()
        conf: = SC. C
        SC. Unlock ()
        Probehandler (W, R, Conf, logger, RH)
    })



In order to prevent data acquisition during overloading, access is secured using the lock mechanism, and the function actually called is in Probehandler.



Func Probehandler (w http. Responsewriter, R *http. Request, C *config. Config, logger log. Logger, rh *resulthistory) {
    modulename: = R.url. Query (). Get ("module")
    if modulename = = "" {
        modulename = "http_2xx"
    }
    module, OK: = C.modules[modulename]
    If!ok {
        http. Error (W, FMT. Sprintf ("Unknown module%q", ModuleName), HTTP. Statusbadrequest)
        return
    }



The function first queries the module parameter for existence and, if it does not exist, uses the Http_2xx module by default (provided that the module is configured in config). When the name is obtained, the config is queried. If there is a corresponding module in the modules, the 400 error is returned if not.



var timeoutseconds float64
if V: = R.header.get ("X-prometheus-scrape-timeout-seconds"); v! = ""
    {var err errortimeoutseconds, err = StrConv. Parsefloat (V, +)
    if err! = Nil {
        http. Error (W, FMT. Sprintf ("Failed to parse timeout from Prometheus header:%s", err), HTTP. Statusinternalservererror)
        return
    }
}
if timeoutseconds = = 0 {
    timeoutseconds = ten
}
If module. Timeout.seconds () < timeoutseconds && module. Timeout.seconds () > 0 {
    timeoutseconds = module. Timeout.seconds ()
}



If the time-out of the acquisition is specified in the header, this value is used as the time-out probe, and if no setting is set to 10s, both take the minimum value.



CTX, Cancel: = context. Withtimeout (context. Background (), time. Duration (Timeoutseconds*float64 (time. Second))
defer cancel ()
R = R.withcontext (CTX)



The context is used to manage timeouts, and when the timeout time arrives, directly closes any query connections and returns.



Probesuccessgauge: = Prometheus. Newgauge (Prometheus. gaugeopts{
    Name: ' probe_success ', help
    : ' Displays whether or not ' the probe was a success ',
}
Probedurationgauge: = Prometheus. Newgauge (Prometheus. gaugeopts{
    Name: "Probe_duration_seconds", help
    : "Returns How long the probe took to complete in seconds",
})
params: = R.url. Query ()
target: = params. Get ("target")
if target = = "" {
    http. Error (W, "Target parameter is missing", http. Statusbadrequest)
    return
}



Define the basic acquisition metrics, which are output in the results, regardless of any configuration module, to record the success and acquisition time of the two indicators. Gets the target parameter, and exits if no target is set.



Prober, OK: = Probers[module. Prober]
    if!ok {
        http. Error (W, FMT. Sprintf ("Unknown prober%q", module. Prober), HTTP. Statusbadrequest)
        return
    }



The specific module configuration is obtained based on the Prober name defined in the configuration. For example, Prober DNS is queried for DNS processing functions in probers. To perform the subsequent operation.



    Start: = time. Now ()
    registry: = Prometheus. Newregistry ()
    registry. Mustregister (Probesuccessgauge)
    registry. Mustregister (Probedurationgauge)
    Success: = Prober (CTX, Target, module, registry, SL)
    Duration: = time. Since (Start). Seconds ()
    probedurationgauge.set (duration)
    If success {
        Probesuccessgauge.set (1) level
        . Info (SL). Log ("msg", "Probe succeeded", "Duration_seconds", duration)
    } else {level
        . Error (SL). Log ("msg", "Probe failed", "Duration_seconds", duration)
    }



The specific query logic first registers two previously defined indicator objects and performs the actual fetch operation (Prober) to pass the timeout context, query object, and Loger parameters. The corresponding indicator data is set by the return value and time.





In addition, each module has different indicator content, which is defined in different module acquisition processes. specific acquisition (DNS module)





For different modules have their own implementation of the acquisition module, here only the DNS module acquisition implementation of the code to see the specific implementation:



 func Probedns (CTX context. Context, target string, module CONFIG. Module, registry *prometheus. Registry, logger log. Logger) bool {var dialprotocol string Probednsanswerrrsgauge: = Prometheus. Newgauge (Prometheus. gaugeopts{Name: "Probe_dns_answer_rrs", Help: "Returns number of entries in the answer resource record Li St ",}) Probednsauthorityrrsgauge: = Prometheus. Newgauge (Prometheus. gaugeopts{Name: "Probe_dns_authority_rrs", Help: "Returns number of entries in the Authority resource Rec Ord list ",}) Probednsadditionalrrsgauge: = Prometheus. Newgauge (Prometheus. gaugeopts{Name: "Probe_dns_additional_rrs", Help: "Returns number of entries in the additional resource R Ecord list ",}) registry. Mustregister (Probednsanswerrrsgauge) registry. Mustregister (Probednsauthorityrrsgauge) registry. Mustregister (Probednsadditionalrrsgauge) 



The function begins by defining and registering three gauge objects, three of which are used to store the results of the query (Answer records, authoritative records, and number of other records)





Depending on the actual configuration such as protocol, in addition Targer if the containing port is separate host and port, the default port is 53. The Go-dns library (GITHUB.COM/MIEKG/DNS) is implemented to send DNS requests based on configuration.



Probednsanswerrrsgauge.set (float64 (len response. Answer))
Probednsauthorityrrsgauge.set (float64 (len response. Ns))
Probednsadditionalrrsgauge.set (float64 (len response. Extra)))



Finally update the indicator information. And if a separate validation is configured, the defined valid function is performed to perform the validation information for the configuration, such as verifying that the response information meets the requirements



Level. Info (Logger). Log ("msg", "Validating Answer RRs")
if!validrrs (&response. Answer, &module. Dns. Validateanswer, logger) {level
    . Error (Logger). Log ("msg", "Answer RRs validation Failed")
    return False
} level
. Info (Logger). Log ("msg", "Validating Authority RRs")
if!validrrs (&response. Ns, &module. Dns. Validateauthority, logger) {level
    . Error (Logger). Log ("MSG", "Authority RRs validation failed")
    return False
} level
. Info (Logger). Log ("msg", "Validating Additional RRs")
if!validrrs (&response. Extra, &module. Dns. Validateadditional, logger) {level
    . Error (Logger). Log ("msg", "Additional RRs validation Failed")
    return False
}



The specific verification code is as follows:



Func Validrrs (RRs *[]dns. RR, v *config. Dnsrrvalidator, logger log. Logger) bool {//Fail the probe if there is no RRs of a given type, but a regexp match is required//(i.e. Fail
    Ifnotmatchesregexp is set). If Len (*rrs) = = 0 && len (v.failifnotmatchesregexp) > 0 {level. Error (Logger). Log ("msg", "Fail_if_not_matches_regexp specified but no RRs returned") return to False} for _, RR: = range * RRs {level. Info (Logger). Log ("msg", "Validating rr", "RR", RR) for _, Re: = Range V.failifmatchesregexp {match, err: = RegExp. Matchstring (Re, RR. String ()) if err! = nil {level. Error (Logger). Log ("msg", "Error matching RegExp", "RegExp", Re, "err", err) return false} if MA tch {level. Error (Logger). Log ("msg", "RR matched RegExp", "RegExp", Re, "RR", RR) return False}} for _ , Re: = Range V.failifnotmatChesregexp {match, err: = RegExp. Matchstring (Re, RR. String ()) if err! = nil {level. Error (Logger). Log ("msg", "Error matching RegExp", "RegExp", Re, "err", err) return false} if!m Atch {level. Error (Logger).
    Log ("msg", "RR did not match regexp", "RegExp", Re, "RR", RR) return false}}   
 return true}


In addition, Black_exporter supports reload through the Web and overloads of configuration files by sending signal. The overloaded mode of the signal is implemented by starting a separate goroutine, capturing the signal, and performing the overloaded operation:



Hup: = Make (chan os. Signal)
RELOADCH: = Make (chan chan error)
Signal. Notify (Hup, Syscall. SIGHUP)
go func () {for
    {
        Select {case
        <-hup:
            If err: = SC. Reloadconfig (*configfile); Err! = Nil {level
                . Error (Logger). Log ("msg", "Error reloading config", "err", err)
                continue
            } level
            . Info (Logger). Log ("msg", "Reloaded config file") Case
        rc: = <-reloadch:
            If err: = SC. Reloadconfig (*configfile); Err! = Nil {level
                . Error (Logger). Log ("msg", "Error reloading config", "err", err)
                RC <-Err
            } else {level
                . Info (Logger). Log ("msg", "Reloaded config file")
                RC <-Nil
            }}}
()



When triggered by the web, by sending a signal to the pipeline, the overloaded operation in the standalone Goroutine is implemented, and the other collection work is blocked during overloading.



    http. Handlefunc ("/-/reload",
        func (w http). Responsewriter, R *http. Request) {
            if R.method! = "POST" {
                w.writeheader (http. statusmethodnotallowed)
                FMT. fprintf (W, "This endpoint requires a POST request.\n")
                return
            }

            rc: = Make (chan error)
            reloadch <-rc< C9/>if ERR: = <-rc; Err! = Nil {
                http. Error (W, FMT. Sprintf ("Failed to reload Config:%s", err), HTTP. Statusinternalservererror)
            }
        })



For standard go run environment detection via promhttp. Handler direct implementation.





http. Handle ("/metrics", promhttp. Handler ())





This is the specific implementation of Black_exporter, and the implementation of the other modules is not described here.