Filter data using HttpRequest

I'm new to the ELK stack and I'd like to know if there's a way to filter the logs in filebeat using a processor that checks some values from an external REST API, and based on the response, it drops or sends the events to elasticsearch. I tried using the script processor and an XMLHttpRequest inside it, but it doesn't seem to work, probably due to the implementation of ECMAScript that is used.
Does anyone have a suggestion?

U can use the httpjson input, HTTP JSON input | Filebeat Reference [7.14] | Elastic

Thanks for your answer, but I wasn't clear explaining my situation. I use filebeat to collect the logs outputted from a machine's syslog. I need to be able to filter those messages based on the state of the REST APIs that are generated from a server that is not linked in any way to the first machine.
Hope this explains the issue better.

Gotcha. Can u provide a sample of what you're trying to do?

Here's the idea. Drop or pass each new syslog based on the values of the JSON response fetched from the server.

    path: "${path.config}/modules.d/*.yml"
    reload.enabled: false

    - http://elasticsearch:9200
  index: "logs-rpi-%{+yyyy.MM.dd}"

  name: "log"
  pattern: "log-*"
  enabled: false

setup.ilm.enabled: false

  - type: "log"
    enabled: true
    scan_frequency: 1s
      - "/var/log/testLog/machine.log" #this is the file where the machine's syslog are sent to

  - script:
      lang: javascript
      id: my_filter
      source: >
        var xmlhttp = new XMLHttpRequest();
        var url = "Server URL";

        xmlhttp.onreadystatechange = function() {
            if (this.readyState == 4 && this.status == 200) {
                var answer = JSON.parse(this.responseText);
                if (answer.state == "offline")
        }"GET", url, true);

  - add_fields:
      target: tag
        client_id: client_1
        color: red
        key_1: value_1

Ya, as far as I'm tracking the XMLHttpRequest() function is not available. The only JavaScript is the basic string/object manipulation functions. Unfortunately I don't think u r a going to be able to do what u want with Filebeat. Maybe look at logstash to see if it has a processor that could do what u want.

1 Like

Okay. Thanks!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.