Filter data using HttpRequest

Elastic Stack Beats
1

Hi!
I'm new to the ELK stack and I'd like to know if there's a way to filter the logs in filebeat using a processor that checks some values from an external REST API, and based on the response, it drops or sends the events to elasticsearch. I tried using the script processor and an XMLHttpRequest inside it, but it doesn't seem to work, probably due to the implementation of ECMAScript that is used.
Does anyone have a suggestion?
Thanks!

2

U can use the httpjson input, HTTP JSON input | Filebeat Reference [7.14] | Elastic

3

Thanks for your answer, but I wasn't clear explaining my situation. I use filebeat to collect the logs outputted from a machine's syslog. I need to be able to filter those messages based on the state of the REST APIs that are generated from a server that is not linked in any way to the first machine.
Hope this explains the issue better.

4

Gotcha. Can u provide a sample of what you're trying to do?

5

Here's the idea. Drop or pass each new syslog based on the values of the JSON response fetched from the server.

---
filebeat.config:
  modules:
    path: "${path.config}/modules.d/*.yml"
    reload.enabled: false

output.elasticsearch:
  hosts:
    - http://elasticsearch:9200
  index: "logs-rpi-%{+yyyy.MM.dd}"

setup.template:
  name: "log"
  pattern: "log-*"
  enabled: false

setup.ilm.enabled: false

filebeat.inputs:
  - type: "log"
    enabled: true
    scan_frequency: 1s
    paths:
      - "/var/log/testLog/machine.log" #this is the file where the machine's syslog are sent to

processors:
  - script:
      lang: javascript
      id: my_filter
      source: >
        
        var xmlhttp = new XMLHttpRequest();
        var url = "Server URL";

        xmlhttp.onreadystatechange = function() {
            if (this.readyState == 4 && this.status == 200) {
                var answer = JSON.parse(this.responseText);
                if (answer.state == "offline")
                  event.Cancel();
            }
        }
        xmlhttp.open("GET", url, true);
        xmlhttp.send();

  - add_fields:
      target: tag
      fields:
        client_id: client_1
        color: red
        key_1: value_1
6

Ya, as far as I'm tracking the XMLHttpRequest() function is not available. The only JavaScript is the basic string/object manipulation functions. Unfortunately I don't think u r a going to be able to do what u want with Filebeat. Maybe look at logstash to see if it has a processor that could do what u want.

1 Like
7

Okay. Thanks!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.