Send “raw log” and “filter log” from single server to elastic server

Elastic Stack Elasticsearch
1

Hi
Need to Send “raw log” and “filter log” from single server to elastic server, but in different index.

Like this:
Host1 (rawlog) > elastic (rawindex)
Host1 (filter log) > elastic (filterindex)

Is it possible?

Any idea?
Thanks

2

By raw log you mean the message without any parsing?

You probably can do that, but it depends on how you are sending your data to Elasticsearch, if you are using Logstash it is pretty easy, if you are not using Logstash you may complicate a little.

3

@leandrojmp yes i mean raw log without any filtering, i have two choise, filebeat and logstash.
Log size in 24 hours size about 80GB (from 3 servers, each server 30GB).

Which is more sustainable for this scenario ?

4

Filebeat and Logstash both parse logs and stores individual log lines in Elasticsearch. You can choose to keep the original data in the log event together with the extracted fields, but you usually do not have separate indices for the raw events and extracted fields.

5

As Christian said in the other answer, you normally do not use a separated index to store the unparsed raw log line, you keep it in the same index where you can have the parsed log line and the unparsed log line.

If this suits your needs, you can use any of the tools, if you really want to have a different index to store the unparsed log line, then you can only use Logstash as Filebeat only supports one output.

6

@leandrojmp as I mention need to store them on different index.
is there any limitation for output in filebeat or logstash? e.g: multiple output for same log but in different index?

7

This is very unusual so i would like to understand the reasoning behind it so there is no misunderstanding. You can technically do this using Logstash, but i do not think it is easy (maybe not even posible) using Filebeat.

8

Filebeat only supports one output, Logstash supports multiple outputs and is way more flexibel than Filebeat.

To do this in Logstash you will need to use multiple pipelines with the pipelines.yml file and configure it to use the pipeline-to-pipeline communicaton.

You will need these 3 pipelines:

main.conf

input {
    your data input
}
output {
    pipeline {
        send_to => "raw"
    }
    pipeline {
        send_to => "filtered"
    }
}

Then you will have the raw.conf and filtered.conf pipelines.

raw.conf

input {
    pipeline {
        address => "raw"
    }
}
output {
    elasticsearch {
        your elastic configuration for the raw data
    }
}

and

filtered.conf

input {
    pipeline {
        address => "filtered"
    }
}
filter {
    your filters to parse and enrich your message
}
output {
    elasticsearch {
        your elastic configuration for the raw data
    }
}

Your pipelines.yml will have the following configuration:

- pipeline.id: main
  path.config: "/path/to/main.conf"

- pipeline.id: raw
  path.config: "/path/to/raw.conf"

- pipeline.id: filtered
  path.config: "/path/to/filtered.conf"

Basically you are receiving your data in the main pipeline and send the same message to the other two pipelines where you can apply different filters and send to differente destinations.

But as @Christian_Dahlqvist said, this is not usual since you already can keep the raw message in the same index, keep in mind that this approach will use more disk as you are storing things twice.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.