How to take beats inputs in multiple codecs in logstash

Elastic Stack Logstash
1

I've got a set of systems that I'll be deploying filebeats to, and I need it to monitor some text sources (like /var/log/secure) and some json sources (JSON-ified Apache logs).

I've got the following on the filebeats side:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/httpd/logstash_access_log
      input_type: log
      document_type: beat
    -
      paths:
        - /var/log/secure
      input_type: log
      document_type: message
: 
output:
  logstash:
    hosts: ["192.168.77.68:5043"]

And then, on the logstash host (these are two different systems -- we're using filebeats on clients and having them ship to a local logstash node for filtering & forwarding back to our main location):

input {
    beats {
        port => 5043
        codec => "json"
    }
}

I need to be able to see the plain text entries as well as the json ones. What's the right way to go about this? Is it even possible, or do I need to have my filebeats agents forward one data type only to the logstash aggregator for processing & filtering? We're just in the exploratory stages of figuring out how we want to make everything work.

Thanks,

Bret

2

You either need multiple beats inputs with different codec configurations (and then you'll need to run two Filebeat processes too) or you can conditionally use the json filter if the input has a certain type (or just looks like JSON).

3

I am tagging them with different types -- how does that work in the input section? Do I wrap the 'codec => foo' in a conditional, or does that need to occur at a higher level?

4

You can't have any conditionals in the input section because there's no event context to evaluate conditions in, but the input events themselves should contain metadata that you can use.

5

(I've only spent about 48 hours with these tools so far, so I thank you for your patience).

So I need to not apply the codec in the input section, right? Is there a way to apply it later, in a filter or during the output?

This was working extremely cleanly when I was just sending the Apache logs in json format. Adding other logs has muddied it a bit and I'm trying to keep the code as clean (and brief) as possible.

6

"json filter". I just realized what you were saying and I can now read json as plain and convert it via filter.

Tack så mycket!

7

I have similiar issue with multiline codec in beat inputs . multiline filter is deprecated. How to handle 2 files from a single filebeat instance , where one file required multiline codec and another not.

8

@sumitkgaur, please start a new thread for your unrelated question.