How to parse single line for different outputs

Elastic Stack Logstash
1

Hi,

Does somebody know how to parse single line from a file and parse it for different outputs? For example: input is a log file, outputs are elasticsearch indices with different templates. I need to parse every line and save it into the first index and some of files which has a promo code (like "?promo=wteaewfsthser") I need to put to another index. I think it's possible to use two logstash instances (correct me if I'm wrong please). But I want to know is it possible to use single instance of logstash and one configuration file?

Thanks,
Igor

2

You probably want https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals

ie you create your grok pattern, and then if an event contains [list of things you want], then send to output A and B!

3

Hi Mark,

Thanks for your response, but the problem within the document structure. For the output A I need IP, Date, HTTP Method, Request, Response code, Referrer. For the output B I need only Date, Referrer, and Promo.

Thanks,
Igor

4

Ahh ok.

Then you probably want to clone the initial event and then do your manipulation to it, tag it separately and then use conditionals.

1 Like
5

Yes, that's exactly what I was looking for!!! Thanks a log Mark.:+1:

6

Hi Mark,

I found that the clone filter is what I needed. But also I found that I cannot use conditions to determine either event is original or cloned. Here's my configuration file:

input {
    file {
        path => "access.log"
        start_position => beginning
    }
}
filter {
    clone {
        clones => ["cloned"]
    }
    if "cloned" == type {
        mutate {
            add_tag => ["cloned"]
        }
    }
}
output {
    stdout {codec => json}
}

The tag cloned will not be added to any event. Do you know any solution? Or maybe the configuration is incorrect? I use Logstash 1.4.5.

Thanks,
Igor

7

Try putting the add tag in the clone section - https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html#plugins-filters-clone-add_tag

8

yes, i've tried but still the same result. Also I've tried to put new field. No success.

9

I think the docs imply that the array in the above needs to be the type of the origin event.

Try adding tags => [ "cloned" ] to the input?

10

now event not cloned