How to parse single line for different outputs

(Igor Belykh) #1


Does somebody know how to parse single line from a file and parse it for different outputs? For example: input is a log file, outputs are elasticsearch indices with different templates. I need to parse every line and save it into the first index and some of files which has a promo code (like "?promo=wteaewfsthser") I need to put to another index. I think it's possible to use two logstash instances (correct me if I'm wrong please). But I want to know is it possible to use single instance of logstash and one configuration file?


(Mark Walkom) #2

You probably want

ie you create your grok pattern, and then if an event contains [list of things you want], then send to output A and B!

(Igor Belykh) #3

Hi Mark,

Thanks for your response, but the problem within the document structure. For the output A I need IP, Date, HTTP Method, Request, Response code, Referrer. For the output B I need only Date, Referrer, and Promo.


(Mark Walkom) #4

Ahh ok.

Then you probably want to clone the initial event and then do your manipulation to it, tag it separately and then use conditionals.

(Igor Belykh) #5

Yes, that's exactly what I was looking for!!! Thanks a log Mark.:+1:

(Igor Belykh) #6

Hi Mark,

I found that the clone filter is what I needed. But also I found that I cannot use conditions to determine either event is original or cloned. Here's my configuration file:

input {
    file {
        path => "access.log"
        start_position => beginning
filter {
    clone {
        clones => ["cloned"]
    if "cloned" == type {
        mutate {
            add_tag => ["cloned"]
output {
    stdout {codec => json}

The tag cloned will not be added to any event. Do you know any solution? Or maybe the configuration is incorrect? I use Logstash 1.4.5.


(Mark Walkom) #7

Try putting the add tag in the clone section -

(Igor Belykh) #8

yes, i've tried but still the same result. Also I've tried to put new field. No success.

(Mark Walkom) #9

I think the docs imply that the array in the above needs to be the type of the origin event.

Try adding tags => [ "cloned" ] to the input?

(Igor Belykh) #10

now event not cloned

(system) #11