Keep one logstash conf file rather having multiple logstash conf file!

Elastic Stack Logstash
1

Suppose I have two sets of data i.e. user & products. I have created two different indexes for them on ES. Now I want to use one logstash conf file that will have some condition to differentiate the data and send them to their related index ie users data to user index and product data to product index.

is it even possible or I am daydreaming!!!

Cheers @inandi

2

Yes, you can use conditionals in the output section. Like this

output {
    if (some condition) {
        elasticsearch {
             first configuration
        }
    } else {
        elasticsearch {
             second configuration
        }
    }
}

Or, if the only difference is the index name, and the name of the index is based on a field in the event then you can use a sprintf reference.

output {
    elasticsearch {
        index => "%{someField}-%{+YYYY.MM}"
    }
}
1 Like
3

thanks, @Badger for your reply. I am posting my sample logstash conf file

input {
  file {
    path => "/var/www/html/elastic/user_data.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"        
  }
}

filter { }

output {
    elasticsearch {
      hosts => ["http://192.168.0.1:9200/"] 
      index => "user_index"
      document_id => "%{userid}"
      template =>"/var/www/html/elastic/user_template.json"
      template_name=>"datauser"
      template_overwrite => true
  }
}

now suppose I have another JSON file which contains product data and i want to insert that into product_index, so How to write this logstash file?

Thanks @inandi

4

If you want to process both files in the same logstash configuration then you can add a second file input. Then in the filter section test the path field on the event

if [path] =~ "user_data.json" {
    mutate { add_field => { "filetype" => "user" } }
} else {
    mutate { add_field => { "filetype" => "product" } }
}

Then you can use a conditional in the output section

output {
    if [filetype] == "user" {
        elasticsearch {
             first configuration
        }
    } else {
        elasticsearch {
             second configuration
        }
    }
}
5

exactly I wanted something like this. Cheers @Badger :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.