How to configure different indexes in logstash

Elastic Stack Logstash
21

@magnus. the host in output used here is coming from logs parsed from filebeats?and the index "beats_sit-%{+YYYY.MM.dd}" in output does this index template beats_sit-* has to be first created ?

22

the host in output used here is coming from logs parsed from filebeats?

You mean the host field? Yes, that should be the Filebeat host that sends the data.

and the index "beats_sit-%{+YYYY.MM.dd}" in output does this index template beats_sit-* has to be first created ?

Yes, you should probably have an index template that matches the name of your indexes.

23

Thanks magnus for reply.so in this case "beats_dev-%{+YYYY.MM.dd}" index name . What index template I should create and how ? Can you post the index for this -beats_dev-%{+YYYY.MM.dd} and its path location in logstash?

24

if [host] in ["example1.sit.com", "example2.sit.com"] ...

the host here is coming after filebeat parsing? so here host contains the host name where the respective event log is coming from filebeat client ?my confusion is in my json logs itself i have a field called hostname .any way I can use the hostname from the logs ?

25

What index template I should create and how ?

You can make a copy of the existing template file (which you'll find in the Logstash installation directory), adjust the template field that controls which indexes it applies to, and configure Logstash's elasticsearch output to use it.

Another way of obtaining the current template is by fetching it directly from ES via the get index template REST API. The template uploaded by Logstash is by default named "logstash".

Since you are a beginner it would be easier if you just stuck to the defaults and called your index logstash-*, e.g. logstash-beats_dev-2017.06.12. Once you learn more you can start customizing things.

the host here is coming after filebeat parsing? so here host contains the host name where the respective event log is coming from filebeat client ?

I believe so, yes. That should be easy to verify, shouldn't it?

my confusion is in my json logs itself i have a field called hostname .any way I can use the hostname from the logs ?

Yes, of course.

26

ok thanks for your reply magnus. Here is what i am going to do-

1.i took the default template from \logstash-5.4.1\vendor\bundle\jruby\1.9\gems\logstash-output-elasticsearch-7.3.1-java\lib\logstash\outputs\elasticsearch\elasticsearch template es 2x.json

  1. i am going to use this default template and rename this json file as mytemplate.json and inside that going to rename existing "template" : "logstash-" to "template" :"logstash-beats_dev-".

  2. then in logstash output
    {
    elasticsearch {
    hosts => ["elasticserver:9200"]
    template => "/logstash/logstash-5.4.1/templates/mytemplate.json"
    template_name => "logstash-beats_dev-*"
    template_overwrite => true
    }
    stdout{}
    }

Please tell me if i am missing i any steps here or I need to do anything else.thanks a lot for patience

27

i am going to use this default template and rename this json file as mytemplate.json and inside that going to rename existing "template" : "logstash-" to "template" :"logstash-beats_dev-".

If your index names match logstash-* then you don't need to create your own index template.

then in logstash output
{
elasticsearch {
hosts => ["elasticserver:9200"]
template => "/logstash/logstash-5.4.1/templates/mytemplate.json"
template_name => "logstash-beats_dev-*"
template_overwrite => true
}

But you're not setting the index name to anything so you're getting the default, logstash-%{+YYYY.MM.DD}.

28

ok .where should i set the index name as logstash-beats_dev ? my requirement is that I should redirect all events to index logstash-beats_dev* based on hostname ?if different hostname=sit or hostname=test it should create respective index like logstash-beats_sit* or logstash-beats_test*.....?

Can you please elaborate how it works?thanks for your patience

29

ok .where should i set the index name as logstash-beats_dev ?

The elasticsearch output's index option sets the index name.

my requirement is that I should redirect all events to index logstash-beats_dev* based on hostname ?if different hostname=sit or hostname=test it should create respective index like logstash-beats_sit* or logstash-beats_test*.....?

elasticsearch {
  ...
  index => "logstash-beats_%{host}-%{+YYYY.MM.dd}"
}

This assumes that

  • the field host contains the name of the host that generated the event and
  • that you have a very small number of hosts since you otherwise will create a large number of indexes.
30

ok thanks for reply.Please correct me -or let me know if anything more is needed

Will the logstash conf get the {host} field from filebeats automatically ?if yes then the index would be
logstash-beats_dev.com-2017.12.06 ?

What index name I have to use in kibana UI so as to separate the logs viewing based on logstash-beats_dev.com-* or logstash-beats_test.com-* .....

So my logstash.conf is

input {
beats {
port => 5044
}
}

filter{
json{
source => "message"
target => "tweet"
}
}

output {
elasticsearch {
hosts => ["elasticserver:9200"]
index => "logstash-beats_%{host}-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

filebeats.yml from various hosts -

  • input_type: log

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • C:/Users....

#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

hosts: ["localhost:5044"]

31

Will the logstash conf get the {host} field from filebeats automatically ?

I think so, but as I said you can easily verify it yourself.

if yes then the index would be logstash-beats_dev.com-2017.12.06 ?

Yes, if host contains "dev.com" and the event is from the future (Dec 6, 2017).

What index name I have to use in kibana UI so as to separate the logs viewing based on logstash-beats_dev.com-* or logstash-beats_test.com-* .....

You can set up multiple index patterns that each match a different index name prefix.

32

Hi all,
this is an old topic, but i need an help on it.

I have the following configuration:

input {
beats {
port => "5043"
}
}
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} "]
match => ["message", "%{HTTPDERROR_DATE:timestamp} (%{NUMBER:bytes}) "]
match => ["message", "%{TIME:timestamp} %{LOGLEVEL:level}"]
}
date {
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSS", "HH:mm:ss,SSS", "EEE MMM dd HH:mm:ss YYYY" ]
timezone => "Europe/Rome"
target => "@timestamp"
}
mutate {
remove_field => [ "timestamp","offset","level","@version","input_type","beat","type","tags","sort","fields" ]
}

}
output {
if [host] in ["olinux7-template.farm.eurobet-italia.local"]{
elasticsearch {
hosts => [ "172.16.19.216:9200", "172.16.19.217:9200", "172.16.19.218:9200", "172.16.19.220:9200" ]
user => elastic
password => changeme
manage_template => false
index => "prova-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
else {
elasticsearch {
hosts => [ "172.16.19.216:9200", "172.16.19.217:9200", "172.16.19.218:9200", "172.16.19.220:9200" ]
user => elastic
password => changeme
manage_template => false
index => "sito-gioco-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}

But, the index prova-%{........} is never created.
can i use the "host" keyword in the if statement?
If not, can you help me with some tips?

Thanks,
BR

33

Please start a new thread for your problem.

34

Hi, solved adding "fields" in filebeat config and using it in the logstash output with if statement,

BR

35

input {
beats {
port => 5044
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => ["https://localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

After running "service logstash start" new index is not created.Please Help.

36

Please open a new thread if your own and provide more details, e.g. output from log files.

37

Thank you.created in thread.Please have look.

38

@magnusbaeck : I wanted to put an "if-else" block in my logstash conf file to send data in discrete indices to ES on the basis of IPs,but its not working in my case.Please have a look into my conf file:

input {
beats {
port => 5044
}
}

output {
if [host] == "http://192.168.43.154:9200"
{
elasticsearch {
index => "travlogs"

}

}
}