How to know the log comme from which file

Elastic Stack Logstash
1

Hi,

I am using filebeat,logstash,ES and kibana 5.6.3. I want to be able to see the log source file name in kibana, but i am not being able to to. i have all information but do not have the file source name , the log are coming from. Please can anyone help me.

In kibana i want a line "file"=> log-12.01.2000.log for example for each log. here is my logstach conf file.

input {
beats {
port => 5044
host => "localhost"
}
}

filter {
#(?<queue_id>[0-9A-F]{10,11})
#%{TIMESTAMP_ISO8601:FixedFormatISOInternationalDate}

grok {
match => { "message" => ["(?(\d{4})-(\d{2})-(\d{2}).(\d{2}):(\d{2}):(\d{2}).(\d{3})) %{INT:EventId} %{UUID:ActivityId} %{DATA:UserName} %{NOTSPACE:TransactionIsolationLevel} %{NOTSPACE:TransactionLocalIdentifier} %{NOTSPACE:TransactionDistributedIdentifier} %{NOTSPACE:TransactionStatus} %{NOTSPACE:severity } %{GREEDYDATA:data}"] }
}

mutate {
add_field => {
"[@metadata][Source]" => "%{source}"
"[@metadata][Input_type]" => "%{input_type}"
"[@metadata][Timestamp]" => "%{@timestamp}"
"[@metadata][Tags]" => "%{tags}"
}
}

mutate {
remove_field => ["message", "source"]
}
}

output {
if "_grokparsefailure" not in [tags] {
elasticsearch {
hosts => ["10.184.161.66","10.184.161.67"]
user => elastic
password => "Cs_24Z*-;u3WXMzwk]66"
codec => json
index => opera_index
}
}

}

-- what i have in kibana see attachment.

Thank you.

2017-10-25_11h29_19
2017-10-25_11h29_19.png1040×748 34.3 KB

2

The filename is in the source field that you're removing.

3

Thank you mahnusbaek, i am going to try this. must i delete the index or il will just get update with this field automatically ? thank you.

4

All new events will get the source field.

5

magnusbaeck, i still cannot see the new field named "source", i have restarted the logstash and filebeat services.

6

Remove all your current filters and use a stdout { codec => rubydebug } output to dump all incoming events to the Logstash log. What does an example event look like?

7

what do you think about this :

input { stdin { } }

filter {
grok {
match => { "message" => ["(?(\d{4})-(\d{2})-(\d{2}).(\d{2}):(\d{2}):(\d{2}).(\d{3})) %{INT:EventId} %{UUID:ActivityId} %{DATA:UserName} %{NOTSPACE:TransactionIsolationLevel} %{NOTSPACE:TransactionLocalIdentifier} %{NOTSPACE:TransactionDistributedIdentifier} %{NOTSPACE:TransactionStatus} %{NOTSPACE:severity } %{GREEDYDATA:data}"] }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

output {
elasticsearch {
hosts => ["10.184.161.66","10.184.161.67"]
user => elastic
password => "Cs_24Z*-;u3WXMzwk]66"
index => opera_index

}

stdout { codec => rubydebug }
}

here is an event in a log file :

2017-09-05 01:00:08.092 131 00000000-0000-0000-0000-000000000000 user (null) (null) (null) (null) Information this is a message

8

Keep your beats input. It's the Filebeat messages we want to look at.

2017-09-05 01:00:08.092 131 00000000-0000-0000-0000-000000000000 user (null) (null) (null) (null) Information this is a message

I want to see the messages processed by Logstash, written to the Logstash log. Not the input messages.

9

Hi magnusbaek, your recommandation of removing "source" in the remove_field 's line works. I just have to wait some time to see the source field. thank you the problem is resolved.

best.