I have set custom name in filebeat input section. However, in logstash configuration file it was unable to be use
Filebeat config file
- type: log
enabled: true
paths:
...
fields:
- name_of_index: group-1
- name_of_log: apache
Logstash.conf
input {
beats {
port => 5044
...
type => "%{[field][name_of_log]}"
}
}
filter {
if "%{[field][name_of_log]}" in ["apache"] {
...
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[fields][name_of_index]}-%{+YYYY-MM-dd}"
}
}
output in elasticsearch
%[fields][name_of_index]-2020-01-05
I not sure which parr it is wrong able to help in this?
stephenb
(Stephen Brown)
January 9, 2021, 5:25pm
2
looks like you have typos...
In filebeat no -
See Here
fields:
name_of_index: group-1
name_of_log: apache
The fields are under fields
several places you only have the term field
no s
.
Also you could probably simplify perhaps look here .
input {
beats {
port => 5044
...
type => "%{[fields][name_of_log]}"
}
}
filter {
if [type] == "apache" {
...
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[fields][name_of_index]}-%{+YYYY-MM-dd}"
}
}
hi can i know if i use type, it means table name right?
another thing would like to know whyc below not combining the idnex rather it giving 2 seperate sets of index?
- type: log
enabled: true
paths: /var/log/apache
...
fields:
- name_of_index: group-1
- name_of_log: apache
- type: log
enabled: true
paths: /var/log/message
...
fields:
- name_of_index: group-1
- name_of_log: message
stephenb
(Stephen Brown)
January 10, 2021, 3:10am
4
No type means in the input type see here It is not the index name or type in this case it is log
which means reading a log file
I would suggest perhaps referring to the documentation
Hi i check in logstash input
type => "%{[fields][name_of_log]}"
it is printing the string rather than the value in the output section. so in output it is showing as
"type" : "%{[fields][name_of_log]}".
why there are such behaviour
stephenb
(Stephen Brown)
January 10, 2021, 4:48pm
7
Apologies I was cut and pasting...
At the input you does not appear have access to the fields yet so you can only set static values. (My bad I will need to look closer at that)
So to accomplish what I think you want you would do this..
input {
beats {
port => 5044
}
}
filter {
if [fields][name_of_log] == "apache" {
...
} else if [fields][name_of_log] == "nginx" {
...
} else {
...
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[fields][name_of_index]}-%{+YYYY-MM-dd}"
}
}
stephenb
(Stephen Brown)
January 10, 2021, 5:32pm
8
You could also do it like this which might be a little cleaner
input {
beats {
port => 5044
}
}
filter {
mutate {
add_field => { "type" => "%{[fields][name_of_log]]}" }
}
if [type] == "apache" {
...
} else if [type] == "nginx" {
...
} else {
...
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[fields][name_of_index]}-%{+YYYY-MM-dd}"
}
}
stephenb
(Stephen Brown)
January 10, 2021, 11:20pm
9
I see you opened another thread ...
Hi @navin1093 I did not realize you were trying to set _type
as the poster stated that is not longer supported if you just want to set a normal field named type
you can still do that, but it is just like any other field.
In reality, with the removal of types ( "_type" : "_doc"
always ) , You should think of an index as a table not a database...
system
(system)
Closed
February 7, 2021, 11:20pm
10
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.