Hi
on logstash need to use file as input, output as http.
now question is how can i extract hostname from log filename, here is file name:
/tmp/log.hostname1.20230720
/tmp/log.hostname2.20230720
Any idea
Thanks
Hi
on logstash need to use file as input, output as http.
now question is how can i extract hostname from log filename, here is file name:
/tmp/log.hostname1.20230720
/tmp/log.hostname2.20230720
Any idea
Thanks
Hi. You can try this. It may be wrong but you can adjust it accordingly.
filter {
grok {
match => {
"@source_path" => "%{TIMESTAMP_ISO8601}%{NOTSPACE}%{SPACE}%{GREEDYDATA}"
}
match => {
"@source_path" => "/tmp/log\.hostname%{NOTSPACE:hostname}\.20230720"
}
}
mutate {
add_field => { "hostname" => "%{hostname}" }
}
}
Extract hostname and create it as a field.
If your files have always this name pattern and are always in the same path, it would be easier to use a dissect filter.
Logstash will save the file path in a field named path
or log.file.path
depending if you have ecs compatibility enabled or not.
So the filter would be something like this:
filter {
dissect {
mapping => {
"fieldName" => "/tmp/log.%{[host][hostname]}.%{}"
}
}
}
Here are two more examples using grok and dissect each extracting hostname the path
field as path_hostname
. Also, extracting date as path_date
.
Dissect:
filter {
dissect {
mapping => {
"path" => "/tmp/log.%{path_hostname}.%{path_date}"
}
}
}
Grok:
filter {
grok {
match => {
"path" => "/tmp/log.%{HOSTNAME:path_hostname}.%{WORD:path_date}"
}
}
}
Hopefully, any of these options help.
@leandrojmp @ritchierich @PodarcisMuralis
Neither work for me, probably i miss something, here is more pattern examples:
/tmp/log.hostname1.20230720
/tmp/log.hostname2.20230720
/tmp/log.hostname5.20230720
/tmp/log.hostname6.20230722
/tmp/log.hostname7.20230723
Expected output field:
hostname1 As host
20230720 As date
Any idea?
Thanks
Please share your logstash config and example of the logstash output
input {
file
{
path => "/tmp/log.*.????????"
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => ["*.gz" , "*.bz2" , "*.slice" ]
codec => plain { charset => "UTF-8" }
}
}
filter {
dissect {
mapping => {
"path" => "/tmp/log.%{path_hostname}.%{path_date}"
}
}
}
output
{
http {
url => "%{[URL]}"
http_method => "post"
format => message
message => 'host=%{[path_hostname]},id=%{[id]} trace="%{[trace]}"'
http_compression => true
headers => [
'Authorization', 'Token %{[TOKEN_INFLUX]}'
]
}
stdout { codec => rubydebug }
}
You need to share the output you are getting, without it is not possible to know what may be the error.
You have a stdout output, please share this output.
Another thing is this that I mentioned before
Logstash will save the file path in a field named
path
orlog.file.path
depending if you have ecs compatibility enabled or not.
If you are using Logstash 8, ecs compatibility is enabled by default, so you will not have a path
field, but you will have [log][file][path]
, so you need to use this field.
@leandrojmp I’m using logstash 8.9.1
Try these
"path" => "/tmp/log.%{path_hostname}.%{path_date}"
"log.file.path" => "/tmp/log.%{path_hostname}.%{path_date}"
"[log][file][path]" => "/tmp/log.%{path_hostname}.%{path_date}"
Still not work.
As mentioned before, you need to share the output you are getting.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.