Extract hostname from log file name

Elastic Stack Logstash
1

Hi
on logstash need to use file as input, output as http.

now question is how can i extract hostname from log filename, here is file name:
/tmp/log.hostname1.20230720
/tmp/log.hostname2.20230720

Any idea
Thanks

2

Hi. You can try this. It may be wrong but you can adjust it accordingly.

filter {
  grok {
    match => {
      "@source_path" => "%{TIMESTAMP_ISO8601}%{NOTSPACE}%{SPACE}%{GREEDYDATA}"
    }
    match => {
      "@source_path" => "/tmp/log\.hostname%{NOTSPACE:hostname}\.20230720"
    }
  }

  mutate {
    add_field => { "hostname" => "%{hostname}" }
  }
}

Extract hostname and create it as a field.

3

If your files have always this name pattern and are always in the same path, it would be easier to use a dissect filter.

Logstash will save the file path in a field named path or log.file.path depending if you have ecs compatibility enabled or not.

So the filter would be something like this:

filter {
    dissect {
        mapping => {
            "fieldName" => "/tmp/log.%{[host][hostname]}.%{}"
        }
    }
}
4

@Indeed2000

Here are two more examples using grok and dissect each extracting hostname the path field as path_hostname. Also, extracting date as path_date.

Dissect:

filter { 

  dissect {
        mapping => {
            "path" => "/tmp/log.%{path_hostname}.%{path_date}"
        }
    }
}

Grok:

filter { 

  grok {
    match => {
      "path" => "/tmp/log.%{HOSTNAME:path_hostname}.%{WORD:path_date}"
    }

  }
}

Hopefully, any of these options help.

5

@leandrojmp @ritchierich @PodarcisMuralis
Neither work for me, probably i miss something, here is more pattern examples:

/tmp/log.hostname1.20230720
/tmp/log.hostname2.20230720
/tmp/log.hostname5.20230720
/tmp/log.hostname6.20230722
/tmp/log.hostname7.20230723

Expected output field:
hostname1 As host
20230720 As date

Any idea?
Thanks

6

Please share your logstash config and example of the logstash output

7

@ritchierich


input {
  
  file 
  {
    path => "/tmp/log.*.????????"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    exclude => ["*.gz" , "*.bz2" , "*.slice" ]
    codec => plain { charset => "UTF-8" }
  }
 
} 




filter { 

  dissect {
        mapping => {
            "path" => "/tmp/log.%{path_hostname}.%{path_date}"
        }
    }

}


output 
{

  http {
    url => "%{[URL]}"
    http_method => "post"
    format => message
    message => 'host=%{[path_hostname]},id=%{[id]} trace="%{[trace]}"'

    http_compression => true
    headers => [
      'Authorization', 'Token %{[TOKEN_INFLUX]}'
    ]
  }


  stdout { codec => rubydebug }

    }
8

You need to share the output you are getting, without it is not possible to know what may be the error.

You have a stdout output, please share this output.

Another thing is this that I mentioned before

Logstash will save the file path in a field named path or log.file.path depending if you have ecs compatibility enabled or not.

If you are using Logstash 8, ecs compatibility is enabled by default, so you will not have a path field, but you will have [log][file][path], so you need to use this field.

9

@leandrojmp I’m using logstash 8.9.1

Try these

"path" => "/tmp/log.%{path_hostname}.%{path_date}"

"log.file.path" => "/tmp/log.%{path_hostname}.%{path_date}"

"[log][file][path]" => "/tmp/log.%{path_hostname}.%{path_date}"

Still not work.

10

As mentioned before, you need to share the output you are getting.