Problem indexing the year

Elastic Stack Logstash
1

Hello, I am new to this whole ELK thing. Im an intern working on indexing a bunch of syslogs into elastic using logstash. The only problem is that the logs do not contain the year which I also want to index for each mapping. The year is only available on the file name. I am using the stdin plugin to read the logs into logstash since they are .gz files. I am aware I can use the file plugin with the read mode but this, to my understanding, deletes the log and relogs it even with the log setting. This is not okay with my superiors. My question is, what would be the best way to get the year from the file name and add it to each mapping. My current config is below.

input {
    stdin {
    }
}
filter {
     grok {
    match  => {"message" => "%{SYSLOGTIMESTAMP:timestamp}\s+[0-9\.\/]+\s+dhcpd\[[0-9]+\]:\s+(DHCPACK) %{WORD} %{IP:ipaddress} %{WORD} %{MAC:macaddress} %{GREEDYDATA}"}
    match  => {"message" => "%{SYSLOGTIMESTAMP:timestamp}\s+[0-9\.\/]+\s+dhcpd\[[0-9]+\]:\s+(DHCPACK) %{WORD} %{IP:ipaddress} \(%{MAC:macaddress}\) %{GREEDYDATA}"}
 }
}


output{
if "_grokparsefailure" in [tags] {
  } else {
elasticsearch {
     hosts => "http://localhost:9200"
     index => "dhcp"
}

  }
}
2

If I had to do that I would do something like

for F in ... ; do
    gunzip -c $F | awk -v F=$F '{print F $0}' | logstash ...
done

to prefix every line with the filename.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.