Add Logstash host name to JSON formatted logstream

Elastic Stack Logstash
1

I've got a pipeline: JSON formatted log file -> Filebeat -> Logstash -> ES. I'd like to add the Logstash host name to the document before they are sent along to ES. I took a stab at it:

default-pipeline.conf

input {
    beats {
        port => 5044
        codec => "json"
    }
}
filter {
  mutate {
    add_field => { "logstash_host" => "%{host}" }
  }
}

output {
  amazon_es {
    hosts => ["vpc-xxxxxx.us-east-1.es.amazonaws.com"]
    protocol => https
    codec => plain
    region => "us-east-1"
    manage_template => false
    index => "xxxxxx-logs-%{+YYYY.MM.dd}"
  }
  elasticsearch {
    hosts => ["http://xxxxxxxxx:9200"]
    index => "xxxxx-logs-%{+YYYY.MM.dd}"
    manage_template => false
  }
}

And what shows up in Kibana is:

logstash_host {"name":"<hostname of system where log was sent from>"}

So, of course, two problems:

  1. The logstash_host value isn't being converted to a JSON object
  2. The hostname that is populated isn't the correct one.

My searches have not been fruitful, how would I go about doing this?

2 
logstash_host {"name":"<hostname of system where log was sent from>"}

Please copy/paste from Kibana's JSON tab (visible when you expand an event). I want to see what the event really looks like.

See How can I get the logstash hostname for how to get the Logstash hostname.

3

Here's the line from the JSON tab of a document:

"logstash_host": "{\"name\":\"ec2-xxx-xx-x-xxxxx-xxx-10-xx-1-68\"}",

I glanced at the logstash hostname link, and that makes sense, my only concern is does it make that socket call for every event is processed?

4

You can use a json filter to parse that JSON string, but if that's not the hostname you're interested in perhaps it doesn't matter.

I glanced at the logstash hostname link, and that makes sense, my only concern is does it make that socket call for every event is processed?

Yes, but I'd expect it to be a cheap operation. Otherwise you should be able to fetch the hostname once when the filter initializes and reuse that value for each event. Something like this might work:

ruby {
  init => "
    require 'socket'
    @@hostname = Socket.gethostname
  "
  code => "event.set('host', @@hostname)"
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.