# Messages from gelf input not ending-up in ES
**URL:** https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291
**Category:** Logstash
**Created:** [November 12, 2018, 1:15pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291 "2018-11-12T13:15:52Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@plex1030](https://discuss.elastic.co/u/plex1030)
#### Post date: [November 12, 2018, 1:15pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/1 "2018-11-12T13:15:52Z")
Hi,
I was hoping to use the gelf input plugin to receive messages and put them into ES.
This is my config:
```
input {
gelf {
port => 12201
type => gelf
host => "0.0.0.0"
}
}
...
output {
file {
path => "/home/myuser/gelfoutput/gelf_output.txt"
}
}
output {
elasticsearch {
hosts => ["https://mycloud:9243"]
#index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "xxxx"
password => "xxxx"
document_id => "%{[@metadata][fingerprint]}"
}
}
```
Strangely the messages are successfully being written to my file output but not to ES. I don't see any errors in the logstash log-files.
Any ideas ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 12, 2018, 1:20pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/2 "2018-11-12T13:20:23Z")
Do you have a fingerprint metadata field in all your documents? Is noting being written to the current (default) `logstash-*` index?
---
### Author:  [@plex1030](https://discuss.elastic.co/u/plex1030)
#### Post date: [November 12, 2018, 1:22pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/3 "2018-11-12T13:22:47Z")
I do:
```
filter {
fingerprint {
source => ["@timestamp","message"]
target => "[@metadata][fingerprint]"
method => "SHA256"
key => "xxx"
}
}
```
There is nothing from the gelf input written to that index, at least nothing that I could see in ES. I have another filebeats input and those messages are showing up in ES.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 12, 2018, 1:32pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/4 "2018-11-12T13:32:46Z")
If Logstash can not write to Elasticsearch, it will stop and retry until successful. If you are constantly seeing new self data being written to the file, it should therefore also be going into Elasticsearch. How do you identify data coming in through the gelf plugin?
---
### Author:  [@plex1030](https://discuss.elastic.co/u/plex1030)
#### Post date: [November 12, 2018, 1:35pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/5 "2018-11-12T13:35:17Z")
That's what I'd have expected but it's just simply not showing up.
I'm just looking at the Discover page in Kibana and while all other messages are shown, the ones from gelf are not.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 12, 2018, 1:36pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/6 "2018-11-12T13:36:45Z")
Try adding a tag or field in the gelf input and then filter on this in the discover screen. The Discover screen only shows a sample, so if the amount of data coming in via the gelf plugin is small it might easily get missed.
---
### Author:  [@plex1030](https://discuss.elastic.co/u/plex1030)
#### Post date: [November 12, 2018, 1:38pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/7 "2018-11-12T13:38:48Z")
We're just getting started with logstash, so our input volume is very very low. Not a single message is being shown for the last 15 Minutes and I've sent multiple gelf messages 1 minute ago.
When I widen the windows to Today I can see all other messages from filebeats.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 12, 2018, 1:40pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/8 "2018-11-12T13:40:06Z")
Then I would recommend enabling debug logging in Logstash to see if there is any error reported.
---
### Author:  [@plex1030](https://discuss.elastic.co/u/plex1030)
#### Post date: [November 12, 2018, 1:47pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/9 "2018-11-12T13:47:16Z")
Good idea!
Here's what I see when I send a gelf message:
```
[2018-11-12T14:45:43,131][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-11-12T14:45:43,131][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-11-12T14:45:43,903][DEBUG][logstash.pipeline] filter received {"event"=>{"source_host"=>"10.122.64.20", "level"=>6, "version"=>"1.1", "host"=>"hostname", "@timestamp"=>2018-11-12T13:45:43.874Z, "message"=>"hello gelf", "@version"=>"1", "type"=>"gelf"}}
[2018-11-12T14:45:43,904][DEBUG][logstash.filters.grok] Running grok filter {:event=>#}
[2018-11-12T14:45:43,905][DEBUG][logstash.filters.grok] Event now: {:event=>#}
[2018-11-12T14:45:43,906][DEBUG][logstash.pipeline] output received {"event"=>{"source_host"=>"10.122.64.20", "level"=>6, "version"=>"1.1", "host"=>"hostname" "tags"=>["_grokparsefailure"], "@timestamp"=>2018-11-12T13:45:43.874Z, "message"=>"hello gelf", "@version"=>"1", "type"=>"gelf"}}
[2018-11-12T14:45:43,967][DEBUG][logstash.outputs.file] File, writing event to file. {:filename=>"/home/iniuser/gelfoutput/gelf_output.txt"}
[2018-11-12T14:45:43,968][DEBUG][logstash.outputs.file] Starting stale files cleanup cycle {:files=>{"/home/iniuser/gelfoutput/gelf_output.txt"=>#>}}
[2018-11-12T14:45:43,968][DEBUG][logstash.outputs.file] 0 stale files found {:inactive_files=>{}}
[2018-11-12T14:45:44,114][DEBUG][logstash.outputs.file] Starting flush cycle
[2018-11-12T14:45:44,115][DEBUG][logstash.outputs.file] Flushing file {:path=>"/home/iniuser/gelfoutput/gelf_output.txt", :fd=>#>}
[2018-11-12T14:45:46,115][DEBUG][logstash.outputs.file] Starting flush cycle
[2018-11-12T14:45:46,115][DEBUG][logstash.outputs.file] Flushing file {:path=>"/home/iniuser/gelfoutput/gelf_output.txt", :fd=>#>}
[2018-11-12T14:45:46,609][DEBUG][logstash.pipeline] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#"}
[2018-11-12T14:45:48,116][DEBUG][logstash.outputs.file] Starting flush cycle
[2018-11-12T14:45:48,116][DEBUG][logstash.outputs.file] Flushing file {:path=>"/home/iniuser/gelfoutput/gelf_output.txt", :fd=>#>}
[2018-11-12T14:45:48,134][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-11-12T14:45:48,134][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
```
I don't see it say anything about ES but I'm not sure why that would be. No errors though.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 12, 2018, 1:50pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/10 "2018-11-12T13:50:06Z")
You do not seem to have showed your complete config. Might there be an issue in the parts we have not seen, e.g. conditionals?
---
### Author:  [@plex1030](https://discuss.elastic.co/u/plex1030)
#### Post date: [November 12, 2018, 1:54pm UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/11 "2018-11-12T13:54:22Z")
Thank you for bearing with me 🙂
My complete config1:
```
input {
beats {
port => 5044
host => "0.0.0.0"
}
}
filter {
grok {
match => { "message" => ["\A%{TIMESTAMP_ISO8601}\s-\s[a-zA-Z0-9]+\s-\s%{LOGLEVEL:log-level}\s-\sBestelltnummer:\s(?[0-9]{9}),\sILN:\s(?[0-9]{13}),\sKundenNr\. (?[0-9]{6}),\s(?[0-9]{1,3})\sST,\sArtNr.:\s(?[0-9]{13,14})", "\A%{TIMESTAMP_ISO8601}\s-\s[a-zA-Z0-9]+\s-\s%{LOGLEVEL:log-level}\s-\s%{DATA}:\s(?%{GREEDYDATA})", "\A%{TIMESTAMP_ISO8601}\s-\s[a-zA-Z0-9]+\s-\s%{LOGLEVEL:log-level}\s-\s"] }
}
mutate {
convert => { "stueckzahl" => "integer" }
}
#fingerprint {
# source => ["@timestamp","message"]
# target => "[@metadata][fingerprint]"
# method => "SHA256"
# key => "xxx"
#}
#ruby {
# code => "event.set('@metadata[tsprefix]', event.get('@timestamp').to_i.to_s(16))"
#}
}
filter {
fingerprint {
source => ["@timestamp","message"]
target => "[@metadata][fingerprint]"
method => "SHA256"
key => "xxxxx"
}
}
output {
elasticsearch {
hosts => ["https://mycloud:9243"]
#index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "xxx"
password => "xxx"
document_id => "%{[@metadata][fingerprint]}"
}
}
```
In a 2nd config-file:
```
input {
gelf {
port => 12201
type => gelf
host => "0.0.0.0"
}
}
output {
file {
path => "/home/myuser/gelfoutput/gelf_output.txt"
}
}
```
---
### Author:  [@plex1030](https://discuss.elastic.co/u/plex1030)
#### Post date: [November 14, 2018, 9:26am UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/12 "2018-11-14T09:26:41Z")
Any ideas ? I'm kinda lost here.
---
### Author:  [@plex1030](https://discuss.elastic.co/u/plex1030)
#### Post date: [November 14, 2018, 10:17am UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/13 "2018-11-14T10:17:02Z")
Okay....I've decided to try and put the messages from gelf in a different index. Turns out this works which probably means ES is rejecting the messages because of some field-mismatch.
How do I find out what exactly is causing this, without getting any errors ? Because the ES logs (in the cloud interface) are not showing anything other than INFOs on snapshots.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 12, 2018, 10:17am UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291/14 "2018-12-12T10:17:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.