Logstash get data but Kibana can't show it!

Elastic Stack Logstash
1

Hello.
I'm using "Winlogbeat for sending Windows event Log to my ELK. Logstash get data:

$ curl -XGET 'http://localhost:9200/_cat/indices?v'

yellow open   beat-2017.08.22       Ez89QEvaR8207QD06iY3jw   5   1          8

My logstash config:

input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch {
#    action => "%{[@metadata][action]}"
#    document_id => "%{[@metadata][_id]}"
    hosts => ["172.30.9.20"]
    #index => "index_name"
   index => "beat-%{+YYYY.MM.dd}"
  }
}

And:
beat

How can I troubleshooting it?

Thank you.

2

The problem occurs if you dont chose "time based data" properly when creating the index. If you dont want to use timestamp chose it. Only if you chose properly it will work. Otherwise, discover will not show anything.

3

How to chose “time based data”? Can you show me how can I solve it? If you mean time in Kibana then I must say that it is a Log for today and as you see, I selected "Today".

4

When you create a new index in kibana, i think you created one as i can see in the figure you attached. Did you chose
"Time filter field name"?? If you have any field which you want to act as timefilter chose from the drop down list. Otherwise, chose "I dont want to use to timefilter". If you dont do this index will be created but discover will not show up any data. Hth.

5

beat-2
I can't find any option "“I dont want to use to timefilter" !!! Can you address it?
In your opinion, Is my config OK? How can I clear cache?

6

Uncheck index contains time-based events. To clear cache you can do it like this

curl -XPOST 'http://localhost:9200/_cache/clear'

I also do the following:

curl -XDELETE 'http://localhost:9200/_all'

Should resolve the issue. Key is to chose no timefilter.

7

both commands tell me:

{"acknowledged":true}

"tcpdump" tell me I have data stream:

# tcpdump -n -s 0 -i enp4s0 -X tcp port 5044 and host 172.30.10.15
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp4s0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:39:10.765514 IP 172.30.10.15.49258 > 172.30.9.20.lxi-evntsvc: Flags [S], seq 4228250198, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
	0x0000:  4500 0034 01a8 4000 7f06 8ebc ac1e 0a0f  E..4..@.........
	0x0010:  ac1e 0914 c06a 13b4 fc05 fa56 0000 0000  .....j.....V....
	0x0020:  8002 2000 1935 0000 0204 05b4 0103 0308  .....5..........
	0x0030:  0101 0402                                ....
10:39:13.766075 IP 172.30.10.15.49258 > 172.30.9.20.lxi-evntsvc: Flags [S], seq 4228250198, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
	0x0000:  4500 0034 01a9 4000 7f06 8ebb ac1e 0a0f  E..4..@.........
	0x0010:  ac1e 0914 c06a 13b4 fc05 fa56 0000 0000  .....j.....V....
	0x0020:  8002 2000 1935 0000 0204 05b4 0103 0308  .....5..........
	0x0030:  0101 0402                                ....
10:39:19.768996 IP 172.30.10.15.49258 > 172.30.9.20.lxi-evntsvc: Flags [S], seq 4228250198, win 8192, options [mss 1460,nop,nop,sackOK], length 0
	0x0000:  4500 0030 01aa 4000 7f06 8ebe ac1e 0a0f  E..0..@.........
	0x0010:  ac1e 0914 c06a 13b4 fc05 fa56 0000 0000  .....j.....V....
	0x0020:  7002 2000 2d44 0000 0204 05b4 0101 0402  p...-D..........

but I guess it is not working properly. I mean is that I sense data can't receive :frowning:

8

Sorry, still not working. I am not an expert here. But, faced similar issue and solved it like i mentioned here. Maybe you need to wait for experts comment now.

9

Thank you. The problem is that this forum is so slow.

10

Not really. But there are many questions to answer. I am tagging some people who helped in my problems. @magnusbaeck @fbaligand.

11

@magnusbaeck @fbaligand can offer any help?

12

Up to me, the main problem is that you have a wrong index mapping.
Question : why do you put logstash between beats and elasticsearch ?
Beats can directly send data to elasticsearch. And when it does that, it puts the right template mapping for your data.
If you use Winlogbeat, it could be relevant to import specific Winlogbeat elasticsearch mapping template and kibana dashboards.

Steps 4 and 6 here :
https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html

13

My index is "beat" as you see in config file.
Thus, I don't need "Logstash". OK, I disabled it.
but if I want to use a diagram like below then what should I do:
"Winlogbeat -> Logstash -> Elasticsearch -> Kibana".

14

If you don't need document transformation to another output than elasticsearch, Logstash is useless.

That said, you have to set this in Logstash elasticsearch output :
manage_template => false

And to import manually elasticsearch index template for winlogbeat.

All of this is explained here :
https://www.elastic.co/guide/en/beats/winlogbeat/current/config-winlogbeat-logstash.html

Globally I advice you to read carefully getting started doc for winlogbeat which is very well done and explain you all the possibilities.

15

I configured it correctly as URL said.
Where I did "manage_template => false" ?

16

Where I said in my previous comment :
In Logstash elasticsearch output

17

I never set it!!!

18

OK, to be more clear :

  • by default, Logstash elasticsearch output puts its own index template into elasticsearch. A template that is relevant with Logstash logs.
  • in your case, Logstash template is not relevant because it is winlogbeat data.
  • If you wish winlogbeat data is correctly ingested into elasticsearch, you have to put winlogbeat template into elasticsearch
  • you have to disable template management in Logstash elasticsearch output using manage_template => false
  • finally, if you think I said "you did set this setting", you misunderstand me. Sorry to tell that, but when I say "You have to", it means "You must" (it is synonym). So yes, you have not set this setting, and if you want that it works, you must set this setting.
  • globally, all that I said is explained in winlogbeat "getting started" :
    https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-getting-started.html
19

Thus, I must put "manage_template => false" into my "winlogbeat" configuration?

20

No, just in Logstash.