My packetbeat events don't show up in Kibaba

Hi,
I once use the follwing ELK + packetbeat stack before, the packetbeat events showed up in Kibana as expected.
packet:1.0.0-beta2
elasticsearch: 1.4.4
logstash: 1.4.2
redis: 3.0.1
Kibana:4.1.1

Now I upgrade the ELK & packetbeat, The packetbeat doesn't show up in Kibana now.
packetbeat: 1.0.1
elasticsearch: 2.1.0
Kibana: 4.3.0

Here is the print screen , no results found.

I query in the Elasticsearch, the packetbeat indexes are available.
[root@i-a623ee17 ~]# curl '10.10.0.168:9200/_cat/indices?v' | grep packetbeat
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4845 100 4845 0 0 10248 0 --:--:-- --:--:-- --:--:-- 10243
green open packetbeat-2015.12.30 5 1 157199 0 168.9mb 84.6mb
green open packetbeat-2015.12.31 5 1 151899 0 160.9mb 80.3mb
green open packetbeat-2015.12.24 5 1 160891 0 173.5mb 86.7mb
green open packetbeat-2015.12.25 5 1 151637 0 160.4mb 80.1mb
green open packetbeat-2015.12.22 5 1 120476 0 139.8mb 69.9mb
green open packetbeat-2015.12.23 5 1 154092 0 166.8mb 83.3mb
green open packetbeat-2015.12.28 5 1 151771 0 160.7mb 80.4mb
green open packetbeat-2015.12.29 5 1 152689 0 162.3mb 81.1mb
green open packetbeat-2015.12.26 5 1 151437 0 160.5mb 80.3mb
green open packetbeat-2015.12.27 5 1 151578 0 160.9mb 80.4mb
green open packetbeat-2015.12.21 5 1 92282 0 101.7mb 50.8mb
green open packetbeat-2015.12.20 5 1 90493 0 98.9mb 49.5mb
green open .packetbeat-topology 5 1 2 0 29kb 14.5kb
green open packetbeat-2016.01.01 5 1 152914 0 162.2mb 81.1mb
green open packetbeat-2016.01.02 5 1 152788 0 162mb 81mb
green open packetbeat-2016.01.03 5 1 152660 0 161.3mb 80.7mb
green open packetbeat-2015.12.18 5 1 44453 0 44.3mb 22.1mb
green open packetbeat-2016.01.04 5 1 19793 0 21.8mb 10.9mb
green open packetbeat-2015.12.19 5 1 90342 0 98.6mb 49.3mb

I also load the packetbeat index template using the following command.
curl -XPUT 'http://localhost:9200/_template/packetbeat' -d@/etc/packetbeat/packetbeat.template.json

I am not sure if the kibana 4.3.0 is compatible with packetbeat 1.0.1, or if there are some mistake in my setup.

I will be grateful if you can help me to figure out the root cause. Thank you.

We currently recommend to use the following pattern for packetbeat in Kibana: [packetbeat-]YYYY.MM.DD Can you try if it works with this pattern? To be honest, I would think it should not make a difference and your setup should work as expected, but it is worth a try.

Is your Kibana instance connecting to the same elasticsearch cluster as packetbeat?

Hi Rufin,
I try the pattern [packetbeat-]YYYY.MM.DD in Kibana, just as you said, it doesn't work.
I am sure my kibana is connecting to the same elasticsearch cluster, I store the logstash events and packetbeat events in the same ES, my logstash events show up properly, but the packetbeat events doesn't show up in Kibana.

Are there any other suggestions to debug this issue?

The thing that bugs me is that you were even able to setup the pattern. If there are no docs, as far as I know you can't setup the pattern in Kibana. So it seems like Kibana knows about the docs but they don't show up?

Yes, it seems the Kibana knows about the packetbeat docs, please refer to the following print screen.

But they don't show up in the 'Discover' tab.

I notice you have live demo of packetbeat, http://demo.elastic.co/packetbeat/#/dashboard/Packetbeat-Dashboard?_g=()
Can you please let me know the version of Kibana, packetbeat, ES?

The events from filebeat 1.0.01 show up under discovery but not the one from packetbeat 1.0.1? The part I don't get is that there could be incompatibility with the dashboards but the discovery page is mainly showing a list of events.

Could you try to select in Kibana last month instead of last 5 years?

It might be that just a refresh of the index pattern fixes the issue. This can be done under the Settings page and it forces Kibana to get the latest fields of the index pattern.

In case this doesn't fix the issue, I suggest to use the load script that we provide to create the index patterns for all the beats with the latests fields. More details can be found here.

Hi @ruflin & @monica,
I am able to search the packetbeat events in the Kibana now. The problem is that I load the old version of index template packetbeat.template.json(1.0.0-beta2).
The issue is gone when I load the 1.0.1 packetbeat.template.json

As you can see the differences of index template as below, the 1.0.1 packetbeat.template.json have the date type @timestamp.

I deeply appreciate all your effort helping walk out of this issue. Happy New Year to you.

@Augustin_Chen I'm really happy that you got it working
@monica Thanks for helping out here

1 Like