Packetbeat data not going from ES to Kibana

Elastic Stack Beats
1

Hi guys,

Not sure if this is the best category, as it may be either an ElasticSearch or Kibana issue, but since I followed the guide found here: https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-getting-started.html I think it's fair to create it here.

As mentioned, I followed the guide provided by Elastic, skipping the ES and Kibana installs as they were already installed. I tried installing Packetbeat on the ELK server, and also another server hosting mongodb.

When I followed the step which asks you to test it via creating a HTTP request ( "curl http://www.elastic.co/ > /dev/null" ), I did this, and then did the curl ( "curl -XGET 'http://localhost:9200/packetbeat-*/_search?pretty' ") however no HTTP queries were returned by Elastic. I thought it was weird, so I installed it on the server hosting MongoDB, and again didn't see the HTTP query. What I did see however, was data from my MongoDB server - so this indicated it was working to me . Here's an example:

ubuntu@proxy01:/var/log/elasticsearch$ curl -XGET 'http://localhost:9200/packetbeat-*/_search?pretty'
{
"took" : 11,
"timed_out" : false,
"_shards" : {
"total" : 20,
"successful" : 20,
"failed" : 0
},
"hits" : {
"total" : 628816,
"max_score" : 1.0,
"hits" : [ {
"_index" : "packetbeat-2015.09.04",
"_type" : "mongodb",
"_id" : "AU-WuXSOCLSXQAff_ybn",
"_score" : 1.0,
"_source":{"bytes_in":151,"bytes_out":90,"client_ip":"10.2.194.215","client_port":45066,"client_proc":"","client_server":"mongo-primary.QA.DEV.LOCAL","count":1,"ip":"10.2.194.216","method":"otherCommand","mongodb":{"cursorId":0,"fullCollectionName":"local.$cmd","numberReturned":1,"numberToReturn":1,"numberToSkip":0,"startingFrom":0},"port":27017,"proc":"","query":"local.$cmd.otherCommand().limit(1)","resource":"local.$cmd","responsetime":0,"server":"","shipper":"mongo-primary.QA.DEV.LOCAL","status":"OK","timestamp":"2015-09-04T04:59:38.896Z","type":"mongodb"}

When I continued the guide, following the steps for creating the packetbeat-* index pattern within Kibana, I couldn't see any data. Even after the weekend. So, following some troubleshooting advice with a nice man named "Warkolm" in IRC, I was able to further verify data in ES

ubuntu@proxy01:~$ curl localhost:9200/_cat/indices
yellow open logstash-2015.09.04 5 1 167712 0 81.5mb 81.5mb
yellow open packetbeat-2015.09.06 5 1 195938 0 25.4mb 25.4mb
yellow open logstash-2015.09.02 5 1 243716 0 47mb 47mb
yellow open logstash-2015.09.01 5 1 240887 0 45.2mb 45.2mb
yellow open packetbeat-2015.09.05 5 1 195951 0 25.5mb 25.5mb
yellow open packetbeat-2015.09.04 5 1 143022 0 65.3mb 65.3mb
yellow open logstash-2015.08.31 5 1 307528 0 86.8mb 86.8mb
yellow open .packetbeat-topology 5 1 2 0 12.8kb 12.8kb
yellow open packetbeat-2015.09.07 5 1 83439 0 22mb 22mb
yellow open logstash-2015.09.03 5 1 144903 0 64.4mb 64.4mb
yellow open logstash-2015.08.12 5 1 18 0 154kb 154kb
yellow open logstash-2015.08.28 5 1 182024 0 39.5mb 39.5mb
yellow open .kibana 1 1 3 1 14.1kb 14.1kb

Output of ES logfile:

ubuntu@proxy01:/var/log/elasticsearch$ cat logstash.log
[2015-09-07 00:00:02,988][INFO ][cluster.metadata ] [smoker] [packetbeat-2015.09.07] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [default]
[2015-09-07 00:00:03,843][INFO ][cluster.metadata ] [smoker] [packetbeat-2015.09.07] update_mapping [mongodb] (dynamic)
[2015-09-07 06:33:49,082][INFO ][cluster.metadata ] [smoker] [packetbeat-2015.09.07] update_mapping [http] (dynamic)

And here is a screenshot from Kibana:

http://puu.sh/k2uwU/9319f1c8e8.png

pbeatss.png1624×943 47.5 KB

At this point i'm quite stumped, so any further advice would be great

2

Just to add, after adding the index filter into KB ad then heading to discover, it just sits there.

I note that your hostname is proxy01, does that imply there is a proxy between KB and ES?

Happy to help :stuck_out_tongue:

1 Like
3

So data seems to be in Elasticsearch, but it's not shown by Kibana, right? Maybe try opening the developer console in your browser and see if it reports any Javascript errors.

4

I tried deleting and re-creating the index pattern packetbeat-* with timestamp, no dice though..

@warkolm - no, just the name of the host. And I am so sorry for calling you a lady!!! It was a long day

@tudor - there are some js errors relating to "timestamp" being a missing field. How can @timestamp be missing? Isnt it provided by ES? The error occurs each time I click "discover"

5

@timestamp is not provided by ES, it's just added by Logstash. Packetbeat uses timestamp, without the @. You should try deleting the index pattern in Kibana and add it again using timestamp for the timestamp field. If timestamp without the @ is not offered, I recommend going through: stop packetbeat, delete all packetbeat-* indexes, make sure the packetbeat template is loaded, start packetbeat.

6

@tudor that's exactly what I had done already - I did it again just for good measure and it did offer "timestamp" and not "@timestamp". It's still not showing up in Discover though.. It's kind of weird because the indices do match the packetbeat-* pattern thats created ..

7

Do you still see the JS exceptions talking about missing @timestamp? Maybe we've moved to another error now.

8

I have the similar issue, I can see the packetbeat-* indexes in ES, but the Kibana is now showing any packetbeat events. I use the following stacks:
packetbeat: 1.0.0
ES: 2.1.0
Kibana: 4.3.0

This is what I get from the ES.
[root@i-6d31fcdc ~]# curl -XGET 'http://10.10.0.65:9200/packetbeat-/_search?pretty'
{
"took" : 20,
"timed_out" : false,
"_shards" : {
"total" : 35,
"successful" : 35,
"failed" : 0
},
"hits" : {
"total" : 651145,
"max_score" : 1.0,
"hits" : [ {
"_index" : "packetbeat-2015.12.18",
"_type" : "http",
"_id" : "AVG0YK-uqvqHQIKqybYs",
"_score" : 1.0,
"_source":{"@timestamp":"2015-12-18T09:16:53.097Z","beat":{"hostname":"i-1e3ffbae","name":"i-1e3ffbae"},"bytes_in":455,"bytes_out":725,"client_ip":"10.10.1.76","client_port":11430,"client_proc":"","client_server":"","count":1,"direction":"in","http":{"code":200,"content_length":534,"phrase":"OK"},"ip":"10.10.2.83","method":"GET","params":"","path":"/auth-services/oauth/validatetoken","port":80,"proc":"nginx","query":"GET /auth-services/oauth/validatetoken","real_ip":"10.10.0.152","request":"GET /auth-services/oauth/validatetoken HTTP/1.1\r\nhost: auth-services-pw-dev-internal.mse-esp.com\r\nAccept: text/plain, application/json, application/+json, /\r\nAuthorization: Bearer160e0d7e-611a-443f-97cc-f924af6c5bf0\r\nCache-Control: no-cache\r\nnonce: 963e0180af86a91998cdb5d889ce447e\r\nPragma: no-cache\r\nts: 1450430212862\r\nUser-Agent: Java/1.7.0_91\r\nX-Forwarded-For: 10.10.0.152\r\nX-Forwarded-Port: 443\r\nX-Forwarded-Proto: https\r\nConnection: keep-alive\r\n\r\n","response":"HTTP/1.1 200 OK\r\nServer: nginx\r\nDate: Fri, 18 Dec 2015 09:16:53 GMT\r\nContent-Type: application/json\r\nContent-Length: 534\r\nConnection: keep-alive\r\nCache-Control: no-store\r\nPragma: no-cache\r\n\r\n","responsetime":4,"server":"i-1e3ffbae","status":"OK","tags":["pw-dev","pw","tomcat","auth-services","packetbeat"],"type":"http"}
}

I used following stacks before, the Kibana shows the packetbeat events welll.
packetbeat : 1.0.0-beta2
Redis:2.8.9
Logstash:1.4.2
ES: 1.4.4
Kibana: 4.1.1