Kibana4 not working with documents send via python into ElasticSearch

Elastic Stack Kibana
1

Hello,

I am trying to implement an ElasticSearch server to store and display analysis of https answer.
What I'm doing is that I send all the documents via python with the following code :

i = 0
for answer in answers:
  query = json.dumps(answer)
  url = "http://localhost:9200/piccolo-" + str(answer["campaign"]) + "/answer/" + str(i)
  response = requests.post(url, data=query)
  i += 1
return redirect("/", code=302)

which send a packet like this to the ElasticSearch server :

`#@l#FdnV+

U
UPOST /piccolo-0/answer/1023 HTTP/1.1 
Host: localhost:9200 
Content-Length: 584 
User-Agent: python-requests/2.9.1 
Connection: keep-alive 
Accept: */* 
Accept-Encoding: gzip, deflate 
 
{"trust_flag": "trusted", "alert_type": "", "ordered": 0, "campaign": 0, "grade": "D", "ip": "104.20.77.29", "type_str": "TLS 1.2 Handshake (49195)", "complete": 1, "chain_hash": "174bb3c4415d932d291569cd123a9f9f58ec16d9", "version": 771, "answer_type": 21, "type": "TLS 1.2 Handshake (49195)", "ciphersuite": 49195, "timestamp_str": "2015-12-16 10:01:36", "timestamp": 1450260096, "alert_level": "", "date": "2015-12-16 10:01:36", "_timestamp": 1450260096, "name": "txxx.com", "n_transvalid": 2, "rfc": 1, "cipherphrase": "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "chain_length": 6}

And when I try to get the document with the rest API, I get something like this :

{
  "_index": "piccolo-0",
  "_type": "answer",
  "_id": "1",
  "_version": 1,
  "_timestamp": 1450260096,
  "found": true,
  "_source": {
    "trust_flag": "trusted",
    "alert_type": "",
    "ordered": 1,
    "campaign": 0,
    "grade": "B",
    "ip": "23.3.13.42",
    "type_str": "TLS 1.2 Handshake (49199)",
    "complete": 1,
    "chain_hash": "5e66cc799c18d0c05e339f9314e904527cf1e8f7",
    "version": 771,
    "answer_type": 21,
    "type": "TLS 1.2 Handshake (49199)",
    "ciphersuite": 49199,
    "timestamp_str": "2015-12-16 10:01:36",
    "timestamp": 1450260096,
    "alert_level": "",
    "date": "2015-12-16 10:01:36",
    "_timestamp": 1450260096,
    "name": "target.com",
    "n_transvalid": 0,
    "rfc": 0,
    "cipherphrase": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
    "chain_length": 3
  }
}

Which is what I want, but I didn't find the way to visualize all those documents into Kibana4.

Thanks in advance for all the help you will provide me :slightly_smiling:

2

What do you mean here, how do you want to visualise them?

3

Sorry, what I meant is that I just want to visualize them. When I go to kibana, I can choose the default index but I don't get any timestamp choice, and even when I add _timestamp in the metafields, kibana doesn't found any doc.

4

Can you see the timestamp field in the mapping?

5

Here is the mapping that I get in Kibana4 if it is the one you are speaking about.

snapshot3.png1564×654 26.5 KB

Here is the result of this command:

curl -XGET 'http://localhost:9200/piccolo-0/_mapping/answer

and I get

{
  "piccolo-0": {
    "mappings": {
      "answer": {
        "properties": {
          "_timestamp": {
            "type": "date",
            "store": true,
            "format": "strict_date_optional_time||epoch_millis"
          },
          "alert_level": {
            "type": "string"
          },
          "alert_type": {
            "type": "string"
          },
          "answer_type": {
            "type": "long"
          },
          "campaign": {
            "type": "long"
          },
          "chain_hash": {
            "type": "string"
          },
          "chain_length": {
            "type": "long"
          },
          "cipherphrase": {
            "type": "string"
          },
          "ciphersuite": {
            "type": "long"
          },
          "complete": {
            "type": "long"
          },
          "date": {
            "type": "string"
          },
          "grade": {
            "type": "string"
          },
          "ip": {
            "type": "string"
          },
          "n_transvalid": {
            "type": "long"
          },
          "name": {
            "type": "string"
          },
          "ordered": {
            "type": "long"
          },
          "rfc": {
            "type": "long"
          },
          "timestamp": {
            "type": "long"
          },
          "timestamp_str": {
            "type": "string"
          },
          "trust_flag": {
            "type": "string"
          },
          "type": {
            "type": "string"
          },
          "type_str": {
            "type": "string"
          },
          "version": {
            "type": "long"
          }
        }
      }
    }
  }
}
6

You can see the KB has picked the _timestamp field automatically (as evidenced by the little clock icon beside the field name).

When you go into discover, did you set the time range correctly?

7

I tried to put the largest range possible. But even with 5 years it doesn't work.

8

And you changed the index pattern, on the left near the top.

9

I have only one index pattern....

Here is a screenshot I took, I took a very big time range but still nothing

capture d'écran1.png1347×627 51.8 KB

and all the piccolo-* index are selected

10

What's the output from _cat/indices show?

11

Here is the output :

"yellow open piccolo-13012016 5 1 2242 0 1.1mb 1.1mb 
yellow open .kibana          1 1    2 0   5kb   5kb 
"

It's not piccolo-0 anymore but it is the same structure.

12

I think I will give up and go with logstash instead. Thanks again for all the help you gave me :slightly_smiling: