Kibana shows only last updated _type

Elastic Stack Kibana
1

I have a index which includes two types of _type .

However , on Kibana , I am only able to see 1 type.
This index has 2 types, text and binary.

kibana-image1.PNG239×651 14.9 KB

I have indexed two different types from different source in below order.

1 index document using index function of python-api

es.index( index=get_index_name(fp) + get_current_date(), doc_type=attachment_type, body=doc)

above get_index_returns this

"attachment-web-"

2 index document using logstash.

elasticsearch {
  hosts         => ["http://127.0.0.1:9200"]
  index         => "attachment-web-%{[@metadata][index_date]}"
  document_type => "text"

  user          => "elastic"
  password      => "elastic"
}

Both use below template.

{
  "template" : "index-web-*",
  "mappings" : {
    "web" : {
      "properties" : {
        "access_time"    : { "type" : "date", "format" : "yyyy-MM-dd HH:mm:ss" },
        "source_ip"      : { "type" : "keyword" },
        "source_mac"     : { "type" : "keyword" },
        "web_host"       : { "type" : "keyword" },
        "web_host_mac"   : { "type" : "keyword" },
        "url"            : { "type" : "keyword" },
        "content_type"   : { "type" : "keyword" },
        "req_method"     : { "type" : "keyword" },
        "resp_body_size" : { "type" : "integer" },
        "stored_file"    : { "type" : "keyword" }
      }
    }
  }
}

According to elasticsearch.log , looks like the mapping is updated when the indexing by logstash

[2017-03-03T12:21:28,485][INFO ][o.e.c.m.MetaDataMappingService] [IL9zTP3] [attachment-web-2017.03.03/2T7Ehp65Tgi-Qn81pKxv8Q] create_mapping [binary]
[2017-03-03T12:24:54,722][INFO ][o.e.c.m.MetaDataCreateIndexService] [IL9zTP3] [attachment-mail-2017.03.03] creating index, cause [auto(index api)], templates [attachment-mail], shards [5]/[1], mappings [text]
[2017-03-03T12:24:54,915][INFO ][o.e.c.m.MetaDataMappingService] [IL9zTP3] [attachment-mail-2017.03.03/i-FSgDwZQDeQy91rImYz_w] create_mapping [binary]
[2017-03-03T12:25:08,726][INFO ][o.e.c.m.MetaDataMappingService] [IL9zTP3] [attachment-web-2017.03.03/2T7Ehp65Tgi-Qn81pKxv8Q] update_mapping [text]
[2017-03-03T12:25:09,341][INFO ][o.e.c.m.MetaDataMappingService] [IL9zTP3] [attachment-mail-2017.03.03/i-FSgDwZQDeQy91rImYz_w] update_mapping [text]

Is this update_mapping causing Kibana to not show type binary? If yes , How can I avoid the update_mapping?

2

UPDATE

Documents both updated by python api and logstash have identicle field names .

Document from logstash

{
  "_index": "attachment-web-2017.03.03",
  "_type": "text",
  "_id": "AVqSM4649z4yE_g9EcBj",
  "_score": null,
  "_source": {
    "stored_file": "HTTP_qo7DLR.txt",
    "offset": 982,
    "input_type": "log",
    "source": "/var/tmp/charset/UTF-8/HTTP_qo7DLR.txt",
    "message": "xxxxx"
    "type": "log",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "@timestamp": "2017-03-03T03:25:31.661Z",
    "@version": "1",
    "beat": {
      "hostname": "ElasticServer",
      "name": "ElasticServer",
      "version": "5.2.1"
    },
    "host": "ElasticServer",
    "mongodb_insert_id": "n/a",
    "fields": {
      "encoding": "UTF-8"
    }
  },
  "fields": {
    "@timestamp": [
      1488511531661
    ]
  },
  "sort": [
    1488511531661
  ]
}

Document from python.

{
  "_index": "attachment-web-2017.03.03",
  "_type": "binary",
  "_id": "AVqSMvSm9z4yE_g9EZX6",
  "_score": null,
  "_source": {
    "@timestamp": "2017-03-03T03:24:57.382Z",
    "stored_file": "20170217175801-20.http",
    "mongodb_insert_id": "58b8e20968cc1222bf6cfad2",
    "message": "n/a",
    "source": "/var/tmp/attachment/migration/20170217175801-20.http",
    "beat": {
      "hostname": "n/a",
      "name": "n/a",
      "version": "n/a"
    },
    "offset": 0,
    "input_type": "n/a",
    "tags": [
      "processed_by_script"
    ],
    "fields": {
      "encoding": "binary"
    },
    "type": "binary",
    "@version": "1",
    "host": "n/a"
  },
  "fields": {
    "@timestamp": [
      1488511497382
    ]
  },
  "sort": [
    1488511497382
  ]
}

Any help is appreciated!

3

Hi,

If this is non-production system that you can delete and recreate these indices on I would suggest this;

  1. Delete the index
  2. load one set of your data into the index and check the doc count with _cat/indices?v
  3. load the other set of data into the index and check the doc count again. Did the doc count go up?

You can also get a count by type with something like this;
http://localhost:9200/attachment-web-2017.03.03/text/_count

The other thing to be aware of is that POST can be used to achieve auto-generation of ids whereas a PUT is used when you want to specify an id. I'm pretty sure that neither logstash nor your python index method are using PUT and updating the existing docs, but if there's a common id between them it could be possible to do that.

Regards,
Lee

4

@LeeDr

Thanks for the reply.

I will take a look at it.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.