Daily redefinition of index fields

Elastic Stack Elasticsearch
1

Hi every body, I have been working in a custom index, the first time that I define this index the mapping takes the data types that I need, however , when a new day starts it redefine the fields of the index and puts string as a datatype instead the original datatypes that I had put before, what I'm missing?

This is the way I define the index:

curl -X PUT "localhost:9200/my_index-2018.06.14" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "myindex_log": {
      "properties": {
        "amount":   { "type": "double", "null_value":"0" },
        "budget":   { "type": "double", "null_value":"0"  },
        "cc":       { "type": "integer" },
        "creationTime": { "type": "text" },
        "fm":      { "type": "text" },
        "host":    { "type": "text" },
        "kpi":     { "type": "text" },
        "message": { "type": "text" },
        "mkt":     { "type": "text" },
        "price":   { "type": "double", "null_value":"0" },
        "status":  { "type": "integer" },
        "success": { "type": "boolean" },
        "type":    { "type": "integer" }
      }
    }
  }
}
'

Can someone provide me some guidance?

Thank you.

2

It sounds like what you're looking for is index templates. You'd define your mappings and then set index_patterns to my_index-* or whatever your pattern is

3

Hello shanec.

Thank you very much for your help. Now what I've done is declare the index as first step, to do that y executed this line at Kibana 's Dev Tools:

PUT my_index-2018.06.18?pretty

The answer that I recived into the console is:

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "my_index-2018.06.18"
}

After that I declare my index template as follow:

curl -X PUT "localhost:9200/_template/my_index_template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["my_index-*"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "my_index": {
      "_source": {
        "enabled": true
      },
      "properties": {
        "amount":   { "type": "double", "null_value":"0" },
        "budget":   { "type": "double", "null_value":"0"  },
        "cc":       { "type": "integer" },
        "creationTime": { "type": "text" },
        "fm":      { "type": "text" },
        "host":    { "type": "text" },
        "kpi":     { "type": "text" },
        "message": { "type": "text" },
        "mkt":     { "type": "text" },
        "price":   { "type": "double", "null_value":"0" },
        "status":  { "type": "integer" },
        "success": { "type": "boolean" },
        "type":    { "type": "integer" }
      }
    }
  }
}
'

The answer I received from this transaction is:

{"acknowledged":true}

Then I checked the index with this sentence:

GET /_cat/indices?v

And this is the answer that I got:

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .monitoring-kibana-6-2018.06.18 LDKVSWAKTC-B-6rjzpvCYw   1   0        133            0    132.6kb        132.6kb
yellow open   my_index-2018.06.18             AGrkXoFzRQimfX052GHIJw   5   1          0            0      1.1kb          1.1kb
green  open   .monitoring-es-6-2018.06.18     LBXmfnRkQceYUB6qgP_pyA   1   0       1436          306      1.1mb          1.1mb
green  open   .monitoring-es-6-2018.06.15     vA1yF6blQrSYpO8P52R94w   1   0      12825         3490      5.9mb          5.9mb
green  open   .monitoring-kibana-6-2018.06.15 NxgGQ5XjRkKSumGIrPvrlA   1   0       1832            0    557.1kb        557.1kb

After that I restart elasticsearch and kibana and then I start to send data from my app to the logs that logstash reads, however, Kibana does not show me any data at the discover screen and when I move to the management module and try to define a new index patterns; based on the index that I have defined before, this also does not show me any data.

They way that I ve configured my output in logstash config file is:

output {

     elasticsearch { hosts => ["localhost:9200"]
                     index => "my_index-%{YYYY.MM.dd}"
     }

     stdout { codec => rubydebug }
}

An example of the data that I send to the logs is:

2018-06-01 19:04:51.274 INFO 7267 --- [nio-8080-exec-3] c.a.b.api.controller.UserController : PERSISTED BID: Bid{id=1527897891207313, foreignId=null, amount=400, budget=null, price=745, exchangeSymbol=BTC, liquiditySymbol=EUR, tickerPrice=false, type=1, status=1, creationTime=Fri Jun 01 19:04:51 CDT 2018, updateTime=null, enabled=true, bidReductorList=null, bidParameterList=null, operationList=null, user=com.mycompany.myapp.persistence.vo.User[ id=1526579465576364 ]}

And the form how this data is transformed for elasticsearch is:

{
            "type" => "1",
            "host" => "user-laptop",
              "cc" => "BTC",
              "fm" => "EUR",
         "message" => "2018-06-01 19:04:51.274  INFO 7267 --- [nio-8080-exec-3] c.a.b.api.controller.UserController      : PERSISTED BID: Bid{id=1527897891207313, foreignId=null, amount=400, budget=null, price=745, exchangeSymbol=BTC, liquiditySymbol=MXN, tickerPrice=false, type=1, status=1, creationTime=Fri Jun 01 19:04:51 CDT 2018, updateTime=null, enabled=true, bidReductorList=null, bidParameterList=null, operationList=null, user=com.mycompany.myapp.persistence.vo.User[ id=1526579465576364 ]}",
    "creationTime" => "Fri Jun 01 19:04:51 CDT 2018",
             "mkt" => "false",
        "@version" => "1",
           "price" => "745",
          "status" => "1",
          "succes" => "true",
          "amount" => "400",
          "budget" => "0",
          "kpi" => "buy_limited",
      "@timestamp" => 2018-06-18T16:00:24.254Z
}

Is there any thing that I'm doing wrong or maybe I missed? Thank you very much.

4

You should define your index template before you create the index, not after. Once you've declared an index template, it will be inherited for future indices.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.