Use API of elasticsearch ? Change settings

Beuhlet_Reseau · April 3, 2017, 9:18am

Hello,

I want increase performance of indexing, usage disk of elasticsearch. I don't know elk API

Some questions stay without response (after reading the official doc)

How Use bulk requests (I have for the moment, 2 index per days)
Disable refresh and replicas.
/!\ : I am on test step, so i delete, create index to find the better configuration.

So, i want know how configure this options ? (It's impossible in elasticsearch.yml since version 5+ of elk). I believe we must use api ? But how.

3.Indexing buffer size : By default is take 10% of jvm (so it's not necessary to modify it)

4.Use index template

I don't know how to create index, where create index... I believe we can change refresh interval and replicas by default before creation of any index.

dadoonet · April 3, 2017, 9:39am

Bulk API | Elasticsearch Guide [8.11] | Elastic
Update index settings API | Elasticsearch Guide [8.11] | Elastic
Don't touch it yet. Unless you really know what you are doing. Defaults are generally good.
Index templates | Elasticsearch Guide [8.11] | Elastic

PUT index

No you can change it dynamically. See Index modules | Elasticsearch Guide [8.11] | Elastic

Beuhlet_Reseau · April 3, 2017, 11:54am

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

Can you explain me with whats is bulk ?? What does it really and how use it ?

About analyzers, it increase performance ? Here is an example of line :

149332,AS1,0-diamproxy;1086841;88958;58da02,REQUEST,367,30987,37887,201,350657887623,LEGANT,,0,,,,,,,,0,POS__,,,,,,,,,,0,4238626,ore,,MAX_TO,858992,0

dadoonet · April 3, 2017, 4:45pm

Bulk is explained here: Cheaper in Bulk | Elasticsearch: The Definitive Guide [2.x] | Elastic

About analyzers, it increase performance ?

If you mean indexing performance, I'd say no. It's the opposite. No analysis will mean less work to do.

149332,AS1,0-diamproxy;1086841;88958;58da02,REQUEST,367,30987,37887,201,350657887623,LEGANT,,0,,,,,,,,0,POS__,,,,,,,,,,0,4238626,ore,,MAX_TO,858992,0

Well. It looks like a CSV file. You should do a structured document out of it IMO if you want to be able to query for anything.

I wrote a CSV import with LS tutorial a while ago: https://david.pilato.fr/blog/2015-04-28-exploring-capitaine-train-dataset/

Might help.

Beuhlet_Reseau · April 4, 2017, 8:32am

Yes I use already a csv filter. IT's perfect for this line type.

Just you talking about the definition of mapping ? How to create one for my data (example of line Above) ?

I must go in kibana, devtools ? or in command line on my linux server ? Can you tell me an example of mapping for my line ?

Thank you a lots

PS : To remove empty fields in filter, i must use remove_field ? or in the "template", or with expression as [if field = ""] drop {} ?

dadoonet · April 4, 2017, 9:33am

What does a document look like after it has been parsed by logstash?

Beuhlet_Reseau · April 4, 2017, 9:42am

{
"ca-d-2017.04": {
"aliases": {},
"mappings": {
"cra": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"beat": {
"properties": {
"hostname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"ca_PN": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"cra_AVol": {
"type": "long"
},
"cra_II": {
"type": "long"
},
"cra_ISI": {
"type": "long"
},
"cra_MDN": {
"type": "long"
},
"cra_PType": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},

[...]

Beuhlet_Reseau · April 4, 2017, 9:44am

I use dev tools of kibana. It's simply of use.

Can ia create template from this tools ?

Beuhlet_Reseau · April 4, 2017, 1:39pm

Here how i see things to create template :

What do you think about that ?

dadoonet · April 4, 2017, 1:42pm

Looks correct.

Beuhlet_Reseau · April 5, 2017, 8:16am

Hello,

I am questio about template i have created. I don't want my text fields are cut (analyzed grrr)

Instead of specifying it on each field, I would make a global rule for all my template, this possible? If yes, how?

For now, here my template. What do you think about that ? I create it on kibana devtools :

PUT _template/templatecra
{
"template" : "cra*",
"order" : 0,
"settings" : {
"number_of_shards" : 1 , "refresh_interval": "-1" , "number_of_replicas": 0
},

"mappings" : {
    "cra" : {
      "properties": {
        "source" : { "enabled" : false },
        "beat": { "enabled" : false },
        "@version": { "enabled" : false },
        "name": { "enabled" : false },
        "version": { "enabled" : false },
        "host": { "enabled" : false },
        "input_type": { "enabled" : false },
        "tags": { "enabled" : false },
        "type": { "enabled" : false },
        
        "cra_AN": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
          }
         }
        },
        "cra_ountVol": {
          "type": "long"
        },
        "cra_EI": {
          "type": "long"
        },
        "cra_SI": {
          "type": "long"
        },
        "cra_ISDN": {
          "type": "long"
        },
        "cra_PType": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_Nid": {
          "type": "integer"
        },
        "cra_Sere2": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_Tade": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_biPlan": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_dataVol": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_dataVolO": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_leglDist": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_mobrage": {
          "type": "text"
        },
        "cra_ratType": {
          "type": "short"
        },
        "cra_rType": {
          "type": "byte"
        },
        "cra_servID": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_startTime": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_terv": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_zoD": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
      
        }
      }
  }

}

Beuhlet_Reseau · April 5, 2017, 8:40am

oh i have error ... invisible

"type": "illegal_argument_exception",
"reason": "failed to parse template source",
"caused_by": {
"type": "json_parse_exception",
"reason": "Unexpected character ('}' (code 125)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@3bb8d48c; line: 155, column: 14]"
}
},
"status": 400
}

dadoonet · April 5, 2017, 9:39am

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

If the JSON is incorrect I believe that Kibana console is telling you that with a red cross.

Beuhlet_Reseau · April 5, 2017, 1:08pm

Ok here with code </>.

Where i can put not analyze string type ? (text & keyword from ES 5.x).

So, maybe I don't need metafield as source, score, beats info ... where i can delete few of them ?

    {
    "order": 0,
    "version": 50001,
    "template": "ta-test-edr",
    "settings": {
      "index": {
        "number_of_replicas": 0,
        "number_of_shards" : 1,
        "refresh_interval": "-1"
      }
    },
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "string_fields": {
              "mapping": {
                "norms": false,
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                }
              },
              "match_mapping_type": "string",
              "match": "*"
            }
          }
        ],
        "_all": {
          "norms": false,
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "include_in_all": false,
            "type": "date"
          },
          "geoip": {
            "dynamic": true,
            "properties": {
              "ip": {
                "type": "ip"
              },
              "latitude": {
                "type": "half_float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          },
          "@version": {
            "include_in_all": false,
            "type": "keyword"
          }
        }
      }
    },
    "aliases": {}
  }

system · May 3, 2017, 1:08pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to change the default amount of shards and replicas of indices? Elasticsearch	6	37646	September 4, 2017
Best way to bulk insert? Elasticsearch	13	6396	July 6, 2017
Dec 21, 2017: [EN][Elasticsearch] Knobs to turn for better indexing performance Advent Calendar	1	2147	August 23, 2018
How to change number_of_shards and number_of_replica in elasticseatch template Elasticsearch	6	16464	March 28, 2019
Configuring indices for monitoring using new template API Elasticsearch elastic-stack-monitoring	7	336	September 14, 2021

Use API of elasticsearch ? Change settings

Related Topics