Use API of elasticsearch ? Change settings

Beuhlet_Reseau · April 3, 2017, 9:18am

Hello,

I want increase performance of indexing, usage disk of elasticsearch. I don't know elk API

Some questions stay without response (after reading the official doc)

How Use bulk requests (I have for the moment, 2 index per days)
Disable refresh and replicas.
/!\ : I am on test step, so i delete, create index to find the better configuration.

So, i want know how configure this options ? (It's impossible in elasticsearch.yml since version 5+ of elk). I believe we must use api ? But how.

3.Indexing buffer size : By default is take 10% of jvm (so it's not necessary to modify it)

4.Use index template

I don't know how to create index, where create index... I believe we can change refresh interval and replicas by default before creation of any index.

dadoonet · April 3, 2017, 9:39am

Bulk API | Elasticsearch Guide [8.11] | Elastic
Update index settings API | Elasticsearch Guide [8.11] | Elastic
Don't touch it yet. Unless you really know what you are doing. Defaults are generally good.
Index templates | Elasticsearch Guide [8.11] | Elastic

PUT index

No you can change it dynamically. See Index modules | Elasticsearch Guide [8.11] | Elastic

Beuhlet_Reseau · April 3, 2017, 11:54am

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

Can you explain me with whats is bulk ?? What does it really and how use it ?

About analyzers, it increase performance ? Here is an example of line :

149332,AS1,0-diamproxy;1086841;88958;58da02,REQUEST,367,30987,37887,201,350657887623,LEGANT,,0,,,,,,,,0,POS__,,,,,,,,,,0,4238626,ore,,MAX_TO,858992,0

dadoonet · April 3, 2017, 4:45pm

Bulk is explained here: Cheaper in Bulk | Elasticsearch: The Definitive Guide [2.x] | Elastic

About analyzers, it increase performance ?

If you mean indexing performance, I'd say no. It's the opposite. No analysis will mean less work to do.

149332,AS1,0-diamproxy;1086841;88958;58da02,REQUEST,367,30987,37887,201,350657887623,LEGANT,,0,,,,,,,,0,POS__,,,,,,,,,,0,4238626,ore,,MAX_TO,858992,0

Well. It looks like a CSV file. You should do a structured document out of it IMO if you want to be able to query for anything.

I wrote a CSV import with LS tutorial a while ago: https://david.pilato.fr/blog/2015-04-28-exploring-capitaine-train-dataset/

Might help.

Beuhlet_Reseau · April 4, 2017, 8:32am

Yes I use already a csv filter. IT's perfect for this line type.

Just you talking about the definition of mapping ? How to create one for my data (example of line Above) ?

I must go in kibana, devtools ? or in command line on my linux server ? Can you tell me an example of mapping for my line ?

Thank you a lots

PS : To remove empty fields in filter, i must use remove_field ? or in the "template", or with expression as [if field = ""] drop {} ?

dadoonet · April 4, 2017, 9:33am

What does a document look like after it has been parsed by logstash?

Beuhlet_Reseau · April 4, 2017, 9:42am

{
"ca-d-2017.04": {
"aliases": {},
"mappings": {
"cra": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"beat": {
"properties": {
"hostname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"ca_PN": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"cra_AVol": {
"type": "long"
},
"cra_II": {
"type": "long"
},
"cra_ISI": {
"type": "long"
},
"cra_MDN": {
"type": "long"
},
"cra_PType": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},

[...]

Beuhlet_Reseau · April 4, 2017, 9:44am

I use dev tools of kibana. It's simply of use.

Can ia create template from this tools ?

Beuhlet_Reseau · April 4, 2017, 1:39pm

Here how i see things to create template :

What do you think about that ?

dadoonet · April 4, 2017, 1:42pm

Looks correct.

Beuhlet_Reseau · April 5, 2017, 8:16am

Hello,

I am questio about template i have created. I don't want my text fields are cut (analyzed grrr)

Instead of specifying it on each field, I would make a global rule for all my template, this possible? If yes, how?

For now, here my template. What do you think about that ? I create it on kibana devtools :

PUT _template/templatecra
{
"template" : "cra*",
"order" : 0,
"settings" : {
"number_of_shards" : 1 , "refresh_interval": "-1" , "number_of_replicas": 0
},

"mappings" : {
    "cra" : {
      "properties": {
        "source" : { "enabled" : false },
        "beat": { "enabled" : false },
        "@version": { "enabled" : false },
        "name": { "enabled" : false },
        "version": { "enabled" : false },
        "host": { "enabled" : false },
        "input_type": { "enabled" : false },
        "tags": { "enabled" : false },
        "type": { "enabled" : false },
        
        "cra_AN": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
          }
         }
        },
        "cra_ountVol": {
          "type": "long"
        },
        "cra_EI": {
          "type": "long"
        },
        "cra_SI": {
          "type": "long"
        },
        "cra_ISDN": {
          "type": "long"
        },
        "cra_PType": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_Nid": {
          "type": "integer"
        },
        "cra_Sere2": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_Tade": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_biPlan": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_dataVol": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_dataVolO": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_leglDist": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_mobrage": {
          "type": "text"
        },
        "cra_ratType": {
          "type": "short"
        },
        "cra_rType": {
          "type": "byte"
        },
        "cra_servID": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_startTime": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_terv": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cra_zoD": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
      
        }
      }
  }

}

Beuhlet_Reseau · April 5, 2017, 8:40am

oh i have error ... invisible

"type": "illegal_argument_exception",
"reason": "failed to parse template source",
"caused_by": {
"type": "json_parse_exception",
"reason": "Unexpected character ('}' (code 125)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@3bb8d48c; line: 155, column: 14]"
}
},
"status": 400
}

dadoonet · April 5, 2017, 9:39am

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

If the JSON is incorrect I believe that Kibana console is telling you that with a red cross.

Beuhlet_Reseau · April 5, 2017, 1:08pm

Ok here with code </>.

Where i can put not analyze string type ? (text & keyword from ES 5.x).

So, maybe I don't need metafield as source, score, beats info ... where i can delete few of them ?

    {
    "order": 0,
    "version": 50001,
    "template": "ta-test-edr",
    "settings": {
      "index": {
        "number_of_replicas": 0,
        "number_of_shards" : 1,
        "refresh_interval": "-1"
      }
    },
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "string_fields": {
              "mapping": {
                "norms": false,
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                }
              },
              "match_mapping_type": "string",
              "match": "*"
            }
          }
        ],
        "_all": {
          "norms": false,
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "include_in_all": false,
            "type": "date"
          },
          "geoip": {
            "dynamic": true,
            "properties": {
              "ip": {
                "type": "ip"
              },
              "latitude": {
                "type": "half_float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          },
          "@version": {
            "include_in_all": false,
            "type": "keyword"
          }
        }
      }
    },
    "aliases": {}
  }

system · May 3, 2017, 1:08pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.