Search Error - All Shards failed

After parsing JSON log with the filter I used below

filter {
    if [type] == "vizql_cpp" or [type] == "vizql_tabprotosrv" {
        json {
            source => "message"
        }
        mutate {
            rename => { "v" => "v_%{k}" }
        }
        date {
            match => ["ts", "YYYY-MM-dd|HH:mm:ss", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]
            target => "@timestamp"
        }
    }
}

I had to increase the "index.mapping.total_fields.limit": 100000 as this JSON log file has created more than the default number of 1000 index fields and kept warning Limit of total fields [1000] has been exceeded

There were few warning regarding data type conversions like below.

[2021-02-10T10:48:18,077][WARN ][logstash.outputs.elasticsearch][main][5407fae34cad552531c896a58609701a37967d1fe753424bc18f596bd9c3372e] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"06543072-vizql_cpp2", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x3db19409], :response=>{"index"=>{"_index"=>"06543072-vizql_cpp2", "_type"=>"_doc", "_id"=>"b40yiXcBtCBrnwvsEjrn", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [v_server-startup-options.file.encoding] cannot be changed from type [text] to [ObjectMapper]"}}}}

Ignoring the warning above, the index created and the final result of index pattern in Kibana was 9055 fields in total.

With that index, search keyword 'error' given, throw an search error regardless Lucene or KQL. The error saying 'all shards failed'


I have a single node Elasticsearch.
The health of the index in the summary page is green with the settings as below.


{
"index.blocks.read_only_allow_delete": "false",
"index.priority": "1",
"index.query.default_field": [
"*"
],
"index.refresh_interval": "1s",
"index.write.wait_for_active_shards": "1",
"index.routing.allocation.include._tier_preference": "data_content",
"index.mapping.total_fields.limit": "100000",
"index.mapping.ignore_malformed": "true",
"index.number_of_replicas": "0"
}

What went wrong? Any idea?

The Json log file looks like as below.
{"ts":"2020-12-12T00:00:12.305","pid":6648,"tid":"5578","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"rotate-log","v":{"new-path":"C:\ProgramData\Tableau\Tableau Server\data\tabsvc\logs\vizqlserver\nativeapi_vizqlserver_1-0_2020_12_12_00_00_00.txt","old-path":"C:\ProgramData\Tableau\Tableau Server\data\tabsvc\logs\vizqlserver\nativeapi_vizqlserver_1-0_2020_12_11_00_00_00.txt"}}
{"ts":"2020-12-12T00:00:12.303","pid":6648,"tid":"52b0","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: CPU info: 0%"}
{"ts":"2020-12-12T00:00:12.303","pid":6648,"tid":"52b0","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: Memory info: 1,801,904,128 bytes (current process);14,939,303,936 bytes (Tableau total); 29,661,540,352 bytes (total of all processes); 11 (info count)"}
{"ts":"2020-12-12T00:00:12.304","pid":6648,"tid":"52b0","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"cachingdomparser-getdom-stats","v":{"elements-count":4,"hits":0,"logging-period-in-sec":60,"misses":0}}
{"ts":"2020-12-12T00:01:12.313","pid":6648,"tid":"52b0","sev":"info","req":"-","sess":"-","site":"-","user":"-","k":"msg","v":"Resource Manager: CPU info: 0%"}

Hi and welcome to our community!

I'd need some additional information about this, so what version of the Elastic stack are you using? Could you record and share a HAR file, when you experience the error in Discover? There should be errors in you Elasticsearch log file, could you share them?

Many thx and best,
Matthias