I had Elasticsearch, Logstash and Kibana v7.16.2 with xpack security based login enabled, Yesterday i had upgraded my ELK versions to 8.6.1 using my docker-compose file.
Current problem: My logstash fetched the data but not inserted in to elasticsearch. But no error shown in my ELK containers.
@Rios - I have added stdout { codec => rubydebug{} } then removed all the entries from sincedb_path => "/usr/share/logstash/pipeline/sincedb.txt" and tried, but still i can't see the data in elasticsearch that i have loaded into my logstash.
I can see the below warnings and errors in my logstash container now.
[WARN ][logstash.outputs.elasticsearch][main][13ed313a5675abd23c078edd33a0a3c4be86d627339fd3bf53181d4183411c94] Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>"61975163", :_index=>"employee-data",
"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id '61975163'. Preview of field's value: '{name=8cb532a93e4e}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:416"}}}}
"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id '61959101'. Preview of field's value: '{name=8cb532a93e4e}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:419"}}}}
Are you using somewhere the filed: host?
Can you try with next setting in logstash.yml or if you are using pipelines in pipelines.yml or directly in plugin setting. pipeline.ecs_compatibility: disabled
This is probably the issue, the file input adds the host field.
On version 7 it was added into the field host, on version 8 ecs compatibility is enabled by default and the value of the hostname is added into the host.name field, which will lead to mapping issues.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.