Failed to install template {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://x.x.x.x:9200/_template/ecs-logstash'",

Elastic Stack Logstash
1

Using a default mapping template {:es_version=>7, :ecs_compatibility=>:v8} gives me this error,

[2024-01-23T02:24:07,381][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.13.3) {:es_version=>7}
[2024-01-23T02:24:07,382][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2024-01-23T02:24:07,401][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"modsecurity"}
[2024-01-23T02:24:07,401][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2024-01-23T02:24:07,408][WARN ][logstash.outputs.elasticsearch][main] `template_api => auto` resolved to `legacy` since we are connected to Elasticsearch 7, but will resolve to `composable` the first time it connects to Elasticsearch 8+. We recommend either setting `template_api => legacy` to continue providing legacy-style templates, or migrating your template to the composable style and setting `template_api => composable`. The legacy template API is slated for removal in Elasticsearch 9.
[2024-01-23T02:24:07,409][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:v8}
[2024-01-23T02:24:07,415][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["/tmp/logstash.conf"], :thread=>"#<Thread:0x2e33326d /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-01-23T02:24:07,433][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"}
[2024-01-23T02:24:07,467][ERROR][logstash.outputs.elasticsearch][main] Failed to install template {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://x.x.x.x:9200/_template/ecs-logstash'",

What is ecs-logstash template ? Do I need to create it in Elasticsearch ?

Stack trace said no handler for type [match_only_text] declared on field [text]

 "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:172:in `block in after_successful_connection'"],

 :body=>"{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"no handler for type [match_only_text] declared on field [text]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: no handler for type [match_only_text] declared on field [text]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"no handler for type [match_only_text] declared on field [text]\"}},\"status\":400}"}

logstash.conf


input{
}

filter {
}
output {
 file {
   codec => json
   path => "c:/temp/logstash_out.log"
 }
  elasticsearch {
    hosts => ["x.x,x.x:9200"]
    index => "modsecurity"
    user => "elastic"
    password => "**********"
  }
}

c:/temp/logstash_out.log works perectly fine.

Note, no such index exists in Elasticsearch. And this is modsecurity json logs which has nested mapping. I don't know how to create a valid index for that. I only know to create a simple mapping from api/devtools

PUT /test_index1?pretty
{
    "settings" : {
        "number_of_shards" : 2,
        "number_of_replicas" : 1
    },
    "mappings" : {
        "properties" : {
            "updated_at" : { "type" : "date" }
        }
    }
}
2

It seems that you are using a Logstash version that is not compatible with your Elasticsearch version.

From the logs you shared it seems that your Elasticsearch is version 7.13.3

[2024-01-23T02:24:07,381][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.13.3) {:es_version=>7}

So your Logstash version should not be higher than 7.17, Logstash 8.X is not compatible with Elasticsearch lower than 7.17.

And from this line it seems that you are using Logstash 8.12 since this is the version of the Elasticsearch output in 8.12

/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.2-java

The main issue is that the default template in Logstash 8.12 uses a data type (match_only_text) that does not exist in Elasticsearch 7.13.

You need to downgrade your Logstash to 7.13.3.

1 Like
3

That may be.

But if I remove "index" from logstash.conf, It sends data to default log index and I can create index pattern on log* and view data.

If logstash 8.x is not compatible, how it manages to send data ?

4

Not sure, you would need to provide what your data looks like before logstash and how it was indexed in Elasticsearch.

But as mentioned, Logstash 8 is not compatible with Elasticsearch 7.13.3.

It may work on some cases? Yes, but you should not expect it to work always.

If you use Elasticsearch 7.13.3 you need to use Logstash version 7.13.

5

Try to add below setting to logstash.yaml and restart logstash ,it work for me .
pipeline.ecs_compatibility: disabled

6

Thanks. That helped.

But now I have multifield keyword mapping attached to every field. If I don't put any index there is no keyword mapping.

image
image879×99 9.89 KB

Without any index in logstash, it is going to data stream log-generic-default with index named .ds-logs-generic-default-2024.01.22-000001

How do I disable these multi fields keyword ?

Do I nee dto copy the mapping from datastream index into new index ?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.