How to import template for ES 7.12.0?

wajika · August 20, 2021, 7:14am

I am using ES 7.12.0 and APM 7.8.1. Today, I want to upgrade APM to 7.14.0, but APM cannot automatically create a template, so I want to import it manually and encounter some problems.

./apm-server setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["192.168.10.140:9200"]' -E 'output.elasticsearch.username: elastic' -E  'output.elasticsearch.password: elastic'
Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at http://192.168.10.140:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}]

First, I execute the following command, but return to http401. I need to enter the account and password. There is no mark on how to write the account and password in the document.
docker run docker.elastic.co/apm/apm-server:7.14.0 setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["192.168.10.140:9200"]'

Then I use curl to import JSON. The ES server never returns results. After a period of time, it returns "curl: (52) empty reply from server"

curl -XPUT -u elastic:elastic -H 'Content-Type: application/json' http://192.168.10.140:9200/_template/apm-server-7.14.0 -d@apm-server.template.json

Finally, I used binary import without success.

./apm-server setup --index-management -E output.logstash.enabled=false -c apm.yaml
Exiting: error loading template apm-7.14.0-error: failed to load template: couldn't load template: Put "http://192.168.10.140:9200/_template/apm-7.14.0-error": EOF. Response body:

wajika · August 21, 2021, 7:39am

I found that the path of the template has changed. I modified the path but it still doesn’t work.

curl -XPUT -u elastic:elastic -H 'Content-Type: application/json' http://192.168.10.140:9200/_index_template/apm-server-7.14.0 -d@apm-server.template.json
curl: (52) Empty reply from server

wajika · August 23, 2021, 2:32am

I modified the template format, but there are still errors.
https://paste.ubuntu.com/p/Zv4w5dkw23/

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "composable template [simulate_template_uoxlrnqxr_c8eelk9nsykq] template after composition is invalid"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "composable template [simulate_template_uoxlrnqxr_c8eelk9nsykq] template after composition is invalid",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "invalid composite mappings for [simulate_template_uoxlrnqxr_c8eelk9nsykq]",
      "caused_by" : {
        "type" : "mapper_parsing_exception",
        "reason" : "Failed to parse mapping [_doc]: template must have match, path_match or match_mapping_type set {mapping={type=histogram}}",
        "caused_by" : {
          "type" : "mapper_parsing_exception",
          "reason" : "template must have match, path_match or match_mapping_type set {mapping={type=histogram}}"
        }
      }
    }
  },
  "status" : 400
}

Martijn_Laarman · August 23, 2021, 9:46am

Hi @wajika,

APM still uses legacy index templates so to update the template it would have to use the following API:

http://192.168.10.140:9200/_template/apm-server-7.14.0 whereas your example points to the new Index Templates v2 API from Elasticsearch. Our documentation around export/import templates also still uses the legacy v1 API's:

When you initially went through the CLI setup the format to specify the Elasticsearch username and password was slightly off:

-E 'output.elasticsearch.username: USERNAME' where it should be: -E 'output.elasticsearch.username=USERNAME'

Causing the unauthorized message.

If you want to bootstrap everything through the CLI again you can use the following:

./apm-server setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["https://ELASTICSEARCH_DEPLOYMENT"]' -E 'output.elasticsearch.username=USERNAME' -E 'output.elasticsearch.password=PASSWORD' -E 'setup.template.overwrite=true' -E 'apm-server.ilm.setup.overwrite=true' -e -d "*"

Be aware that templates only apply to newly created indices and won't apply retroactively so you might have to rollover your indices manually.

wajika · August 24, 2021, 12:58am

Thank you for the method, I succeeded.

> ./apm-server setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["https://ELASTICSEARCH_DEPLOYMENT"]' -E 'output.elasticsearch.username=USERNAME' -E 'output.elasticsearch.password=PASSWORD' -E 'setup.template.overwrite=true' -E 'apm-server.ilm.setup.overwrite=true' -e -d "*"

For the api v1 version you mentioned, I tried it, but it didn't work. It can be reproduced, and finally it prompts "[index_template] unknown field [mappings]".

1、./apm-server export template > apm-server.template.json
2、copy the content to kibana dev tools

Later I changed the format to api v2, and the prompt of kibana changed. Although it is still unsuccessful

wajika · August 24, 2021, 1:35am

{"log.level":"error","@timestamp":"2021-08-24T01:19:29.057Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":154},"message":"Failed to connect to backoff(elasticsearch(http://192.168.10.141:9200)): Connection marked as failed because the onConnect callback failed: error loading Elasticsearch template: failed to load template: couldn't load template: 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"No type specified for field [headers]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: No type specified for field [headers]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"No type specified for field [headers]\"}},\"status\":400}. Response body: {\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"No type specified for field [headers]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: No type specified for field [headers]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"No type specified for field [headers]\"}},\"status\":400}","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}

@Martijn_Laarman I tried to upgrade the apm server to 7.14.0, but it didn’t work

wajika · August 25, 2021, 2:00am

{
	"error": {
		"root_cause": [{
			"type": "mapper_parsing_exception",
			"reason": "No type specified for field [headers]"
		}],
		"type": "mapper_parsing_exception",
		"reason": "Failed to parse mapping [_doc]: No type specified for field [headers]",
		"caused_by": {
			"type": "mapper_parsing_exception",
			"reason": "No type specified for field [headers]"
		}
	},
	"status": 400
}

Whether it is manually executing the dsl statement or importing the template in other ways, the error always prompts "Failed to parse mapping [_doc]: No type specified for field [headers]". I checked that the dsl contains type: object. Why? Still showing error?

Martijn_Laarman · August 25, 2021, 11:44am

In your example in the developer session you still called /_index_template and not /_template

You might have competing templates now and in fact v2 templates under /_index_template will take precedence.

What does GET /_cat/templates/*apm*?v&h=* return?

You might have to remove some/all templates and rely only on the CLI command to recreate all templates.

wajika · August 26, 2021, 12:30am

/_index_template and /_template I have used it, the result of importing the template is true, but after the apm server runs, it will prompt an error.

apm server full log
https://paste.ubuntu.com/p/FcYMVH7ZhJ/

Error message：
Failed to parse mapping [_doc]: No type specified for field [headers]

axw · August 26, 2021, 2:05am

I don't know why you're getting that error.

I see this message in the log:

{"log.level":"info","@timestamp":"2021-08-25T01:48:01.445Z","log.origin":{"file.name":"template/load.go","file.line":229},"message":"Existing template will be overwritten, as overwrite is enabled.","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}

Do you have setup.template.overwrite: true set in your apm-server.yml?

If you have setup the template manually, then you should configure APM Server to not overwrite the template, and you should then no longer see these errors.

wajika · August 27, 2021, 5:56am

The command i used:
./apm-server setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["https://ELASTICSEARCH_DEPLOYMENT"]' -E 'output.elasticsearch.username=USERNAME' -E 'output.elasticsearch.password=PASSWORD' -E 'setup.template.overwrite=true' -E 'apm-server.ilm.setup.overwrite=true' -e -d "*"

I modified setup.template.overwrite=false and it worked.
I think it should be the wrong order of my operations that caused this problem.

Thank you for your analysis @axw

wajika · August 27, 2021, 5:57am

In this case, if they are not professional personnel, how do they troubleshoot?

axw · August 29, 2021, 5:48am

In this case, if they are not professional personnel, how do they troubleshoot?

The current approach to managing templates, ingest pipelines, etc. is not ideal, and can be difficult to troubleshoot as you have found. We have been working on something that should help here: Switch to the Elastic APM integration | APM User Guide [8.11] | Elastic

With the new APM integration for Elastic Agent, installation of index templates and pipelines will done by Fleet, managed through Kibana. We expect this to be much more user friendly, and less error prone.

wajika · August 30, 2021, 1:01am

Thank you for your reply.
I will try fleet if I have time.

system · September 27, 2021, 1:01am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.