APM Server has still not connected to Elasticsearch (2, previous was closed)

Elastic Observability APM
1

APM Server version:

  • APM-Server 7.4.1
  • Elasticsearch: 7.4.0
  • Kibana: 7.4.1

I'm having this error, yet I ran:

  • docker exec -it elastic-apm-server bash -c "./apm-server test config"
    Config OK
  • docker exec -it elastic-apm-server bash -c "./apm-server test output"
    elasticsearch: http://elasticsearch:9200...
    parse url... OK
    connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.x.x.x
    dial up... OK
    TLS... WARN secure connection disabled
    talk to server... OK
    version: 7.4.0

But nothing happens. Even, the kibana container sees the apm-server one:

  • docker exec -it kibana-ui bash -c "curl -i http://elastic-apm-server:8200/"
    HTTP/1.1 200 OK
    Content-Type: application/json
    X-Content-Type-Options: nosniff
    Date: Mon, 04 Nov 2019 22:01:52 GMT
    Content-Length: 124

{
"build_date": "2019-10-22T15:25:16Z",
"build_sha": "26c6b9c55d4753495d8aa571a21b8a97f9693b4b",
"version": "7.4.1"
}

So, what's next?

Thanks.

2

Could you describe the problem you are facing a bit more detailed? It would be interesting to know which APM agent you are using, and if you see anything in the agent or the server logs indicating any issues. Also, do you not see any data ingested to Elasticsearch or are only specific data missing, and did you change any of the APM Server configurations, etc.?

3

Hi, Silvia.
Thanks for answering promptly on this regard.

I'm working on the APM setup page, checking APM server availability.

Elastic, Kibana and APM are in three different Docker containers, I tested accesibility from each other and they can access through network (curl, ping, etc.).

Status:

  • Kibana connects and visualizes Elastic documents (dashboards, visualizations, etc)

  • Kibana and APM see each other from inside the containers (but Kibana's interface doesn't), still tries to reach a 'apm-*' index which I had created.

  • I ran testing commands: [ ./apm-server test config, ./apm-server test output ] and both are good (the check ouput is posted in this conversation thread).

Apparently, this steps only works having locally both (Localhost) and does not fit for distributed environment.

Check output:

  • docker exec -it elastic-apm-server bash -c "./apm-server test config"
    Config OK

  • docker exec -it elastic-apm-server bash -c "./apm-server test output"
    elasticsearch: http://spring-zipkin-storage:9200...
    parse url... OK
    connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.x.x.x
    dial up... OK
    TLS... WARN secure connection disabled
    talk to server... OK
    version: 7.4.0
    But nothing happens. Even, the kibana container sees the apm-server one:

  • docker exec -it kibana-ui bash -c "curl -i http://elastic-apm-server:8200/"
    HTTP/1.1 200 OK
    Content-Type: application/json
    X-Content-Type-Options: nosniff
    Content-Length: 124
    {
    "build_date": "2019-10-22T15:25:16Z",
    "build_sha": "26c6b9c55d4753495d8aa571a21b8a97f9693b4b",
    "version": "7.4.1"
    }

apm-server.yml:

apm-server:

host: "0.0.0.0:8200" # ":8200"

concurrent_requests: 5

rum:

enabled: true

queue:

mem:

events: 4096

max_procs: 4

output:

elasticsearch:

hosts: ["http://spring-zipkin-storage:9200"]

index: "apm-%{+yyyy-MM-dd}"

index: "zipkin-span-%{+yyyy-MM-dd}"

setup:

template:

name: "apm"

pattern: "apm-*"

enabled: true

template:

name: "zipkin-span"

pattern: "zipkin-span-*"

enabled: true

kibana:

For APM Agent configuration in Kibana, enabled must be true.

enabled: true

Scheme and port can be left out and will be set to the default (http and 5601).

In case you specify an additional path, the scheme is required: [http://localhost:5601/path](http://localhost:5601/path`).

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601.

host: "elastic-kibana-ui:5601"

Optional protocol and basic auth credentials.

protocol: http

#username: "elastic"

#password: "changeme"

path: /app/kibana

dashboards:

enabled: true

logging:

level: debug

to_files: false

Any clue?

Thanks in advance.

4

Not sure I fully understand the issue - do you have an APM agent set up that is sending data to the APM Server but they are not successfully indexed into Elasticsearch? If so, could you please check the APM Server logs for any errors?

5

I don't have an agent, but I'm sure that step comes after checking APM availability through Kibana's APM setup page.
Else, how can I test an agent if Kibana does not know if an APM server even exists?

Kindly, could you verify if the settings I posted were correct?

Regards

6

Hi @fraballi, I see you have changed the index names. By default indices start with apm*. If this is changed as in your example, you need to change the default APM index prefixes used in Kibana.

E.g. you need to add something like the following to your config/kibana.yml:

apm_oss.indexPattern: "zipkin*"
apm_oss.errorIndices: "zipkin*"
apm_oss.onboardingIndices: "zipkin*"
apm_oss.spanIndices: "zipkin*"
apm_oss.transactionIndices: "zipkin*"
apm_oss.metricsIndices: "zipkin*"

I saw that you changed setup.template.name: zipkin-span but setup.template.pattern: zipkin-span-*. You need to remove the trailing - from the pattern so it matches the name.

Let me know if this solves your issues.

7

Many thanks, Silvia.
I'll give a try.

When apm runs some init scripts, it creates those indexes too?

Regards

8

Hi, Silvia

I tried what you suggested, but also changed the pattern in: apm-server.yml. Now, I have a link in between, but still requires the APM setup completion and seems like apm-server blanks my original zipkin index, even when zipkin is storing events in continuum.

So, I'd like to use Kibana and APM tracing feature with an existing zipkin storage index.

What does it requires to have it working?

setup:

template:

name: "apm"

pattern: "apm-*"

name: "zipkin"

pattern: "zipkin*"

Best regards

9

Could you please specify what you are generally trying to achieve? So far I understood you only wanted to customize index names. This can be done as discussed above, by changing settings in the APM Server and Kibana configuration.

seems like apm-server blanks my original zipkin index, even when zipkin is storing events in continuum.
So, I'd like to use Kibana and APM tracing feature with an existing zipkin storage index.

Are you trying to have APM Server writing to an existing index? If so, you would need to ensure that the Elasticsearch mapping for the index fits the structure that APM Server expects. By default APM Server creates indices and Elasticsearch templates on startup or when running the dedicated setup command. If the used index templates do not fit APM Server data structures, documents cannot be ingested.

10

In case you are trying to use Zipkin agents in combination with the APM UI, this is not expected to work as the APM UI expects data in a specific format.

11

Many thanks Silvia on your answers.

You kindly answered both questions and here goes my feedback:

  • Need to read a Zipkin's index stored in Elastic, from APM server (my company has stored events in a Zipkin's index, as Elastic says it is compatible with OpenTracing specs as Zipkin is, we were leveraging a change to ELK)

  • APM blanks Zipkin's index (seems like as you mentioned, you ingest specific metadata and format in APM indices, so when APM bootstraps templates and custom env, ovewrites Zipkin's index structure and events can not be stored anymore in previous format. We have Zipkin in production and APM server comes after in deployment)

  • Does Elastic have any tool to migrate indices: move OpenTracing data compatible between sources?

  • Does Elastic APM has any API (REST preferably) useful to talk with APM server and save those old tracing documents into ELK APM comatible format?

Best regards.

12

Thanks for the more detailed explanation. I recommend to read the blog post about Distributed Tracing, OpenTracing and Elastic APM, as it explains in detail how distributed tracing works with Elastic APM and where we are when it comes to compatibility with other tracers. I specifically recommend the section about the Elastic OpenTracing Bridge that describes how you can switch from one Opentracing compliant tracer to another one.

Regarding APM blanking Zipkin's index: As long as setup.template.overwrite: false (default setting) I don't see why APM Server should overwrite the index or index templates. While the server does take care of the setup on the first startup, it shouldn't overwrite anything existing by default. Please check your setting in apm-server.yml again to verify you don't have overwrite enabled.
As mentioned before, the APM Server and UI do have a strict data format, so the server will probably encounter issues in ingesting data and the APM UI in showing data when using an index with different data and index template. I very much recommend to use the default APM indices and ingest the data monitored with Elastic APM agents in its own indices.

You can have a look at the APM Server Intake API to figure out which data are sent by the Elastic APM agents.

Unfortunately there is no tooling available that would allow to migrate the indices. Mostly because Elastic APM enriches data with additional information to be able to provide the very rich UI.

13

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.