Unable to see APM agent data on Kibana UI

ES version : 7.16.2
APM Server version: 7.16.2
Java Agent Version : 1.28.4 (elastic-apm-agent-1.28.4.jar)

We are trying to push data from APM Agent,

The APM server shows data is getting published on Elasticsearch :

{"log.level":"debug","@timestamp":"2022-02-15T07:47:33.577Z","log.logger":"Elasticsearch","log.origin":{"file.name":"Elasticsearch/client.go","file.line":232},"message":"PublishEvents: 6 events have been published to Elasticsearch in 9.351447ms.","service.name":"apm-server","ecs.version":"1.6.0"}

But on Kibana UI we do not see data.
We do not see any error on Kibana logs at this specific moment when app server says "events published ".

apm-server.yml:


################################ APM Server ################################

apm-server:
  # Defines the host and port the server is listening on. Use "unix:/path/to.sock" to listen on a unix domain socket.
  host: "0.0.0.0:8200"

  #---------------------------- APM Server - Secure Communication with Agents ----------------------------
  # Enable secure communication between APM agents and the server. By default ssl is disabled.
  ssl:
    enabled: true
    certificate: '<path to cert********com.cert.pem>'
    key: '<path ********com.key>'
    #certificate_authorities: ['<path ********com.root.pem']
    supported_protocols: [TLSv1.2, TLSv1.3]
    #cipher_suites: []
    #curve_types: []

  #---------------------------- APM Server - Agent Configuration ----------------------------

  # When using APM agent configuration, information fetched from Kibana will be cached in memory for some time.
  # Specify cache key expiration via this setting. Default is 30 seconds.
  #agent.config.cache.expiration: 30s

  kibana:
    # For APM Agent configuration in Kibana, enabled must be true.
    #enabled: true

    # Scheme and port can be left out and will be set to the default (`http` and `5601`).
    # In case you specify an additional path, the scheme is required: `http://localhost:5601/path`.
    # IPv6 addresses should always be defined as: `https://[2001:db8::1]:5601`.
    host: "https://r********com:5601"

    # Optional protocol and basic auth credentials.
    protocol: "https"
    username: "elastic"
    password: "********************"

    # Optional HTTP path.
    #path: ""

    # Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.
    ssl.enabled: true
    ssl.verification_mode: full
    ssl.supported_protocols: [TLSv1.2]
    ssl.certificate_authorities: ["/home/couchbase/elastic/apm_on_8200/certs/r***************root.pem"]
    ssl.certificate: "/home/couchbase/elastic/apm_on_8200/certs/rn***********e.com.cert.pem"
    ssl.key: "/home/couchbase/elastic/apm_on_8200/certs/rn*************.com.key"
    #ssl.cipher_suites: []
    #ssl.curve_types: []


#================================ Outputs =================================

# Configure the output to use when sending the data collected by apm-server.

#-------------------------- Elasticsearch output --------------------------
output.elasticsearch:
  hosts: ["https://r********com:9200","https://r********com:9200","https://r********com:9200"]

  # Boolean flag to enable or disable the output module.
 # enabled: true

  # Set gzip compression level.
  #compression_level: 0

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  username: "elastic"
  password: "********"


  # Number of workers per Elasticsearch host.
  #worker: 1

  # Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.
  ssl.enabled: true
  ssl.verification_mode: full
  ssl.supported_protocols: [TLSv1.2]
  ssl.certificate_authorities: ["<<path>>/r********com.root.pem"]
  ssl.certificate: "<<path>>r********com.cert.pem"
  # Client Certificate Key
  ssl.key: "/<<path>>/r********com.key"


#================================= Paths ==================================

# The home path for the apm-server installation. This is the default base path
# for all other path settings and for miscellaneous files that come with the
# distribution.
# If not set by a CLI flag or in the configuration file, the default for the
# home path is the location of the binary.
#path.home:

path.config: <<path>>

path.data: <<path>>

path.logs: <<path>>


#================================= Logging =================================

# Available log levels are: error, warning, info, or debug.
logging.level: debug

# The log files are automatically rotated when the log file size limit is reached.
#logging.to_files: true
logging.files:
  # Configure the path where the logs are written. The default is the logs directory
  # under the home path (the binary location).
  path: <<path>>logs

  # The name of the files where the logs are written to.
  name: apm-server

  # Configure log file size limit. If limit is reached, log file will be
  # automatically rotated.
  rotateeverybytes: 10485760 # = 10MB

  # Number of rotated log files to keep. Oldest files will be deleted first.
  keepfiles: 7

  # The permissions mask to apply when rotating log files. The default value is 0600.
  # Must be a valid Unix-style file permissions mask expressed in octal notation.
  permissions: 0650

  # Enable log file rotation on time intervals in addition to size-based rotation.
  # Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h
  # are boundary-aligned with minutes, hours, days, weeks, months, and years as
  # reported by the local system clock. All other intervals are calculated from the
  # Unix epoch. Defaults to disabled.
  interval: 24h

# Set to true to log messages in json format.
logging.json: true

# Set to true, to log messages with minimal required Elastic Common Schema (ECS)
# information. Recommended to use in combination with `logging.json=true`.
#logging.ecs: true

type or paste code here

$ curl --cacert certs/rn*com.root.pem https://rn***********.com:8200
{
"build_date": "2021-12-18T19:59:06Z",
"build_sha": "24fe620eeff5a19e2133c940c7e5ce1ceddb1445",
"publish_ready": true,
"version": "7.16.2"
}

I see some conflicts on the APM indices

This field has a type conflict
The type of the @timestamp field changes across indices and might not be available for search, visualizations, and other analysis.

apm-7.14.1-error-000001, apm-7.14.1-metric-000001, apm-7.14.1-onboarding-2022.02.11, apm-7.14.1-profile-000001, apm-7.14.1-span-000001, apm-7.14.1-transaction-000001, apm-7.16.2-onboarding-2022.01.18, apm-7.16.2-onboarding-2022.02.10, apm-7.16.2-onboarding-2022.02.11, apm-7.16.2-onboarding-2022.02.12
text
apm-7.16.2-error-000001, apm-7.16.2-metric-000001, apm-7.16.2-transaction-000001

I checked the timestamp fields in the conflicting indices but do not see any issues

#"@timestamp" : "2022-02-15T07:17:42.372Z
GET apm-7.16.2-error-000001/_search

#"@timestamp" : "2022-02-12T00:55:27.807Z",
GET apm-7.16.2-onboarding-2022.02.12/_search

Did you see anything in the APM Server logs that indicate it gets data from the agent?
If not, first thing to do is check the agent logs and go through the agent's troubleshooting page.

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.