# Java APM agent version \`unknown\` and Host \`N/A\` due to missing \`host.hostname\`

**URL:** <https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076>\
**Category:** APM\
**Tags:** java\
**Created:** [November 24, 2021, 1:49pm UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076 "2021-11-24T13:49:31Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\
**Post date:** [November 24, 2021, 1:49pm UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076/1 "2021-11-24T13:49:31Z")

</div>

**Kibana version** : 7.15.0

**Elasticsearch version** : 7.15.0

**APM Server version** : 7.15.0

**APM Agent language and version** : Java 1.27.0 (Java 8)

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:  
`agent.version` has `unknown` value after upgrading to Elastic APM Java Agent `1.27.0`.

![image](https://us1.discourse-cdn.com/elastic/original/3X/c/5/c5c83d66f3b99bece3d0eadf341b5e8f2e9bca22.png)

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [November 24, 2021, 2:07pm UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076/2 "2021-11-24T14:07:26Z")

</div>

Hi @hendry.lim ,

- Do you see that from Kibana only ?
- Can you send a few documents that appear to be sent by the recently upgraded agents ?
- Also, Can you share the agent logs please ? See [procedure in documentation](https://www.elastic.co/guide/en/apm/agent/java/current/trouble-shooting.html#trouble-shooting-logging-procedure) for details.

---

<div class="post-metadata">

**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\
**Post date:** [November 24, 2021, 2:17pm UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076/3 "2021-11-24T14:17:31Z")

</div>

I also noticed that `host.hostname` is missing from the document sent by agent `v1.27.0`, while we had that in `1.26.0`. This affects the APM UI which is displaying `Host N/A` on the JVMs tab.

![image](https://us1.discourse-cdn.com/elastic/original/3X/1/1/115474c8d075b3b3b22103e5db38aa62cb76c8b9.png)

![image](https://us1.discourse-cdn.com/elastic/original/3X/e/b/eb5793f754beb524d9559e52c786b0fd28d43137.png)

Sample doc from another app running Java 11 (some fields were removed) - `1.27.0`

```auto
{
  "container": {
    "id": "9328fd947c04ddc0e17437f42408aeaa42bcf45da557d6688c65c60c2fe4986d"
  },
  "agent": {
    "name": "java",
    "ephemeral_id": "e4ec45b1-ed97-4b6e-bc5a-5b51ec989022",
    "version": "unknown"
  },
  "process": {
    "pid": 1,
    "title": "/opt/java/openjdk/bin/java"
  },
  "processor": {
    "name": "transaction",
    "event": "transaction"
  },
  "url": {
    "path": "/index.html",
    "scheme": "https",
    "port": 443,
    "domain": "xxx",
    "full": "xxx"
  },
  "observer": {
    "hostname": "xxx",
    "id": "c4e5c03a-3714-4b33-bf8e-e7dd958d7bbe",
    "ephemeral_id": "17c9f1f3-5e4b-4898-8f14-d1bc715bc0de",
    "type": "apm-server",
    "version": "7.15.0",
    "version_major": 7
  },
  "trace": {
    "id": "b52dba2843c6af3abf08e9a3a60ffaa1"
  },
  "@timestamp": "2021-11-24T14:13:22.032Z",
  "ecs": {
    "version": "1.11.0"
  },
  "service": {
    "node": {
      "name": "xxx"
    },
    "runtime": {
      "name": "Java",
      "version": "11.0.11"
    },
    "language": {
      "name": "Java",
      "version": "11.0.11"
    }
  },
  "host": {
    "os": {
      "platform": "Linux"
    },
    "ip": "127.0.0.1",
    "name": "xxx",
    "architecture": "amd64"
  },
  "event": {
    "ingested": "2021-11-24T14:13:23.433948504Z",
    "outcome": "success"
  },
  "timestamp": {
    "us": 1637763202032015
  }
}

```

Comparison with `1.26.0`

```auto
{
  "container": {
    "id": "9a1f43470e789214ffae06ba0a29fe3193d6fe80f5e6079e201f8882ed3a18f2"
  },
  "agent": {
    "name": "java",
    "ephemeral_id": "46e72620-6668-486f-af71-54c303ce3f12",
    "version": "1.26.0"
  },
  "observer": {
    "hostname": "xxx",
    "id": "c4e5c03a-3714-4b33-bf8e-e7dd958d7bbe",
    "ephemeral_id": "17c9f1f3-5e4b-4898-8f14-d1bc715bc0de",
    "type": "apm-server",
    "version": "7.15.0",
    "version_major": 7
  },
  "trace": {
    "id": "77d3c6093d678e43ad14bca138a9cffb"
  },
  "ecs": {
    "version": "1.11.0"
  },
  "host": {
    "hostname": "xxx",
    "os": {
      "platform": "Linux"
    },
    "ip": "127.0.0.1",
    "name": "xxx",
    "architecture": "amd64"
  },
  "event": {
    "ingested": "2021-11-24T12:35:13.845309666Z",
    "outcome": "success"
  },
  "process": {
    "pid": 1,
    "title": "/opt/java/openjdk/bin/java"
  },
  "processor": {
    "name": "transaction",
    "event": "transaction"
  },
  "url": {
    "path": "xxx",
    "scheme": "https",
    "port": 443,
    "domain": "xxx",
    "full": "xxx"
  },
  "@timestamp": "2021-11-24T12:35:09.445Z",
  "service": {
    "node": {
      "name": "xxx"
    },
    "name": "xxx",
    "runtime": {
      "name": "Java",
      "version": "11.0.11"
    },
    "language": {
      "name": "Java",
      "version": "11.0.11"
    }
  }
}

```

Logs

```auto
2021-11-24 14:09:40,575 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2021-11-24 14:09:40,651 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM unknown as xxx (null) on Java 11.0.11 Runtime version: 11.0.11+9 VM version: 11.0.11+9 (AdoptOpenJDK) Linux 3.10.0-1160.42.2.el7.x86_64
2021-11-24 14:09:40,652 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_name: 'xxx' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:40,652 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_node_name: 'xxx' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:40,652 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - hostname: 'xxx' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:40,652 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - environment: 'production' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:40,652 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - transaction_max_spans: '2000' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:40,653 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - capture_body: 'ALL' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:40,654 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - trace_methods: '*ServiceImpl#*' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:40,654 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - config_file: '/usr/local/tomcat/conf/elastic-apm-agent.properties' (source: Java System Properties)
2021-11-24 14:09:40,654 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - server_url: 'http://127.0.0.1:8200' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:40,654 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - verify_server_cert: 'false' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:40,654 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'xxx' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:40,654 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - enable_log_correlation: 'true' (source: /usr/local/tomcat/conf/elastic-apm-agent.properties)
2021-11-24 14:09:43,185 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2021-11-24 14:09:43,258 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: { "build_date": "2021-09-16T02:05:39Z", "build_sha": "a183f675ecd03fca4a897cbe85fda3511bc3ca43", "version": "7.15.0"}

```

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [November 25, 2021, 8:20am UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076/4 "2021-11-25T08:20:27Z")

</div>

Thanks !

I'm investigating this today, I'll keep you posted when we have a PR to fix that.

---

<div class="post-metadata">

**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\
**Post date:** [November 25, 2021, 8:37am UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076/5 "2021-11-25T08:37:18Z")

</div>

Thanks for the update.

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [November 25, 2021, 10:41am UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076/6 "2021-11-25T10:41:24Z")

</div>

There are two issues:

- I've just opened a [PR for the agent version](https://github.com/elastic/apm-agent-java/pull/2289)
- some fields are missing in the provided documents by doing a diff (details below)

`host.hostname` appears to be missing in `1.27.0`, which might be related to [this issue](https://github.com/elastic/apm-agent-java/issues/2284).  
Can you provide an obfuscated version of the value you had with `1.26.0` and the provided value of `hostname` agent configuration ? Replacing all alphabetic characters with `x` should be fine.

Also, `service.name` appears to is missing in `1.27.0`, thus I'll have to investigate this too.

---

<div class="post-metadata">

**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\
**Post date:** [November 25, 2021, 10:58am UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076/7 "2021-11-25T10:58:09Z")

</div>

The `service.name` is available in `1.27.0`. No issue there, but yes the `host.hostname` is missing. The `service.name` was missing because I removed it from the sample doc.

You referred to `hostname` agent configuration, were you referring to `service_node_name` agent configuration?

Here is another sample doc for `1.27.0`. I have kept most of the fields intact.

```auto
{
  "container": {
    "id": "9328fd947c04ddc0e17437f42408aeaa42bcf45da557d6688c65c60c2fe4986d"
  },
  "parent": {
    "id": "676d853ebfccf1e1"
  },
  "agent": {
    "name": "java",
    "ephemeral_id": "e4ec45b1-ed97-4b6e-bc5a-5b51ec989022",
    "version": "unknown"
  },
  "source": {
    "ip": "1.1.1.1"
  },
  "observer": {
    "hostname": "xxx",
    "id": "c4e5c03a-3714-4b33-bf8e-e7dd958d7bbe",
    "ephemeral_id": "17c9f1f3-5e4b-4898-8f14-d1bc715bc0de",
    "type": "apm-server",
    "version": "7.15.0",
    "version_major": 7
  },
  "trace": {
    "id": "e516efe8b31e9395f839f152a56e9fe0"
  },
  "ecs": {
    "version": "1.11.0"
  },
  "host": {
    "os": {
      "platform": "Linux"
    },
    "ip": "127.0.0.1",
    "name": "offsite.domain.com",
    "architecture": "amd64"
  },
  "@version": "1",
  "client": {
    "geo": {
      ...
    },
    "ip": "1.1.1.1"
  },
  "event": {
    "ingested": "2021-11-25T10:59:35.860366371Z",
    "outcome": "success"
  },
  "user_agent": {
    "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36",
    "os": {
      "name": "Windows",
      "version": "10",
      "full": "Windows 10"
    },
    "name": "Chrome",
    "device": {
      "name": "Other"
    },
    "version": "96.0.4664.45"
  },
  "timestamp": {
    "us": 1637837971346000
  },
  "process": {
    "pid": 1,
    "title": "/opt/java/openjdk/bin/java"
  },
  "processor": {
    "name": "transaction",
    "event": "transaction"
  },
  "url": {
    "path": "/api/xxx",
    "scheme": "https",
    "port": 443,
    "domain": "crm.domain.com",
    "full": "https://crm.domain.com/api/xxx"
  },
  "@timestamp": "2021-11-25T10:59:31.346Z",
  "service": {
    "node": {
      "name": "offsite.domain.com"
    },
    "environment": "production",
    "framework": {
      "name": "Spring Web MVC",
      "version": "xxx"
    },
    "name": "somename",
    "runtime": {
      "name": "Java",
      "version": "11.0.11"
    },
    "language": {
      "name": "Java",
      "version": "11.0.11"
    }
  },
  "http": {
    "request": {
      "headers": {
        ...
      },
      "method": "GET",
      "socket": {
        "remote_address": "1.1.1.1"
      }
    },
    "response": {
      "headers": {
        ...
      },
      "status_code": 200,
      "finished": true,
      "headers_sent": true
    },
    "version": "1.0"
  },
  "user": {
    "name": "xxx"
  },
  "transaction": {
    "duration": {
      "us": 71422
    },
    "result": "HTTP 2xx",
    "name": "controller#getList",
    "span_count": {
      "dropped": 0,
      "started": 1
    },
    "id": "3c3eaf0ea6c3cd5c",
    "type": "request",
    "sampled": true
  }
}

```

`1.26.0`

```auto
{
  "container": {
    "id": "9a1f43470e789214ffae06ba0a29fe3193d6fe80f5e6079e201f8882ed3a18f2"
  },
  "parent": {
    "id": "5a6a1b27673b681e"
  },
  "agent": {
    "name": "java",
    "ephemeral_id": "46e72620-6668-486f-af71-54c303ce3f12",
    "version": "1.26.0"
  },
  "source": {
    "ip": "1.1.1.1"
  },
  "observer": {
    "hostname": "xxx",
    "id": "c4e5c03a-3714-4b33-bf8e-e7dd958d7bbe",
    "ephemeral_id": "17c9f1f3-5e4b-4898-8f14-d1bc715bc0de",
    "type": "apm-server",
    "version": "7.15.0",
    "version_major": 7
  },
  "trace": {
    "id": "72e2c582d5f4a7a29951d8631b618065"
  },
  "ecs": {
    "version": "1.11.0"
  },
  "host": {
    "hostname": "offsite.domain.com",
    "os": {
      "platform": "Linux"
    },
    "ip": "127.0.0.1",
    "name": "offsite.domain.com",
    "architecture": "amd64"
  },
  "@version": "1",
  "client": {
    "geo": {
      ...
    },
    "ip": "1.1.1.1"
  },
  "event": {
    "ingested": "2021-11-24T11:04:50.303760566Z",
    "outcome": "success"
  },
  "user_agent": {
    "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36",
    "os": {
      "name": "Windows",
      "version": "10",
      "full": "Windows 10"
    },
    "name": "Chrome",
    "device": {
      "name": "Other"
    },
    "version": "96.0.4664.45"
  },
  "timestamp": {
    "us": 1637751879654000
  },
  "process": {
    "pid": 1,
    "title": "/opt/java/openjdk/bin/java"
  },
  "processor": {
    "name": "transaction",
    "event": "transaction"
  },
  "url": {
    "path": "/api/xxx",
    "scheme": "https",
    "port": 443,
    "domain": "crm.domain.com",
    "full": "https://crm.domain.com/api/xxx"
  },
  "@timestamp": "2021-11-24T11:04:39.654Z",
  "service": {
    "node": {
      "name": "offsite.domain.com"
    },
    "environment": "production",
    "framework": {
      "name": "Spring Web MVC",
      "version": "xxx"
    },
    "name": "somename",
    "runtime": {
      "name": "Java",
      "version": "11.0.11"
    },
    "language": {
      "name": "Java",
      "version": "11.0.11"
    }
  },
  "http": {
    "request": {
      "headers": {
        ...
      },
      "method": "GET",
      "socket": {
        "encrypted": true,
        "remote_address": "1.1.1.1"
      }
    },
    "response": {
      "headers": {
        ...
      },
      "status_code": 200,
      "finished": true,
      "headers_sent": true
    },
    "version": "1.0"
  },
  "user": {
    "name": "xxx"
  },
  "transaction": {
    "duration": {
      "us": 43103
    },
    "result": "HTTP 2xx",
    "name": "controller#getList",
    "span_count": {
      "dropped": 0,
      "started": 11
    },
    "id": "d81af3d8a4aff5f5",
    "type": "request",
    "sampled": true
  }
}

```

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [November 25, 2021, 2:58pm UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076/8 "2021-11-25T14:58:02Z")

</div>

After further investigation it seems the issue might be in Kibana that might rely on an obsolete attribute that is no longer populated with Java agent 1.27.0 and later.

I have opened [[APM] Hostname broken in APM / JVM Tab · Issue #119737 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/119737).

---

<div class="post-metadata">

**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\
**Post date:** [November 26, 2021, 2:28am UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076/9 "2021-11-26T02:28:23Z")

</div>

Thanks for the quick response/fix!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 16, 2021, 10:29pm UTC](https://discuss.elastic.co/t/java-apm-agent-version-unknown-and-host-n-a-due-to-missing-host-hostname/290076/10 "2021-12-16T22:29:10Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.