Spring boot can't ship metrics to Logstash with http post


(Robin Guo) #1

Hi Elastic Folks,
Recently we have a case to send spring boot metrics to ES, our java developers have deployed the spring boot app with micrometer library(https://micrometer.io/). basically, with micrometer library, sprint boot can send its metrics to any kind of monitoring system whatever Elasticsearch, Graphite,influxDB, Prometheus etc.

The issue we had like this:

  • spring boot is ok to ship metrics to ES with HTTP post directly
sprint boot(http bulk post) -> ES  (good)
Sprint boot-(http bulk post)> (front:http)Haproxy(backend:http)->ES (good)
  • sprint boot ship metrics to ES have nothing except for HTTP header info while we add logstash in front of ES.
    spring boot(http post) -> (input:http) logstash (output:localfile)  (bad)
    spring boot(http post) -> (input:http) logstash (output:http) ->ES (bad)
    spring boot(http post) -> (front:http) haproxy(backend:http) -(input:http) logstash >ES (bad)

Configuration file:

  • spring boot
190108162407 root@test01 etc # cat application.yml
logging:
  file: /home/deployer/welcome/logs/welcome.log
  level:
    com:
      tlscontact: debug
management:
  server:
    port: 11080
  endpoints:
    web:
      exposure:
        include: health,info,metrics,logfile
  endpoint:
    health:
      show-details: always
  metrics:
    export:
      elastic:
        enabled: true
        host: http://logstash-test1.tls.ad:5055
        index: metrics
        step: 1m

    tags:
      hostname: test01
      client: uk
      dc: fr4
      env: test
      os: Linux
      field_tags: weclome-metrics
  • logstash
#logstash for pipeline filebeat
input {
  http {
    port => 5055
    codec => es_bulk
    response_headers => {
    "Access-Control-Allow-Origin" => "*"
    "Content-Type" => "text/plain"
    "Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type, Accept"
}
  }
}

output {
file {
  path => "/tmp/httpbeat.log"
}

  elasticsearch {
     hosts => ["elastic-test4:9200","elastic-test5:9200"]
    template_overwrite => false
    manage_template => false
    sniffing => false
  }

}

Test result

  • spring boot -> logstash
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19613","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:54:56.932Z","host":"10.0.0.27"}
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19611","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:55:56.947Z","host":"10.0.0.27"}
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19612","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:56:56.933Z","host":"10.0.0.27"}
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19612","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:57:56.934Z","host":"10.0.0.27"}
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19611","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:58:56.961Z","host":"10.0.0.27"}
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19612","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:59:56.935Z","host":"10.0.0.27"}

spring boot ->ES

{
  "_index": "metrics-2019-01",
  "_type": "doc",
  "_id": "j8BjKGgBO8DVodfB-yBo",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-01-07T12:57:47.561Z",
    "name": "process_cpu_usage",
    "type": "gauge",
    "client": "uk",
    "dc": "fr4",
    "env": "test",
    "field_tags": "weclome-metrics",
    "hostname": "test01",
    "os": "Linux",
    "value": 0.0003343852536103158
  },
  "fields": {
    "@timestamp": [
      "2019-01-07T12:57:47.561Z"
    ]
  },
  "sort": [
    1546865867561
  ]
}

{
  "_index": "metrics-2019-01",
  "_type": "doc",
  "_id": "msBjKGgBO8DVodfB-yBo",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-01-07T12:57:47.561Z",
    "name": "jvm_memory_committed",
    "type": "gauge",
    "area": "nonheap",
    "client": "uk",
    "dc": "fr4",
    "env": "test",
    "field_tags": "weclome-metrics",
    "hostname": "test01",
    "id": "Metaspace",
    "os": "Linux",
    "value": 59203584
  },
  "fields": {
    "@timestamp": [
      "2019-01-07T12:57:47.561Z"
    ]
  },
  "sort": [
    1546865867561
  ]
}


(Robin Guo) #2

seems logstash http input plugin is not good to handle http bulk post?
Any good suggestions/comments to fix the issue?

Thanks in advance


(Stephen Goodall) #3

I think the problem might be that it is trying to post to logstash using this URL on the end of the "host" setting:

I'm not sure that logstash would be expecting anything after the port in the host setting.

It might work if you could post the same data to the "http://logstash-test1.tls.ad:5055" URL, without the bit on the end that micrometer is adding, but I don't know that it would be possible out of the box.

Is there a specific reason for not going directly to ES?


(Robin Guo) #4

@StephenGoodall,

Thanks, yeah, you're right.


(system) closed #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.