Hi Elastic Folks,
Recently we have a case to send spring boot metrics to ES, our java developers have deployed the spring boot app with micrometer library(https://micrometer.io/). basically, with micrometer library, sprint boot can send its metrics to any kind of monitoring system whatever Elasticsearch, Graphite,influxDB, Prometheus etc.
The issue we had like this:
- spring boot is ok to ship metrics to ES with HTTP post directly
sprint boot(http bulk post) -> ES (good)
Sprint boot-(http bulk post)> (front:http)Haproxy(backend:http)->ES (good)
- sprint boot ship metrics to ES have nothing except for HTTP header info while we add logstash in front of ES.
spring boot(http post) -> (input:http) logstash (output:localfile) (bad)
spring boot(http post) -> (input:http) logstash (output:http) ->ES (bad)
spring boot(http post) -> (front:http) haproxy(backend:http) -(input:http) logstash >ES (bad)
Configuration file:
- spring boot
190108162407 root@test01 etc # cat application.yml
logging:
file: /home/deployer/welcome/logs/welcome.log
level:
com:
tlscontact: debug
management:
server:
port: 11080
endpoints:
web:
exposure:
include: health,info,metrics,logfile
endpoint:
health:
show-details: always
metrics:
export:
elastic:
enabled: true
host: http://logstash-test1.tls.ad:5055
index: metrics
step: 1m
tags:
hostname: test01
client: uk
dc: fr4
env: test
os: Linux
field_tags: weclome-metrics
- logstash
#logstash for pipeline filebeat
input {
http {
port => 5055
codec => es_bulk
response_headers => {
"Access-Control-Allow-Origin" => "*"
"Content-Type" => "text/plain"
"Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type, Accept"
}
}
}
output {
file {
path => "/tmp/httpbeat.log"
}
elasticsearch {
hosts => ["elastic-test4:9200","elastic-test5:9200"]
template_overwrite => false
manage_template => false
sniffing => false
}
}
Test result
- spring boot -> logstash
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19613","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:54:56.932Z","host":"10.0.0.27"}
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19611","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:55:56.947Z","host":"10.0.0.27"}
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19612","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:56:56.933Z","host":"10.0.0.27"}
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19612","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:57:56.934Z","host":"10.0.0.27"}
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19611","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:58:56.961Z","host":"10.0.0.27"}
{"@version":"1","index":{},"headers":{"http_host":"logstash-test1.tls.ad:5055","x_forwarded_for":"10.0.0.203","http_accept":"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2","content_length":"19612","request_method":"POST","content_type":"application/json","http_version":"HTTP/1.1","request_path":"/metrics-2019-01/doc/_bulk","http_user_agent":"Java/1.8.0_181"},"@timestamp":"2019-01-08T03:59:56.935Z","host":"10.0.0.27"}
spring boot ->ES
{
"_index": "metrics-2019-01",
"_type": "doc",
"_id": "j8BjKGgBO8DVodfB-yBo",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-01-07T12:57:47.561Z",
"name": "process_cpu_usage",
"type": "gauge",
"client": "uk",
"dc": "fr4",
"env": "test",
"field_tags": "weclome-metrics",
"hostname": "test01",
"os": "Linux",
"value": 0.0003343852536103158
},
"fields": {
"@timestamp": [
"2019-01-07T12:57:47.561Z"
]
},
"sort": [
1546865867561
]
}
{
"_index": "metrics-2019-01",
"_type": "doc",
"_id": "msBjKGgBO8DVodfB-yBo",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-01-07T12:57:47.561Z",
"name": "jvm_memory_committed",
"type": "gauge",
"area": "nonheap",
"client": "uk",
"dc": "fr4",
"env": "test",
"field_tags": "weclome-metrics",
"hostname": "test01",
"id": "Metaspace",
"os": "Linux",
"value": 59203584
},
"fields": {
"@timestamp": [
"2019-01-07T12:57:47.561Z"
]
},
"sort": [
1546865867561
]
}