Logstash - Datadog metrics - Unable to send them to Datadog


(David Niceact) #1

Hello everyone,

I am trying to use the Datadog metric plugin for logstash. I have configured it to use a CSV file as an input and then perform the parse with grok. Unfortately I am not able to take these metrics and push them to Datadog using the community supported Datadog metrics plugin.

I would apprecaite any help to get this off the ground.

Thanks!

My logstash file
input {
file {
path => "/opt/server_counters.log.csv"
type => "csv"
}
}

filter {
if [type] == "csv" {
csv {
remove_field => ["message"]
columns => [
"log_timestamp",
"CPU",
"Usage",
"Threads",
"File Descriptors",
"RSS Virtual Bytes",
"Available Bytes",
"Broker Memory Usage",
"Broker Store Usage",
"Total Messages Enqueued",
"Total Messages Dequeued",
"Number of Pending Messages",
"Number of Queues",
"Number of Topics",
"Total CPU Usage",
"Process - IO Read Operations/sec",
"Process - IO Write Operations/sec",
"Disk Stats - Reads Completed Successfully/sec",
"Disk Stats - Writes Completed Successfully/sec",
"App Allocated Bytes",
"Allocator Sys Bytes",
"Allocator Available Bytes",
"Allocator Thread Caches Bytes"
]
convert => {
"CPU" => "float"
"Usage" => "float"
"Threads" => "float"
"File Descriptors" => "float"
"RSS Virtual Bytes" => "float"
"Available Bytes" => "float"
"Broker Memory Usage" => "float"
"Broker Store Usage" => "float"
"Total Messages Enqueued" => "float"
"Total Messages Dequeued" => "float"
"Number of Pending Messages" => "float"
"Number of Queues" => "float"
"Number of Topics" => "float"
"Total CPU Usage" => "float"
"Process - IO Read Operations/sec" => "float"
"Process - IO Write Operations/sec" => "float"
"Disk Stats - Reads Completed Successfully/sec" => "float"
"Disk Stats - Writes Completed Successfully/sec" => "float"
"App Allocated Bytes" => "float"
"Allocator Sys Bytes" => "float"
"Allocator Available Bytes" => "float"
"Allocator Thread Caches Bytes" => "float"
}
}

}
}

output {
datadog_metrics {
api_key => "my-api-key"
metric_name => "metric-name"
metric_type => "gauge"
metric_value => "metric value"
timeframe => 1
}
}


(David Niceact) #2

For example, this is my csv file I am trying to parse in logstash and send to datadog using datadog_metrics plugin.

Process ID=26350,Messaging Broker Name=ip-10-220-2-122,Broker Memory Limit=181665792,Broker Store Limit=1594460092
Time,CPU Usage,Threads,File Descriptors,RSS,Virtual Bytes,Available Bytes,Broker Memory Usage,Broker Store Usage,Total Messages Enqueued,Total Messages Dequeued,Number of Pending Messages,Number of Queues,Number of Topics,Total CPU Usage, Process - IO Read Operations/sec, Process - IO Write Operations/sec, Disk Stats - Reads Completed Successfully/sec, Disk Stats - Writes Completed Successfully/sec,App Allocated Bytes,Allocator Sys Bytes,Allocator Available Bytes,Allocator Thread Caches Bytes
2017-Jan-31 16:09:11.487,2.298,73,305,426479616,3027333120,612290560,0,3,0,0,0,0,0,3.684,,,0,2,57887832,78528512,0,6655960
2017-Jan-31 16:09:21.487,1.400,73,305,427180032,3027333120,613728256,0,3,0,0,0,0,0,2.254,,,0,0,58121600,78528512,0,6965992
2017-Jan-31 16:09:31.487,1.300,73,300,427257856,3027333120,613584896,0,3,0,0,0,0,0,4.574,,,0,2,57615648,78528512,0,7105288


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.