How to parse such log

Elastic Stack Logstash
1

I am getting logs in below format. How can I parse it. if I use msgpack, i am able to see proper results. but i want to parse it in some other way hence need to know what can be done with below mentioned format.

\x8C\xACcounter_name\xBBnetwork.outgoing.bytes.rate\xABresource_id\xDA\u0000Einstance-00000016-700deb56-3785-4e76-a38b-a3b8f33be665-tapb7b9be47-21\xA9timestamp\xBA2018-03-29T07:21:31.237111\xAEcounter_volume\xCB@h\x93+xB\uA7E7user_id\xDA\u0000 817366bc44b646188e12ffc0ac96c784\xB1message_signature\xDA\u0000@502960e5fefc2ce293bcbc4987476ae0d61ed1fcac296c3d02d28bf2f3ef857b\xB1resource_metadata\xDE\u0000\u0018\xADinstance_host\xB9compute3.abc.com\xA5image\xC0\xAAramdisk_id\xC0\xA6flavor\x87\xA4name\xA5Plan1\xA5links\x91\x82\xA4href\xDA\u0000Ehttp://192.168.0.31:8774/flavors/33c137d7-203e-4578-a83b-a86a9f16f36c\xA3rel\xA8bookmark\xA3ram\xCD\u0004\u0000\xA9ephemeral\u0000\xA5vcpus\u0001\xA4disk\u0000\xA2id\xDA\u0000$33c137d7-203e-4578-a83b-a86a9f16f36c\xA9memory_mb\xCD\u0004\u0000\xACdisplay_name\xA5test1\xAAparameters\x80\xA5state\xA6active\xBBOS-EXT-AZ:availability_zone\xA4nova\xA6status\xA6active\xACephemeral_gb\u0000\xA7disk_gb\u0000\xA9kernel_id\xC0\xA4fref\xC0\xA3mac\xB1fa:16:3e:f9:84:50\xADimage_ref_url\xC0\xA7root_gb\u0000\xA4name\xAEtapb7b9be47-21\xA4host\xDA\u000089f9b2ad6a79daef8681b691f5616eec0a8fb960867c59e93e9b085a6\xA9vnic_name\xAEtapb7b9be47-21\xABinstance_id\xDA\u0000$700deb56-3785-4e76-a38b-a3b8f33be665\xADinstance_type\xA5Plan1\xA5vcpus\u0001\xA9image_ref\xC0\xA6source\xA9openstack\xACcounter_unit\xA3B/s\xAAproject_id\xDA\u0000 06e55e2058e14283a63bc64958060c9d\xAAmessage_id\xDA\u0000$cdaf2a56-3321-11e8-9c43-ac1f6b80f4cc\xACcounter_type\xA5gauge

2

Have you tried using the logstash-codec-msgpack plugin?

3

Yes I did and i got it in format like

{
"counter_name" => "network.outgoing.bytes.rate",
"resource_id" => "instance-00000016-700deb56-3785-4e76-a38b-a3b8f33be665-tapb7b9be47-21",
"timestamp" => "2018-03-29T07:12:31.174715",
"counter_volume" => 303.1224128880113,
"user_id" => "817366bc44b646188e12ffc0ac96c784",
"message_signature" => "8f98b04352e8c8570466091f9759acc617aac358b5dae4e7b154d5aa0a41df2f",
"resource_metadata" => {
"instance_host" => "compute3.abc.com",
"image" => nil,
"ramdisk_id" => nil,
"flavor" => {
"name" => "Plan1",
"links" => [
[0] {
"href" => "http://192.168.0.31:8774/flavors/33c137d7-203e-4578-a83b-a86a9f16f36c",
"rel" => "bookmark"
}
],
"ram" => 1024,
"ephemeral" => 0,
"vcpus" => 1,
"disk" => 0,
"id" => "33c137d7-203e-4578-a83b-a86a9f16f36c"
},
"memory_mb" => 1024,
"display_name" => "test1",
"parameters" => {},
"state" => "active",
"OS-EXT-AZ:availability_zone" => "nova",
"status" => "active",
"ephemeral_gb" => 0,
"disk_gb" => 0,
"kernel_id" => nil,
"fref" => nil,
"mac" => "fa:16:3e:f9:84:50",
"image_ref_url" => nil,
"root_gb" => 0,
"name" => "tapb7b9be47-21",
"host" => "9f9b2ad6a79daef8681b691f5616eec0a8fb960867c59e93e9b085a6",
"vnic_name" => "tapb7b9be47-21",
"instance_id" => "700deb56-3785-4e76-a38b-a3b8f33be665",
"instance_type" => "Plan1",
"vcpus" => 1,
"image_ref" => nil
},
"source" => "openstack",
"counter_unit" => "B/s",
"project_id" => "06e55e2058e14283a63bc64958060c9d",
"message_id" => "8bc83b6a-3320-11e8-9a8c-ac1f6b80f4cc",
"counter_type" => "gauge",
"@version" => "1",
"@timestamp" => "2018-03-29T07:12:31.170Z",
"tags" => [],
"type" => "ceilometer",
"host" => "10.9.255.16"
}

now i am unable to convert this into time series data to send it to influxdb. can I use filters to process this log and then send it to influxdb output.

4

What are you trying to do? How is it not working?

I see that the message you posted has a @timestamp property, which is what we use in the Elastic Stack to store the event's timestamp -- if you have two or more messages with a timestamp, then you have a timeseries dataset.

Sure.

A good place to start is the documentation for the logstash-filter-* plugins and the logstash-output-influxdb plugin.

5

here is what I am trying to achieve, I want to push openstack ceilometer data to logstash and from logstash I want to take that into influxdb as timeseries. Input is formatted now using msgpack and I get the data in the format, but I am unable to send it in timeseries format to influx db or to graphite. basically my idea is to use expert tools like ELK or Graphite to do the graphic work for ceilometer data.

6

What specifically is your problem; what have you tried, and how is that not working for you?

7

my end goal is to push all the ceilometer data from openstack to either graphite OR ELK in a format so I can generate dashboard for VMs running on openstack using logstash. the issue I am facing is the data not reaching graphite from logstash in the format required. I can see the output in logstash logs when I do stdout, but it doesn't reach graphite. I also tried using Kibana to generate graphs, but there I am unable to get the data in the integer format, and hence calculations are not happening.

8

You will find that if you post specific examples of log messages and your pipeline configuration, people will be much more likely to be able to help :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.