[Newbie Alert] How would you process this data?

Elastic Stack Logstash
1

I would like to import this data into ELK. Data structure see below.

2

How does your ingestion pipeline looks like? Do you use Filebeat -> Logstash -> Elasticsearch or something else? Do you use the JSON decoding in Filebeat?

3

Yes, that's the pipeline i use. And, as i said, the UniqueID gets represented in ES as a data.UniqueID key field with Host1 as data.

4

Can you share the exact data that you get in elasticsearch compared to what you feed in? And what the expected outcome is?

5

That is a little bit of a problem because i'm not supposed to publish the data structure on the Internet. Let me try to simulate it. This is the data i am pushing in:

This is text that needs to be deleted: {
	"data": {
		"someUniqueNumber": {
			"someHostname": [{
				"receivedTime": 1487696514219,
				"someLabel": someValue,
				"somePropertyList": {
					"someItentifier": someCounter
				}
			}]
		},
		"nextUniqueNumber": {
			"anotherHostname": [{
				"receivedTime": 1487696514250,
				"someLabel": someValue ,
				"somePropertyList": {
					"someIdentifier": someCounter
				    "anotherIdentifier": anotherCounter
				}
			}],
			"anotherHostname": [{
				"receivedTime": 1487696514275,
				"someLabel": someValue,
				"somePropertyList": {
					"someIdentifier": someCounter
				}
			}]
		},
		"nextUniqueNumber": {
			"yetAnotherHostname": [{
				"receivedTime": 1487696514250,
				"someLabel": someValue,
				"somePropertyList": {
					"someIdentifier": someCounter
				}
			}],
			"yetAnotherHostname": [{
				"receivedTime": 1487696514275,
				"someLabel": someValue,
				"somePropertyList": {
					"someIdentifier": someCounter
  				    "anotherIdentifier": anotherCounter
			}
			}]
		},
	"summary": {
		"someHostname": totalForThisHostname,
		"anotherHostname": totalForThisHostname,
		"yetAnotherHostname": totalForThisHostname,
		"yetAnotherHostname": totalForThisHostname
	}
}

As you can see i have some nested objects which i need to handle. Nobody told me this would be easy.

This is my current filter:

filter {
    grok {
        patterns_dir => ["./patterns"]
        match => { "message" => "%{DATA:todelete}: %{GREEDYDATA:message}" }
        remove_field => "todelete"
        overwrite => ["message"]
    }
   json {
       source => "message"
   }

   date {
       match => [ "receivedTime", "UNIX_MS" ]
   }
}

I get the feeling that receivedTime is not transferred to @timestamp because it's in a nested object which cannot be found (yet).

What i am trying to achieve is to analyse the following:

  • Trend of someValue over receivedTime for uniqueNumber.
  • Trend of someValue over receivedTime for uniqueNumber group by Hostname.
  • Graphing someValues over someIdentifier for specific uniqueNumber.

I hope this all makes a little sense.

6

As your conversion is quite unusual, I think you have to write some customer "script" for it. I assume this should be possible with the ruby filter: https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html

Not 100% sure I understand the full problem but it seems the challenge your facing is mainly on the transformation side, means in LS. Should I move this into the LS forum?

7

Yeah, i think it more belongs there. Go ahead and move it.
An, and the problem i am having is that technically it's a valid JSON file, but i have key-value pairs, arrays and lists in the file which make is complex (for a beginner) to find the right conversion into fields.

8

Ok, move this topic to the Logstash forum.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.