[Newbie Alert] How would you process this data?

I would like to import this data into ELK. Data structure see below.

How does your ingestion pipeline looks like? Do you use Filebeat -> Logstash -> Elasticsearch or something else? Do you use the JSON decoding in Filebeat?

Yes, that's the pipeline i use. And, as i said, the UniqueID gets represented in ES as a data.UniqueID key field with Host1 as data.

Can you share the exact data that you get in elasticsearch compared to what you feed in? And what the expected outcome is?

That is a little bit of a problem because i'm not supposed to publish the data structure on the Internet. Let me try to simulate it. This is the data i am pushing in:

This is text that needs to be deleted: {
	"data": {
		"someUniqueNumber": {
			"someHostname": [{
				"receivedTime": 1487696514219,
				"someLabel": someValue,
				"somePropertyList": {
					"someItentifier": someCounter
				}
			}]
		},
		"nextUniqueNumber": {
			"anotherHostname": [{
				"receivedTime": 1487696514250,
				"someLabel": someValue ,
				"somePropertyList": {
					"someIdentifier": someCounter
				    "anotherIdentifier": anotherCounter
				}
			}],
			"anotherHostname": [{
				"receivedTime": 1487696514275,
				"someLabel": someValue,
				"somePropertyList": {
					"someIdentifier": someCounter
				}
			}]
		},
		"nextUniqueNumber": {
			"yetAnotherHostname": [{
				"receivedTime": 1487696514250,
				"someLabel": someValue,
				"somePropertyList": {
					"someIdentifier": someCounter
				}
			}],
			"yetAnotherHostname": [{
				"receivedTime": 1487696514275,
				"someLabel": someValue,
				"somePropertyList": {
					"someIdentifier": someCounter
  				    "anotherIdentifier": anotherCounter
			}
			}]
		},
	"summary": {
		"someHostname": totalForThisHostname,
		"anotherHostname": totalForThisHostname,
		"yetAnotherHostname": totalForThisHostname,
		"yetAnotherHostname": totalForThisHostname
	}
}

As you can see i have some nested objects which i need to handle. Nobody told me this would be easy.

This is my current filter:

filter {
    grok {
        patterns_dir => ["./patterns"]
        match => { "message" => "%{DATA:todelete}: %{GREEDYDATA:message}" }
        remove_field => "todelete"
        overwrite => ["message"]
    }
   json {
       source => "message"
   }

   date {
       match => [ "receivedTime", "UNIX_MS" ]
   }
}

I get the feeling that receivedTime is not transferred to @timestamp because it's in a nested object which cannot be found (yet).

What i am trying to achieve is to analyse the following:

  • Trend of someValue over receivedTime for uniqueNumber.
  • Trend of someValue over receivedTime for uniqueNumber group by Hostname.
  • Graphing someValues over someIdentifier for specific uniqueNumber.

I hope this all makes a little sense.

As your conversion is quite unusual, I think you have to write some customer "script" for it. I assume this should be possible with the ruby filter: https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html

Not 100% sure I understand the full problem but it seems the challenge your facing is mainly on the transformation side, means in LS. Should I move this into the LS forum?

Yeah, i think it more belongs there. Go ahead and move it.
An, and the problem i am having is that technically it's a valid JSON file, but i have key-value pairs, arrays and lists in the file which make is complex (for a beginner) to find the right conversion into fields.

Ok, move this topic to the Logstash forum.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.