# \[Newbie Alert\] How would you process this data?

**URL:** <https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111>\
**Category:** Logstash\
**Created:** [March 10, 2017, 8:39am UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111 "2017-03-10T08:39:45Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Toontje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toontje/32/16100_2.png) [@Toontje](https://discuss.elastic.co/u/Toontje)\
**Post date:** [March 10, 2017, 8:39am UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111/1 "2017-03-10T08:39:45Z")

</div>

I would like to import this data into ELK. Data structure see below.

---

<div class="post-metadata">

**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\
**Post date:** [March 10, 2017, 3:18pm UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111/2 "2017-03-10T15:18:06Z")

</div>

How does your ingestion pipeline looks like? Do you use Filebeat -\> Logstash -\> Elasticsearch or something else? Do you use the JSON decoding in Filebeat?

---

<div class="post-metadata">

**Author:** ![Toontje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toontje/32/16100_2.png) [@Toontje](https://discuss.elastic.co/u/Toontje)\
**Post date:** [March 12, 2017, 11:18am UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111/3 "2017-03-12T11:18:13Z")

</div>

Yes, that's the pipeline i use. And, as i said, the UniqueID gets represented in ES as a data.UniqueID key field with Host1 as data.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 15, 2017, 8:17pm UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111/4 "2017-03-15T20:17:53Z")

</div>

Can you share the exact data that you get in elasticsearch compared to what you feed in? And what the expected outcome is?

---

<div class="post-metadata">

**Author:** ![Toontje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toontje/32/16100_2.png) [@Toontje](https://discuss.elastic.co/u/Toontje)\
**Post date:** [March 22, 2017, 7:46pm UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111/5 "2017-03-22T19:46:41Z")

</div>

That is a little bit of a problem because i'm not supposed to publish the data structure on the Internet. Let me try to simulate it. This is the data i am pushing in:

```
This is text that needs to be deleted: {
	"data": {
		"someUniqueNumber": {
			"someHostname": [{
				"receivedTime": 1487696514219,
				"someLabel": someValue,
				"somePropertyList": {
					"someItentifier": someCounter
				}
			}]
		},
		"nextUniqueNumber": {
			"anotherHostname": [{
				"receivedTime": 1487696514250,
				"someLabel": someValue ,
				"somePropertyList": {
					"someIdentifier": someCounter
				    "anotherIdentifier": anotherCounter
				}
			}],
			"anotherHostname": [{
				"receivedTime": 1487696514275,
				"someLabel": someValue,
				"somePropertyList": {
					"someIdentifier": someCounter
				}
			}]
		},
		"nextUniqueNumber": {
			"yetAnotherHostname": [{
				"receivedTime": 1487696514250,
				"someLabel": someValue,
				"somePropertyList": {
					"someIdentifier": someCounter
				}
			}],
			"yetAnotherHostname": [{
				"receivedTime": 1487696514275,
				"someLabel": someValue,
				"somePropertyList": {
					"someIdentifier": someCounter
  				    "anotherIdentifier": anotherCounter
			}
			}]
		},
	"summary": {
		"someHostname": totalForThisHostname,
		"anotherHostname": totalForThisHostname,
		"yetAnotherHostname": totalForThisHostname,
		"yetAnotherHostname": totalForThisHostname
	}
}

```

As you can see i have some nested objects which i need to handle. Nobody told me this would be easy.

This is my current filter:

```
filter {
    grok {
        patterns_dir => ["./patterns"]
        match => { "message" => "%{DATA:todelete}: %{GREEDYDATA:message}" }
        remove_field => "todelete"
        overwrite => ["message"]
    }
   json {
       source => "message"
   }

   date {
       match => ["receivedTime", "UNIX_MS"]
   }
}

```

I get the feeling that receivedTime is not transferred to @timestamp because it's in a nested object which cannot be found (yet).

What i am trying to achieve is to analyse the following:

- Trend of someValue over receivedTime for uniqueNumber.
- Trend of someValue over receivedTime for uniqueNumber group by Hostname.
- Graphing someValues over someIdentifier for specific uniqueNumber.

I hope this all makes a little sense.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 25, 2017, 7:57pm UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111/6 "2017-03-25T19:57:45Z")

</div>

As your conversion is quite unusual, I think you have to write some customer "script" for it. I assume this should be possible with the ruby filter: [https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html)

Not 100% sure I understand the full problem but it seems the challenge your facing is mainly on the transformation side, means in LS. Should I move this into the LS forum?

---

<div class="post-metadata">

**Author:** ![Toontje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/toontje/32/16100_2.png) [@Toontje](https://discuss.elastic.co/u/Toontje)\
**Post date:** [April 3, 2017, 12:10pm UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111/7 "2017-04-03T12:10:54Z")

</div>

Yeah, i think it more belongs there. Go ahead and move it.  
An, and the problem i am having is that technically it's a valid JSON file, but i have key-value pairs, arrays and lists in the file which make is complex (for a beginner) to find the right conversion into fields.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [April 3, 2017, 8:09pm UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111/8 "2017-04-03T20:09:20Z")

</div>

Ok, move this topic to the Logstash forum.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 1, 2017, 8:10pm UTC](https://discuss.elastic.co/t/newbie-alert-how-would-you-process-this-data/78111/9 "2017-05-01T20:10:09Z")

</div>

Closed on May 1, 2017, 8:10pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.