# Ingesting table-like data

**URL:** <https://discuss.elastic.co/t/ingesting-table-like-data/83771>\
**Category:** Logstash\
**Created:** [April 26, 2017, 9:07pm UTC](https://discuss.elastic.co/t/ingesting-table-like-data/83771 "2017-04-26T21:07:41Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![WiemDW](https://avatars.discourse-cdn.com/v4/letter/w/ba8739/32.png) [@WiemDW](https://discuss.elastic.co/u/WiemDW)\
**Post date:** [April 26, 2017, 9:07pm UTC](https://discuss.elastic.co/t/ingesting-table-like-data/83771/1 "2017-04-26T21:07:41Z")

</div>

Hi,

I am trying to ingest the following file format:

```
<Some header lines>
===================
X 1
   2
Y 1
   3

```

And would like the following document format in Elasticsearch 5.x

```
{'item': 'X', 'value': '1'}
{'item': 'X', 'value': '2'}
{'item': 'Y', 'value': '1'}
{'item': 'Y', 'value': '3'}

```

I am currently testing with filebeat (run-once) + logstash's aggregate plugin, but it feels cumbersome and a bit hacky. Any ideas or hints on how to parse this data format are welcome.

Thanks,  
Wim DW

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 26, 2017, 9:23pm UTC](https://discuss.elastic.co/t/ingesting-table-like-data/83771/2 "2017-04-26T21:23:06Z")

</div>

I moved your question to #logstash.

I'm pretty sure @fbaligand will help 😛

---

<div class="post-metadata">

**Author:** ![WiemDW](https://avatars.discourse-cdn.com/v4/letter/w/ba8739/32.png) [@WiemDW](https://discuss.elastic.co/u/WiemDW)\
**Post date:** [April 26, 2017, 9:37pm UTC](https://discuss.elastic.co/t/ingesting-table-like-data/83771/3 "2017-04-26T21:37:12Z")

</div>

Just want to note that I do not really have restrictions on _how_ the data gets into Elasticsearch. Whether it's through beat, logstash, a plugin, an ES pipeline or a custom script, I'd just prefer the simplest (~ most elegant) option.

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [April 27, 2017, 8:21am UTC](https://discuss.elastic.co/t/ingesting-table-like-data/83771/5 "2017-04-27T08:21:12Z")

</div>

@WiemDW  
Given that you have an input with `type` option filled, here's a Logstash configuration which should answer your need.

```
filter {
	
	grok {
		match => { "message" => ["^%{WORD:item}%{SPACE}%{INT:value}$", "^%{SPACE}%{INT:value}$"] }
	}
	
	if [item] {
		aggregate {
			task_id => "%{type}"
			code => "map['item'] = event.get('item')"
		}
	}
	else {
		aggregate {
			task_id => "%{type}"
			code => "event.set('item', map['item'])"
		}
	}
}
```

---

<div class="post-metadata">

**Author:** ![WiemDW](https://avatars.discourse-cdn.com/v4/letter/w/ba8739/32.png) [@WiemDW](https://discuss.elastic.co/u/WiemDW)\
**Post date:** [May 2, 2017, 8:18am UTC](https://discuss.elastic.co/t/ingesting-table-like-data/83771/6 "2017-05-02T08:18:14Z")

</div>

This works fine (and is a lot less complex as what I currently already had). Thanks!

Any plans to allow this type of aggregation in the ES ingest node as a processor?

Wim DW

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [May 2, 2017, 8:38am UTC](https://discuss.elastic.co/t/ingesting-table-like-data/83771/7 "2017-05-02T08:38:44Z")

</div>

Happy to know it works fine and answers your need 🙂

Concerning ES ingest node, it is not done for this kind of use.  
I mean : ES ingest node is done to process "simple" cases where all lines are processed in the same way.  
For example, there is no if/else statement in ES ingest node.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 30, 2017, 8:53am UTC](https://discuss.elastic.co/t/ingesting-table-like-data/83771/8 "2017-05-30T08:53:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.