Ingesting table-like data

(Wiem DW) #1


I am trying to ingest the following file format:

<Some header lines>
X  1
Y  1

And would like the following document format in Elasticsearch 5.x

{'item': 'X', 'value': '1'}
{'item': 'X', 'value': '2'}
{'item': 'Y', 'value': '1'}
{'item': 'Y', 'value': '3'}

I am currently testing with filebeat (run-once) + logstash's aggregate plugin, but it feels cumbersome and a bit hacky. Any ideas or hints on how to parse this data format are welcome.

Wim DW

(David Pilato) #2

I moved your question to #logstash.

I'm pretty sure @fbaligand will help :stuck_out_tongue:

(Wiem DW) #3

Just want to note that I do not really have restrictions on how the data gets into Elasticsearch. Whether it's through beat, logstash, a plugin, an ES pipeline or a custom script, I'd just prefer the simplest (~ most elegant) option.

(Fabien Baligand) #5

Given that you have an input with type option filled, here's a Logstash configuration which should answer your need.

filter {
	grok {
		match => { "message" => ["^%{WORD:item}%{SPACE}%{INT:value}$", "^%{SPACE}%{INT:value}$"] }
	if [item] {
		aggregate {
			task_id => "%{type}"
			code => "map['item'] = event.get('item')"
	else {
		aggregate {
			task_id => "%{type}"
			code => "event.set('item', map['item'])"

(Wiem DW) #6

This works fine (and is a lot less complex as what I currently already had). Thanks!

Any plans to allow this type of aggregation in the ES ingest node as a processor?

Wim DW

(Fabien Baligand) #7

Happy to know it works fine and answers your need :slight_smile:

Concerning ES ingest node, it is not done for this kind of use.
I mean : ES ingest node is done to process "simple" cases where all lines are processed in the same way.
For example, there is no if/else statement in ES ingest node.

(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.