Ingesting table-like data

[WiemDW]

Hi,

I am trying to ingest the following file format:

<Some header lines>
===================
X  1
   2
Y  1
   3

And would like the following document format in Elasticsearch 5.x

{'item': 'X', 'value': '1'}
{'item': 'X', 'value': '2'}
{'item': 'Y', 'value': '1'}
{'item': 'Y', 'value': '3'}

I am currently testing with filebeat (run-once) + logstash's aggregate plugin, but it feels cumbersome and a bit hacky. Any ideas or hints on how to parse this data format are welcome.

Thanks,
Wim DW

[dadoonet (David Pilato)]

I moved your question to #logstash.

I'm pretty sure @fbaligand will help

[WiemDW]

Just want to note that I do not really have restrictions on how the data gets into Elasticsearch. Whether it's through beat, logstash, a plugin, an ES pipeline or a custom script, I'd just prefer the simplest (~ most elegant) option.

[fbaligand (Fabien Baligand)]

@WiemDW
Given that you have an input with type option filled, here's a Logstash configuration which should answer your need.

filter {
	
	grok {
		match => { "message" => ["^%{WORD:item}%{SPACE}%{INT:value}$", "^%{SPACE}%{INT:value}$"] }
	}
	
	if [item] {
		aggregate {
			task_id => "%{type}"
			code => "map['item'] = event.get('item')"
		}
	}
	else {
		aggregate {
			task_id => "%{type}"
			code => "event.set('item', map['item'])"
		}
	}
}

[WiemDW]

This works fine (and is a lot less complex as what I currently already had). Thanks!

Any plans to allow this type of aggregation in the ES ingest node as a processor?

Wim DW

[fbaligand (Fabien Baligand)]

Happy to know it works fine and answers your need

Concerning ES ingest node, it is not done for this kind of use.
I mean : ES ingest node is done to process "simple" cases where all lines are processed in the same way.
For example, there is no if/else statement in ES ingest node.