# Simple log processing without Logstash
**URL:** https://discuss.elastic.co/t/simple-log-processing-without-logstash/73249
**Category:** Beats
**Tags:** beats-development
**Created:** [January 30, 2017, 5:10pm UTC](https://discuss.elastic.co/t/simple-log-processing-without-logstash/73249 "2017-01-30T17:10:16Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@John16](https://discuss.elastic.co/u/John16)
#### Post date: [January 30, 2017, 5:10pm UTC](https://discuss.elastic.co/t/simple-log-processing-without-logstash/73249/1 "2017-01-30T17:10:16Z")
Hello,
I want to use filebeat to send logs to elasticsearch with simple structure (date in custom format, http code, processing time in ms, query text).
So log line should be parsed and these data should go to different fields in Index.
Also I want to add another field (length of query text in symbols, provided it is in UTF-8 encoding), and I want to truncate the actual text so it fits to 32Kb (because of ES limitation).
As far as I understand I can do all these things in Logstash (even add custom handler written in Ruby).
The question is: is it possible to avoid using logstash at all and achieve these transformations using filebeat only (and possibly in ES using ingest API, pipelining. etc).
Thanks!
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 30, 2017, 9:46pm UTC](https://discuss.elastic.co/t/simple-log-processing-without-logstash/73249/2 "2017-01-30T21:46:25Z")
Why not Logstash?
Filebeat can't do this. Ingest can't either.
---
### Author:  [@John16](https://discuss.elastic.co/u/John16)
#### Post date: [January 31, 2017, 9:42am UTC](https://discuss.elastic.co/t/simple-log-processing-without-logstash/73249/3 "2017-01-31T09:42:05Z")
I feel that Logstash eats too much CPU for simple task (receive data via network, parse line against regexp, post parsed json to ES).
It eats almost the same CPU as ES instance on the same machine. But ES performs complex task (index logs) compared with dumb logstash.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [February 1, 2017, 8:58am UTC](https://discuss.elastic.co/t/simple-log-processing-without-logstash/73249/4 "2017-02-01T08:58:24Z")
If LS is not an option, you should check the Ingest Processors if you can do it their: [https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest-processors.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest-processors.html) But Logstash is the one with the full power for such transformation.
---
### Author:  [@John16](https://discuss.elastic.co/u/John16)
#### Post date: [February 1, 2017, 9:47am UTC](https://discuss.elastic.co/t/simple-log-processing-without-logstash/73249/5 "2017-02-01T09:47:47Z")
I tried to load sample log file into ES vis LS. LS process consumed 2 times more CPU that ES. Is it normal? ES does complex job indexing data. And LS only parses lines against regexp. It feels that LS should be rather light process, but it is not case ☹
---
### Author:  [@tudor](https://discuss.elastic.co/u/tudor)
#### Post date: [February 1, 2017, 10:44am UTC](https://discuss.elastic.co/t/simple-log-processing-without-logstash/73249/6 "2017-02-01T10:44:05Z")
Regexp tend to be CPU expensive. I recommend trying the Logstash dissect filter: [https://www.elastic.co/blog/logstash-dude-wheres-my-chainsaw-i-need-to-dissect-my-logs](https://www.elastic.co/blog/logstash-dude-wheres-my-chainsaw-i-need-to-dissect-my-logs)
---
### Author:  [@John16](https://discuss.elastic.co/u/John16)
#### Post date: [February 4, 2017, 1:35pm UTC](https://discuss.elastic.co/t/simple-log-processing-without-logstash/73249/7 "2017-02-04T13:35:45Z")
Thanks, I will looks at dissect module.
Though the same logic for parsing log lines against regexps written in Python consumes like an order of magnitude less CPU that logstash does. So something looks broken here...
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 4, 2017, 1:35pm UTC](https://discuss.elastic.co/t/simple-log-processing-without-logstash/73249/8 "2017-03-04T13:35:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.