# Parsing the logs **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 17, 2017, 9:16am UTC](https://discuss.elastic.co/t/parsing-the-logs/108095 "2017-11-17T09:16:15Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Sniipe](https://avatars.discourse-cdn.com/v4/letter/s/e56c9b/32.png) [@Sniipe](https://discuss.elastic.co/u/Sniipe)\ **Post date:** [November 17, 2017, 9:16am UTC](https://discuss.elastic.co/t/parsing-the-logs/108095/1 "2017-11-17T09:16:15Z")
Hi, I went to a devOpsDays event and saw ELK for the first time. I've it installed locally to test it out. Its working with filebeat =\> ElasticSearch =\> Kibana. I want to parse the message because it has the useful information I need. It looks like this: 2017-10-24 17:20:07,581 [19] INFO TestFunctionService START isFlagTrue I'd love to have the message "split up" into available fields like this: {TIMESTAMP} {THREAD} {ERROR\_TYPE} {FUNCTION\_TYPE} {MESSAGE} How would I go about doing this correctly? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 17, 2017, 9:30am UTC](https://discuss.elastic.co/t/parsing-the-logs/108095/2 "2017-11-17T09:30:11Z")
You can do this through an [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/ingest.html) in Elasticsearch or by introducing Logstash into your pipeline. ---
**Author:** ![Sniipe](https://avatars.discourse-cdn.com/v4/letter/s/e56c9b/32.png) [@Sniipe](https://discuss.elastic.co/u/Sniipe)\ **Post date:** [November 17, 2017, 9:55am UTC](https://discuss.elastic.co/t/parsing-the-logs/108095/3 "2017-11-17T09:55:24Z")
Thanks Christian, I will look into the ingest pipeline now. I was hoping to keep things simple and thought that by adding logstash it would add more unnecessary complexity. From the diagram on the ELK homepage it looked like I could have beats or logstash and that beats was lighter... If I fail with ingest pipeline I will install logstash. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 17, 2017, 9:56am UTC](https://discuss.elastic.co/t/parsing-the-logs/108095/4 "2017-11-17T09:56:39Z")
Ingest pipelines have a limited set of processors, but do support grok, so you should be fine. ---
**Author:** ![Sniipe](https://avatars.discourse-cdn.com/v4/letter/s/e56c9b/32.png) [@Sniipe](https://discuss.elastic.co/u/Sniipe)\ **Post date:** [November 21, 2017, 11:01pm UTC](https://discuss.elastic.co/t/parsing-the-logs/108095/5 "2017-11-21T23:01:52Z")
Thanks Christian, so, I've managed to use Grok to format the messages I've had into something that outputs well (using the debugger - I've yet to integrate it). During the process I've noticed that the logging I have is all over the place. I was wondering if there is a good standard perhaps in place to create the log files? For example is there a log4net helper function which perhaps takes in multiple parameters. I will probably end up creating one, I was just wondering if some standard method with a standard ingest pipleline existed? ---
**Author:** ![Sniipe](https://avatars.discourse-cdn.com/v4/letter/s/e56c9b/32.png) [@Sniipe](https://discuss.elastic.co/u/Sniipe)\ **Post date:** [November 30, 2017, 11:27am UTC](https://discuss.elastic.co/t/parsing-the-logs/108095/6 "2017-11-30T11:27:32Z")
I'm really struggling with this. It seems so simple, but there aren't any good examples. I have a log.txt which now looks like this sample: 2017-11-30 11:06:25,933 [20] INFO DAL.DBManager "FunctionName":"Connect" 2017-11-30 11:06:25,940 [20] INFO DAL.DBManager "FunctionName":"ExecuteDataReader", "Query":"pkg\_general.get\_number\_of\_players" 2017-11-30 11:06:29,704 [20] INFO DAL.DBManager "FunctionName":"Dispose" Currently in kibana the whole log comes up under Message but I need to break out message to being: LogLevel, Class, FunctionName, Query I don't know what file to modify. I've gone through a few grok examples and I can get the output I desire. I don't know where to implement it. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 30, 2017, 6:38pm UTC](https://discuss.elastic.co/t/parsing-the-logs/108095/7 "2017-11-30T18:38:22Z")
One normally uses Logstash or Elasitcsearch Ingest node to apply the grok parsing. ---
**Author:** ![Sniipe](https://avatars.discourse-cdn.com/v4/letter/s/e56c9b/32.png) [@Sniipe](https://discuss.elastic.co/u/Sniipe)\ **Post date:** [December 11, 2017, 11:18am UTC](https://discuss.elastic.co/t/parsing-the-logs/108095/8 "2017-12-11T11:18:46Z")
Hi Steffen, I will remove filebeat and restart with logstash. Thanks for the reply. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 8, 2018, 11:19am UTC](https://discuss.elastic.co/t/parsing-the-logs/108095/9 "2018-01-08T11:19:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.