# How to find out which logshatsh filter is best for your data **URL:** \ **Category:** Logstash\ **Created:** [July 2, 2018, 8:54am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153 "2018-07-02T08:54:52Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Ambuj\_Saxena](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Ambuj\_Saxena](https://discuss.elastic.co/u/Ambuj_Saxena)\ **Post date:** [July 2, 2018, 8:54am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/1 "2018-07-02T08:54:52Z")
Hi, I am new with ELK, I start learning with ELK and saw some issue with my own data. That time I realized that I am using the wrong filter that is why I am not getting desirable output. This is the sample of my data, Anyone can tell which filter I have to use > 35858786,5-17-18 19:54,RA05,2200110035,2.3.6,10,30 > 35858787,5-17-18 19:54,RL26,2200110037,2.3.6,ERR,78 > 35858788,5-17-18 19:54,RA05,2200110037,2.3.6,20,78 > 35858789,5-17-18 19:54,RA05,1100110274,2.3.5,20,15 > 35858790,5-17-18 19:54,RB03,1100110192,2.2.8,5B00504CE3A4,20.3 > 35858792,5-17-18 19:54,RV02,1100110085,2.3.6,02004931057F,OFF > 35858794,5-17-18 19:54,RB03,4400110018,2.3.6,3F001AC5FF1F,3 > 35858796,5-17-18 19:54,RV02,4400110029,2.3.6,5A009954B92E,OFF > 35858798,5-17-18 19:54,RA05,2200110035,2.3.6,10,30 > 35858799,5-17-18 19:54,RA05,1100110081,2.3.6,20,110 > 35858800,5-17-18 19:54,RA05,1100110251,2.3.a,2,50 > 35858801,5-17-18 19:54,RB03,4400110031,2.3.6,5A009966FB5E,10 > 35858803,5-17-18 19:54,RA05,2200110035,2.3.6,10,30 > 35858804,5-17-18 19:54,RV02,1100110192,2.2.8,1E002CE5E83F,OFF > 35858806,5-17-18 19:54,RB03,4400110015,2.3.6,51004D96FB71,6.6 > 35858808,5-17-18 19:54,RB03,1100110107,2.3.1,CHINTPURNI07,0.2 > 35858810,5-17-18 19:54,RA05,1100110107,2.3.1,10,30 > 35858811,5-17-18 19:54,RA05,1100110189,2.3.5,20,81 > 35858812,5-17-18 19:54,RV02,1100110285,2.3.5,1E002D151E38,OFF/\> ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 2, 2018, 10:31am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/2 "2018-07-02T10:31:31Z")
It depends on the expected outcome, but the easiest way to split this data into multiple fields is to use a csv filter. ---
**Author:** ![Ambuj\_Saxena](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Ambuj\_Saxena](https://discuss.elastic.co/u/Ambuj_Saxena)\ **Post date:** [July 2, 2018, 10:47am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/3 "2018-07-02T10:47:57Z")
@magnusbaeck I used CSV filter but I find **dateparsefailure** this error for some logs, maybe because some string size was not same or they have null also. I would like to recognize the pattern so now what I should do? ``` input { file { type => logs path => ["D:\Ambuj\type 0\118\May_2018\Client_4_May_2018_16_31.csv"] start_position => "beginning" } } filter { csv{ columns => ["log_id","date","msg_string","ID","Version","data","List"] separator => "," } date { match => ["date", "MM-dd-yy hh:mm"] target => "date" } mutate { convert => { "log_id" => "integer" "ID" => "integer" } } } output { elasticsearch { hosts => "localhost" index => "error-5" } stdout {codec => rubydebug} } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 2, 2018, 11:15am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/4 "2018-07-02T11:15:07Z")
When the date filter fails it'll log a message that points to where the problem is. ---
**Author:** ![Ambuj\_Saxena](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Ambuj\_Saxena](https://discuss.elastic.co/u/Ambuj_Saxena)\ **Post date:** [July 2, 2018, 12:44pm UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/5 "2018-07-02T12:44:51Z")
For some logs date filter work perfectly and for some logs it wouldn't. And for pattern recognizing which filter is would work for me ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 2, 2018, 1:35pm UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/6 "2018-07-02T13:35:41Z")
> For some logs date filter work perfectly and for some logs it wouldn't. Sure, that's a possibility. Again, check the Logstash log for clues. > And for pattern recognizing which filter is would work for me What pattern recognition do you need? ---
**Author:** ![Ambuj\_Saxena](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Ambuj\_Saxena](https://discuss.elastic.co/u/Ambuj_Saxena)\ **Post date:** [July 3, 2018, 7:30am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/7 "2018-07-03T07:30:52Z")
@magnusbaeck For logstash log path : > ~path\logstash\log Pattern: I would like to find time to fail pattern One more problem I am facing once I parse the data from logstash and create the index pattern after some time data size will decrease and after reboot the system I have again purse the data, what I am doing wrong , I should wait till all data parse? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2018, 7:34am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/8 "2018-07-03T07:34:39Z")
> Pattern: I would like to find time to fail pattern I have no idea what you mean by that. ---
**Author:** ![Ambuj\_Saxena](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Ambuj\_Saxena](https://discuss.elastic.co/u/Ambuj_Saxena)\ **Post date:** [July 3, 2018, 7:39am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/9 "2018-07-03T07:39:18Z")
this is the log of machine, every log related to some sensor so I want to know when this machine will stop working, what kind of pattern it has ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2018, 8:43am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/10 "2018-07-03T08:43:26Z")
So you want Logstash to analyze the values emitted by a sensor and correlate that with failures of some kind? Logstash doesn't have that kind of analysis capabilities built-in but it should be able to help you parse files to extract the sensor values. ---
**Author:** ![Ambuj\_Saxena](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Ambuj\_Saxena](https://discuss.elastic.co/u/Ambuj_Saxena)\ **Post date:** [July 3, 2018, 9:32am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/11 "2018-07-03T09:32:33Z")
No.. I dont want logstash to analyze data. I want to know which filter is best for me to find pattern weather CSV of gork or something else so I can do what I desire (Time to fail patter). If I am using proper filter to parse data into ES that will give me freedom to analyze logs. Parsing data in ES is first step. I just want to do that right ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2018, 9:54am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/12 "2018-07-03T09:54:42Z")
> I want to know which filter is best for me to find pattern weather CSV of gork or something else so I can do what I desire (Time to fail patter). If you give an example of the input data you have and the desired result it'll be much easier to help. I've looked at your CSV example at the top but I have no idea how to extract any "time to fail" stuff from it. ---
**Author:** ![Ambuj\_Saxena](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Ambuj\_Saxena](https://discuss.elastic.co/u/Ambuj_Saxena)\ **Post date:** [July 3, 2018, 10:14am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/13 "2018-07-03T10:14:18Z")
@magnusbaeck Thanks man for helping me here. really appreciated In my very first post I post the input data. If you still require more on let me know I ll share the data with you ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2018, 10:59am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/14 "2018-07-03T10:59:44Z")
I'm losing my patience here. Yes, as I said I know what the input data looks like, but what is Logstash supposed to do with it? If you can't describe with words and examples what you want to accomplish it's impossible for us to help you translate that into a Logstash configuration. ---
**Author:** ![Ambuj\_Saxena](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Ambuj\_Saxena](https://discuss.elastic.co/u/Ambuj_Saxena)\ **Post date:** [July 3, 2018, 11:40am UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/15 "2018-07-03T11:40:53Z")
@magnusbaeck my device suppose to send RA05 string on specific time period but after some time randomly (may be 15 days or longer may be 2 hours) it will stop sending RA05 string. So I want to analyze what kind of pattern here is, in the mean while regular work is going on and every log would be there. May be possible, there is some kind of pattern that we cant see, some sensor is tripping or sending some data that it will cause non working of machine, I want to relate those instance. Even possible sensor A sending some data and then sensor B sending some data than machine stop working so is there any relation I want to find out that. Time line may be there or may be only sensor data or may be something else. Note : how to find machine is working or even in idle condition, that time RA05 is there with respective period than machine working otherwise we have to count that instance a Time to fail instance. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2018, 1:33pm UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/16 "2018-07-03T13:33:16Z")
As I said, Logstash doesn't have that kind of analysis capabilities built-in but it should be able to help you parse files to extract the sensor values. Over and out. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 31, 2018, 1:33pm UTC](https://discuss.elastic.co/t/how-to-find-out-which-logshatsh-filter-is-best-for-your-data/138153/17 "2018-07-31T13:33:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.