# Need help on logstash.conf for Wildfly logs **URL:** \ **Category:** Logstash\ **Created:** [June 27, 2018, 7:09am UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549 "2018-06-27T07:09:15Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Chandana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandana/32/90168_2.png) [@Chandana](https://discuss.elastic.co/u/Chandana)\ **Post date:** [June 27, 2018, 7:09am UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/1 "2018-06-27T07:09:15Z")
I converted the wildfly logs to logstash readable format (Json). Here is how my log looks like > {"@timestamp":"2018-06-22T18:36:55.138+0530","@message":"2018-06-22 18:36:55,138 INFO [com.vcl.util.LogTracer] (default task-2) - [T:293148959518596\_w4qpV6V4Oq|L:null|LU:null] - 192.168.3.51 | /login.do | ? | ? | 5559\r","@source":"stdout","@source\_host":"xxx-laptop","@fields":{"timestamp":1529672815138,"level":"INFO","line\_number":0,"class":"org.jboss.stdio.AbstractLoggingWriter","method":"write"},"@tags":["UNKNOWN"]} I am trying to parse these logs using the below logshash.conf file ``` input{ file{ path => "D:/elkstack/QfundTestLogs/*.log" start_position => beginning } } filter{ grok{ match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} \[%{DATA:class}\] - \[%{DATA:message}\] - %{IP:ip} %{GREEDYDATA:message}"} } json{ source => "message" target => "parsedJson" remove_field=>["message"] } } output{ elasticsearch{ hosts => "localhost:9200" index => "logstash-%{+YYY.MM.dd}" } } ``` It would be really great if someone can help me write a correct logstash.conf . ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 27, 2018, 8:39am UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/2 "2018-06-27T08:39:29Z")
As a start use `codec => json` in your file input. ---
**Author:** ![Chandana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandana/32/90168_2.png) [@Chandana](https://discuss.elastic.co/u/Chandana)\ **Post date:** [June 27, 2018, 11:09am UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/3 "2018-06-27T11:09:24Z")
Many Thanks @magnusbaeck. I tried using codec =\> json , no luck. I dont see anything in Kibana, I think my log is not getting parsed. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 27, 2018, 11:10am UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/4 "2018-06-27T11:10:28Z")
Use a `stdout { codec => rubydebug }` while debugging your inputs and your filters. Once that looks okay you can reenable your elasticsearch output. ---
**Author:** ![Chandana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandana/32/90168_2.png) [@Chandana](https://discuss.elastic.co/u/Chandana)\ **Post date:** [June 27, 2018, 11:18am UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/5 "2018-06-27T11:18:54Z")
@magnusbaeck I see below output . ``` [2018-06-27T16:43:31,495][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"D:/elkstack/logstash-6.2.3/modules/fb_apache/configuration"} [2018-06-27T16:43:31,515][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"D:/elkstack/logstash-6.2.3/modules/netflow/configuration"} [2018-06-27T16:43:31,766][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-06-27T16:43:32,610][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.2.3"} [2018-06-27T16:43:33,426][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [2018-06-27T16:43:39,275][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2018-06-27T16:43:40,221][INFO][logstash.pipeline] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#"} [2018-06-27T16:43:40,373][INFO][logstash.agent] Pipelines running {:count=>1, :pipelines=>["main"]} { "@tags" => [ [0] "UNKNOWN" ], "path" => "D:/elkstack/QfundTestLogs/wildfly_logstash.log", "@timestamp" => 2018-06-22T13:06:49.717Z, "@message" => "2018-06-22 18:36:49,717 INFO [com.vcl.util.LogTracer] (default task-2) - [T:293148959518596_w4qpV6V4Oq|L:null|LU:null] - LookUp : select : Query Elapsed time0.02sec\r", "@source" => "stdout", "@fields" => { "line_number" => 0, "method" => "write", "timestamp" => 1529672809717, "class" => "org.jboss.stdio.AbstractLoggingWriter", "level" => "INFO" }, "@version" => "1", "tags" => [ [0] "_grokparsefailure" ], "host" => "xxx-LAPTOP", "@source_host" => "yyy-laptop" } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 27, 2018, 11:35am UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/6 "2018-06-27T11:35:39Z")
Okay, so Logstash is at least able to read from the file. If Logstash has problems sending to Elasticsearch it'll tell you about it in its own logfile. Cranking up the loglevel will give additional details. I just saw this in your configuration: ``` index => "logstash-%{+YYY.MM.dd}" ``` There's a "Y" missing here. ---
**Author:** ![Chandana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandana/32/90168_2.png) [@Chandana](https://discuss.elastic.co/u/Chandana)\ **Post date:** [June 27, 2018, 11:39am UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/7 "2018-06-27T11:39:51Z")
I Added the Y over there. Thank You. > [@Chandana](#): > > [0] "\_grokparsefailure" It is saying that \_grokparsefailure . How do I solve this ?? Do I have make any changes to the pattern? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 27, 2018, 12:04pm UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/8 "2018-06-27T12:04:27Z")
To start with the grok filter you posted earlier attempts to parse the `message` field but your events have no field with that name. ---
**Author:** ![Chandana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandana/32/90168_2.png) [@Chandana](https://discuss.elastic.co/u/Chandana)\ **Post date:** [June 27, 2018, 12:06pm UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/9 "2018-06-27T12:06:37Z")
> [@Chandana](#): > > "@message" =\> "2018-06-22 18:36:49,717 INFO [com.vcl.util.LogTracer] (default task-2) - [T:293148959518596\_w4qpV6V4Oq|L:null|LU:null] - LookUp : select : Query Elapsed time0.02sec\r", I am a bit puzzled, are you talking about this by any chance? I have a field named message in my logs. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 27, 2018, 12:20pm UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/10 "2018-06-27T12:20:38Z")
> I have a field named message in my logs. Nope. As you can see from the stdout output the field is named `@message`. ---
**Author:** ![Chandana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandana/32/90168_2.png) [@Chandana](https://discuss.elastic.co/u/Chandana)\ **Post date:** [June 27, 2018, 12:46pm UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/11 "2018-06-27T12:46:06Z")
> match =\> { "@message" =\> "%{TIMESTAMP\_ISO8601:timestamp} %{LOGLEVEL:log-level} [%{DATA:class}] %{DATA:text} - [%{DATA:kvpairs}] - %{GREEDYDATA:message}"} I changed the match in grok to this and trying to parse the log ` > {"@message":"2018-06-22 19:36:49,717 INFO [com.vcl.util.LogTracer] (default task-2) - [T:293148959518596\_w4qpV6V4Oq|L:null|LU:null] - LookUp : select : No of rows returned:0\r","@source":"stdout","@source\_host":"xxx-laptop","@fields":{"timestamp":1529672809717,"level":"INFO","line\_number":0,"class":"org.jboss.stdio.AbstractLoggingWriter","method":"write"},"@tags":["UNKNOWN"]} ` Am I missing any thing with the pattern match? Can you please help me to match the pattern. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 27, 2018, 1:27pm UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/12 "2018-06-27T13:27:22Z")
I can't spot anything OTOH. Build your expression gradually by starting with the very simplest expression (`^%{TIMESTAMP_ISO8601:timestamp}`). Eventually it'll break and then you've narrowed things down. ---
**Author:** ![Chandana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandana/32/90168_2.png) [@Chandana](https://discuss.elastic.co/u/Chandana)\ **Post date:** [June 28, 2018, 9:55am UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/13 "2018-06-28T09:55:22Z")
Many Thanks @magnusbaeck 🙂 I follwed your suggestion, I broke down and I could succeed parsing the log. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 26, 2018, 9:55am UTC](https://discuss.elastic.co/t/need-help-on-logstash-conf-for-wildfly-logs/137549/14 "2018-07-26T09:55:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.