Grok doesn't split fields properly

[kishorramanan]

Log file:

INFO  2017-02-24 14:25:12,612 ScriptEngines.java:191 - Loaded groovy ScriptEngine
WARN  2017-02-24 14:25:12,814 DseSessionOpProcessor.java:295 - Exception processing a script on request [RequestMessage{, manageTransaction=true, scriptEvaluationTimeout=0}}].
java.lang.UnsupportedOperationException: Could not find an index to answer query clause and graph.allow_scan is disabled: ((label = role))
                at com.datastax.bdp.graph.impl.query.graph.GraphQueryExecutorImpl.execute(GraphQueryExecutorImpl.java:215) ~[dse-graph-5.0.4.jar:5.0.4]
                at com.datastax.bdp.graph.impl.query.graph.GraphQueryExecutorImpl.execute(GraphQueryExecutorImpl.java:56) ~[dse-graph-5.0.4.jar:5.0.4]

Logstash Version: 2.4

Conf file:

filter {
      grok {
          match => {
              "message" => "%{LOGLEVEL:Level}  %{GREEDYDATA:message}"
          }
      }
  }

Output:

Time                         Level 
March 2nd 2017, 01:22:49.658  -  
March 2nd 2017, 01:22:49.562  -  
March 2nd 2017, 01:22:49.543 WARN 

How do I create the filter?

Note: Grok Debugger gave me the expected results

{
  "Level": [
    [
      "WARN"
    ]
  ],
  "message": [
    [
      " 2017-02-24 14:25:12,814 DseSessionOpProcessor.java:295 - Exception processing a script on request "
    ]
  ]
}

[jkuang (Jimmy Kuang)]

I'm not sure what grok filter you're expecting. Please elaborate.

For example:

1. Input to Logstash:

WARN March 2nd 2017, 01:22:49.543

2. Logstash > ES with grok filter to drop or add or change ....

3. Expected ES Output:

Time                         Level 
March 2nd 2017, 01:22:49.658  -  
March 2nd 2017, 01:22:49.562  -  
March 2nd 2017, 01:22:49.543 WARN

[kishorramanan]

@jkuang

3. Expected ES Output:

Exected Output is to split logs into fields

Time Loglevel Message

But while passing GROK, I get unwanted lines in the Loglevel fields[quote="jkuang, post:2, topic:77047"]
Time Level
March 2nd 2017, 01:22:49.658 -
March 2nd 2017, 01:22:49.562 -
March 2nd 2017, 01:22:49.543 WARN
[/quote]

I dont want any hypen in my log output

[jkuang (Jimmy Kuang)]

Try the following:

input {

#stdin {}
 file {
    sincedb_path => "/tmp/sincedb"
    path => "/tmp/apache6.log"
    type => "apache-access"  # a type to identify those logs (will need this later)
    start_position => "beginning"
    codec => multiline {
      pattern => "^%{LOGLEVEL}"
      negate => true
      what => "previous"
    }
  }

}

 filter {
  mutate {
    gsub => [ "message", "r", "" ]
  }
  grok {
    match => [ "message", "(?m)%{LOGLEVEL:severity}  %{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:message}" ]
    overwrite => [ "message" ]
  }
  date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss,SSS" ]
  }
}

output {
  #stdout { codec => rubydebug }
          elasticsearch {
                action => "index"
                index => "grok-test"
                document_type => "mytest"
                manage_template => "false"
                #user => logstash_internal
                #password => changeme
        }

}