Logstash not sending logs

[anudeepmk]

I am newbie to ELK, I am trying to send the SAP hybris logs to elasticsearch from logstash on the server

     input {
      file {
           path => "/hybris/log/hybris*.log"
         type => "hybris"
         start_position => "beginning"
     }

    file {
         path => "/hybris/log/access*.log"
         type => "hybris_access"
         start_position => "beginning"
      }
   }

   filter {
      if [type] == "hybris" {
                    grok {
                           match => [
                             "message",
                             "^\s*%{TIMESTAMP_ISO8601:logdate}\s*\[%{LOGLEVEL:loglevel} \|\[%{IP:client_ip}\] \|%
     {WORD:sessionid}\|%{NOTSPACE:classname}\]"
                           ]
                    }
            date {
                   match => [ "logdate", "ISO8601" ]
                   remove_field => ["logdate"]
            }
            multiline {
                   pattern => "^\s*%{TIMESTAMP_ISO8601}"
                   negate => true
                   what => "previous"
                   periodic_flush => true
            }
      }

       if [type] == "hybris_access" {
          grok {
               match => [
                 "message",
                 "^%{IP:client_ip}.*\[%{DATA:logdate}\].* %{WORD:sessionid}$"
               ]
           }

        date {
          # [16/Mar/2016:17:51:17 +0100]
          match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ]
          remove_field => ["logdate"]
           }
       }
       }

      output {
          elasticsearch {
            hosts => "<elkIP>:9200"
            index => "hybrislogs"
         }
       }

I see the below logs from logstash

{:timestamp=>"2017-12-13T12:58:22.675000+0000", :message=>"Defaulting pipeline worker threads to 1 because there are some filters that might not work with multiple worker threads", :count_was=>4, :filters=>["multiline"], :level=>:warn}

please advise, Am I doing anything wrong?

Versions:
logstash - logstash-2.2.4
kibana-4.4.2-1.x86_64
elasticsearch-2.4.6-1.noarch

[anudeepmk]

any suggetions on this?

[magnusbaeck (Magnus Bäck)]

1. Remove your multiline filter.
2. Replace the elasticsearch output with a stdout { codec => rubydebug } output to reduce complexity and simplify debugging.
3. Read the file input documentation. Pay attention to what's said about sincedb.
4. Read some of the countless previous topics about "Logstash's file input not reading files".
5. When you have Logstash reading input lines introduce a multiline codec to deal with multiline messages. Do not use a multiline filter.
6. When multiline messages are correctly dealt with, reintroduce the elasticsearch output.

[anudeepmk]

@magnusbaeck, thank you for the suggestion, I have now installed latest version of logstash, I read about sincedb, is it necessary to mention the sincedb path of the input file?
Also should I always use the filebeat and then send to logstash or I can directly use logstash and send it to ES?
I have only 2 hybris servers and I need to setup logstash on both of them and process and then forward to ES..
Below is the logs I need to send to ES

INFO | jvm 1 | main | 2017/12/15 11:23:46.139 | INFO [hybrisHTTP2] [] [StartPickingAction] Generating Pack Label for consignment 00005004
INFO | jvm 1 | main | 2017/12/15 11:23:46.139 | INFO [hybrisHTTP2] [] [DefaultPrintMediaService] Generating media for template: [PickLabelDocumentTemplate] and item associated with business process: [NonFoodConsignmentProcessModel]

please advise,
1)what filters can I use?
2)should I use multiline codec in the input section of logstash?

is there any predefined pattern for hybris logs in Logstash?