Unknown error in Filebeat

Elastic Stack Beats
1

I am using Filebeats to fetch my Log files(XML log file). It was fetching the logs and after some time I am getting this below error and my Filebeats stops automatically.

goroutine 51620 [semacquire]:
sync.runtime_Semacquire(0xc82011c8c4)
        /usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*Mutex).Lock(0xc82011c8c0)
        /usr/local/go/src/sync/mutex.go:82 +0x1c4
log.(*Logger).Output(0xc82011c8c0, 0x4, 0xc918926600, 0x503d, 0x0, 0x0)
        /usr/local/go/src/log/log.go:147 +0x8b
github.com/elastic/beats/libbeat/logp.send(0x4, 0x7, 0x932ed8, 0x5, 0x941360, 0xb, 0xc918867210, 0x1, 0x1)
        /go/src/github.com/elastic/beats/libbeat/logp/log.go:59 +0x1a5
github.com/elastic/beats/libbeat/logp.debugMessage(0x3, 0x939a10, 0x7, 0x941360, 0xb, 0xc918867210, 0x1, 0x1)
        /go/src/github.com/elastic/beats/libbeat/logp/log.go:50 +0xee
github.com/elastic/beats/libbeat/logp.MakeDebug.func1(0x941360, 0xb, 0xc918867210, 0x1, 0x1)
        /go/src/github.com/elastic/beats/libbeat/logp/log.go:74 +0x71
github.com/elastic/beats/libbeat/publisher.PrintPublishEvent(0xc89ad2f980)
        /go/src/github.com/elastic/beats/libbeat/publisher/publish.go:109 +0x295
github.com/elastic/beats/libbeat/publisher.(*client).annotateEvent(0xc820015bc0, 0xc89ad2f980)
        /go/src/github.com/elastic/beats/libbeat/publisher/client.go:114 +0x17f
github.com/elastic/beats/libbeat/publisher.(*client).PublishEvents(0xc820015bc0, 0xc8ae84c000, 0x800, 0x800, 0xc87a9a2420, 0x2, 0x2, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/client.go:99 +0x86
github.com/elastic/beats/filebeat/beat.(*syncLogPublisher).Start.func1(0xc8482a81c0)
        /go/src/github.com/elastic/beats/filebeat/beat/publish.go:103 +0x3c1
created by github.com/elastic/beats/filebeat/beat.(*syncLogPublisher).Start
        /go/src/github.com/elastic/beats/filebeat/beat/publish.go:113 +0x58

goroutine 51621 [semacquire]:
sync.runtime_Semacquire(0xc82011c8c4)
        /usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*Mutex).Lock(0xc82011c8c0)
        /usr/local/go/src/sync/mutex.go:82 +0x1c4
log.(*Logger).Output(0xc82011c8c0, 0x4, 0xc9194b4640, 0x9d, 0x0, 0x0)
        /usr/local/go/src/log/log.go:147 +0x8b
github.com/elastic/beats/libbeat/logp.send(0x4, 0x7, 0x932ed8, 0x5, 0x9ad7e0, 0x25, 0xc8d8468b58, 0x1, 0x1)
        /go/src/github.com/elastic/beats/libbeat/logp/log.go:59 +0x1a5
github.com/elastic/beats/libbeat/logp.debugMessage(0x3, 0x9482b0, 0x9, 0x9ad7e0, 0x25, 0xc8d8468b58, 0x1, 0x1)
        /go/src/github.com/elastic/beats/libbeat/logp/log.go:50 +0xee
github.com/elastic/beats/libbeat/logp.Debug(0x9482b0, 0x9, 0x9ad7e0, 0x25, 0xc8d8468b58, 0x1, 0x1)
        /go/src/github.com/elastic/beats/libbeat/logp/log.go:69 +0x66
github.com/elastic/beats/filebeat/harvester.(*logFileReader).Read(0xc8d9dd8bd0, 0xc8cb61a000, 0x4000, 0x4000, 0x4000, 0x0, 0x0)
        /go/src/github.com/elastic/beats/filebeat/harvester/reader.go:138 +0xdba
github.com/elastic/beats/filebeat/harvester/encoding.(*LineReader).advance(0xc8d9dd8c40, 0x0, 0x0)
        /go/src/github.com/elastic/beats/filebeat/harvester/encoding/reader.go:118 +0x43e
github.com/elastic/beats/filebeat/harvester/encoding.(*LineReader).Next(0xc8d9dd8c40, 0x0, 0x0, 0x0, 0xa, 0x0, 0x0)
        /go/src/github.com/elastic/beats/filebeat/harvester/encoding/reader.go:67 +0x5c
github.com/elastic/beats/filebeat/harvester/processor.LineSource.Next(0xc8d9dd8c40, 0x0, 0xc800000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/filebeat/harvester/processor/processor.go:58 +0x6d
github.com/elastic/beats/filebeat/harvester/processor.(*timeoutProcessor).Next.func1(0xc843ab6080)
        /go/src/github.com/elastic/beats/filebeat/harvester/processor/timeout.go:52 +0x4c
created by github.com/elastic/beats/filebeat/harvester/processor.(*timeoutProcessor).Next
        /go/src/github.com/elastic/beats/filebeat/harvester/processor/timeout.go:58 +0xa6

Have any one faced this issue.
Below is my filebeat yml config file.

filebeat:
  prospectors:
    -
      paths:
         - /errors/20[0-9][0-9]/logs-*.xml
        
      document_type: elmah_log
      ignore_older: 72h
      registry_file: /myfile/logs/.filebeat
      multiline:
       pattern: '<\/response>'
       negate: true
       match: before
input_type: xml
backoff: 8s
output: 
  logstash:     
     hosts: ["localhost:17603"]
console:
    pretty: true

Please correct me if I missed anything.

2

you running filebeat in debug mode? The method failing is for debug message generation (which is turned off by default). How do you run filebeat?

Any filebeat logs you have to share?

Can you confirm if formatting in forum at fault, or is input_type, backoff and console really at these indent levels in your config file?

You config is basically interpreted as:

filebeat:
  prospectors:
    -
      paths:
         - /errors/20[0-9][0-9]/logs-*.xml
      document_type: elmah_log
      ignore_older: 72h
      multiline:
       pattern: '<\/response>'
       negate: true
       match: before
output: 
  logstash:     
     hosts: ["localhost:17603"]
3

I am running filebeat in below mentioned way,

/bin/filebeat -c /config/filebeat.yml -e -v "*".

After running it throws the above message.

4

Is the stack-trace complete as is? Some more logs with full stack-trace would be helpfull. Trace is somewhere from within standard library (have never seen this before)

what does the final "*". stand for? I don't think it's used. You run this command from console or somewhere in background? The '-e' option enables output to stdout and stderr. As error happens somewhere in stdlib I wonder if file descriptor is still open. What happens if you run filebeat with /bin/filebeat -c /config/filebeat.yml -v ? Have you removed console output from your config file?

5

You running filebeat in terminal and missing logs in scrollback buffer? If so try /bin/filebeat -c /config/filebeat.yml -e -v |& tee x.log to get full log written to file x.log.

6

I am, running the command in console only. which console output your talking about. You mean in Logstash config file?

7

the full filebeat console output. The stack-trace looks kinda incomplete to me.