Parsing logback log files with filebeat and sending them to Elasticsearch

Jemli_Fathi · May 5, 2016, 2:11pm

Hi, i have a java application with logback for the log configuration and i want to parse my application log files, so they become more useful and to send them to ES, log files will be stored in a directory called logs and this is the log entry format used with logback:

%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n

I need to know the simplest way of doing this task (including the configuration file to use), i thought, i can do that using filebeat but filebeat cannot parse log entries.

Thank you.

andrewkroh · May 6, 2016, 12:08am

You can use Filebeat to ship the data to Logstash where you can apply a grok filter to parse the log line. Then ship the data to Elasticsearch from Logstash. The Beats documentation has an example of how to configure Filebeat to send to Logstash.

In Elasticsearch version 5 (there's an alpha release now) there is a new feature called Ingest Node that enables you to grok the message without the use of Logstash. See https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html

Jemli_Fathi · May 6, 2016, 12:40am

I tried this configuration file with filebeat:

filebeat:
	prospectors:
		-
		 paths:
         	- /home/fathi/IdeaProjects/fouras/logs*.log
         input_type: log
         fields:
        	context: fouras
        	version: 0
         document_type: fouras_logs
         scan_frequency: 120s
         multiline:
         	pattern: ^\[
    registry_file: /var/lib/filebeat/registry
    output:
    	elasticsearch:
    		hosts: ["localhost:9200"]
    		index: "fouras_logs"
  shipper:
  logging:
    files:
      path: /var/log/mybeat
      rotateeverybytes: 10485760 # = 10MB

But, it's not working

andrewkroh · May 6, 2016, 1:28am

Your indentations are wrong and because it's YAML that matters. Also tabs are not allowed in YAML files.

Try this;

filebeat:
  registry_file: /var/lib/filebeat/registry
  prospectors:
    - paths:
        - /home/fathi/IdeaProjects/fouras/logs*.log
      input_type: log
      fields:
        context: fouras
        version: 0
      document_type: fouras_logs
      scan_frequency: 120s
      multiline:
        pattern: ^\[

output:
  elasticsearch:
    hosts: ["localhost:9200"]
    index: "fouras_logs"

shipper:

logging:
  to_files: true
  files:
    path: /var/log/filebeat
    rotateeverybytes: 10485760 # = 10MB

Jemli_Fathi · May 6, 2016, 10:40am

Sorry, but the indentations are wrong because i copied it from /etc/filebeat/filebeat.yml, but the original file is correctly indented

http://www.mediafire.com/download/hporwhls4av6ppw/filebeat.yml

Actually, i modified that existing configuration file /etc/filebeat/filebeat.yml and restarted filebeat service. That is how to configure Filebeat, right?

andrewkroh · May 6, 2016, 12:52pm

Can you elaborate on the problem? Are there errors?

The config file you uploaded looks OK.

Jemli_Fathi · May 6, 2016, 1:19pm

I modified the configuration file and i started filebeat service using sudo service filebeat start, and i got * filebeat is running

But, it sounds like filebeat is not considering the configuration file, actually even log files under /var/log/mybeat was not created.

andrewkroh · May 6, 2016, 1:22pm

Increase the logging level. By default it's set to error and the file only gets created when something is logged.

logging:
  level: debug     # or info if you want less verbose output
  to_files: true
  files:
    path: /var/log/filebeat
    rotateeverybytes: 10485760 # = 10MB

Jemli_Fathi · May 6, 2016, 1:25pm

I got this:

* Restarting Sends log files to Logstash or directly to Elasticsearch. filebeat                                                                                2016/05/06 13:24:14.630402 beat.go:135: DBG  Initializing output plugins
2016/05/06 13:24:14.630485 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/05/06 13:24:14.630603 client.go:265: DBG  ES Ping(url=http://localhost:9200, timeout=1m30s)
2016/05/06 13:24:14.646024 client.go:274: DBG  Ping status code: 200
2016/05/06 13:24:14.646119 outputs.go:119: INFO Activated elasticsearch as output plugin.
2016/05/06 13:24:14.646172 publish.go:232: DBG  Create output worker
2016/05/06 13:24:14.646296 publish.go:274: DBG  No output is defined to store the topology. The server fields might not be filled.
2016/05/06 13:24:14.646416 publish.go:288: INFO Publisher name: fathi-HP-Pavilion-g6-Notebook-PC
2016/05/06 13:24:14.647094 async.go:78: INFO Flush Interval set to: 1s
2016/05/06 13:24:14.647162 async.go:84: INFO Max Bulk Size set to: 50
2016/05/06 13:24:14.647213 async.go:92: DBG  create bulk processing worker (interval=1s, bulk size=50)
2016/05/06 13:24:14.647400 beat.go:147: INFO Init Beat: filebeat; Version: 1.1.2
                                                                         [ OK ]

And nothing on Elasticsearch side

andrewkroh · May 6, 2016, 1:31pm

Is there anything in your registry file at /var/lib/filebeat/registry? You should probably delete it before each test to ensure that filebeat re-ships logs that it has already read.

I expect a lot more output in the log file if you are running with level: debug. Is the above output from the log?

Jemli_Fathi · May 6, 2016, 1:40pm

/var/lib/filebeat/registry is containing {}

log file content: filebeat

The only error is this:

ERR Stop Harvesting. Unexpected encoding line reader error: unknown matcher type:

Jemli_Fathi · May 6, 2016, 1:49pm

Few lines of my log files to be processed using Filebeat:

[2016-05-06 14:47:33.808 [main] DEBUG o.s.w.c.s.StandardServletEnvironment - Adding [servletConfigInitParams] PropertySource with lowest search precedence]
[2016-05-06 14:47:33.837 [main] DEBUG o.s.w.c.s.StandardServletEnvironment - Adding [servletContextInitParams] PropertySource with lowest search precedence]
[2016-05-06 14:47:33.837 [main] DEBUG o.s.w.c.s.StandardServletEnvironment - Adding [systemProperties] PropertySource with lowest search precedence]
[2016-05-06 14:47:33.837 [main] DEBUG o.s.w.c.s.StandardServletEnvironment - Adding [systemEnvironment] PropertySource with lowest search precedence]
[2016-05-06 14:47:33.837 [main] DEBUG o.s.w.c.s.StandardServletEnvironment - Initialized StandardServletEnvironment with PropertySources [servletConfigInitParams,servletContextInitParams,systemProperties,systemEnvironment]]
[2016-05-06 14:47:34.126 [main] INFO  com.datcom.fouras.FourasApplication - Starting FourasApplication on fathi-HP-Pavilion-g6-Notebook-PC with PID 13441 (/home/fathi/IdeaProjects/fouras/target/classes started by fathi in /home/fathi/IdeaProjects/fouras)]
[2016-05-06 14:47:34.127 [main] INFO  com.datcom.fouras.FourasApplication - No active profile set, falling back to default profiles: default]
[2016-05-06 14:47:35.620 [main] INFO  o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@1e730495: startup date [Fri May 06 14:47:35 CET 2016]; root of context hierarchy]
[2016-05-06 14:47:39.468 [main] INFO  o.s.b.f.s.DefaultListableBeanFactory - Overriding bean definition for bean 'beanNameViewResolver' with a different definition: replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration; factoryMethodName=beanNameViewResolver; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/ErrorMvcAutoConfiguration$WhitelabelErrorViewConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter; factoryMethodName=beanNameViewResolver; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/WebMvcAutoConfiguration$WebMvcAutoConfigurationAdapter.class]]]

andrewkroh · May 6, 2016, 2:09pm

You are missing the match key from your multiline config. For example:

multiline:
    pattern: ^\[
    negate: true
    match: after

Reference: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#multiline

In the next major version we are working on improving configuration validation so that we can catch these errors with -configtest.

Jemli_Fathi · May 6, 2016, 2:10pm

It's working, i forget this:

multiline:
pattern: ^[
negate: true
match: after

instead of:

multiline:
pattern: ^[

Thank you very much, i appreciate it

Tri_Nguyen1 · December 16, 2016, 6:00pm

Hi Andrew and Jemli,

I am a newbie to Filebeat and Elastic Stack.
I am also trying to do something similar to you, Jemli.

Where did you specify the grok format expression for parsing the Java log entries?
Likewise, don't we need to specify the transformation to JSON?!
I assume that we need something like:

output:
elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
template.name: "filebeat"
template.path: "filebeat.template.json"

Topic		Replies	Views
Parsing-log-files-with-filebeat-and-sending-them-to-elasticsearch Beats filebeat	2	10058	February 9, 2018
Moving from ELK to EFK Beats docker , filebeat	3	1203	November 26, 2019
Filebeat to Elasticsearch grok example Beats filebeat	2	3555	May 26, 2020
Filebeats to logstash _grokparsefailure, direct to ES is fine Beats	2	1353	July 5, 2017
Filebeat logs are not being send to logstash Beats filebeat	5	525	April 23, 2024

Parsing logback log files with filebeat and sending them to Elasticsearch

Related topics