Can i send multiple line log directly to elastic search using file beat?

consider this scenario -

This is log file (Directly taken from example conf files in file beat reference)

Exception in thread "main" java.lang.NullPointerException
at com.example.myproject.Book.getTitle(Book.java:16)
at com.example.myproject.Author.getBookTitles(Author.java:25)
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

This is the filebeat.yml multiline settings

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after

the output is an elastic search.

On analysing elastic search using kibana I found out that each line is stored in a separate document.

Can I store multiple lines in a message field of a document?

Multiline messages are stored in the same document. The example configuration you have pulled from our website is correct and multiple lines are aggregated into a single document.
Could you please share your whole configuration formatted using </>? There might be a whitespace problem somewhere in your config which causes the problem.

filebeat.yml

filebeat.inputs:


- type: log


    - /home/oracle/example3.aud

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after



output.elasticsearch:
  
  hosts: ["IPadress:9200"]



  index: "file_beat_example3"


setup.template.name: "file_beat_example3"
setup.template.pattern: "file_beat*"

content in example3 file

Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

Kibana output

Your input configuration seems incorrect. The input needs to be enabled. Also, multiline settings need to be at the level of the input.

The correct format:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/oracle/example3.aud
  multiline.pattern: '^[[:space:]]'
  multiline.negate: false
  multiline.match: after
1 Like

Thankyou for the help now I am getting mapping error. here is the full filebeat.yml file.
and previously also enabled was set to true.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/oracle/example4.aud


             multiline.pattern: '^[[:space:]]'
             multiline.negate: false
             multiline.match: after


output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["IP:9200"]



  index: "file_beat_example4"


setup.template.name: "file_beat_example4"
setup.template.pattern: "file_beat*"

Now I am getting the error
mapping values are not allowed in this context when i am running filebeats

image

error at line number 22 mapping values are not allowed in this context

Could you copy the exact error message here? Also, is there any error in the logs of Elasticsearch?

Sure
Exiting: error loading config file: yaml: line 22: mapping values are not allowed in this context

The indentation is still off. The section multiline needs to be in the same column as the input. See my configuration snippet. I think your editor is interfering with the indentation of the config. You need to use spaces and do not let your editor mix tabs and spaces.

1 Like

thank, this solved the problem. I really appreciate your patience. Thank you again.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.