Can i send multiple line log directly to elastic search using file beat?

[Mayurbiw (Mayur Khatri)]

consider this scenario -

This is log file (Directly taken from example conf files in file beat reference)

Exception in thread "main" java.lang.NullPointerException
at com.example.myproject.Book.getTitle(Book.java:16)
at com.example.myproject.Author.getBookTitles(Author.java:25)
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

This is the filebeat.yml multiline settings

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after

the output is an elastic search.

On analysing elastic search using kibana I found out that each line is stored in a separate document.

Can I store multiple lines in a message field of a document?

[kvch (Noémi Ványi)]

Multiline messages are stored in the same document. The example configuration you have pulled from our website is correct and multiple lines are aggregated into a single document.
Could you please share your whole configuration formatted using </>? There might be a whitespace problem somewhere in your config which causes the problem.

[Mayurbiw (Mayur Khatri)]

filebeat.yml

filebeat.inputs:


- type: log


    - /home/oracle/example3.aud

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after



output.elasticsearch:
  
  hosts: ["IPadress:9200"]



  index: "file_beat_example3"


setup.template.name: "file_beat_example3"
setup.template.pattern: "file_beat*"

content in example3 file

Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

Kibana output

[kvch (Noémi Ványi)]

Your input configuration seems incorrect. The input needs to be enabled. Also, multiline settings need to be at the level of the input.

The correct format:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/oracle/example3.aud
  multiline.pattern: '^[[:space:]]'
  multiline.negate: false
  multiline.match: after

[Mayurbiw (Mayur Khatri)]

Thankyou for the help now I am getting mapping error. here is the full filebeat.yml file.
and previously also enabled was set to true.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/oracle/example4.aud


             multiline.pattern: '^[[:space:]]'
             multiline.negate: false
             multiline.match: after


output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["IP:9200"]



  index: "file_beat_example4"


setup.template.name: "file_beat_example4"
setup.template.pattern: "file_beat*"

Now I am getting the error
mapping values are not allowed in this context when i am running filebeats

error at line number 22 mapping values are not allowed in this context

[kvch (Noémi Ványi)]

Could you copy the exact error message here? Also, is there any error in the logs of Elasticsearch?

[Mayurbiw (Mayur Khatri)]

Sure
Exiting: error loading config file: yaml: line 22: mapping values are not allowed in this context

[kvch (Noémi Ványi)]

The indentation is still off. The section multiline needs to be in the same column as the input. See my configuration snippet. I think your editor is interfering with the indentation of the config. You need to use spaces and do not let your editor mix tabs and spaces.

[Mayurbiw (Mayur Khatri)]

thank, this solved the problem. I really appreciate your patience. Thank you again.