Filebeat to send application logs to elasticsearch

[jacobzh (Jacob)]

Hi,
I want to use filebeat to ship developers application logs to elasticsearch running in a developers servers. Developers application generates a log file in a particular folder. Whenever the log file reaches a configured size, then a new log file is started like app.log.$date1, app.log.$date2, app.log.$date3 etc. Can I make filebeat read new log files whenever the new log file is started.
Most important I have to see content of the application log files. Right now I can see only message.
Also, is there a possibility I can only send a subset of traces from these application log files to elasticsearch or does it require logstash to do this?

[Mario_Castro]

Use the log input https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html

If you need send some metadata, you can use tags

[jacobzh (Jacob)]

Hi Mario,

Thank you for quick replay. Below my filebeat configuration:

=========================== Filebeat inputs =============================

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

- type: log

Change to true to enable this input configuration.

enabled: true

Paths that should be crawled and fetched. Glob based paths.

paths:

paths:

/var/log/localhost.log

/var/log/server.log_y*

The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

multiline.pattern: '^['

Defines if the pattern set under pattern should be negated or not. Default is false.

multiline.negate: true

Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

that was (not) matched before or after or as long as a pattern is not matched based on negate.

Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

multiline.match: after

============================= Filebeat modules ===============================

filebeat.config.modules:

Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

Set to true to enable config reloading

reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:

index.number_of_shards: 3

#index.codec: best_compression

#_source.enabled: false

#============================== Kibana =====================================

Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

This requires a Kibana endpoint configuration.

setup.kibana:

Kibana Host

Scheme and port can be left out and will be set to the default (http and 5601)

In case you specify and additional path, the scheme is required: http://localhost:5601/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

host: "x.x.53.201:5601"

#-------------------------- Elasticsearch output ------------------------------

output.elasticsearch:

Array of hosts to connect to.

hosts: ["x.x.53.201:9200"]

What I can see from Kibana:

[Mario_Castro]

@jacobzh

/var/log/localhost.log

Use a wildcard /var/log/*.log

[jacobzh (Jacob)]

After I used wild card /var/log/*.log, I can’t see a message and files content.

[Mario_Castro]

Ok, that doesn't make sense It's a typical wildcard, I thought it was implicit, you can try /var/log/* to ingest all files in that folder (just to double check, I mean in the path object )

[fadjar340 (Fadjar Tandabawana)]

Better change the log files name such as: $date1.app.log $date2.app.log
So, the configuration of filebeat is simpler. just /var/log/*.log

Another suggestion, if you want to put the message contents of the apps log to the meaningfull fields, use logstash to handle the specific log format, also, if there are not syslog format, better put into another folder, such as /var/log/specificapps/*.log

Then use grok from logstash filter to dissect the message content into fields in the index.

Regards,
Fadjar Tandabawana

[jacobzh (Jacob)]

I changed path to /var/log/*. Same result. I can see only /var/log/syslog and /var/log/auth.log file. No messages from my files.

[jacobzh (Jacob)]

Hi Fadjar,

I changed path to /var/log/awsapi/*.log and used logstash to handle specific log format. Noting happened and it getting worth:

I can’t see even message with my files and loosed the server(awsapi02) for dashboard:

[fadjar340 (Fadjar Tandabawana)]

Did you already add the path into the filebeat conf?

[jacobzh (Jacob)]

Yes, I did

Thank you,

Jacob Zhitnitsky
St. System Administrator
Folio Financial, Inc
zhitnitskyj@foliofinancial.com
work: 703-245-4813
cell: 571-405-1043

Home | Folio Financial

Thank you,

Jacob Zhitnitsky
Sr. System Administrator
Folio Financial, Inc
zhitnitskyj@foliofinancial.com
work: 703-245-4813
cell: 571-405-1043

Home | Folio Financial

Please do not use email to transmit orders for securities or for other time-sensitive messages. Broker-dealer securities products and services are offered only through Folio Investments, Inc. Member FINRA/SIPC. Folio Investments, Inc. conducts business under various trade names, including: Folio Financial, Folio Institutional, Folio Investing, FolioFirst, VIAFolio, Folio Client, and DirectIndexer. Folio Investments, Inc. is a subsidiary of Folio Financial, Inc. and an affiliate of First Affirmative Financial Network, LLC and Folio Research, LLC. Investment advisory services are provided through First Affirmative Financial Network, LLC and Folio Research, LLC.