Create index based in filename in Elasticsearch

(Vijayakrishna Gade) #1

Hi,

We have new log files get created every time when we run load test on jenkins machine.

Ex:
Load test1 creates file called: /jenkins/var/lib/simulation-1.log
Load test2 Creates file: jenkins/var/lib/simulation-2.log.
Load testx Creates file: jenkins/var/lib/simulation-x.log.

Like the above we will generate more then 10 log files each day, i can create index called "simulation-*.log", it includes all the logs in single index.

we want to have separate index get created on Elastic machine every time when there is new simulation file with build number, so that we will filter data in kibana based on each simulation log file.

Please suggest me how can we send those logs from Jenkins machine to Elastic machine separately, Thanks.

(Mark Walkom) #2

Why not just create a filter for each filename?

(Vijayakrishna Gade) #3

Filter in Kibana Dashboard?

(Mark Walkom) #4

Yep.

(Vijayakrishna Gade) #5

it means elastic will create a separate index based on file name automatically?

(Mark Walkom) #6

No, it means you don't need to worry about that complexity, and you can simply filter to the file you want.

(Vijayakrishna Gade) #7

how can filter data with build number, only file name contains build number not logs in the file, how can i filter in kibana?

i want to separate logs based on build number.

(Mark Walkom) #8

How are you sending data to Elasticsearch?

(Vijayakrishna Gade) #9

I am using fluentd for Kubernetes logs, Filebeat for Nginx and kafka logs.

in the same way either i use filebeat or logstash to forward logs from jenkins machine to Elasticsearch, i haven't tried before that make sure clarify recommended way.

(system) closed #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.