Logs from different sources in single txt file, to be stored in different indices using logstash

[Divyank_Mahalle]

Hi,

I am having txt file consist of 3 different logs coming from 3 sources collated into single text file.

I want to separate out logs and stored them in a respective index.

Sample Logs-

10/12/2021 8:54:00 AM,MSGID: <..........> Mobile_Number:<.............>Job_no:<...............>User_id:<......> mstrGateWay <..........>
10/12/2021 8:54:03 AM,strDisplay: Message Id  : <..........>Done Date   : <..........>STAT        : <..............>mstrGateWay: <................>

I got this link but still I need to confirm if this use case is possible or not.

https://www.elastic.co/blog/using-logstash-to-split-data-and-send-it-to-multiple-outputs

[stephenb (Stephen Brown)]

Hi @Divyank_Mahalle

Yes it is possible, you will just need to determine the field or fields that will determine which index you want to route your logs to.

Sorting to different indices is a very common use case / pattern.

[Mustafa_NAJIB (Mustafa)]

HELLO,
i want to complet the reply of Mr @stephenb
by this examlpe that you can use as your output

if [field1] == "..." {   # or field1 exist
      elasticsearch {
          hosts => [ "...", "...", "...", "...", "..." ]
          index => "index1"
          ssl => true
          ...
		  ...
      }
     if [field2] == "..." {   # or field2 exist
        elasticsearch {
          hosts => [ "...", "...", "...", "...", "..." ]
          index => "index2"
          ssl => true
          ...
		  ...
        }
      }
    }

[Divyank_Mahalle]

Thanks @stephenb and @Mustafa_NAJIB for clearing the query