Send multiple output 1 at a time to another pipeline in Logstash

Elastic Stack Logstash
1

Hi - I am very new to logstash. Is there a way on how I can send the output into another pipeline one at a time, with the unique field "somefield"? "somefield" values are from UUID function.

CODE

    #input ODX files from local path file
    input {
     file { path => "/Users/sample.odx" 
    	start_position => "beginning"
    	sincedb_path => "NUL" 
    	codec => multiline {
    		pattern => "^#----- Start Task -----"
    		negate => true
    		what => "previous"
    		max_lines => 4000
    		}
    	}
    }

    filter{
    	mutate { add_field => {"somefield" => " "} }
    	uuid {
    		target => "somefield"
    		overwrite => true
    	}
    }
    #output JSON File in local
    output{
    	pipeline {
    		send_to => [data_transform]
    	}
    }

current output:

    {
      "@version": "1",
          "@timestamp": "2020-03-14T02:36:25.441Z",
          "message": "#Money Exchange Rate\r\n250\r\n#Date of Export\r\nWed Jan 20 09:40:12 2018\r\n#Current Exchange\r\nSGDUSD\r\n#MoneyCounter\r\n207     \r",
          "tags": [
            "multiline"
          ],
          "host": "BD001A2A",
      "somefield": "c657e-65e2-4312-a015-e61"
    }{
     "@version": "1",
          "@timestamp": "2020-03-14T02:36:25.497Z",
          "message": "#----- Start Task -----\r\n\r\n#Location\r\nOrchard Road\\Singapore\\01ACVB\\125\r\n#Counter ID\r\n7008\r",
          "tags": [
            "multiline"
          ],
          "host": "BD001A2A",
      "somefield": "c9c6a2-fb81-435f-8a31-3728"
    }{
     "@version": "1",
          "@timestamp": "2020-03-14T02:37:02.809Z",
          "message": "#----- Start Task -----\r\n\r\n#Location\r\nDotonburi\\Japan\\98CVBS\\112\r\n#Counter ID\r\n9001\r",
          "tags": [
            "multiline"
          ],
          "host": "BD001A2A",
      "somefield": "28a5c-0cd7-4b89-bb7f-fff"
    }

Thanks in advance!

2

I do not understand what the problem is nor what you are asking. Could you please explain a bit more?

3

Sure. I would like to send the message output 1 at a time in another pipeline, [data_transform]. Notice that there are 3 messages in the output.

Currently the output is being sent everything to another pipeline:

    {
      "@version": "1",
          "@timestamp": "2020-03-14T02:36:25.441Z",
          "message": "#Money Exchange Rate\r\n250\r\n#Date of Export\r\nWed Jan 20 09:40:12 2018\r\n#Current Exchange\r\nSGDUSD\r\n#MoneyCounter\r\n207     \r",
          "tags": [
            "multiline"
          ],
          "host": "BD001A2A",
      "somefield": "c657e-65e2-4312-a015-e61"
    }{
     "@version": "1",
          "@timestamp": "2020-03-14T02:36:25.497Z",
          "message": "#----- Start Task -----\r\n\r\n#Location\r\nOrchard Road\\Singapore\\01ACVB\\125\r\n#Counter ID\r\n7008\r",
          "tags": [
            "multiline"
          ],
          "host": "BD001A2A",
      "somefield": "c9c6a2-fb81-435f-8a31-3728"
    }{
     "@version": "1",
          "@timestamp": "2020-03-14T02:37:02.809Z",
          "message": "#----- Start Task -----\r\n\r\n#Location\r\nDotonburi\\Japan\\98CVBS\\112\r\n#Counter ID\r\n9001\r",
          "tags": [
            "multiline"
          ],
          "host": "BD001A2A",
      "somefield": "28a5c-0cd7-4b89-bb7f-fff"
    }

each message should be sent to pipeline 1 at a time.

output 1: send to pipeline [data_transform], then perform necessary filters.

    {
      "@version": "1",
          "@timestamp": "2020-03-14T02:36:25.441Z",
          "message": "#Money Exchange Rate\r\n250\r\n#Date of Export\r\nWed Jan 20 09:40:12 2018\r\n#Current Exchange\r\nSGDUSD\r\n#MoneyCounter\r\n207     \r",
          "tags": [
            "multiline"
          ],
          "host": "BD001A2A",
      "somefield": "c657e-65e2-4312-a015-e61"
    }

output 2:send to pipeline [data_transform], then perform necessary filters.

    {
     "@version": "1",
          "@timestamp": "2020-03-14T02:36:25.497Z",
          "message": "#----- Start Task -----\r\n\r\n#Location\r\nOrchard Road\\Singapore\\01ACVB\\125\r\n#Counter ID\r\n7008\r",
          "tags": [
            "multiline"
          ],
          "host": "BD001A2A",
      "somefield": "c9c6a2-fb81-435f-8a31-3728"
    }

thanks

4

Logstash handles prr RT icesdingbin small batches. Why do they need to be sent one at a time?

5

as per requirement, each message should be processed, transformed and outputted to a json object. basically, i have a single input, but should have multiple output.

6

The output seems to have 3 separate events. How do you output these? What is wrong with this? I am still not sure I understand the issue.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.