HTTP output for stadtpuls.com IoT platform

Elastic Stack Logstash
#1

Currently I have a working logstash configuration which stores JSON based input data from my IoT data from TTN via output into my Elasticsearch node. Now I would extract some data (e.g. temperature, pressure) and send only the field measurements via http output to the stadtpuls.com API. Currently I use the following filter statement among many others:

  ### BEGIN stadtpuls.com encoder
      if ( [dev_id] == "akiot-htcc-ab01-umweltsensor01") {
          mutate {
            add_field => {"measurements" => ["%{temperature}","%{pressure}"]}
          }
        }
      ### END stadtpuls.com encoder

But how can I prevent that the http output sends the entire JSON based event document to the API?

#2

You could use a prune filter with a whitelist_names options. If you have multiple outputs you might need pipeline-to-pipeline communications with a forked-path pattern.

#3

Thanks for the advice with the forked-pattern. Now it is working, I defined one input and two output pipelines. How could I suppress the app_id tag from output processing, if I use that tag currently for suppressing unneeded output?

Currently pipeline definition: pipelines.yml

- pipeline.id: input
  queue.type: persisted
  path.config: "/etc/logstash/conf.d/input.conf"

- pipeline.id: buffered-es
  queue.type: persisted
  path.config: "/etc/logstash/conf.d/es.conf"

- pipeline.id: buffered-stadtpuls
  queue.type: persisted
  path.config: "/etc/logstash/conf.d/stadtpuls.conf"

/etc/logstash/conf.d/stadtpuls.conf:

input { pipeline { address => "stadtpuls" } }

filter {
# process data for stadtpuls
      mutate {
        add_field => [ "app_id", "%{[end_device_ids][application_ids][application_id]}" ]
      } # mutate

     # measurements[temperature,pressure,humidity,p10,p25,voltage]
      mutate {
       add_field => {
          "measurements" => [
             ......
         ]
     }
       }

     mutate { remove_field =>   ["app_id","@version","@timestamp","host","received_at","_source","headers","correlation_ids","uplink_message","end_device_ids"] }

} # filter

output {
     if ([app_id] == "adlerkiez-pm-sensor-01") {
         http {
            .....
        }
    } # if app_id

Is it possible to filter not desired input at an earlier stage in the pipeline processing chain?

#4

You unconditionally remove the [app_id] field in the second mutate, so it will not exist when the event reaches the output section.

If you want to use the app_id in the output section but do not want it sent to the output then use [@metadata][app_id]. Then you can test it in the conditional but it will not be sent on with the event.

I hope that helps. It is not entirely clear what you are asking.

#5

This was the right solution, thanks. Now my pipeline is working and sends only what I want.

#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.