# How to add a for loop and if condition in logstash input plugin
**URL:** https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888
**Category:** Logstash
**Created:** [October 31, 2018, 4:38pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888 "2018-10-31T16:38:30Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Bharath\_Pusuluri](https://discuss.elastic.co/u/Bharath_Pusuluri)
#### Post date: [October 31, 2018, 4:38pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/1 "2018-10-31T16:38:30Z")
My requirement is sa follows:
I am running the following python script in input field. The Python script contains a curl command and outputting that data as input (JSON format).
input {
exec {
type =\> "apps"
command =\> "python /usr/share/logstash/pythontest.py"
interval =\> "60"
codec =\> "json"
}
}
Now my concern is the curl command running by active server and sometimes it converts as standby. So I would like to right a If condition in input plugin. If the result of the command executed contains message saying "This is a standby RM", it need to run another script immediately and this need to be put in a for loop or while loop to run for atleast 20 times. Will it be possible to read the data and write a logic in Input plugin?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 31, 2018, 4:50pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/2 "2018-10-31T16:50:03Z")
It is not possible to have conditional logic within a plugin. Why do you not modify your Python script to handle this instead?
---
### Author:  [@Bharath\_Pusuluri](https://discuss.elastic.co/u/Bharath_Pusuluri)
#### Post date: [October 31, 2018, 5:00pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/3 "2018-10-31T17:00:38Z")
#! /usr/bin/python
import subprocess
proc = subprocess.Popen(["curl", "hostname1:8088/ws/v1/cluster/apps?state=RUNNING"], stdout=subprocess.PIPE)
(out, err) = proc.communicate()
print out
The above is my current python code which is executing as part of the logstash input plugin as defined above. Now, sometimes the above curl command returns a message saying hostname1 is a standby server and during that time I should move on to other server hostname2
Exact Message - This is standby RM. The redirect url is: /ws/v1/cluster/apps?state=RUNNING
I need to filter this message passing to elasticsearch output, but the later part of execution to be processed as it is.
import subprocess
proc = subprocess.Popen(["curl", "hostname2:8088/ws/v1/cluster/apps?state=RUNNING"], stdout=subprocess.PIPE)
(out, err) = proc.communicate()
print out
In order to satisfy this condition, I have tried the following. Now the problem is the input plugin reading the first output and also the second output. So thought I can put some filter in input plugin, I think I can try for adding filter. What is your idea in this scenario of curl commands with active and standby.
#! /usr/bin/python
import subprocess
proc = subprocess.Popen(["curl", "hostname1:8088/ws/v1/cluster/apps?state=RUNNING"], stdout=subprocess.PIPE)
(out, err) = proc.communicate()
print out
if "standby RM" in out:
proc = subprocess.Popen(["curl", "hostname2:8088/ws/v1/cluster/apps?state=RUNNING"], stdout=subprocess.PIPE)
(out, err) = proc.communicate()
print out
print "test succesful"
---
### Author:  [@Bharath\_Pusuluri](https://discuss.elastic.co/u/Bharath_Pusuluri)
#### Post date: [October 31, 2018, 5:04pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/4 "2018-10-31T17:04:16Z")
my input and filter looks like this..Another problem is I am not able to filter the remove\_field =\> ["amContainerLogs" ,"trackingUrl"] loading to elasticsearch.
input {
exec {
type =\> "apps"
command =\> "python /usr/share/logstash/pythontest.py"
interval =\> "60"
codec =\> "json"
}
}
filter {
if [type] == "apps"
{
json {
source =\> "message"
remove\_field =\> ["amContainerLogs" ,"trackingUrl"]
}
split { field =\> "[apps]" }
split { field =\> "[apps][app]"
remove\_field =\> ["command" ,"tags" ,"apps.app.amContainerLogs" ,"apps.app.trackingUrl"]
}
split { field =\> "[apps][app][resourceRequests]" }
}
}
---
### Author:  [@Bharath\_Pusuluri](https://discuss.elastic.co/u/Bharath_Pusuluri)
#### Post date: [October 31, 2018, 5:23pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/5 "2018-10-31T17:23:58Z")
Can you help Christian, how to filter the text coming before the JSON output and also how to filter any JSON data columns.
Actually I tried the using JSON filter remove\_field but still not working as expected
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 31, 2018, 5:31pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/6 "2018-10-31T17:31:53Z")
It would be easier if you could show what the data looks like and what the goal is.
---
### Author:  [@Bharath\_Pusuluri](https://discuss.elastic.co/u/Bharath_Pusuluri)
#### Post date: [October 31, 2018, 5:41pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/7 "2018-10-31T17:41:09Z")
JSON format data looks like below
\
\
\application\_1540397431401\_63966\
\prdodsapp\
\
distcp: oozie:action:T=java:W=falcon-dr-fs-workflow:A=dr-replication:ID=0019347-180619181447685-oozie-oozi-W
\
\default\
\RUNNING\
\UNDEFINED\
\81.26881\
\ApplicationMaster\
\
[http://hostname:8088/proxy/application\_1540397431401\_63966/](http://hostname:8088/proxy/application_1540397431401_63966/)
\
\
\1540397431401\
\MAPREDUCE\
\oozie-82feffab11fb930af53ee1d8ca7db26c\
\1541005780354\
\0\
\1751383\
\
[http://hostname:8042/node/containerlogs/container\_e81\_1540397431401\_63966\_01\_000001/prdodsapp](http://hostname:8042/node/containerlogs/container_e81_1540397431401_63966_01_000001/prdodsapp)
\
\hostname:8042\
\12960\
\4\
\4\
\44958791\
\13870\
\0.4974943\
\0.26367188\
\0\
\0\
\0\
\0\
\
\
\3240\
\1\
\
\
\0\
\
\0\
\
\true\
\\*\
\
\
\
\3240\
\1\
\
\
\0\
\
\20\
\
\true\
\\*\
\
\NOT\_START\
\
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 31, 2018, 5:42pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/8 "2018-10-31T17:42:31Z")
That looks a lot like XML and not JSON.
---
### Author:  [@Bharath\_Pusuluri](https://discuss.elastic.co/u/Bharath_Pusuluri)
#### Post date: [October 31, 2018, 5:49pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/9 "2018-10-31T17:49:50Z")
when we run the curl in linux system, it looks like this
{"apps":{"app":[{"id":"application\_1540397431401\_63966","user":"prdodsapp","name":"distcp: oozie:action:T=java:W=falcon-dr-fs-workflow:A=dr-replication:ID=0019347-180619181447685-oozie-oozi-W","queue":"default","state":"RUNNING","finalStatus":"UNDEFINED","progress":84.73618,"trackingUI":"ApplicationMaster","trackingUrl":"[http://hostname.com:8088/proxy/application\_1540397431401\_63966/","diagnostics":"","clusterId":1540397431401,"applicationType":"MAPREDUCE","applicationTags":"oozie-82feffab11fb930af53ee1d8ca7db26c","startedTime":1541005780354,"finishedTime":0,"elapsedTime":2249376,"amContainerLogs":"http://hostname:8042/node/containerlogs/container\_e81\_1540397431401\_63966\_01\_000001/prdodsapp","amHostHttpAddress":"hostname:8042","allocatedMB":12960,"allocatedVCores":4,"runningContainers":4,"memorySeconds":51412664,"vcoreSeconds":15862,"queueUsagePercentage":0.4974943,"clusterUsagePercentage":0.26367188,"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"resourceRequests":[{"capability":{"memory":3240,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":0},"relaxLocality":true,"resourceName":"](http://hostname.com:8088/proxy/application_1540397431401_63966/%22,%22diagnostics%22:%22%22,%22clusterId%22:1540397431401,%22applicationType%22:%22MAPREDUCE%22,%22applicationTags%22:%22oozie-82feffab11fb930af53ee1d8ca7db26c%22,%22startedTime%22:1541005780354,%22finishedTime%22:0,%22elapsedTime%22:2249376,%22amContainerLogs%22:%22http://hostname:8042/node/containerlogs/container_e81_1540397431401_63966_01_000001/prdodsapp%22,%22amHostHttpAddress%22:%22hostname:8042%22,%22allocatedMB%22:12960,%22allocatedVCores%22:4,%22runningContainers%22:4,%22memorySeconds%22:51412664,%22vcoreSeconds%22:15862,%22queueUsagePercentage%22:0.4974943,%22clusterUsagePercentage%22:0.26367188,%22preemptedResourceMB%22:0,%22preemptedResourceVCores%22:0,%22numNonAMContainerPreempted%22:0,%22numAMContainerPreempted%22:0,%22resourceRequests%22:%5B%7B%22capability%22:%7B%22memory%22:3240,%22virtualCores%22:1%7D,%22nodeLabelExpression%22:%22%22,%22numContainers%22:0,%22priority%22:%7B%22priority%22:0%7D,%22relaxLocality%22:true,%22resourceName%22:%22)_"},{"capability":{"memory":3240,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":20},"relaxLocality":true,"resourceName":"_"}],"logAggregationStatus":"NOT\_START"}
---
### Author:  [@Bharath\_Pusuluri](https://discuss.elastic.co/u/Bharath_Pusuluri)
#### Post date: [October 31, 2018, 6:02pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/10 "2018-10-31T18:02:25Z")
Updated another set
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 28, 2018, 6:02pm UTC](https://discuss.elastic.co/t/how-to-add-a-for-loop-and-if-condition-in-logstash-input-plugin/154888/11 "2018-11-28T18:02:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.