# Usign ruby to update local Linux environment variable **URL:** \ **Category:** Logstash\ **Created:** [January 3, 2018, 3:36pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948 "2018-01-03T15:36:39Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\ **Post date:** [January 3, 2018, 3:36pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/1 "2018-01-03T15:36:39Z")
I am ingesting some data with Logstash, with Kafka as an input. Is it possible with ruby, to configure a filter, which will update a Linux environment variable with what I am getting from Kafka? Lets say I will get something like below with Kafka: version\_number = 1 I want ruby to put this as an environment variable on my system, so I can then use environment filter plugin in Logstash, to fetch this value when I need it? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 3, 2018, 7:11pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/2 "2018-01-03T19:11:11Z")
This isn't straight-forward to do, and an environment variable is most likely not what you want to use. If you explain in greater detail what you're trying to do it'll be easier to make a suggestion. ---
**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\ **Post date:** [January 4, 2018, 8:43am UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/3 "2018-01-04T08:43:46Z")
My use case is that I have a single pipeline, where: 1. first input gets data from Kafka, a field called version\_number. Output is like: version\_number=1 This version number will be sent only once. If it changes, it will be sent once with new version number such as: version\_number=2 2)second input is listening for ActiveMQ data and this data is sent continuously. What I want to achieve is for the ActiveMQ bus to contain the version\_number from Kafka, so I can write it to ElasticSearch. From my understanding: - Logstash does not have plugin for Kafka poller, which would return continuously the version\_number - I cannot use Logstash aggregate filter as this would only get the version\_number once in the output, not continuously I want to keep a configuration where I have one pipeline, if possible. I was thinking I could do: -Get version\_number from Kafka and save it as a environment variable (with ruby maybe)? -Pass this version\_number to my ActiveMQ output with the environment filter plugin. I tested the environment filter plugin and it seems to work well, the only trick is to pass this Kafka output to Logstash correctly, so I can use it. I only tested with an environment variable I set myself, manually. Any suggestions? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 4, 2018, 9:02am UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/4 "2018-01-04T09:02:22Z")
You can have a kafka input that listens for version\_number messages and writes those to a file that the translate filter can read. The ActiveMQ part of your pipeline could then have a translate filter that reads the file. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 4, 2018, 1:13pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/5 "2018-01-04T13:13:43Z")
I answered you before with the same suggestion as Magnus - [here](https://discuss.elastic.co/t/environment-variable-in-logstash/112978/2?u=guyboertje) I specified two pipelines to keep the two processing paths clean and separate. LS 6.x can use multiple pipelines in the same LS instance. Get two pipelines working first, then two can converge them into one by using if conditional blocks to separate the specific flows. ---
**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\ **Post date:** [January 4, 2018, 2:04pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/6 "2018-01-04T14:04:47Z")
Thank you both for your answer. My concern is that, the the version\_number I actually have on the Kafka topic, already comes from an xml file. So I would retrieve my needed value from xml file, put it on Kafka topic and then again place it another file. Is this the neatest way to achieve what I need? So my full workflow would look like this: 1. I have an input configuration where I have an xml file, from which I get the version\_number and put it on Kafka topic. The way my version\_number will update is by a user placing a file within a specific directory and Logstash will check the directory and extract the version\_number from the file, which will then be placed on bus. 2. I get the version\_number and put it in another file, lets say version\_number.csv 3. I enhance my ActiveMQ data with version\_number, which comes from version\_number.csv , which comes from my primary xml file. Would it be possible to: 1. Get version\_number from xml and place it on Kafka topic. 2. Get the version\_number from Kafka topic and put it into redis as a key value. 3. Read the key value from redis and enhance my ActiveMQ output with it? Or is this not going to work either? ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 4, 2018, 2:08pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/7 "2018-01-04T14:08:28Z")
OK, so you did not tell us about the LS that reads the XML file and puts it in Kafka. This means you can eliminate Kafka entirely. Have the 1st LS instance read the XML file and write the CSV file directly. Have the 2nd LS (ActiveMq) use the translate filter to periodically read the CSV file. ---
**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\ **Post date:** [January 4, 2018, 3:29pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/8 "2018-01-04T15:29:53Z")
Thanks so much for your suggestion, I think this will simplify my workflow. I will follow your advice! ---
**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\ **Post date:** [January 4, 2018, 4:30pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/9 "2018-01-04T16:30:27Z")
So I have the following working- 1. I can read my xml file and output the version\_number as follows: output { csv { path =\> "/tmp/version\_number.csv" create\_if\_deleted =\> true fields =\> ["version\_number", "@timestamp"] } My csv file contains: issue 01 01,2018-01-04T15:05:27.429Z 2)Now I am trying to get this into my ActiveMQ output. I have used the translate plugin in my filter as follows: translate { field =\> "version\_number" refresh\_interval =\> "1" destination =\> "version\_number" add\_field =\> {"version\_number" =\> "version\_number"} dictionary\_path =\> '/tmp/version\_number.csv' } No field appears though! What am I doing wrong? Basically I would just like to add the version\_number field. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 4, 2018, 6:41pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/10 "2018-01-04T18:41:20Z")
I think you should use the exec filter as I explained in your original post because the LHS of the pair needs to be a field **value** that is found in the event in the `field` that the translate filter is set to use. But what is the value in the field `version_number` in the event before you do the translate? In my previous reply I said you should have a "constant" field value. Meaning you should add a field in the activemq input e.g. `version_number -> 'to_be_sourced'` then the CSV file that you overwrite (with the exec output) has the contents `to_be_sourced,issue 01 01`. The translate filter will translate the **value** `to_be_sourced` into whatever the CSV file has as the RHS of the pair. So initially the CSV file is `to_be_sourced,issue 01 01` then later it becomes `to_be_sourced,issue 01 02` and even later it is `to_be_sourced,issue 02 11` - the LHS remains the same. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 4, 2018, 6:48pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/11 "2018-01-04T18:48:29Z")
You can still use your CSV output solution but then you also need to add a field say `constant_out` with a value of `to_be_sourced` then the csv output has `fields => ["constant_out", "version_number"]` This will yield the required CSV file where the RHS of the pair changes but the LHS remains the same. ---
**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\ **Post date:** [January 5, 2018, 10:51am UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/12 "2018-01-05T10:51:58Z")
Ok, I think I am getting there! What I have in my single config file is: 1. Two inputs, one for file (xml) and one for jms(active mq). In my xml input, I watch the particular file that I need version\_number from. I have not added anything else new. In my jms input, I have added: add\_field =\> { "version\_number" =\> "to\_be\_sourced" } 2. In filter, I have two conditions, if type= activemq, I process activemq data. There I have added: translate { field =\> "version\_number" refresh\_interval =\> "1" destination =\> "current-version" dictionary\_path =\> '/tmp/version\_number.csv' } In my file filter, I have added a new field as follows: mutate { add\_field =\> { "constant\_out" =\> "to\_be\_sourced" } 3)my output is for two types again, file + activemq In fily type output, I have following configuration csv { path =\> "/tmp/version\_number.csv" create\_if\_deleted =\> true fields =\> ["constant\_out", "version\_number"] } in activemq output, I write the output to ElasticSearch as needed. This almost works! I have the following field added in my ActiveMQ output: "current-version" =\> "issue 123456789" and this gets written to Elastic as required! I tested it a bit and see: 1. If I append the csv file manually where I store version\_number, Logstash pics up the version number and appends my ActiveMQ output- great!! 2. If I copy a new xml file, which has a new version number, in the place of the old xml file, my new file doesnt get picked up it seems. The version in the csv file stays the same. 3. If I stop and start my pipeline, Logstash picks up new version from my replaced xml file. I see you mentioned exec plugin I havent used. Maybe this is my problem. How do I ensure that Logstash processes changed xml file(with exact same name, location but different version number inside file) so my csv file gets updated? NOTE: I have noticed that when I append the xml file manually, sometimes my change gets picked up and my new version\_number value gets put in the csv file.Same when I move the file from filexyz to originalfile. It works some of the time, but not always. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 5, 2018, 3:36pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/13 "2018-01-05T15:36:09Z")
Looking at [this line](https://github.com/logstash-plugins/logstash-filter-translate/blob/master/lib/logstash/filters/translate.rb#L180) the actual refresh is not done until two things happen - 1) an event comes along, 2) enough time has elapsed. In my original reply to you I advised that this latency could be a problem - but at least it works in principle. I also advised that you may need to switch the translate filter out for the jdbc\_streaming filter and use a SQLite local db. In this case you would use the exec output and the SQLite CLI to update a single column, single record table. As the jdbc\_streaming filter does a "live" call to the DB it will pick up the new value immediately provided you [turn off caching](https://github.com/logstash-plugins/logstash-filter-jdbc_streaming/blob/master/docs/index.asciidoc#use_cache). Also, because you are using a single LS instance pipeline you will have to "prime" the db with the first version number because you can't start the XML read pipeline a few minutes before the ActiveMQ one. ---
**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\ **Post date:** [January 7, 2018, 6:45pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/14 "2018-01-07T18:45:04Z")
Just out of interest.... If I store my key value of version\_number in redis, can I use some filter to read this value and enrich my ActiveMQ data with this version coming from redis? ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 8, 2018, 9:06am UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/15 "2018-01-08T09:06:11Z")
😄 I had thought about using redis, unfortunately we don't have an enhancement plugin that uses redis. I even considered a Redis JDBC driver, but that is a very untested route which may lead to more problems than its worth. We could build a Redis lookup plugin but it would not happen soon. Seems like a good fit to me. How do you see it being used? Redis key sourced from event where the Redis value is a simple single value or a JSON object or a CSV string? ---
**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\ **Post date:** [January 8, 2018, 2:57pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/16 "2018-01-08T14:57:44Z")
That would sound great if in the future there could be an enhancement plugin for Redis! In my case a single value would be enough I believe. But a JSON object could be a nice option as well. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 5, 2018, 2:57pm UTC](https://discuss.elastic.co/t/usign-ruby-to-update-local-linux-environment-variable/113948/17 "2018-02-05T14:57:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.