# "hash=event\['field'\].to\_hash" instead "hash = event.to\_hash"

**URL:** <https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169>\
**Category:** Logstash\
**Created:** [June 23, 2015, 9:42am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169 "2015-06-23T09:42:08Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 23, 2015, 9:42am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/1 "2015-06-23T09:42:09Z")

</div>

Is it possible...

I want to traverse through events against specific field. something like:  
hash = event['field1'].to\_hash  
hash.each { |event['field1'], v| puts event['field1'] if v == hash.values.max }

Any help?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 23, 2015, 10:01am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/2 "2015-06-23T10:01:38Z")

</div>

The named parameters passed to the iteration block must be identifiers and can't be expressions. What's wrong with

```
field1_hash = event['field1'].to_hash
field1_hash.each { |k, v| puts event['field1'] if v == field1_hash.values.max }

```

or even

```
field1_hash = event['field1'].to_hash
field1_hash.each_value { |v| puts event['field1'] if v == field1_hash.values.max }

```

since you don't appear to care about the hash key?

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 23, 2015, 10:40am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/3 "2015-06-23T10:40:03Z")

</div>

> [@magnusbaeck](#):
>
> field1\_hash = event['field1'].to\_hash  
> field1\_hash.each\_value { |v| puts event['field1'] if v == field1\_hash.values.max }

I am getting below exception when using it..  
Exception in filterworker {"exception"=\>#NoMethodError: undefined method `to_hash' for 31:Fixnum, "backtrace"=>["(ruby filter code):2:in `register'", "org/jruby/RubyProc.java:271:in `call'", "/opt/Log/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-0.1.5/lib/logstash/filters/ruby.rb:37:in `filter'", "/opt/Log/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/filters/base.rb:162:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/Log/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/filters/base.rb:159:in `multi_filter'", "(eval):302:in `filter\_func'", "/opt/Log/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:219:in `filterworker'", "/opt/Log/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:156:in `start\_filters'"], :level=\>:error}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 23, 2015, 11:07am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/4 "2015-06-23T11:07:40Z")

</div>

The `field1` field obviously isn't a hash or something that can be converted to a hash, it's a numerical value.

With more information about what your messages look like and what you want in the end it'll be easier to help.

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 23, 2015, 11:19am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/5 "2015-06-23T11:19:23Z")

</div>

Okay let me explain what I want step by step:

1. XML file contains following fields:  
Array Fields:  
xpath =\> ["/response/data/run/firstView/videoFrames/frame[_]/time/text()","Time\_FV"]  
xpath =\> ["/response/data/run/firstView/videoFrames/frame[_]/image/text()","Image\_FV"]  
xpath =\> ["/response/data/run/firstView/videoFrames/frame[\*]/VisuallyComplete/text()","Progress\_FV"]  
and Non Array fields  
xpath =\> ["/response/data/median/firstView/visualComplete/text()","FV\_visualComplete"]  
xpath =\> ["/response/data/median/firstView/lastVisualChange/text()","FV\_lastVisualChange"]  
xpath =\> ["/response/data/median/firstView/loadTime/text()","FV\_loadTime"]  
xpath =\> ["/response/data/median/firstView/fullyLoaded/text()","FV\_fullyLoaded"]  
xpath =\> ["/response/data/median/firstView/SpeedIndex/text()","FV\_SpeedIndex"]
2. I have multiple XML files with above fields
3. I parsed them and output to ELK
4. When I query found file processed as a new message and for non array fields it is fine.
5. If you remember in my last blog i had asked about splitting the array field; it was for the same kind of processing. BTW for single file i was able achieve what i wanted from these array field.  
But when you have multiple file the same config will not work. For you reference Config for Single XML File.  
input {  
file {  
path =\> "/opt/Log/WebPageTestfinal1/RUN\*/\*\_XML\_WebpageSummary.xml"  
start\_position =\> "beginning"   
}  
}

filter {  
if [message] =~ "^\<?xml .\*" {  
drop {}  
}  
multiline {  
pattern =\> "^\</response\>"  
negate =\> true  
what =\> "next"  
}

xml {  
source =\> "message"  
target =\> "videoFrames"  
store\_xml =\> false  
xpath =\> [

```
"/response/data/run/firstView/videoFrames/frame[*]/time/text()","Time_FV",
"/response/data/run/firstView/videoFrames/frame[*]/image/text()","Image_FV",
"/response/data/run/firstView/videoFrames/frame[*]/VisuallyComplete/text()","Progress_FV",

"/response/data/run/repeatView/videoFrames//frame[*]/time/text()","Time_RV",
"/response/data/run/repeatView/videoFrames/frame[*]/image/text()","Image_RV",
"/response/data/run/repeatView/videoFrames/frame[*]/VisuallyComplete/text()","Progress_RV",
		
]
	}

ruby {
	code => "
		##Finding max for splitting event for that many number of times
		
		x= [event['Time_FV'].length, event['Time_RV'].length]
		max= x.max
		if event['Time_FV'].length==max
			event['flag']='FV'
		end
		if event['Time_RV'].length==max
			event['flag']='RV'
		end
					
		"
		
    }
	if [flag]=="FV" {
		split { field => "Time_FV" } 
		} 
	if [flag]=="RV" {
		split { field => "Time_RV" } 
		}	

ruby {
	code => "
			my_variable = ENV['mycount3']
			if my_variable.nil?
				ENV['mycount3']=0.to_s
				counter=0
			else
				counter=ENV['mycount3'].to_i
				counter=counter+1
				ENV['mycount3']=counter.to_s
			end

			if event['flag'] != 'FV'
				tfv=event['Time_FV']
				event['TimeFV']=tfv[counter]
			end
			if event['flag'] != 'RV'	
				trv=event['Time_RV']
				event['TimeRV']=trv[counter]
			end
			pfv=event['Progress_FV']
			prv=event['Progress_RV']	
							
			event['ProgressFV']=pfv[counter]
			event['ProgressRV']=prv[counter]
			
			##Extracting test id, run id and url from file name	
            filename = File.basename(event['path'], '.*')
			value = filename.split('_')
			event['Test_Id'] = value[0]
			event['Run_Id'] = value[1] + '_' + value[2] + '_' + value[3]
			event['URL1'] = value[4]			
							
			"			
			}

if [flag]=="FV" {
		mutate {rename => { "Time_FV" => "TimeFV" }}
		} 
if [flag]=="RV" {
		mutate {rename => { "Time_RV" => "TimeRV" }}
		}	 
mutate {convert => ["ProgressFV", "integer"]}
mutate {convert => ["TimeFV", "float"]}
mutate {convert => ["ProgressRV", "integer"]}
mutate {convert => ["TimeRV", "float"]}

#mutate { remove_field => ["Time_FV"]}
#mutate { remove_field => ["Time_RV"]}
#mutate { remove_field => ["Progess_FV"]}
#mutate { remove_field => ["Progess_RV"]}

```

}

output {  
elasticsearch {  
action =\> "index"  
host =\> "172.27.155.109"  
index =\> "logstash-xml1%{+YYYY.MM.dd}"  
workers =\> 1  
}  
stdout { codec =\> json }  
}

```
indent preformatted text by 4 spaces
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 23, 2015, 12:59pm UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/6 "2015-06-23T12:59:18Z")

</div>

> If you remember in my last blog i had asked about splitting the array field; it was for the same kind of processing. BTW for single file i was able achieve what i wanted from these array field.  
> But when you have multiple file the same config will not work.

So this is what you're really asking about? If so, please explain what "will not work" means. If not, please explain what your question is.

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 23, 2015, 1:19pm UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/7 "2015-06-23T13:19:36Z")

</div>

Please excuse me for being not so clear:

trying to explain consider that i have two xml with fields  
File1:

\<?Median\> \<?visulaComplete.... \<?Average\> \<?visulaComplete.... \<?Run\> \<?ID \<?FV.. \<?time\>0\<?/time\> \<?time\>200\<?/time\> \<?RV \<?time\>300\<?/time\> File2: Is replica \<?Median\> \<?visulaComplete.... \<?Average\> \<?visulaComplete.... \<?Run\> \<?ID \<?FV.. \<?time\>0\<?/time\> \<?time\>200\<?/time\> \<?RV \<?time\>300\<?/time\> --When parsed I get two new messages --For "visulaComplete" File it is fine and I am able to store them as I want them to be presented on Kibana Graph. -- "Time" field creates an array and stores all values in single field "Time\_FV" & "Time\_RV" from both parsed files. To present it on graph not seeing any distinction due to its array field. - For seeing the distinction I need to split these array values into different messages (Each split message containing "Time\_FV" & "Time\_RV" field along with single items from array] This is what i was trying to achieve. Basically doing data modelling for storing data in a way to draw desired graph. As per above config: using RUBY code I got success but complexity increased when having multiple files to process. Hope I am clear to you now.

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 24, 2015, 5:14am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/8 "2015-06-24T05:14:25Z")

</div>

Can Logstash suffice my need?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 24, 2015, 5:32am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/9 "2015-06-24T05:32:02Z")

</div>

I don't see why the number of input files would in any way matter here. What isn't working? Why is it more complex to support more files?

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 24, 2015, 6:09am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/10 "2015-06-24T06:09:08Z")

</div>

When file path changes it overwrites the values...

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 24, 2015, 6:39am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/11 "2015-06-24T06:39:12Z")

</div>

Sorry, I don't understand what you mean. What values are overwritten?

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 24, 2015, 9:10am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/12 "2015-06-24T09:10:29Z")

</div>

Is it possible to process each file sequentially....

I mean file1 as an input parse----filter processing ----output elasticsearch; then another file2 as an input parse----filter processing ----output elastic search

rather doing  
dir path as an file input ---parse both the file----filter processing of both the file and then ---output elasticsearch.

can we control currently my config parses all files prsent in DIR:  
input {  
file {  
path =\> "/opt/Log/WebPageTestfinal7/RUN\*/\*\_XML\_WebpageSummary.xml"  
start\_position =\> "beginning"   
}  
}

then runs processing on all messages in one go and then output to elasticsearch.

Is it possible to parse-\> process-\>output files one by one ? If yes then how? If i parse file one by one my existing config will work.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 24, 2015, 2:50pm UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/13 "2015-06-24T14:50:45Z")

</div>

You could run separate Logstash instances to completely separate the processing.

But again, what you're describing doesn't sound normal. With more information about what values are being overwritten we can help you.

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 25, 2015, 6:01am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/14 "2015-06-25T06:01:25Z")

</div>

Am I being so difficult to state problem...my bad.

Use case:  
I have WebPageTest Results for multiple URL Performance Tests and for Multiple Runs.

I get them in Folder Structure

WebPageTest/RUN1432621877157  
WebPageTest/RUN1432621608713

XML Data:

![|nullxnull](https://us1.discourse-cdn.com/elastic/original/2X/7/7bb4c3bea4574bfe9c5f73a4b97acc5541a06d2c.png)

Note: Please change file extension from png to xml.

I want to push them into Elasticsearch using same INDEX in such a way to get below charts at Kibana.

```
                            1 Line chart Visual Progress (FV & RV)
                            2 Bar chart Timings (FV & RV) (Median)
                                                            -Visually Complete 
                                                            -Last Visual Change
                                                            -Load Time (onload)
                                                            -Load Time (Fully Loaded)
                                                            -Speed Index
                                                            -Time to First Byte
                                                            -Time to Title
                            3 Score Board | bar Chart (FV &RV) (Median)
                                            <?score_cache>71</?score_cache>
                                            <?score_cdn>28</?score_cdn>
                                            <?score_gzip>100</?score_gzip>
                                            <?score_cookies>-1</?score_cookies>
                                            <?score_keep-alive>100</?score_keep-alive>
                                            <?score_minify>-1</?score_minify>
                                            <?score_combine>100</?score_combine>
                                            <?score_compress>-1</?score_compress>
                                            <?score_etags>-1</?score_etags>
                            4 Bar chart on CPU consumtion (FV &RV) (Median)
                                            <?docCPUms>3057.62</?docCPUms>
                                            <?fullyLoadedCPUms>4976.432</?fullyLoadedCPUms>
                                            <?docCPUpct>85</?docCPUpct>
                                            <?fullyLoadedCPUpct>70</?fullyLoade

```

"For chart 2 to 4 Logstash worked perfectly to parse my data and push to ELK"

Using Below Config.  
 ![|nullxnull](https://us1.discourse-cdn.com/elastic/original/2X/4/42d58a6fe4692b70a94ef991bb173eeac1addd38.png)

Note: Please change file extension from png to conf.

In attached XML Time FirstView & RepeatView data are coming from array elements hence when parsing all values from respective fields being stored as list in ELK.  
Being in List not sufficing desired chart as expecting variations of Progress against time for both FirstView & RepeatView.

For Data Modelling I need to parse the RUN files one by one which i can see as work around rather parsing all run files and processing them and pushing to ELK at once.

Is there any suggestions for handling/processing mutiple XML files or Best Practice.

Thanks Saket.

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 25, 2015, 6:02am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/15 "2015-06-25T06:02:58Z")

</div>

Is there any way to share files with you for reference and better understanding of problem.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 25, 2015, 6:06am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/16 "2015-06-25T06:06:03Z")

</div>

You're still not explaining what you mean when you say that values are being overwritten. Sorry, I'm out of patience here. Maybe someone else help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:36am UTC](https://discuss.elastic.co/t/hash-event-field-to-hash-instead-hash-event-to-hash/24169/17 "2017-07-06T05:36:32Z")

</div>