I ran into an issue where the JSON sent to Elastic Search was being encoded as a float. This appears to a normal Golang solved by creating a json.NewDecoder and setting .UseNumber(). Fine enough, the JSON I'm parsing now has the numbers decoded as int64. If I marshal them manually, I get the integer value, however, when I use publisher.PublishEvent they are marshalled to Elastic Search as floats again (ie: 2.200074e+06 instead of 2200074).
Something is happening between me calling PublishEvent and the event being marshalled again out to Elastic Search. But I'm at a loss as to where to look.
I should add that numbers at the top level of my event do get marshalled out as integers, its number in nested objects that get marshalled as floats again. I'm guessing it has something to with ConvertToGenericEvent and MarshalUnmarshall.
Are you sure, you're using the right types? Testing with go playground using int64 works ok for me.
You can run beats with -e -v -d '*' on command line to print all debug information to stdout. The publisher pipeline will print a json document of original event before forwarding event to actual outputs. Is this one using floats or integers?
Keep in mind, in case you didn't set up an actual mapping, elasticsearch will try to learn the types being used. In case you did publish floats in past, all integers published to same index will be converted to floats within elasticsearch, due to the (dynamically generated) mapping is asking for floats. When testing, drop index. Even better install mapping before sending to elasticsearch. Having the mapping requiring and integer (consider using long), elasticsearch will convert the value to int at ingestion time. Still better use int64 instead of float64 in your code, to make sure the conversation not adding any accidental routing errors.
Like I said, if I marshall them manually (say in a scratch project, or on the playgroun) it works - its when goes through the PublishEvent path that it doesn't work. But I found the issue.
My JSON log files have nested objects. So my top level one is a common.MapStr, but the nested objects are map[string]interface{}'s. PublishEvent re-marshals certain types out to a string and reparses them. This is converting my json.Numbers (or int64's) back into doubles as it doesn't use UseNumber.
Answer: Transform all map[string]interface{}'s in my object to common.MapStr. MapStr's don't go through the marshall/unmarshall before finally being marshalled to elastic search, so the number stay as integers.
Yeah, I noticed this too. libbeat should cast map[string]interface{} to a common.MapStr instead of running the marshal/unmarshal on it. The marshal/unmarshal function has the downside of losing type information for numbers (which always becoming float64) and arrays (which always become []interface{} because JSON allows for mixed type arrays).
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.