Kv variable with multiple types

Elastic Stack Logstash
#1

Hi,

I have used kv filter in logstash configuration file. A variable "jobname" created by kv originally had the type of integer according to its value,

"jobname=40000"

but now in some documents the values of the variable are in type of string

"jobname=c148a"

Elasticsearch complains this and drops all the documents with the value of "jobname" in string. How can I solve this issue?

Thanks,

#2

You will need to convert the field to a string in LS, then it will always treated as such in ES.

#3

Thanks for your response Mark. I have converted the field to a string in LS but ES is still dropping the documents. Maybe because in ES the field is still integer? How can I change the type of the field in ES?

#4

You will need to drop the index and recreate it so that it has the correct mapping applied to the field, you cannot convert it in an existing index.

#5

Thanks for the information. I will try that. In the new index created today the type of the field has become string and the error disappeared.

#6

I've got a new problem now: same field in a log has multiple values with multiple types. Is there anyway to apply kv filter to such logs?

For example, in the following log, fields such as jobfs_local and mem have different values with different types. elasticsearch/logstash complain if I apply kv filter to such logs:

02/24/2016 15:54:30;0080;walltime_for_job: Resources:select=4:ncpus=16:mpiprocs=16:mem=34359738368:job_tags=normal:jobfs_local=104857600,mpiprocs=64,uuid=6eadfce3-481f-43d4-983c-74a0e4ff82be,ncpus=64,nodect=4,jobprio=8092.4810,jobfs_local=419430400b,HT=0,wd=0,mem=137438953472b,jobfs=419430400b,walltime=00:20:00,place=free

#7

As I mentioned before - You will need to convert the field to a string in LS, then it will always treated as such in ES.

#8