Hi, I don't know, if this topic has been already discussed, but I didn't find an appropriate one - hence, I'm creating a new one.
I've got an nginx log whose partial representation is as follows. First, the record stating that the request finished successfully:
... 1 1.615 0.0 0.003 - - - -
And the next one that finished with an error:
... 2 - - 0.003 10 Too many requests
The fields' names are as follows (from left to right):
revision of running code
backend duration
post-backend duration
pre-backend duration
error code
error description
The field separator is the tab char.
When the request finishes successfully, my current nginx config writes the "-" char, indicating that there were no errors with the request (at least in the provided partial log). Otherwise, it writes the appropriate description in the field.
The problem I have is with the data types for the duration of request processing. I would like logstash to cast the number into the float data type, and the "-" char - to string data type. The float data type would be used to perform calculations of mean times and so on.
So far, I was able to develop such a pattern for the durations in the above-provided log (example for backend duration):
\t(%{NUMBER:backend_duration:float}|-)\t
However, the problem is, when I have "-" in the given field, logstash will simply omit this field and won't create it in the elastiscsearch database. On the other hand, the "-" char is hardly a number
Is there a possibility to perform this kind of "dynamic data typing" in logstash? For example, if there's a number data type, cast it into float; if there's "-", leave it as string - it would be done conditionally, based on what kind of data logstash finds in the given field. I read the docs, but I didn't the appropriate solution to my problem.
If I didn't provide enough information, please tell me so - I will try to provide as much info as I can, if someone can help me deal with this issue