Multiple grok pattern from ingest pipeline works but can't see the data in kibana

[hungl99]

Managed to get simulation works with multiple patterns but can't see the data in kibana.

I added two new fields (log.controller & log.controllerMessage to index template. index pattern in kibana shows the 2 new fields but they can't be searchable and that pattern with controller doesn't work as there are no docs with that pattern in Kibana

{
"pipeline": {
"description" : "parse multiple formats MP app logs",
"processors": [
{
"gsub": {
"field": "message",
"pattern": "\n",
"replacement": "\\n"
}
},
{
"grok": {
"field": "message",
"trace_match": true,
"patterns":["%{TIMESTAMP_ISO8601:log.datetime} %{DATA:log.level} \[%{DATA:.log.thread}\] controller:\|%{DATA:log.controller}\| %{GREEDYDATA:log.controllerMessage}",
"%{TIMESTAMP_ISO8601:log.datetime} %{DATA:log.level} \[%{DATA:log.thread}\] %{GREEDYDATA:log.message}"
],
"on_failure": [
{
"set":{
"field": "ingestError",
"value": "{{ _ingest.on_failure_message }}"
}
},
{
"set":{
"field": "log.datetime",
"value": "{{ @timestamp }}"
}
},
{
"date": {
"field":"log.datetime",
"target_field":"log.datetime",
"formats": ["yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"]
}
}
]
}
}
]
},
"docs":[
{
"_source": {"message":"2017-10-04 18:21:55 ERROR [29] controller:|DM.Marketing.Platform.Deal.V1.Controllers.EligibilityController| DM.Marketing.Platform.Deal.V1.Domain.DeskingProviderNotFoundException: No desking provider has been assigned to this deal (9aac0c8e-a099-449e-9119-c79dbf937abc)."}

},
{
"_source": {"message":"2017-10-04 20:24:36 ERROR [18] controller:|DM.Marketing.Platform.Deal.V1.Controllers.DealJacketExtensionController| System.AggregateException: One or more errors occurred. ---> System.Exception: GetAsync Exception, Url: api/vehicle/v2/inventory?bac=112093&vin=1GNSCBKC1FR507889&sellingsource=13&bfc=01&sourceZip=&destinationZip=&webId=&type=New&userId=, Response Content: "}
},
{
"_source": {"message":
"2017-09-27 03:21:40 ERROR [28] Application Error System.Web.HttpException (0x80004005): The controller for path '/2014/products/9' was not found or does not implement IController."}
},
{
"_source": {"message":"2017-10-09 04:00:48 ERROR [27] GetAsync Exception, Url: api/vehicle/v2/inventory?bac=112093&vin=1GNSCBKC1FR507889&sellingsource=13&bfc=01&sourceZip=&destinationZip=&webId=&type=new&userId= System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found). \n at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() "}
}
]
}

[Christian_Dahlqvist]

Did you select the log.datetime field as your timestamp field when you created the index pattern?

[hungl99]

Yes I did. In fact, the index pattern has been working fine with this grok pattern below. I added the second pattern, delete the old index pattern and created a new one selecting log.datetime in kibana. The two new fields show up but they are not searchable.

"%{TIMESTAMP_ISO8601:log.datetime} %{DATA:log.level} \[%{DATA:log.thread}\] %{GREEDYDATA:log.message}"

[Christian_Dahlqvist]

I think I might have misunderstood the issue. It sounds like you are able to see the records in the Discover application, which I thought was the problem. What is the mapping for these fields you are having an issue with?

[hungl99]

Yes and No. Yes I am still able to see the records for the 2nd pattern ("%{TIMESTAMP_ISO8601:log.datetime} %{DATA:log.level} \[%{DATA:log.thread}\] %{GREEDYDATA:log.message}") this pattern was already existing before I added this pattern ("%{TIMESTAMP_ISO8601:log.datetime} %{DATA:log.level} \[%{DATA:.log.thread}\] controller:\|%{DATA:log.controller}\| %{GREEDYDATA:log.controllerMessage}"). However, I am NOT able to see any records from the new pattern. I guess there are issues with the new fields because they are not searchable comparing to the rest.

[hungl99]

I guess my two new fields doesn't show up in Discovery because I just added the two new fields to my existing index template by overwriting the existing on with a new one which has the two new fields. Elasticsearch Documentation says that "changing the mapping would mean invalidating already indexed documents" Not sure if it mean changing existing fields or adding new fields would affect existing docs? What should I do with existing docs and have new index with new fields added?

[hungl99]

_"Other than where documented, existing type and field mappings cannot be updated. Changing the mapping would mean invalidating already indexed documents. Instead, you should create a new index with the correct mappings and reindex your data into that index.." How do I do it?