I have a database that has multiple data fields. A date field when published, and date fields for start and stop times tracking total times of the transaction. In kibana, i notice the default discover page does not have a small histogram at the top, and when i build dashboards, I cannot filter the time, it just shows all my events. Do I have to tell elasticsearch a default date field?
This sounds like those fields aren't mapping to a date type properly (possibly they are just mapped as strings). Can you check your mapping to make sure those fields have type: date?
No, they are definitely mapped as type date. I noticed on my other data, that only have one date type field, that in settings, indices, and by field name, it has a little clock icon, that when I mouse over, it says, "This field represent the time that events occurred".
My new data, does not have a field that has that icon.
I did just figure out if i create the index from kibana, rather than my curl mapping statements, I can select a "time field name". Is there a way I can assign that via curl mappings?
Could you share your curl mapping statements?
Here are my curl/invoke statements. They are in powershell format:
Invoke-RestMethod http://localhost:9200/m1aps -Method Delete
Invoke-RestMethod http://localhost:9200/m1aps -Method Put
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"APSOperations":{"properties":{"Published":{"type":"date","format":"MM/dd/yyyy HH:mm:ss"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"APSOperations":{"properties":{"StartTime":{"type":"date","format":"MM/dd/yyyy HH:mm:ss"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"APSOperations":{"properties":{"StopTime":{"type":"date","format":"MM/dd/yyyy HH:mm:ss"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"APSOperations":{"properties":{"insDt":{"type":"date","format":"yyyy-MM-dd HH:mm:ss.SSS||MM/dd/yyyy HH:mm:ss||MM/dd/yyyy hh:mm:ss aa"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"Text":{"type":"string","index":"not_analyzed"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"SuspectMessage":{"type":"string","index":"not_analyzed"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"SuspectMessageType":{"type":"string","index":"not_analyzed"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"TotalTimeInSeconds":{"type":"integer","index":"not_analyzed"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"Ticks":{"type":"integer","index":"not_analyzed","store":true}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"EligibilityStatus":{"type":"string","index":"not_analyzed"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"MessageTypeName":{"type":"string","index":"not_analyzed"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"SerializedMessage":{"type":"string","index":"not_analyzed"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"StackTrace":{"type":"string","index":"not_analyzed"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"SuspectMessage":{"type":"string","index":"not_analyzed"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"SuspectMessageType":{"type":"string","index":"not_analyzed"}}}}'
Invoke-RestMethod http://localhost:9200/m1aps/_mapping/ApsOperations?ignore_conflicts=true -Method Put -Body '{"ApsOperations":{"properties":{"Payer":{"type":"string","index":"not_analyzed"}}}}'
I think I got it now by selecting the time field name in kibana index settings, but I am still trying to figure out the following:
- how to modify my curl mapping statements to handle date type if value is null
- how to complete delete/remove documents with a specific field name
Did you resolve this? My data has multiple fields with type date too and when I create the index-pattern in kibana I don't get any of these options in the drop down and I am hence unable to filter on time in my graphs and dashboard. I think it is a similar issue so did you solve this - could you let me know how?
I don't remember. But I think it is easier to have indexes and mappings defined prior to inserting any data. Use of templates is great too, but I am not very knowledgeable in creating them yet.