Filebeat sending json date field as text

DougC · October 13, 2018, 6:04pm

Hi. I've been trying to teach myself the Elastic Stack by trying to index data generated by speedtest-cli on my local Ubuntu shell.

When I use Logstash to send the results to Elasticsearch the timestamp field comes through as a date, but when I use Filebeat it comes through as text.

Elasticsearch is set to allow indexes to be created automatically.

Am I able to use a processors to change how the field is sent to Elastic? I had a look at the decode_json_fields processor and did not have a lot of joy.

Thank you.

My Filebeat config file, minus the comments.

filebeat.inputs:
- type: log
  enabled: true
  close_eof: true
  paths:
    - /path/to/directory/**/*.txt
  json.keys_under_root: true
  json.add_error_key: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:

output.elasticsearch:
  hosts: ["localhost:9200"]

Christian_Dahlqvist · October 14, 2018, 7:13pm

What does your data look like? What is inserted into Elasticsearch?

DougC · October 15, 2018, 5:08pm

Here is an example of the json.

{
"download": 128271430.47441846, 
"upload": 11931160.290369328, 
"ping": 40.093, 
"server": {
 "url": "http://awebaddress.net/speedtest/upload.php", 
 "lat": "0.0", 
 "lon": "-0.0", 
 "name": "A Town", 
 "country": "United Kingdom", 
 "cc": "GB", 
 "sponsor": "A Company Name", 
 "id": "00000", 
 "host": "awebaddess.net:8080", 
 "d": 00.0, 
 "latency": 40.093
 }, 
"timestamp": "2018-10-15T16:35:01.947110Z", 
"bytes_sent": 15892480, 
"bytes_received": 160557574, 
"share": null, 
"client": {
 "ip": "0.0.0.0", 
 "lat": "0.0", 
 "lon": "-0.0", 
 "isp": "My ISP", 
 "isprating": "0.0", 
 "rating": "0", 
 "ispdlavg": "0", 
 "ispulavg": "0", 
 "loggedin": "0", 
 "country": "GB"
 }
}

The timestamp is the only field that I am worried about for now - in Elastic it is a text string.

Everything else is coming through fine as text or numbers.

Christian_Dahlqvist · October 15, 2018, 5:14pm

Elasticsearch can only handle millisecond timestamps (3 decimals) which may be why it does not recognise it as a date. You may therefore need to do some further processing on it, e.g. with an ingest pipeline.

DougC · October 15, 2018, 6:40pm

Thank you very much. Lesson learnt.

DougC · October 15, 2018, 7:16pm

Sorry to be that guy, but I just sent in the same json directly into a new index via curl and Elasticsearch has picked it up as a data (minus the last 3 decimals) AOK.

How is the curl command different from how Filebeat is doing it?

$ curl -H 'Content-Type: application/json' -XPOST http://localhost:9200/json/doc -d @201810151955.txt

{"_index":"json","_type":"doc","_id":"PrwieWYB3hSPeK7iZZr6","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

Christian_Dahlqvist · October 15, 2018, 7:18pm

I am not sure I understand. What does that document look like in Elasticsearch? What is the mapping?

Elasticsearch will not allow you to change the mapping for a field in an index, so once you have changed the format you need to index into a new index for the mapping to change.

DougC · October 15, 2018, 7:29pm

Thanks for your patience and replies. Just trying to understand how it works with simple commands and examples before upgrading to proper ingest methods with Logstash and/or Ingest pipelines.

Sorry for not being clear. All my "json files" are ASCII text, with very long lines. I used curl with the above options to send the "json file" into a brand new index in my single-node Elastic 6.4.2 cluster.

The brand new index has the following mappings.

{
  "json": {
    "aliases": {},
    "mappings": {
      "doc": {
        "properties": {
          "bytes_received": {
            "type": "long"
          },
          "bytes_sent": {
            "type": "long"
          },
          "client": {
            "properties": {
              "country": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "ip": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "isp": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "ispdlavg": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "isprating": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "ispulavg": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "lat": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "loggedin": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "lon": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "rating": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "download": {
            "type": "float"
          },
          "ping": {
            "type": "float"
          },
          "server": {
            "properties": {
              "cc": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "country": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "d": {
                "type": "float"
              },
              "host": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "id": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "lat": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "latency": {
                "type": "float"
              },
              "lon": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "sponsor": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "url": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "timestamp": {
            "type": "date"
          },
          "upload": {
            "type": "float"
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1539630588951",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "a0F6LYIUTqGfpN7s0Cmj5A",
        "version": {
          "created": "6040299"
        },
        "provided_name": "json"
      }
    }
  }
}

Christian_Dahlqvist · October 15, 2018, 7:30pm

What does the corresponding document look like?

DougC · October 15, 2018, 8:07pm

Sorry to not answer your question but I think I see an issue with my setup and understanding.

As you suggested looking at the index mappings I looked under that option for the first time and I see a lot, lot more mappings. Much more than I expected. Digging around that I see a couple of mappings for timestamp that are not date fields - for example one under kafka like

"timestamp": {
 "type": "keyword",
 "ignore_above": 1024

and mysql

"timestamp": {
 "type": "keyword",
 "ignore_above": 1024

Could that be (one of) my problems?

Christian_Dahlqvist · October 15, 2018, 8:11pm

If that is in the same index it could indeed.

DougC · October 15, 2018, 8:23pm

Looks like that's exactly the problem I ran into.
I changed the following in my filebeat.yml

setup.template.name: "myfilebeat"
setup.template.pattern: "myfilebeat-*"

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: "myfilebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

and the timestamp field from my "json" has come through as

"timestamp": {
 "type": "date"

Thank you very much.

system · November 12, 2018, 8:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Using index template to read a field as date Beats filebeat	5	4213	September 19, 2018
Filebeat new timestamp field Beats filebeat	3	2601	May 29, 2018
Problem with JSON logs Beats filebeat	3	558	March 9, 2019
Parsing the date and sending to ElasticSearch Logstash	13	7841	May 26, 2017
How to send Json logs to Elastic Search using File Beats without extra fields Beats filebeat	5	4016	August 6, 2020

Filebeat sending json date field as text

Related Topics