Filebeat sending json date field as text

Elastic Stack Beats
1

Hi. I've been trying to teach myself the Elastic Stack by trying to index data generated by speedtest-cli on my local Ubuntu shell.

When I use Logstash to send the results to Elasticsearch the timestamp field comes through as a date, but when I use Filebeat it comes through as text.

Elasticsearch is set to allow indexes to be created automatically.

Am I able to use a processors to change how the field is sent to Elastic? I had a look at the decode_json_fields processor and did not have a lot of joy.

Thank you.

My Filebeat config file, minus the comments.

filebeat.inputs:
- type: log
  enabled: true
  close_eof: true
  paths:
    - /path/to/directory/**/*.txt
  json.keys_under_root: true
  json.add_error_key: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:

output.elasticsearch:
  hosts: ["localhost:9200"]
2

What does your data look like? What is inserted into Elasticsearch?

3

Here is an example of the json.

{
"download": 128271430.47441846, 
"upload": 11931160.290369328, 
"ping": 40.093, 
"server": {
 "url": "http://awebaddress.net/speedtest/upload.php", 
 "lat": "0.0", 
 "lon": "-0.0", 
 "name": "A Town", 
 "country": "United Kingdom", 
 "cc": "GB", 
 "sponsor": "A Company Name", 
 "id": "00000", 
 "host": "awebaddess.net:8080", 
 "d": 00.0, 
 "latency": 40.093
 }, 
"timestamp": "2018-10-15T16:35:01.947110Z", 
"bytes_sent": 15892480, 
"bytes_received": 160557574, 
"share": null, 
"client": {
 "ip": "0.0.0.0", 
 "lat": "0.0", 
 "lon": "-0.0", 
 "isp": "My ISP", 
 "isprating": "0.0", 
 "rating": "0", 
 "ispdlavg": "0", 
 "ispulavg": "0", 
 "loggedin": "0", 
 "country": "GB"
 }
}

The timestamp is the only field that I am worried about for now - in Elastic it is a text string.

Everything else is coming through fine as text or numbers.

4

Elasticsearch can only handle millisecond timestamps (3 decimals) which may be why it does not recognise it as a date. You may therefore need to do some further processing on it, e.g. with an ingest pipeline.

5

Thank you very much. Lesson learnt. :slight_smile:

6

Sorry to be that guy, but I just sent in the same json directly into a new index via curl and Elasticsearch has picked it up as a data (minus the last 3 decimals) AOK.

How is the curl command different from how Filebeat is doing it?

$ curl -H 'Content-Type: application/json' -XPOST http://localhost:9200/json/doc -d @201810151955.txt

{"_index":"json","_type":"doc","_id":"PrwieWYB3hSPeK7iZZr6","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
7

I am not sure I understand. What does that document look like in Elasticsearch? What is the mapping?

Elasticsearch will not allow you to change the mapping for a field in an index, so once you have changed the format you need to index into a new index for the mapping to change.

8

Thanks for your patience and replies. Just trying to understand how it works with simple commands and examples before upgrading to proper ingest methods with Logstash and/or Ingest pipelines.

Sorry for not being clear. All my "json files" are ASCII text, with very long lines. I used curl with the above options to send the "json file" into a brand new index in my single-node Elastic 6.4.2 cluster.

The brand new index has the following mappings.

{
  "json": {
    "aliases": {},
    "mappings": {
      "doc": {
        "properties": {
          "bytes_received": {
            "type": "long"
          },
          "bytes_sent": {
            "type": "long"
          },
          "client": {
            "properties": {
              "country": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "ip": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "isp": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "ispdlavg": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "isprating": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "ispulavg": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "lat": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "loggedin": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "lon": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "rating": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "download": {
            "type": "float"
          },
          "ping": {
            "type": "float"
          },
          "server": {
            "properties": {
              "cc": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "country": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "d": {
                "type": "float"
              },
              "host": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "id": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "lat": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "latency": {
                "type": "float"
              },
              "lon": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "sponsor": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "url": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "timestamp": {
            "type": "date"
          },
          "upload": {
            "type": "float"
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1539630588951",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "a0F6LYIUTqGfpN7s0Cmj5A",
        "version": {
          "created": "6040299"
        },
        "provided_name": "json"
      }
    }
  }
}
9

What does the corresponding document look like?

10

Sorry to not answer your question but I think I see an issue with my setup and understanding.

As you suggested looking at the index mappings I looked under that option for the first time and I see a lot, lot more mappings. Much more than I expected. Digging around that I see a couple of mappings for timestamp that are not date fields - for example one under kafka like

"timestamp": {
 "type": "keyword",
 "ignore_above": 1024

and mysql

"timestamp": {
 "type": "keyword",
 "ignore_above": 1024

Could that be (one of) my problems? :slight_smile:

11

If that is in the same index it could indeed.

12

Looks like that's exactly the problem I ran into.
I changed the following in my filebeat.yml

setup.template.name: "myfilebeat"
setup.template.pattern: "myfilebeat-*"

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: "myfilebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

and the timestamp field from my "json" has come through as

"timestamp": {
 "type": "date"

Thank you very much. :smile:

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.