Log4j2 to Kibana

Elastic Stack Kibana
1

Hello,

I am using Log4j2 to export the log fie to JSON format. Some of the lines are:

{
  "timeMillis" : 1455729263723,
  "thread" : "main",
  "level" : "INFO",
  "loggerName" : "org.eclipse.jetty.util.log",
  "message" : "Logging initialized @1764ms",
  "endOfBatch" : false,
  "loggerFqcn" : "org.eclipse.jetty.util.log.Slf4jLog"
}
{
  "timeMillis" : 1455729263848,
  "thread" : "main",
  "level" : "INFO",
  "loggerName" : "org.eclipse.jetty.server.Server",
  "message" : "jetty-9.2.14.v20151106",
  "endOfBatch" : false,
  "loggerFqcn" : "org.eclipse.jetty.util.log.Slf4jLog"
}
{
  "timeMillis" : 1455729264005,
  "thread" : "main",
  "level" : "INFO",
  "loggerName" : "/hhhh",
  "message" : "Initializing log4j from [C:\\Development\\workspace\log4j.properties]",
  "endOfBatch" : false,
  "loggerFqcn" : "org.eclipse.jetty.util.log.Slf4jLog"
}

Now I have Json log file and would like to create index and get some visualizations from this log file. As far as I know, I have 2 options:
1. create index using LogStash or
2. directly post that file using curl. I prefer second option since I already have json log file. So that I ran following command:

It DOES NOT create index If I see the list of indices. But if I run following command:

It posts first one since I declared "/1" at the end of the command. But even if I ran last one, Kibana could not find indices but index was created if I list them. So my questions are:

  1. Does Kibana have some requirements for JSON format? (There must be a field called @timestamp etc)
  2. To post json file to ES, can I specify path to the file?
  3. Should I use LogStash instead of directly posting JSON file to the ES? Which one is the best way to do it?

Any advice would be appreciated. Thanks

2

Hi Bilguun,

It seems you are trying to do a Bulk index. I would first just create an index:

curl -XPOST localhost:9200/text?pretty

(Or you could add this request to a bulk index file.)

Then, I would create a bulk index file according to the spec in the link above (if you choose to do this without logstash).

3

Thanks Shelby, if I use bulk indexing, Can ES get newly added lines to the log files on the fly? Or should I do it manually every time?

4

Bilguun,

Not sure I understand what you mean by adding lines to log files. Do you mean updates to an indexed document? If so, then yes, you can update docs with bulk API, but the action is changed from index to update. Check out the documentation in the link I sent earlier.