Ingesting JSON Data Samples w/ Logstash

Elastic Stack Logstash
21

It is finding no files that match the value of @path. Could be a permissions issue.

Also @file_completed_log_path should be a file, not a directory.

22

Badger please don't be upset with me, but it was a misspelling of the file name. :man_facepalming:

23

Do you know how I can fix this?

JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character (',' (code 44)): Expected space separating root-level values
24

Look at the [message] field. Is it valid JSON?

25

There are @message within the JSON, but they aren't necessarily used as a field. It's apart of the string in the field.

26

What is in the [message] field?

27

When I grep for it this is what I find and I see the comma that the error points out.

cat train_features_0.json | grep '"message"'
      "message",
28

Good news, is that I only found one occurrence.

29

No, I meant on stdout. But what you posted there indicates the problem. A json codec does not keep accumulating lines until it has a complete JSON object, it parses each line as an object. What you need is a multiline code to combine all the lines for one object into a single event, then parse it using a json filter.

If you want to ingest the whole file as one event see Merge Multiline json into single line json using codec multiline plugin - #2 by Badger. If a file contains multiple objects then this might work.

30

I was able to finally get data in, but now I'm having issues with my license and elasticsearch can't be reached.

31 
elasticsearch]$ uncaught exception in thread [main]
java.lang.IllegalStateException: failed to obtain node locks, tried [[/home/elastic/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:292)
	at org.elasticsearch.node.Node.<init>(Node.java:368)
	at org.elasticsearch.node.Node.<init>(Node.java:278)
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
	at org.elasticsearch.cli.Command.main(Command.java:79)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
For complete error details, refer to the log at /home/elastic/elasticsearch/logs/dev.log
32

Whenever I make some progress, I get stumped.

33

I suggest you ask a new question. This one has covered way too many issues.

1 Like
34

I was able to finally get data in.

35

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.