Elasticsearch output plugin does not write to elasticsearch

Elastic Stack Logstash
1

Hello!

I've just upgraded logstash from 2.2 to 5.5. I have the current version of the elasticsearch output plugin installed (8.8.2)

Logstash no longer outputs anything to elasticsearch with the same config file. The only thing it does is create the index. I've tested this with numerous index names, and without an index config so it creates a logstash index.

No documents reach elasticsearch.

Configuration: (using ElasticSearch 5.5 cluster in AWS)

input {
mongodb {
uri => 'mongo connection string'

#temp database info for the input plugin. The dir has to exist but the plugin makes the .db file.
placeholder_db_dir => '/opt/logstash-mongodb'
placeholder_db_name => 'logstash_sqlite.db'

#regex list of collections to index.
collection => 'collection regex'

batch_size => 300
}
}

output {
elasticsearch {
hosts => ["aws-es-cluster-endpoint"]
index => "test"

#flush_size => 50

#commented out for troubleshooting.
#document_id => "%{[mongo_id]}"

#document_type => "%{[_class]}"
}

stdout { codec => rubydebug { metadata => true } }
}

What I've done:

  • Verified output comes in through the rubydebug stdout - tons of it.
  • DEBUG logs come from mongo, but not from the elasticsearch plugin
  • re-installed the es output plugin, and even upgraded it.
  • rebooted the machine
  • run the logstash config test utility and it says my config is fine.
  • I know it can talk to ES because the index is being created.
  • commented out all non-essential config options

The only thing I've done is upgraded logstash and thus the es output plugin as well

Any help is appreciated.

2

You need to understand something fundamental about Logstash's Elasticsearch output plugin: It doesn't create indices. Ever. It does, however, send a request to Elasticsearch that is, in effect, "Please put these documents in the index named logstash-YYYY.MM.dd". Elasticsearch receives the request, and if the index does not exist, creates the index, and then indexes the documents into it. What this means is that if the index is created, something got sent. Whether it got indexed or not is a different story.

I'm thinking that you might have a mapping conflict, and that's why the documents aren't making it into the index. You should update your mapping template (if you're using one) to reflect the changes.

Was Logstash 2.2 indexing to Elasticsearch 5.5?

2 Likes
3

I understand that, my point was only that logstash is connecting to elasticsearch and something is happening, so a live connection is not the problem.

Yes logstash 2.2 was indexing to Elasticsearch 5.5 using the ES output plugin - it was an older version of the plugin though, not sure which version.

I don't see what the mapping has to do with it. I am not setting any mapping, and any mapping that gets created would be the default that gets created or applied when an index is created. I could be a little ignorant here, but how would a default mapping or nonexistent mapping cause a document to not get indexed when it was indexed just fine by the old plugin?

Also, why aren't there any ES DEBUG logs being output?

Thanks for any additional help!

4

Unless you disable the feature, Logstash uploads a template by default, though it will only apply to logstash-YYYY.MM.dd indices. The index template for Logstash 2.2 would have mapping defaults tuned to Elasticsearch 2.x. Elasticsearch 5.x has a completely different set of mappings (e.g. fieldname.raw changed in favor of fieldname.keyword). That collision could explain the problem, but it might be a red herring.

Do you only have a single node? Did you dial up the Elasticsearch log level? Have you tried running Logstash with debug logging?

5

About the logstash logging: I'm running it on the commandline which throws these weird errors about not being able to find a config file in the paths that actually contain the config file. Researching this showed me that this is only because I'm running logstash on the commandline and not as a service. Logstash will default to using DEBUG logging if it cannot find the config file. I'm not sure if this means that the input or output plugins will default to the same level however. Do you know if the plugins can have a different log level set than logstash itself?

Regarding Mappings: I found that I have mappings for indexes that exist. I've been deleting the indexes that logstash creates between attempts, and apparently the mappings go along with them.

I looked at the Templates using: GET /_template?pretty=true"

There was an old template for logstash. I deleted the template with DELETE _template/{{template-name}},

And it still doesn't actually send data to Elasticsearch, although *elasticsearch is still creating the index when the docs are being sent.

Any additional help appreciated!

-Jeff

6

You should turn up debug logging in both Logstash and Elasticsearch. If mappings are being created, then Elasticsearch is at least getting an index request from Logstash. The question now becomes, "what's it doing, if it's not indexing?"

7

Sorry I might be a little dense here - the central documentation for the ES output plugin doesn't seem to have any information about special debug logging. In fact the page doesn't even have the word 'debug' on the page at all. I see the option for 'failure_type_logging_whitelist' and 'enable_metric', but they both default to being more verbose already.

Thanks for any additional help.

8

You'd have to set log.level: debug in your logstash.yml, and rootLogger.level = debug in log4j2.properties in your Elasticsearch configuration (/etc/elasticsearch, or the config directory wherever you installed it).

Then look in the logs afterward. I'd start with the Logstash logs first.

9

Two quick rants about logstash 5.x.

[red herring]
I was able to fix my issues with logstash not being able to find the config files, you need to run it under the user 'logstash'. This issue that was closed could have resulted in a code change to at least give a message or warning that makes sense - "This error may be because you're not running logstash as the logstash user or as a service"

[red herring]
More very relevant complaints about the logstash 5.x documentation: Logstash 5 not running

I've modified the config files as suggested.

[red herring]
Now its complaining that it can't find a pipeline config even though I'm specifying it with a -f option... wow this is getting a tad ridiculous. I'll log a bug on the project page. For now I fixed it by putting my config in /etc/logstash/conf.d

Here is the warning I'm receiving:

{"type"=>"mapper_parsing_exception", "reason"=>"Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."

At least now I'm getting some feedback from ES, although the ruby output plugin doesn't show the mongo output when I'm running logstash as the logstash user/as a service.

Thanks for your guidance! Hopefully logstash 5.x updates will add more relevant error messages and make it a little easier to use, with a consistent behavior across commandline & config options.

10

Sorry you had a hard time. 5.x is a major version upgrade, and clearly we can do better at helping users transition.

11

Fixed with:

filter {
mutate {
remove_field => [ "_id" ]
}
}

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.