Logstash not sending logs to Elasticsearch

Elastic Stack Logstash
1

Hello,

I have been trying for some time to send a simple log to Elasticsearch and after trying a very simple example, the logs are not been sent to Elasticsearch from logstash.

Services: In same server for this test
Operative Sytem: Centos 7
The logstash version is: 7.17.1
The Elasticsearch version is: 7.17.1

/etc/logstash/conf.d

input {
file {
path => "/var/log/Elasticsearch/Elasticsearch.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {

}
output {
Elasticsearch {
hosts => ["localhost:9200"]
}
}

/var/log/logstash/logstash-plain.log

[2022-03-18T11:33:29,161][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-03-18T11:33:30,690][INFO ][org.reflections.Reflections] Reflections took 118 ms to scan 1 urls, producing 119 keys and 417 values
[2022-03-18T11:33:32,042][INFO ][logstash.outputs.Elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["//localhost:9200"]}
[2022-03-18T11:33:32,540][INFO ][logstash.outputs.Elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2022-03-18T11:33:32,889][WARN ][logstash.outputs.Elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2022-03-18T11:33:32,908][INFO ][logstash.outputs.Elasticsearch][main] Elasticsearch version determined (7.17.1) {:es_version=>7}
[2022-03-18T11:33:32,913][WARN ][logstash.outputs.Elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2022-03-18T11:33:33,037][INFO ][logstash.outputs.Elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2022-03-18T11:33:33,113][INFO ][logstash.outputs.Elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2022-03-18T11:33:33,311][INFO ][logstash.outputs.Elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-03-18T11:33:33,337][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x15acb961 run>"}
[2022-03-18T11:33:34,573][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.23}
[2022-03-18T11:33:34,664][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-03-18T11:33:34,761][INFO ][filewatch.observingtail ][main][2b6c69038f817ebf29690e5d479fe4c6e56f482b9d6cc052978d217447903269] START, creating Discoverer, Watch with file and sincedb collections
[2022-03-18T11:33:34,771][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}

/var/log/Elasticsearch/Elasticsearch.log

[2022-03-18T01:30:00,079][INFO ][o.e.x.m.MlDailyMaintenanceService] [ip-.eu-west-2.compute.internal] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2022-03-18T10:23:44,010][INFO ][o.e.c.m.MetadataIndexTemplateService] [ip-.eu-west-2.compute.internal] adding template [logstash] for index patterns [logstash-]
[2022-03-18T10:23:44,189][INFO ][o.e.c.m.MetadataCreateIndexService] [ip-.eu-west-2.compute.internal] [logstash-2022.03.18-000001] creating index, cause [api], templates [logstash], shards [1]/[1]
[2022-03-18T10:23:44,522][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [ip-.eu-west-2.compute.internal] adding index lifecycle policy [logstash-policy]
[2022-03-18T10:23:44,603][INFO ][o.e.x.i.IndexLifecycleTransition] [ip-.eu-west-2.compute.internal] moving index [logstash-2022.03.18-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [logstash-policy]
[2022-03-18T10:23:44,671][INFO ][o.e.x.i.IndexLifecycleTransition] [ip-.eu-west-2.compute.internal] moving index [logstash-2022.03.18-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [logstash-policy]
[2022-03-18T10:23:44,726][INFO ][o.e.x.i.IndexLifecycleTransition] [ip-.eu-west-2.compute.internal] moving index [logstash-2022.03.18-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [logstash-policy]
[2022-03-18T10:23:55,371][INFO ][o.e.c.r.a.DiskThresholdMonitor] [ip-*.eu-west-2.compute.internal] low disk watermark [85%] exceeded on [r51WwHrKTE-VK6UCAaR4IA][ip-*8.eu-west-2.compute.internal][/var/lib/Elasticsearch/nodes/0] free: 1.1gb[14.1%], replicas will not be assigned to this node

Any help would be really appreciated : )

2

I add an additional note: From my understanding there is nothing being writen in Elasticsearch, not sure if I am missing anything in the output section. Networking issue shouldn't be if it is the same server.

curl localhost:9200/global2-*/_search?pretty
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" :
}
}

3

What is the return for:

curl localhost:9200/logstash-*/_search?pretty

You are not setting an index in your logstash output, so it will use the default logstash-* index.

Also, do you have the file elasticsearch.log and the user running logstash has access to it? Normally the Elasticsearch log has the name of the cluster and only the elasticsearch user (and root) has access to it.

4

Hello,
May be the issue of permission mode chmod xxx (linux) of Elasticsearch.log check it out... I think logstash not taken any input such as Elasticsearch.log... it is not able to load any file in the pipeline of logstash

5

Hello

The command returns:

curl localhost:9200/global2-*/_search?pretty
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" :
}
}

Is any better sample I could use for testing? The first thing I want to see is that is working?

Is the config file configured correctly? :slight_smile:

It is true it can be a permissions issue.
Aobut the logstash config has the following permissions:

/etc/logstash/conf.d
-rw-r--r-- 1 root root 447 Mar 18 13:46 logstash.conf

I think the elasticsearchlog maybe was giving permission issues, how I would be able to read any file with the logstash user?

Any similar simple example to understand these first steps would help a lot.

Thank you for your help.

6

Your index is not global2-*, you didn't specified an index name in your logstash output, so logstash will use the default name which is logstash-*.

You need to run:

curl localhost:9200/logstash-*/_search?pretty

Also, the permissions that can give you an issue is not the logstash config, is the Elasticsearch log config, You need to check that.

Do you have a file named elasticsearch.log? What are the permissions for that file?

7

Additional info

otal 52 /etc/logstash/
drwxr-xr-x 3 root root 156 Mar 18 11:34 .
drwxr-xr-x 95 root root 8192 Mar 17 17:28 ..
drwxr-xr-x 2 root root 27 Mar 18 13:46 conf.d
-rw-r--r-- 1 root root 2129 Feb 23 22:19 jvm.options
-rw-r--r-- 1 root root 7437 Feb 23 22:19 log4j2.properties
-rw-r--r-- 1 root root 342 Feb 23 22:19 logstash-sample.conf
-rw-r--r-- 1 root root 12841 Mar 17 17:28 logstash.yml
-rw-r--r-- 1 root root 285 Feb 23 22:19 pipelines.yml
-rw------- 1 root root 1696 Feb 23 22:19 startup.options

/var/log/logstash
drwxr-xr-x 2 logstash root 187 Mar 18 14:36 .
drwxr-xr-x 11 root root 4096 Mar 18 09:48 ..
-rw-r--r-- 1 logstash logstash 10811 Mar 18 13:48 logstash-deprecation.log
-rw-r--r-- 1 logstash logstash 0 Mar 18 10:10 logstash-json.log
-rw-r--r-- 1 logstash logstash 35090 Mar 18 13:48 logstash-plain.log
-rw-r--r-- 1 root root 35090 Mar 18 14:36 logstash-plain-test.log
-rw-r--r-- 1 logstash logstash 0 Mar 18 10:10 logstash-slowlog-json.log
-rw-r--r-- 1 logstash logstash 0 Mar 18 10:10 logstash-slowlog-plain.log

8

curl localhost:9200/global2-*/_search?pretty
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" :
}
}

I have changed the config file to process /var/logstash/logstash-plain.log and maybe has done something now perhaps with the successful 1?
About the previous message and the permissions is anything that is not ok as the config files are root?
(They came by default like this during the installation process)

9

You are not looking into the correct index.

What is global2-*?

Your logstash configuration will send data to the index logstash-*, not global2-*.

You need to run curl localhost:9200/logstash-*/_search?pretty to see if you have data in that index.

10

Hi leandrojmp,

the command was run and I got this:

curl localhost:9200/logstash-*/_search?pretty
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" :
}
}

Just the successful stays in 1, I don't know if it is doing something else or just stopped sending data to Elasticsearch : |

11

check it for more info...

12

Thank you Kunal, my problem now is that there is something taking all the space in Elasticsearch and is not the logstash log file that is being parsed

exceed the high disk watermark when these relocations are complete
[2022-03-23T17:40:06,161][INFO ][o.e.c.r.a.DiskThresholdMonitor] [ip-10-201-1-208.eu-west-2.compute.internal] high disk watermark [90%] no longer exceeded on [r51WwHrKTE-VK6UCAaR4IA][ip-10-201-1-208.eu-west-2.compute.internal][/var/lib/Elasticsearch/nodes/0] free: 879.3mb[10.7%], but low disk watermark [85%] is still exceeded
[2022-03-23T17:41:06,169][INFO ][o.e.c.r.a.DiskThresholdMonitor] [ip-10-201-1-208.eu-west-2.compute.internal] low disk watermark [85%] no longer exceeded on [r51WwHrKTE-VK6UCAaR4IA][ip-10-201-1-208.eu-west-2.compute.internal][/var/lib/Elasticsearch/nodes/0] free: 1.3gb[17.2%]
[2022-03-23T17:42:36,183][INFO ][o.e.c.r.a.DiskThresholdMonitor] [ip-10-201-1-208.eu-west-2.compute.internal] low disk watermark [85%] exceeded on [r51WwHrKTE-VK6UCAaR4IA][ip-10-201-1-208.eu-west-2.compute.internal][/var/lib/Elasticsearch/nodes/0] free: 897.4mb[10.9%], replicas will not be assigned to this node

Any reason why Elasticsearch space gets very low so quickly? or a way to find out from where is coming this issue? once Elasticsearch surpases the 85% stops getting more new indexes.
Logstash seems not to be the reason. : /

Thanks a lot for your help

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.