Index not being created in Elastic search - S3 as input -

Elastic Stack Logstash
1

please find the trace .. iam testing out S3 as input have Cloudtrail events in S3 buckets from all accounts. trying to ingest to ES. LS doesnt create index.

2

My conf file

input {
s3 {
bucket => "xyzzz-cloudtrail-logs"
aws_credentials_file => "/root/logstash-5.3.0/config/aws_credentials.yaml"
prefix => "AWSLogs/192006145812/CloudTrail"
add_field => { source => gzfiles }
codec => cloudtrail {}
sincedb_path => "/dev/null"
type => "cloud_trail_log"
interval => 30
region => "us-east-1"
}

}
output {
stdout { codec => json_lines }
elasticsearch {
hosts => ["host:9200"]
action =>"index"
index => "logstash-%{+YYYY.MM.dd}"
}
}

3

[2017-12-31T09:58:58,799][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [blackbox-a] failed to put mappings on indices [[[cloudtrail_logstash_2/PffmrcHhQEG9sW192v9NqQ]]], type [cloudtrail]
java.lang.IllegalArgumentException: Limit of total fields [1000] in index [cloudtrail_logstash_2] has been exceeded
at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:604) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:420) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:336) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:268) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:311) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.5.1.jar:5.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

4

You will need to increase the limit or trim your document size.

5

thanks for quick reply. Sorry for confusion. lIMIT OF TOTAL FIELDS TO 1000 was addressed and i still dont see index being created. Above is my configuration file and the debug trace i provided.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.