Hi
I was using filebeat 7.7.1 with Elasticsearch 7.8.1 and had no problem with collecting iis and nginx logs.
But, when I upgraded filebeat from 7.7.1 to 7.14.0 I got these errors logs from Elasticsearch.
{"type": "server", "timestamp": "2021-09-06T03:19:33,235Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "heimdall-storage-prod1", "node.name": "heimdall-storage-prod1-es-hot-2", "message": "path: /_ingest/pipeline/filebeat-7.14.0-nginx-access-pipeline, params: {id=filebeat-7.14.0-nginx-access-pipeline}", "cluster.uuid": "l84hoNj0TaiFLfyQYedx7Q", "node.id": "8oTIBWjLSB6zzx6Nesmvyw" ,
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [heimdall-storage-prod1-es-master-1][10.244.188.161:9300][cluster:admin/ingest/pipeline/put]",
"Caused by: org.elasticsearch.ElasticsearchException: org.elasticsearch.common.xcontent.XContentParseException: [1:2] [script] unknown field [description]",
"at org.elasticsearch.ExceptionsHelper.convertToElastic(ExceptionsHelper.java:66) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:322) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:441) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:398) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:336) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:74) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.ingest.IngestService.validatePipeline(IngestService.java:322) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.ingest.IngestService.putPipeline(IngestService.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.action.ingest.PutPipelineTransportAction.lambda$masterOperation$0(PutPipelineTransportAction.java:83) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:633) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1163) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:213) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:205) [elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:117) [elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) [elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:73) [transport-netty4-client-7.8.1.jar:7.8.1]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at java.lang.Thread.run(Thread.java:832) [?:?]",
"Caused by: java.lang.IllegalArgumentException: [1:2] [script] unknown field [description]",
"at org.elasticsearch.common.xcontent.ObjectParser.lambda$errorOnUnknown$2(ObjectParser.java:102) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]",
"at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:297) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]",
"at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:256) ~[elasticsearch-x-content-7.8.1.jar:7.8.1]",
So, I tried to reconfigure pipeline using the following code, I got same error.
However It worked witout any error with Elasticsearch 7.13.1
Is there any hints for this problem
docker run docker.elastic.co/beats/filebeat:7.14.0 setup --pipelines --modules nginx,iis \
-E logging.level=debug \
-E output.elasticsearch.hosts=["elasticsearchurl:9200"] \
-E output.elasticsearch.protocol=https \
-E output.elasticsearch.username=elastic \
-E output.elasticsearch.password=password \
-E output.elasticsearch.ssl.verification_mode=none
Exiting: 1 error: error loading pipeline for fileset nginx/ingress_controller: couldn't load pipeline: couldn't load json. Error: 500 Internal Server Error: {"error":{"root_cause":[{"type":"exception","reason":"org.elasticsearch.common.xcontent.XContentParseException: [1:2] [script] unknown field [description]","processor_type":"script"}],"type":"exception","reason":"org.elasticsearch.common.xcontent.XContentParseException: [1:2] [script] unknown field [description]","processor_type":"script","caused_by":{"type":"illegal_argument_exception","reason":"[1:2] [script] unknown field [description]"}},"status":500}. Response body: {"error":{"root_cause":[{"type":"exception","reason":"org.elasticsearch.common.xcontent.XContentParseException: [1:2] [script] unknown field [description]","processor_type":"script"}],"type":"exception","reason":"org.elasticsearch.common.xcontent.XContentParseException: [1:2] [script] unknown field [description]","processor_type":"script","caused_by":{"type":"illegal_argument_exception","reason":"[1:2] [script] unknown field [description]"}},"status":500}