hi all,
kindly, i installed filebeat, elasticsearch and kibana in my VMware; notably i am not using logstash. After that, i installed X-pack for kibana and elasticsearch,when i start elasticsearch and kibana, kibana console is up and seems ok. But, the point is there is no index in visualize part of kibana console.Actually, i did not set any configuration about x-pack and authentication for filebeat, and just the configurations had been done for elasticsearch and kibana. The filebeat log is as following, i have no idea that how can i set the related configurations for filebeat. Any advise would be so appreciated. thanks
2018-12-26T16:51:01+03:30 INFO retryer: send wait signal to consumer
2018-12-26T16:51:01+03:30 INFO done
2018-12-26T16:51:15+03:30 INFO Non-zero metrics in the last 30s:
beat.info.uptime.ms=30009 beat.memstats.gc_next=4830496 beat.memstats.memory_alloc=4009736 beat.memstats.memory_total=8764760 filebeat.events.active=1440 filebeat.events.added=1444 filebeat.events.done=4 filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.config.module.running=0 libbeat.config.reloads=1 libbeat.output.read.bytes=1850 libbeat.output.type=elasticsearch libbeat.output.write.bytes=605 libbeat.pipeline.clients=2 libbeat.pipeline.events.active=1434 libbeat.pipeline.events.filtered=10 libbeat.pipeline.events.published=1434 libbeat.pipeline.events.retry=100 libbeat.pipeline.events.total=1444 registrar.states.current=2 registrar.states.update=4 registrar.writes=4
2018-12-26T16:51:17+03:30 ERR Failed to connect: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
2018-12-26T16:51:17+03:30 INFO retryer: send unwait-signal to consumer
2018-12-26T16:51:17+03:30 INFO done
2018-12-26T16:51:17+03:30 INFO retryer: send wait signal to consumer
2018-12-26T16:51:17+03:30 INFO done
but still i cannot see any indices in the "index patterns" part of kibana console (i have been logged in using elastic (admin) user).
also i use following command to check the existence of index which still there is no "filebeat" index
GET /_cat/indices
could you please advise me about this issue?
Also, i checked the logs of elasticsearch which everything seems ok and there is no error
[2018-12-29T12:24:09,727][INFO ][o.e.g.GatewayService ] [9sIG28T] recovered [14] indices into cluster_state
[2018-12-29T12:24:09,706][ERROR][o.e.x.s.a.e.NativeUsersStore]
[9sIG28T] failed to retrieve built in user [elastic] info
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.security][doc][reserved-user-elastic]: routing [null]]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:186) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:95) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:59) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:103) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:188) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:182) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:176) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:164) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:190) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:166) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:195) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:228) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:239) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:193) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:147) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:116) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:165) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$2(SecurityActionFilter.java:117) ~[?:?]
at org.elasticsearch.xpack.security.SecurityContext.executeAsUser(SecurityContext.java:105) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:116) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:115) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:497) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:73) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:589) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:213) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:98) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.doAuthenticateAndCache(CachingUsernamePasswordRealm.java:170) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:109) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:94) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:284) ~[?:?]
at org.elasticsearch.xpack.common.IteratingActionListener.run(IteratingActionListener.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:320) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$9(AuthenticationService.java:257) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:266) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$null$0(AuthenticationService.java:201) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:239) ~[x-pack-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:197) ~[x-pack-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:228) ~[x-pack-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:239) [x-pack-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:193) [x-pack-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:147) [x-pack-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:99) [x-pack-6.1.1.jar:6.1.1]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:69) [x-pack-6.1.1.jar:6.1.1]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.1.1.jar:6.1.1]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:497) [transport-netty4-6.1.1.jar:6.1.1]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-6.1.1.jar:6.1.1]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-6.1.1.jar:6.1.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
[2018-12-29T12:24:09,790][ERROR][o.e.x.s.a.e.ReservedRealm] [9sIG28T] failed to retrieve password hash for reserved user [elastic]
[2018-12-29T12:24:09,791][INFO ][o.e.x.s.a.AuthenticationService]
[9sIG28T] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-12-29T12:24:09,852][ERROR][o.e.x.s.a.e.NativeUsersStore] [9sIG28T] failed to retrieve built in user [elastic] info
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.security][doc][reserved-user-elastic]: routing [null]]
if it helps, i changed the password of elastic previously using following command:
/bin/x-pack/setup-passwords interactive
i am new in x-pack, i used elastic user and its password in filebeat.yml (elasticsearch.out part) at first, when it doesn't work i changed it according role and user defined in console.
Also, i used elastic and its password in kibana.yml file, recently i figure out that it is not needed to use elastic user for communication between kibana and elasticsearch; so i change it as kibana user and its password.
i think maybe the elastic user and its password cached in filebeat and kibana, so in order to prevent of confusion due to using elastic user and its password in filebeat.yml and kibana.yml, i change the password of elastic user in kibana console. therefore, now the filebeat and kibana are using different users and passwords and also password of elastic user has been changed too
now, situation is as following:
1- kibana is up, and kibana user is used in kibana.yml (elasticsearch.username and password part)
2- i can login into kibana console using elastic user and its password
3- elastic user is defined in (curl -u elastic 'http://localhost:9200/_xpack/security/_authentication?pretty') but in the elasticsearch logs, it seems that authentication of elastic has been terminated
4- role and user have been defined in kibana console and the username and password of defined user is used in filebeat.yml file, but still i can see (failed to connect: 401 unauthorized error) in filebeat log
I do the steps again, it seems after starting filebeat (which gives us missing authentication error), elasticsearch log have the (failed to retrieve password hash for reserved user [elastic]) error.
could any one advise me about this problem?
version of my elasticsearch, filebeat,x-pack and kibana is 6.1.1. i tried another version of filebeat (6.5) and used following command in filebeat.yml:
xpack.monitoring:
enabled: true
by adding this line into filebeat.yml, filebeat (v6.5) can stash logs to elasticsearch(v6.1.1) and i can see the index in the discover part of console, therefore the problem is not elasticsearch side and is filebeat side. but it doesn't work in filebeat v6.1 because it doesn't support xpack.monitoring.
I want to use filebeat version 6.1.1 because my elasticsearch and kibana version is 6.1.1, according all these explanations, how can i handle this issue?
Could you maybe briefly describe what the actual problem was after all and what steps did you take to resolve this in case this will be helpful for other members of the community facing similar issues?
hi loannis,
Elasticseach could not verify the authentication information of filebeat. Actually, the filebeat writer user and its role had been define in Kibana console and the same defined in filebeat.yml, but something seems wrong and these information in filebeat side and elasticsearch side seems different. after analysis different situations, i figured out that the problem is in filebeat side and the there is something wrong in the way of my username and password definition in filebeat.yml. according to above posts, the filebeat.yml was as following:
after searching, it has been cleared that, filebeat is searching for "output.elasticsearch.username" and "output.elasticsearch.password" for authentication information. as above filebeat.yml, there are no "output.elasticsearch.username" and "output.elasticsearch.password" , actually username and password information is under "setup.template:" which should be under " output.elasticsearch:
". To handle this issue, i changed the filebeat.yml as following which solved the problem:
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.