Logs are not displaying in order

Elastic Stack Logstash
21

I have added the timestamp like below.
date{
match => {"timestamp" , "ISO8601"}
}
I dont find the difference.

Your events don't even have a timestamp field so the pattern doesn't matter, but let's get the multiline codec working first.

22

Confusion....:innocent:

where i would find out the stdout output?

Regards
Raja

23

Cool..
As per your suggestion I have made Multiline codec but still seeing the output same.

Regards
Raja

24

I would recommend creating a small file that contains a few events and use this while debugging. Remove the elastic search plugin and start Logstash in the foreground. Stdout output should now appear on your screen.

25

Hi Christian,

I have done fresh setup with x-pack because i pulled the elk docker image into my server. Now i am seeing the stdout logs below.

[2017-03-30T09:00:28,675][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x22591b7a URL:http://localhost:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

Even this is there i am able to see the logs in Kibana.

Regards
Raja

26

Please do what we ask and remove the elasticsearch output from your configuration. We only want to see the result of the stdout { codec => rubydebug } output.

27

Hi Magnus,

Its a docker image so we have taken elkx completed container due to which it came with x-pack as well. I removed everything and made the following configuration.

input {
file{
path=> "/etc/logstash/*.log"
}
}
filter{
}
output{
stdout { codec => rubydebug }
}

Now i am seeing below error but i am able to see the logs.

[2017-03-30T09:55:08,155][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-03-30T09:55:11,860][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2017-03-30T09:55:11,990][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x78eeee50 URL:http://elastic:xxxxxx@localhost:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2017-03-30T09:55:16,992][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2017-03-30T09:55:17,112][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x3e5cc2ee URL:http://elastic:xxxxxx@localhost:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

Regards
Raja

28

You clearly haven't removed all elasticsearch outputs from your configuration. Are you pointing Logstash to a directory with configuration files? Keep in mind that Logstash reads all files in that directory.

29

There is only one file i.e., POC.conf file which I have set the path in logstash.yml file

30

Well, as the error message clearly shows Logstash somehow picks up an elasticsearch output. Exactly what does your logstash.yml look like (format the file as preformatted text)? What's Logstash's command line (check with e.g. ps aux)?

31

path.config: /etc/logstash/conf.d/Pipeline

Only above property i have changed to assign the path rest all configuration commented. As i told x-pack already installed by default since it is docker container.

32

So if you have /etc/logstash/conf.d/Pipeline in your logstash.yml, what's this POC.conf you're talking about?

33

The POC.conf having below.

input {
file{
path=> "/etc/logstash/*.log"
}
}
filter{
}
output{
stdout { codec => rubydebug }
}

34

Yes, but what makes Logstash use your POC.conf when you're listing /etc/logstash/conf.d/Pipeline in logstash.yml? Is the latter a directory? If yes, are there any other files in that directory? And again, what's Logstash's command line? Please answer all our questions. It's getting very tiresome to task the same question again and again.

1 Like
35

what makes Logstash use your POC.conf when you're listing /etc/logstash/conf.d/Pipeline in logstash.yml?

Its a conf file where it can read the configuration.

are there any other files in that directory?

No other files listed there. I have created only one file i.e., POC.conf.

what's Logstash's command line?
I am not understanding of this question. If i am not wrong i am using bash shell.

Regards
Raja

36

Its a conf file where it can read the configuration.

You mean it's a directory?

I am not understanding of this question. If i am not wrong i am using bash shell.

Use ps aux and locate the Logstash process.

37

Yes, its a directory.

-bash-4.2# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 125564 4044 ? Ss Mar14 0:25 /usr/lib/system
root 2 0.0 0.0 0 0 ? S Mar14 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Mar14 0:20 [ksoftirqd/0]

38

Surely you have more processes than those three?

39

Yes there are near to 100 processes

kibana 12220 0.3 1.0 1062980 84044 pts/1 Sl 04:27 0:08 /usr/share/kiba
root 42214 0.0 0.4 877624 33464 ? Sl Mar30 0:05 nautilus --no-d
root 50736 0.0 0.5 2427460 41044 ? Sl Mar21 1:08 /opt/IBM/ibm-ja
root 53412 0.0 0.0 361172 3904 ? Ssl Mar30 0:00 /usr/sbin/rsysl
logstash 56599 1.2 6.1 3653792 498544 ? SNsl Mar29 85:37 /usr/bin/java -
elastic+ 61876 0.0 29.6 4792968 2421760 ? Ssl Mar29 6:59 /bin/java -Xms2

40

Then please show the line for the Logstash process. PID 56599, it looks like. Show the whole line and not the truncated one above.