Denormalize data within a log file

Hi Guy,

We can skip this one. I have rebuilt the container. We'll see if I can break it again.

All was going well until mid-day yesterday. I had sucked most of the files into ES with a reasonably good pipeline.conf. Then bad things started to happen in the front-end. Initially, Kibana showed a locked out login page. With a message like 'Talk to your administrator.' After some stopping and restarting, I saw a bright red bar in the browser at localhost:5601, with message, "Kibana did not load properly. Check the server output for more information.". After more experiments, even that no longer appears.

I'm running the stack in the standard docker build on Mac OSX.

I've done some tests. Restarting Kibana never actually gets it running. Tried starting it both as a service and just running the bin/kibana executable.

netstat -natp | grep 5601 returns nothing.

curl result when trying to hit 5601:

root@4b04dc80a457:/# curl -XGET 'http://localhost:5601/'
curl: (7) Failed to connect to localhost port 5601: Connection refused

Elasticsearch is running:

root@4b04dc80a457:/# curl -XGET 'http://localhost:9200/'
{
  "name" : "AsuTt_m",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "5wN3uC3MTXeKsQ3IvEYKYA",
  "version" : {
    "number" : "6.3.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "eb782d0",
    "build_date" : "2018-06-29T21:59:26.107521Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Here's the verbose error output when starting from bin/kibana

root@4b04dc80a457:/# /opt/kibana/bin/kibana > /var/log/kibana/stdout.txt
    root@4b04dc80a457:/# cat /var/log/kibana/stdout.txt 
    {"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/node_modules/x-pack","message":"Found plugin at /opt/kibana/node_modules/x-pack"}
    {"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/src/core_plugins/console","message":"Found plugin at /opt/kibana/src/core_plugins/console"}

     <snipped - many plugins found>

    {"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/src/core_plugins/tagcloud","message":"Found plugin at /opt/kibana/src/core_plugins/tagcloud"}
    {"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/src/core_plugins/tile_map","message":"Found plugin at /opt/kibana/src/core_plugins/tile_map"}
    {"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/src/core_plugins/timelion","message":"Found plugin at /opt/kibana/src/core_plugins/timelion"}
    {"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/src/core_plugins/vega","message":"Found plugin at /opt/kibana/src/core_plugins/vega"}
    {"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["info","optimize"],"pid":419,"message":"Optimizing and caching bundles for ml, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, dashboardViewer, apm and kibana. This may take a few minutes"}
    {"type":"ops","@timestamp":"2018-07-26T15:19:34Z","tags":[],"pid":419,"os":{"load":[0.54296875,0.30810546875,0.26904296875],"mem":{"total":4139020288,"free":1063202816},"uptime":465537},"proc":{"uptime":7.073,"mem":{"rss":235671552,"heapTotal":191827968,"heapUsed":127149536,"external":236281},"delay":64.58339202404022},"load":{"requests":{},"concurrents":{"5601":0},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 121.3MB uptime: 0:00:07 load: [0.54 0.31 0.27] delay: 64.583"}
    {"type":"ops","@timestamp":"2018-07-26T15:19:39Z","tags":[],"pid":419,"os":{"load":[0.57958984375,0.31982421875,0.2734375],"mem":{"total":4139020288,"free":1001254912},"uptime":465542},"proc":{"uptime":12.082,"mem":{"rss":296873984,"heapTotal":252522496,"heapUsed":206178264,"external":1599328},"delay":239.71886104345322},"load":{"requests":{},"concurrents":{"5601":0},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 196.6MB uptime: 0:00:12 load: [0.58 0.32 0.27] delay: 239.719"}
    {"type":"ops","@timestamp":"2018-07-26T15:19:47Z","tags":[],"pid":419,"os":{"load":[0.64453125,0.3427734375,0.28125],"mem":{"total":4139020288,"free":952623104},"uptime":465550},"proc":{"uptime":19.877,"mem":{"rss":343646208,"heapTotal":295858176,"heapUsed":202410656,"external":4309377},"delay":48.28747296333313},"load":{"requests":{},"concurrents":{"5601":0},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 193.0MB uptime: 0:00:20 load: [0.64 0.34 0.28] delay: 48.287"}

My google searches haven't yielded anything helpful. Maybe you can suggest some paths to explore.

Best,

Jim

I'm tempted to just pull our precious pipeline and beats configuration files out of the container, and start over with a fresh, new container!

Dissect has your back, after the mapping setting add:

    convert_datatype => {
      "[qso][freq]" => "int"
    }

RE: Kibana, you should post a comment in the Kibana Discuss channel. https://discuss.elastic.co/c/kibana

Thanks for all your help, Guy . . . I'm moving along pretty well now. Now attacking the query learning curve . . .

Best,

Jim

Hey Jim,

Its been a pleasure. Very interesting project. I'm happy to have helped.

Guy

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.