# PacketBeat timestamp

**URL:** <https://discuss.elastic.co/t/packetbeat-timestamp/84581>\
**Category:** Beats\
**Tags:** packetbeat\
**Created:** [May 4, 2017, 2:43pm UTC](https://discuss.elastic.co/t/packetbeat-timestamp/84581 "2017-05-04T14:43:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![gcorgne](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@gcorgne](https://discuss.elastic.co/u/gcorgne)\
**Post date:** [May 4, 2017, 2:43pm UTC](https://discuss.elastic.co/t/packetbeat-timestamp/84581/1 "2017-05-04T14:43:20Z")

</div>

hi,  
We oversee a network interface that is close to 2 gigabits per second.  
All our servers are interfaced at 10 Gbps. Currently, we have only one node elastic search: 4x16 core and 256 Gb of RAM.  
We do not understand why flow seems to be ingested in blocks and not live.

 ![](https://us1.discourse-cdn.com/elastic/original/3X/b/a/bae3b7cee3402974faac726a4bf9fa0f3798688d.png)  
If you have an idee...  
thanks  
Gaby

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [May 4, 2017, 3:00pm UTC](https://discuss.elastic.co/t/packetbeat-timestamp/84581/2 "2017-05-04T15:00:51Z")

</div>

Flow events are send based on the configured [`period`](https://www.elastic.co/guide/en/beats/packetbeat/current/configuration-flows.html#_period). You get intermediate updates about the flow every N seconds. Then you get a [final](https://www.elastic.co/guide/en/beats/packetbeat/current/exported-fields-flows_event.html#_final) event when the flow completes.

You can decrease the period for a more live view.

---

<div class="post-metadata">

**Author:** ![gcorgne](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@gcorgne](https://discuss.elastic.co/u/gcorgne)\
**Post date:** [May 5, 2017, 6:45am UTC](https://discuss.elastic.co/t/packetbeat-timestamp/84581/3 "2017-05-05T06:45:50Z")

</div>

I think we have an other problem. our configuration period is  
period: 1s  
when the traffic is slow, it's ok but when it increases, it's ingested in blocks like this :

 ![](https://us1.discourse-cdn.com/elastic/original/3X/1/7/173c0079722360035f06599bdf580660fdbc306d.png)

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [May 5, 2017, 4:29pm UTC](https://discuss.elastic.co/t/packetbeat-timestamp/84581/4 "2017-05-05T16:29:37Z")

</div>

I wonder if there are issues with the output keeping up with the amount of data being sent by Packetbeat. Are you going directly to Elasticsearch? Maybe you are getting some bulk rejections? Anything in the logs (there a 30 second periodic metric dump in the beat log)? Anything interesting from `GET _nodes/stats` in ES?

---

<div class="post-metadata">

**Author:** ![gcorgne](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@gcorgne](https://discuss.elastic.co/u/gcorgne)\
**Post date:** [May 9, 2017, 6:27am UTC](https://discuss.elastic.co/t/packetbeat-timestamp/84581/5 "2017-05-09T06:27:45Z")

</div>

Hi Andrew,

"I wonder if there are issues with the output keeping up with the amount  
of data being sent by Packetbeat. Are you going directly to Elasticsearch?"

Yes

"Maybe you are getting some bulk rejections? Anything in the logs  
(there a 30 second periodic metric dump in the beat log)?"

Bulk ?  
/2017-05-09T08:23:51+02:00 INFO Error publishing events (retrying):  
temporary bulk send failure/

We have this info every 30s :  
/2017-05-09T08:21:12+02:00 INFO Non-zero metrics in the last 30s:  
beat.memstats.gc\_next=1331232 beat.memstats.memory\_alloc=-37092840  
beat.memstats.memory\_total=229741960  
output.elasticsearch.events.acked=17808  
output.elasticsearch.publishEvents.call.count=359  
output.elasticsearch.read.bytes=183295  
output.elasticsearch.write.bytes=11037292 output.events.acked=17808  
output.write.bytes=11037292 publisher.events.count=17653  
publisher.queue.messages.count=30//  
/  
Anything interesting from |GET \_nodes/stats| in ES?"

I join the output

---

<div class="post-metadata">

**Author:** ![gcorgne](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@gcorgne](https://discuss.elastic.co/u/gcorgne)\
**Post date:** [May 9, 2017, 6:31am UTC](https://discuss.elastic.co/t/packetbeat-timestamp/84581/6 "2017-05-09T06:31:18Z")

</div>

* * *

## { "\_nodes" : { "total" : 1, "successful" : 1, "failed" : 0 }, "cluster\_name" : "squid-eple", "nodes" : { "OmG2KhT3QDC7tTXXAySLhw" : { "timestamp" : 1494311032698, "name" : "elastic", "transport\_address" : "127.0.0.1:9300", "host" : "127.0.0.1", "ip" : "127.0.0.1:9300", "roles" : ["master", "data", "ingest"], "indices" : { "docs" : { "count" : 741900111, "deleted" : 13 }, "store" : { "size\_in\_bytes" : 416233754146, "throttle\_time\_in\_millis" : 0 }, "indexing" : { "index\_total" : 112629752, "index\_time\_in\_millis" : 141861806, "index\_current" : 0, "index\_failed" : 0, "delete\_total" : 0, "delete\_time\_in\_millis" : 0, "delete\_current" : 0, "noop\_update\_total" : 0, "is\_throttled" : false, "throttle\_time\_in\_millis" : 0 }, "get" : { "total" : 25, "time\_in\_millis" : 16, "exists\_total" : 25, "exists\_time\_in\_millis" : 16, "missing\_total" : 0, "missing\_time\_in\_millis" : 0, "current" : 0 }, "search" : { "open\_contexts" : 0, "query\_total" : 678210, "query\_time\_in\_millis" : 1930111, "query\_current" : 0, "fetch\_total" : 405289, "fetch\_time\_in\_millis" : 4370404, "fetch\_current" : 0, "scroll\_total" : 0, "scroll\_time\_in\_millis" : 0, "scroll\_current" : 0, "suggest\_total" : 0, "suggest\_time\_in\_millis" : 0, "suggest\_current" : 0 }, "merges" : { "current" : 0, "current\_docs" : 0, "current\_size\_in\_bytes" : 0, "total" : 219414, "total\_time\_in\_millis" : 204593627, "total\_docs" : 2295167592, "total\_size\_in\_bytes" : 1208995640184, "total\_stopped\_time\_in\_millis" : 0, "total\_throttled\_time\_in\_millis" : 12909675, "total\_auto\_throttle\_in\_bytes" : 10660626671 }, "refresh" : { "total" : 1843832, "total\_time\_in\_millis" : 67139879, "listeners" : 0 }, "flush" : { "total" : 974, "total\_time\_in\_millis" : 922366 }, "warmer" : { "current" : 0, "total" : 1844694, "total\_time\_in\_millis" : 742127 }, "query\_cache" : { "memory\_size\_in\_bytes" : 0, "total\_count" : 187483, "hit\_count" : 54, "miss\_count" : 187429, "cache\_size" : 0, "cache\_count" : 103, "evictions" : 103 }, "fielddata" : { "memory\_size\_in\_bytes" : 0, "evictions" : 0 }, "completion" : { "size\_in\_bytes" : 0 }, "segments" : { "count" : 6706, "memory\_in\_bytes" : 1207499371, "terms\_memory\_in\_bytes" : 1037726383, "stored\_fields\_memory\_in\_bytes" : 108435888, "term\_vectors\_memory\_in\_bytes" : 0, "norms\_memory\_in\_bytes" : 6369664, "points\_memory\_in\_bytes" : 16441476, "doc\_values\_memory\_in\_bytes" : 38525960, "index\_writer\_memory\_in\_bytes" : 36215380, "version\_map\_memory\_in\_bytes" : 113612, "fixed\_bit\_set\_memory\_in\_bytes" : 0, "max\_unsafe\_auto\_id\_timestamp" : 1494115208511, "file\_sizes" : { } }, "translog" : { "operations" : 1611641, "size\_in\_bytes" : 1396188410 }, "request\_cache" : { "memory\_size\_in\_bytes" : 367567, "evictions" : 0, "hit\_count" : 108536, "miss\_count" : 13114 }, "recovery" : { "current\_as\_source" : 0, "current\_as\_target" : 0, "throttle\_time\_in\_millis" : 0 } }, "os" : { "timestamp" : 1494311033144, "cpu" : { "percent" : 7, "load\_average" : { "1m" : 20.81, "5m" : 16.77, "15m" : 11.68 } }, "mem" : { "total\_in\_bytes" : 271019524096, "free\_in\_bytes" : 137014861824, "used\_in\_bytes" : 134004662272, "free\_percent" : 51, "used\_percent" : 49 }, "swap" : { "total\_in\_bytes" : 0, "free\_in\_bytes" : 0, "used\_in\_bytes" : 0 } }, "process" : { "timestamp" : 1494311033145, "open\_file\_descriptors" : 2003, "max\_file\_descriptors" : 65536, "cpu" : { "percent" : 3, "total\_in\_millis" : 330584720 }, "mem" : { "total\_virtual\_in\_bytes" : 473911775232 } }, "jvm" : { "timestamp" : 1494311033147, "uptime\_in\_millis" : 335999627, "mem" : { "heap\_used\_in\_bytes" : 15031914264, "heap\_used\_percent" : 44, "heap\_committed\_in\_bytes" : 33984610304, "heap\_max\_in\_bytes" : 33984610304, "non\_heap\_used\_in\_bytes" : 129616528, "non\_heap\_committed\_in\_bytes" : 138563584, "pools" : { "young" : { "used\_in\_bytes" : 931525520, "max\_in\_bytes" : 3001090048, "peak\_used\_in\_bytes" : 3001090048, "peak\_max\_in\_bytes" : 3001090048 }, "survivor" : { "used\_in\_bytes" : 118319576, "max\_in\_bytes" : 375128064, "peak\_used\_in\_bytes" : 375128064, "peak\_max\_in\_bytes" : 375128064 }, "old" : { "used\_in\_bytes" : 13982069168, "max\_in\_bytes" : 30608392192, "peak\_used\_in\_bytes" : 13982069168, "peak\_max\_in\_bytes" : 30608392192 } } }, "threads" : { "count" : 235, "peak\_count" : 243 }, "gc" : { "collectors" : { "young" : { "collection\_count" : 16740, "collection\_time\_in\_millis" : 1250246 }, "old" : { "collection\_count" : 1, "collection\_time\_in\_millis" : 370 } } }, "buffer\_pools" : { "direct" : { "count" : 819, "used\_in\_bytes" : 1111517508, "total\_capacity\_in\_bytes" : 1111517507 }, "mapped" : { "count" : 14377, "used\_in\_bytes" : 415438488151, "total\_capacity\_in\_bytes" : 415438488151 } }, "classes" : { "current\_loaded\_count" : 11521, "total\_loaded\_count" : 11521, "total\_unloaded\_count" : 0 } },

---

<div class="post-metadata">

**Author:** ![gcorgne](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@gcorgne](https://discuss.elastic.co/u/gcorgne)\
**Post date:** [May 9, 2017, 6:32am UTC](https://discuss.elastic.co/t/packetbeat-timestamp/84581/7 "2017-05-09T06:32:15Z")

</div>

* * *

```
  "thread_pool" : {
    "bulk" : {
      "threads" : 32,
      "queue" : 0,
      "active" : 5,
      "rejected" : 50796,
      "largest" : 32,
      "completed" : 11906818
    },
    "fetch_shard_started" : {
      "threads" : 1,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 64,
      "completed" : 440
    },
    "fetch_shard_store" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "flush" : {
      "threads" : 2,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 5,
      "completed" : 1823
    },
    "force_merge" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "generic" : {
      "threads" : 4,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 4,
      "completed" : 34144
    },
    "get" : {
      "threads" : 25,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 25,
      "completed" : 25
    },
    "index" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "listener" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "management" : {
      "threads" : 5,
      "queue" : 0,
      "active" : 1,
      "rejected" : 0,
      "largest" : 5,
      "completed" : 431476
    },
    "refresh" : {
      "threads" : 10,
      "queue" : 0,
      "active" : 1,
      "rejected" : 0,
      "largest" : 10,
      "completed" : 21738340
    },
    "search" : {
      "threads" : 49,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 49,
      "completed" : 1219097
    },
    "snapshot" : {
      "threads" : 0,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 0,
      "completed" : 0
    },
    "warmer" : {
      "threads" : 4,
      "queue" : 0,
      "active" : 0,
      "rejected" : 0,
      "largest" : 4,
      "completed" : 1844809
    }
  },
  "fs" : {
    "timestamp" : 1494311033149,
    "total" : {
      "total_in_bytes" : 1266243043328,
      "free_in_bytes" : 836408279040,
      "available_in_bytes" : 772063252480,
      "spins" : "true"
    },
    "data" : [
      {
        "path" : "/home/elastic/nodes/0",
        "mount" : "/home (/dev/mapper/tallud--vg-home)",
        "type" : "ext4",
        "total_in_bytes" : 1266243043328,
        "free_in_bytes" : 836408279040,
        "available_in_bytes" : 772063252480,
        "spins" : "true"
      }
    ],
    "io_stats" : {
      "devices" : [
        {
          "device_name" : "dm-4",
          "operations" : 112820992,
          "read_operations" : 53148,
          "write_operations" : 112767844,
          "read_kilobytes" : 6197576,
          "write_kilobytes" : 1575305684
        }
      ],
      "total" : {
        "operations" : 112820992,
        "read_operations" : 53148,
        "write_operations" : 112767844,
        "read_kilobytes" : 6197576,
        "write_kilobytes" : 1575305684
      }
    }
  },
  "transport" : {
    "server_open" : 0,
    "rx_count" : 0,
    "rx_size_in_bytes" : 0,
    "tx_count" : 0,
    "tx_size_in_bytes" : 0
  },
  "http" : {
    "current_open" : 73,
    "total_opened" : 6592
  },
  "breakers" : {
    "request" : {
      "limit_size_in_bytes" : 20390766182,
      "limit_size" : "18.9gb",
      "estimated_size_in_bytes" : 0,
      "estimated_size" : "0b",
      "overhead" : 1.0,
      "tripped" : 0
    },
    "fielddata" : {
      "limit_size_in_bytes" : 20390766182,
      "limit_size" : "18.9gb",
      "estimated_size_in_bytes" : 0,
      "estimated_size" : "0b",
      "overhead" : 1.03,
      "tripped" : 0
    },
    "in_flight_requests" : {
      "limit_size_in_bytes" : 33984610304,
      "limit_size" : "31.6gb",
      "estimated_size_in_bytes" : 30262,
      "estimated_size" : "29.5kb",
      "overhead" : 1.0,
      "tripped" : 0
    },
    "parent" : {
      "limit_size_in_bytes" : 23789227212,
      "limit_size" : "22.1gb",
      "estimated_size_in_bytes" : 30262,
      "estimated_size" : "29.5kb",
      "overhead" : 1.0,
      "tripped" : 0
    }
  },
  "script" : {
    "compilations" : 0,
    "cache_evictions" : 0
  },
  "discovery" : {
    "cluster_state_queue" : {
      "total" : 0,
      "pending" : 0,
      "committed" : 0
    }
  },
  "ingest" : {
    "total" : {
      "count" : 3957832,
      "time_in_millis" : 12146,
      "current" : 0,
      "failed" : 0
    },
    "pipelines" : {
      "xpack_monitoring_2" : {
        "count" : 0,
        "time_in_millis" : 0,
        "current" : 0,
        "failed" : 0
      },
      "proxyecole" : {
        "count" : 3957832,
        "time_in_millis" : 12146,
        "current" : 0,
        "failed" : 0
      },
      "test_proxyecole" : {
        "count" : 0,
        "time_in_millis" : 0,
        "current" : 0,
        "failed" : 0
      },
      "squid-eple" : {
        "count" : 0,
        "time_in_millis" : 0,
        "current" : 0,
        "failed" : 0
      }
    }
  }
}

```

## } }

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [May 9, 2017, 1:35pm UTC](https://discuss.elastic.co/t/packetbeat-timestamp/84581/8 "2017-05-09T13:35:01Z")

</div>

1s for flows is very small. The idea of flows is to 'compress' the information by computing counters/summaries to be published at a higher interval. By decreasing the period from 10s to 1s, you multiplied the amount of events by 10... On a GB interface. All flows are reported at exactly the same point in time. That is, counters of multiple flows can be summed together.

The message `/2017-05-09T08:23:51+02:00 INFO Error publishing events (retrying): temporary bulk send failure/` indicates a bulk publish being rejected by ES. Have you checked ES logs? Reduce events to be published and/or introduce some queuing (e.g. redis) to buffer up events. On ES side you can also try to tune? E.g. number of shards/replicas. Note: when using some queue between packetbeat and ES, you want the consumer to have a chance to catch up. When traffic is high all the time, try to reduce amount of data being generate or tune/improve ingestion.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 6, 2017, 1:46pm UTC](https://discuss.elastic.co/t/packetbeat-timestamp/84581/9 "2017-06-06T13:46:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.