aws-kinesis-agent错误


宪法

  • 常见模式

    • 从AP服务器部署并发送到kinesis-firehose，并存储在S3中

Untitled.png

环境

  • aws-kinesis-agent-1.1.2-1.amzn1.noarch

事件

  • 在构建时没有问题，但是在旋转获取目标日志时发生了错误。
  • 目标日志

    • /var/app/current/log/production.log

      • 日志

/var/log/aws-kinesis-agent/aws-kinesis-agent.log

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Initialization logs can be found in /tmp/aws-kinesis-agent.20170301165118.initlog
Unhandled exception during startup.
java.lang.IllegalStateException: Expected the service to be RUNNING, but the service has FAILED
    at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:285)
    at com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:229)
    at com.google.common.util.concurrent.AbstractExecutionThreadService.awaitRunning(AbstractExecutionThreadService.java:197)
    at com.amazon.kinesis.streaming.agent.Agent.startUp(Unknown Source)
    at com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54)
    at com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
    at com.google.common.util.concurrent.MoreExecutors$2.execute(MoreExecutors.java:849)
    at com.google.common.util.concurrent.AbstractIdleService$2.doStart(AbstractIdleService.java:50)
    at com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:174)
    at com.google.common.util.concurrent.AbstractIdleService.startAsync(AbstractIdleService.java:135)
    at com.amazon.kinesis.streaming.agent.Agent.main(Unknown Source)
Caused by: java.lang.IllegalStateException
    at com.google.common.base.Preconditions.checkState(Preconditions.java:158)
    at com.amazon.kinesis.streaming.agent.tailing.TrackedFileRotationAnalyzer.findCurrentOpenFileAfterTruncate(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.updateCurrentFile(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.initialize(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.FileTailer.initialize(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.FileTailer.startUp(Unknown Source)
    at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:55)
    at com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
    at java.lang.Thread.run(Thread.java:745)
FATAL: Thread FileTailer[fh:************:/var/app/current/log/production.log] threw an unrecoverable error. Aborting application

您可以看到在旋转过程中发生了一些事情。

原因

  • Kinesis Agent记录在每个文件模式的检查点/ var / run / aws-kinesis-agent /检查点中，如果跟踪目标是logrotate和copytruncate，则与Kinesis代理记录的检查点的一致性是正确的，似乎发生了错误因为它无法删除。

/etc/logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.webapp.conf

1
2
3
4
5
6
7
8
9
10
11
/var/app/containerfiles/logs/* {
 size 10M
 rotate 5
 missingok
 compress
 notifempty
 copytruncate
 dateext
 dateformat %s
 olddir /var/app/containerfiles/logs/rotated
 }

对策

  • 更改日志输出目的地
  • 更改copytruncate以复制或创建
  • 在filePattern中的目标文件末尾添加*

    • 补充

    • 可以通过删除/ var / run / aws-kinesis-agent / checkpoints重新启动aws-kinesis-agent
    • 由于这是一个问题，请期待更新

      • LogRotate#50上的异常