logrotate对日志压缩转储

在开发任务比较紧急的情况下,很容易会把功能优先级提高,而把可靠性能力放低,这都无可厚非,但是还是需要有计划的将可靠性问题排在计划中。本文写作的目的就是因为日志功能的可靠性能力一拖再拖导致最终发生了影响功能的情况,回顾时进行总结,以免后续再发生类似情况。

引子

日志对于一个正常的项目来说是必不可少的,不管是开发过程中的调试、后面的运维。理论上来说为了更好的运维要打尽可能多的日志,这就会有一个问题:日志打的太多而不设置大小限制有可能会把磁盘占潢,导致一些其他的问题,有可能进程无法运行甚至down机。个人项目很难出现这种情况,主要是个人项目很少长时间运行。

当然这种问题当前也有很多成熟的日志框架像log4j/logback等等(这里的项目是使用java写的,所以这里只举了java语言的日志框架)都可以解决。但是有一些日志是这些日志框架无法管理的,像java进程的crash日志、java进程的所有线程调用栈日志(一般打到console中,可以设置打印到文件中)。

这种日志框架无法管理的日志,有两种方法可以解决:1,自己实现一个日志框架;2,使用第三方工具,例如本文要说的logrotate工具。

logrotate

logrotate工具是一个类Unix系统下的日志压缩转储工具。可以通过用户自定义配置规则实现对日志的压缩转储。确保日志一个上限。

对于logrotate工具的介绍网上有很多文章,这里记录一下logrotate的配置项及相应说明,以方便后面查看

命令参数

这里有一个注意事项:logrotate有一个-f参数,看描述是强制执行一次logrotate动作,但是这个动作经过测试在root用户和非root用户下执行有区别:非root用户下执行时,不管配置文件中有没有配置size(就是达到多大的时候进行压缩)选项,配置的不管多大好像都是不生效的,去掉-f选项就OK了,root用户下没有这个问题。在网上搜索了很久没也找到原因。这个在实际使用时要注意下。

常用的方法:logrotate <logrotate_config_file> -s <logrotate_status_file>,详细的参数说明如下

-d, --debug
Turns on debug mode and implies -v. In debug mode, no changes will be made to the logs or to the logrotate state file.
-f, --force
Tells logrotate to force the rotation, even if it doesn't think this is necessary. Sometimes this is useful after adding new entries to a logrotate config file, or if old log files have been removed by hand, as the new files will be created, and logging will continue correctly.
-m, --mail <command>
Tells logrotate which command to use when mailing logs. This command should accept two arguments: 1) the subject of the message, and 2) the recipient. The command must then read a message on standard input and mail it to the recipient. The default mail command is /bin/mail -s.
-s, --state <statefile>
Tells logrotate to use an alternate state file. This is useful if logrotate is being run as a different user for various sets of log files. The default state file is /var/lib/logrotate.status.
--usage
Prints a short usage message.
--?, --help
-Prints help message.
-v, --verbose
Turns on verbose mode.

logrotate配置项

logrotate配置文件中可以使用的配置项有很多,组合起来使用可以满足用户很多场景的需要,并且还可以执行用户提供的脚本(或者命令)。详细的描述如下:

Here is more information on the directives which may be included in a logrotate configuration file:

compress
Old versions of log files are compressed with gzip(1) by default. See also nocompress.
compresscmd
Specifies which command to use to compress log files. The default is gzip. See also compress.
uncompresscmd
Specifies which command to use to uncompress log files. The default is gunzip.
compressext
Specifies which extension to use on compressed logfiles, if compression is enabled. The default follows that of the configured compression command.
compressoptions
Command line options may be passed to the compression program, if one is in use. The default, for gzip(1), is "-9" (maximum compression).
copy
Make a copy of the log file, but don't change the original at all. This option can be used, for instance, to make a snapshot of the current log file, or when some other utility needs to truncate or parse the file. When this option is used, the create option will have no effect, as the old log file stays in place.

copytruncate
Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one. It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the create option will have no effect, as the old log file stays in place.
create mode owner group
Immediately after rotation (before the postrotate script is run) the log file is created (with the same name as the log file just rotated). mode specifies the mode for the log file in octal (the same as chmod(2)), owner specifies the user name who will own the log file, and group specifies the group the log file will belong to. Any of the log file attributes may be omitted, in which case those attributes for the new file will use the same values as the original log file for the omitted attributes. This option can be disabled using the nocreate option.
daily
Log files are rotated every day.

dateext
Archive old versions of log files adding a daily extension like YYYYMMDD instead of simply adding a number. The extension may be configured using the dateformat option.
dateformat format_string
Specify the extension for dateext using the notation similar to strftime(3) function. Only %Y %m %d and %s specifiers are allowed. The default value is -%Y%m%d. Note that also the character separating log name from the extension is part of the dateformat string. The system clock must be set past Sep 9th 2001 for %s to work correctly. Note that the datestamps generated by this format must be lexically sortable (i.e., first the year, then the month then the day. e.g., 2001/12/01 is ok, but 01/12/2001 is not, since 01/11/2002 would sort lower while it is later). This is because when using the rotate option, logrotate sorts all rotated filenames to find out which logfiles are older and should be removed.
delaycompress
Postpone compression of the previous log file to the next rotation cycle. This only has effect when used in combination with compress. It can be used when some program cannot be told to close its logfile and thus might continue writing to the previous log file for some time.
extension ext
Log files with ext extension can keep it after the rotation. If compression is used, the compression extension (normally .gz) appears after ext. For example you have a logfile named mylog.foo and want to rotate it to mylog.1.foo.gz instead of mylog.foo.1.gz.
ifempty
Rotate the log file even if it is empty, overriding the notifempty option (ifempty is the default).
include file_or_directory
Reads the file given as an argument as if it was included inline where the include directive appears. If a directory is given, most of the files in that directory are read in alphabetic order before processing of the including file continues. The only files which are ignored are files which are not regular files (such as directories and named pipes) and files whose names end with one of the taboo extensions, as specified by the tabooext directive. The include directive may not appear inside a log file definition.
mail address
When a log is rotated out-of-existence, it is mailed to address. If no mail should be generated by a particular log, the nomail directive may be used.
mailfirst
When using the mail command, mail the just-rotated file, instead of the about-to-expire file.
maillast
When using the mail command, mail the about-to-expire file, instead of the just-rotated file (this is the default).
maxage count
Remove rotated logs older than <count> days. The age is only checked if the logfile is to be rotated. The files are mailed to the configured address if maillast and mail are configured.
minsize size
Log files are rotated when they grow bigger than size bytes, but not before the additionally specified time interval (daily, weekly, monthly, or yearly). The related size option is similar except that it is mutually exclusive with the time interval options, and it causes log files to be rotated without regard for the last rotation time. When minsize is used, both the size and timestamp of a log file are considered.
missingok
If the log file is missing, go on to the next one without issuing an error message. See also nomissingok.
monthly
Log files are rotated the first time logrotate is run in a month (this is normally on the first day of the month).
nocompress
Old versions of log files are not compressed. See also compress.
nocopy
Do not copy the original log file and leave it in place. (this overrides the copy option).

nocopytruncate
Do not truncate the original log file in place after creating a copy (this overrides the copytruncate option).
nocreate
New log files are not created (this overrides the create option).
nodelaycompress
Do not postpone compression of the previous log file to the next rotation cycle (this overrides the delaycompress option).
nodateext
Do not archive old versions of log files with date extension (this overrides the dateext option).
nomail
Don't mail old log files to any address.

nomissingok
If a log file does not exist, issue an error. This is the default.
noolddir
Logs are rotated in the same directory the log normally resides in (this overrides the olddir option).
nosharedscripts
Run prerotate and postrotate scripts for every log file which is rotated (this is the default, and overrides the sharedscripts option). The absolute path to the log file is passed as first argument to the script. If the scripts exit with error, the remaining actions will not be executed for the affected log only.
noshred
Do not use shred when deleting old log files. See also shred.
notifempty
Do not rotate the log if it is empty (this overrides the ifempty option).
olddir directory
Logs are moved into directory for rotation. The directory must be on the same physical device as the log file being rotated, and is assumed to be relative to the directory holding the log file unless an absolute path name is specified. When this option is used all old versions of the log end up in directory. This option may be overridden by the noolddir option.
postrotate/endscript
The lines between postrotate and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) after the log file is rotated. These directives may only appear inside a log file definition. Normally, the absolute path to the log file is passed as first argument to the script. If sharedscripts is specified, whole pattern is passed to the script. See also prerotate. See sharedscripts and nosharedscripts for error handling.
prerotate/endscript
The lines between prerotate and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) before the log file is rotated and only if the log will actually be rotated. These directives may only appear inside a log file definition. Normally, the absolute path to the log file is passed as first argument to the script. If sharedscripts is specified, whole pattern is passed to the script. See also postrotate. See sharedscripts and nosharedscripts for error handling.
firstaction/endscript
The lines between firstaction and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) once before all log files that match the wildcarded pattern are rotated, before prerotate script is run and only if at least one log will actually be rotated. These directives may only appear inside a log file definition. Whole pattern is passed to the script as first argument. If the script exits with error, no further processing is done. See also lastaction.
lastaction/endscript
The lines between lastaction and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) once after all log files that match the wildcarded pattern are rotated, after postrotate script is run and only if at least one log is rotated. These directives may only appear inside a log file definition. Whole pattern is passed to the script as first argument. If the script exits with error, just an error message is shown (as this is the last action). See also firstaction.
rotate count
Log files are rotated count times before being removed or mailed to the address specified in a mail directive. If count is 0, old versions are removed rather than rotated.
size size
Log files are rotated only if they grow bigger then size bytes. If size is followed by k, the size is assumed to be in kilobytes. If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100Gare all valid.
sharedscripts
Normally, prerotate and postrotate scripts are run for each log which is rotated and the absolute path to the log file is passed as first argument to the script. That means a single script may be run multiple times for log file entries which match multiple files (such as the /var/log/news/* example). If sharedscripts is specified, the scripts are only run once, no matter how many logs match the wildcarded pattern, and whole pattern is passed to them. However, if none of the logs in the pattern require rotating, the scripts will not be run at all. If the scripts exit with error, the remaining actions will not be executed for any logs. This option overrides the nosharedscripts option and implies create option.
shred
Delete log files using shred -u instead of unlink(). This should ensure that logs are not readable after their scheduled deletion; this is off by default. See also noshred.

shredcycles count
Asks GNU shred(1) to overwite log files count times before deletion. Without this option, shred's default will be used.
start count
This is the number to use as the base for rotation. For example, if you specify 0, the logs will be created with a .0 extension as they are rotated from the original log files. If you specify 9, log files will be created with a .9, skipping 0-8. Files will still be rotated the number of times specified with the count directive.
tabooext [+] list
The current taboo extension list is changed (see the include directive for information on the taboo extensions). If a + precedes the list of extensions, the current taboo extension list is augmented, otherwise it is replaced. At startup, the taboo extension list contains .rpmorig, .rpmsave, ,v, .swp, .rpmnew, ~, .cfsaved and .rhn-cfg-tmp-*.
weekly
Log files are rotated if the current weekday is less than the weekday of the last rotation or if more than a week has passed since the last rotation. This is normally the same as rotating logs on the first day of the week, but it works better if logrotate is not run every night.

yearly

Log files are rotated if the current year is not the same as the last rotation.

定时执行

这个可以配合linux的cron工具实现定时执行logrotate,具体的实现可以看下cron相关的内容。

附常见的配文件

# sample logrotate configuration file
compress

/var/log/messages {
    rotate 5
    weekly
    postrotate
        /usr/bin/killall -HUP syslogd
    endscript
}

"/var/log/httpd/access.log" /var/log/httpd/error.log {
    rotate 5
    mail [www@my.org](mailto:www@my.org)
    size 100k
    sharedscripts
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}

/var/log/news/* {
    monthly
    rotate 2
    olddir /var/log/news/old
    missingok
    postrotate
        kill -HUP 'cat /var/run/inn.pid'
    endscript
    nocompress
}</pre>

最后编辑于
©著作权归作者所有,转载或内容合作请联系作者
  • 序言:七十年代末,一起剥皮案震惊了整个滨河市,随后出现的几起案子,更是在滨河造成了极大的恐慌,老刑警刘岩,带你破解...
    沈念sama阅读 158,117评论 4 360
  • 序言:滨河连续发生了三起死亡事件,死亡现场离奇诡异,居然都是意外死亡,警方通过查阅死者的电脑和手机,发现死者居然都...
    沈念sama阅读 66,963评论 1 290
  • 文/潘晓璐 我一进店门,熙熙楼的掌柜王于贵愁眉苦脸地迎上来,“玉大人,你说我怎么就摊上这事。” “怎么了?”我有些...
    开封第一讲书人阅读 107,897评论 0 240
  • 文/不坏的土叔 我叫张陵,是天一观的道长。 经常有香客问我,道长,这世上最难降的妖魔是什么? 我笑而不...
    开封第一讲书人阅读 43,805评论 0 203
  • 正文 为了忘掉前任,我火速办了婚礼,结果婚礼上,老公的妹妹穿的比我还像新娘。我一直安慰自己,他们只是感情好,可当我...
    茶点故事阅读 52,208评论 3 286
  • 文/花漫 我一把揭开白布。 她就那样静静地躺着,像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上,一...
    开封第一讲书人阅读 40,535评论 1 216
  • 那天,我揣着相机与录音,去河边找鬼。 笑死,一个胖子当着我的面吹牛,可吹牛的内容都是我干的。 我是一名探鬼主播,决...
    沈念sama阅读 31,797评论 2 311
  • 文/苍兰香墨 我猛地睁开眼,长吁一口气:“原来是场噩梦啊……” “哼!你这毒妇竟也来了?” 一声冷哼从身侧响起,我...
    开封第一讲书人阅读 30,493评论 0 197
  • 序言:老挝万荣一对情侣失踪,失踪者是张志新(化名)和其女友刘颖,没想到半个月后,有当地人在树林里发现了一具尸体,经...
    沈念sama阅读 34,215评论 1 241
  • 正文 独居荒郊野岭守林人离奇死亡,尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 30,477评论 2 244
  • 正文 我和宋清朗相恋三年,在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 31,988评论 1 258
  • 序言:一个原本活蹦乱跳的男人离奇死亡,死状恐怖,灵堂内的尸体忽然破棺而出,到底是诈尸还是另有隐情,我是刑警宁泽,带...
    沈念sama阅读 28,325评论 2 252
  • 正文 年R本政府宣布,位于F岛的核电站,受9级特大地震影响,放射性物质发生泄漏。R本人自食恶果不足惜,却给世界环境...
    茶点故事阅读 32,971评论 3 235
  • 文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹,春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 26,055评论 0 8
  • 文/苍兰香墨 我抬头看了看天上的太阳。三九已至,却和暖如春,着一层夹袄步出监牢的瞬间,已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 26,807评论 0 194
  • 我被黑心中介骗来泰国打工, 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留,地道东北人。 一个月前我还...
    沈念sama阅读 35,544评论 2 271
  • 正文 我出身青楼,却偏偏与公主长得像,于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子,可洞房花烛夜当晚...
    茶点故事阅读 35,455评论 2 266

推荐阅读更多精彩内容

  • Android 自定义View的各种姿势1 Activity的显示之ViewRootImpl详解 Activity...
    passiontim阅读 170,563评论 25 707
  • Spring Boot 参考指南 介绍 转载自:https://www.gitbook.com/book/qbgb...
    毛宇鹏阅读 46,353评论 6 343
  • 下午不知道该干什么,于是跑去教室自习。之前自习都是去图书馆自习室的,但可能我这种人不适合自习室,它总是会给我一种无...
    王一玉阅读 401评论 0 0
  • 九月有30天,我的机械表现在显示的日历是10.1。 按它的步调的话,我在等待12号,那天去干嘛呢?像一个傻子一样。...
    Ryub阅读 163评论 0 1
  • ➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣➣ 一、起因 1,我们在使用mac...
    ForestSen阅读 1,487评论 18 30