最小化的配置文件
input{}
input {
file {
path => "E:/software/logstash-1.5.4/logstash-1.5.4/data/*"
}
}
filter {
}
output {
stdout {}
}
当然也可以监听多个目标文件：
input {
file {
path => ["E:/software/logstash-1.5.4/logstash-1.5.4/data/*","F:/test.txt"]
}
}
filter {
}
output {
stdout {}
}
文件的路径名需要时绝对路径，并且支持globs写法。
其他的配置
另外，处理path这个必须的项外，file还提供了很多其他的属性：
input {
file {
#监听文件的路径
path => ["E:/software/logstash-1.5.4/logstash-1.5.4/data/*","F:/test.txt"]
#排除不想监听的文件
exclude => "1.log"
#添加自定义的字段
add_field => {"test"=>"test"}
#增加标签
tags => "tag1"
#设置新事件的标志
delimiter => "\n"
#设置多长时间扫描目录，发现新文件
discover_interval => 15
#设置多长时间检测文件是否修改
stat_interval => 1
#监听文件的起始位置，默认是end
start_position => beginning
#监听文件读取信息记录的位置
sincedb_path => "E:/software/logstash-1.5.4/logstash-1.5.4/test.txt"
#设置多长时间会写入读取的位置信息
sincedb_write_interval => 15
}
}
filter {
}
output {
stdout {}
}
其中值得注意的是：
1 path
是必须的选项，每一个file配置，都至少有一个path
2 exclude
是不想监听的文件，logstash会自动忽略该文件的监听。配置的规则与path类似，支持字符串或者数组，但是要求必须是绝对路径。
3 start_position
是监听的位置，默认是end，即一个文件如果没有记录它的读取信息，则从文件的末尾开始读取，也就是说，仅仅读取新添加的内容。对于一些更新的日志类型的监听，通常直接使用end就可以了；相反，beginning就会从一个文件的头开始读取。但是如果记录过文件的读取信息，这个配置也就失去作用了。
4 sincedb_path
这个选项配置了默认的读取文件信息记录在哪个文件中，默认是按照文件的inode等信息自动生成。其中记录了inode、主设备号、次设备号以及读取的位置。因此，如果一个文件仅仅是重命名，那么它的inode以及其他信息就不会改变，因此也不会重新读取文件的任何信息。类似的，如果复制了一个文件，就相当于创建了一个新的inode，如果监听的是一个目录，就会读取该文件的所有信息。
5 其他的关于扫描和检测的时间，按照默认的来就好了，如果频繁创建新的文件，想要快速监听，那么可以考虑缩短检测的时间。
6 add_field
就是增加一个字段，例如：
file {
add_field => {"test"=>"test"}
path => "D:/tools/logstash/path/to/groksample.log"
start_position => beginning
}
7 tags
用于增加一些标签，这个标签可能在后续的处理中起到标志的作用
8 delimiter
是事件分行的标志，如果配置成123,那么就会如下所示。这个选项，通常在多行事件中比较有用。
This plugin supports the following configuration options:
Setting | Input type | Required | Default value |
No | | ||
No | | ||
No | | ||
No | | ||
No | | ||
No | |||
No | | ||
Yes | |||
No | |||
No | | ||
string, one of | No | | |
No | | ||
No | |||
No |
Detailsedit
add_fieldedit
- Value type is hash
- Default value is
{}
Add a field to an event
close_olderedit
- Value type is number
- Default value is
3600
If this option is specified, the file input closes any files that remain unmodified for longer than the specified timespan in seconds. The default is 1 hour
codecedit
- Value type is codec
- Default value is
"plain"
The codec used for input data. Input codecs are a convenient method for decoding your data before it enters the input, without needing a separate filter in your Logstash pipeline.
delimiteredit
- Value type is string
- Default value is
"\n"
set the new line delimiter, defaults to "\n"
discover_intervaledit
- Value type is number
- Default value is
15
How often (in seconds) we expand the filename patterns in the path option to discover new files to watch.
excludeedit
- Value type is array
- There is no default value for this setting.
Exclusions (matched against the filename, not full path). Filename patterns are valid here, too. For example, if you have
path => "/var/log/*"
You might want to exclude gzipped files:
exclude => "*.gz"
ignore_olderedit
- Value type is number
- Default value is
86400
If this option is specified, when the file input discovers a file that was last modified before the specified timespan in seconds, the file is ignored. After it’s discovery, if an ignored file is modified it is no longer ignored and any new data is read. The default is 24 hours.
pathedit
- This is a required setting.
- Value type is array
- There is no default value for this setting.
The path(s) to the file(s) to use as an input. You can use filename patterns here, such as /var/log/*.log. If you use a pattern like /var/log/**/*.log, a recursive search of /var/log will be done for all *.log files. Paths must be absolute and cannot be relative.
You may also configure multiple paths. See an example on the Logstash configuration page.
sincedb_pathedit
- Value type is string
- There is no default value for this setting.
Path of the sincedb database file (keeps track of the current position of monitored log files) that will be written to disk. The default will write sincedb files to some path matching $HOME/.sincedb* NOTE: it must be a file path and not a directory path
sincedb_write_intervaledit
- Value type is number
- Default value is
15
How often (in seconds) to write a since database with the current position of monitored log files.
start_positionedit
- Value can be any of:
beginning,end - Default value is
"end"
beginning.
This option only modifies "first contact" situations where a file is new and not seen before, i.e. files that don’t have a current position recorded in a sincedb file read by Logstash. If a file has already been seen before, this option has no effect and the position recorded in the sincedb file will be used.
stat_intervaledit
- Value type is number
- Default value is
1
How often (in seconds) we stat files to see if they have been modified. Increasing this interval will decrease the number of system calls we make, but increase the time to detect new log lines.
tagsedit
- Value type is array
- There is no default value for this setting.
Add any number of arbitrary tags to your event.
This can help with processing later.
typeedit
- Value type is string
- There is no default value for this setting.
Add a type field to all events handled by this input.
Types are used mainly for filter activation.
The type is stored as part of the event itself, so you can also use the type to search for it in Kibana.
If you try to set a type on an event that already has one (for example when you send an event from a shipper to an indexer) then a new input will not override the existing type. A type set at the shipper stays with that event for its life even when sent to another Logstash server.
