リポジトリ追加
# vi /etc/yum.repos.d/td.repo
[treasuredata] name=TreasureData baseurl=http://packages.treasure-data.com/redhat/$basearch gpgcheck=0
インストール
# yum -y install td-agent
設定ファイル修正
# cp /etc/td-agent/td-agent.conf /etc/td-agent/td-agent.conf.org # vi /etc/td-agent/td-agent.conf
<source> type tail format apache path /var/log/httpd/access_log tag apache.access </source> <match apache.access> type file path /var/log/fluent/access_log </match>
ログファイル保存場所
# mkdir /var/log/fluent # chown td-agent:td-agent /var/log/fluent
パーミッション変更（グループに読取・実行権限付与）
# chmod g+rx /var/log/httpd
Fluentdスタート
# /etc/init.d/td-agent start Starting td-agent: [ OK ]
アクセスログ確認
# tail /var/log/fluent/access_log.20130131.b4d49ecf89504b484
2013-01-31T16:28:03-08:00 apache.access {"host":"192.168.26.1","user":"-","method":"GET","path":"/pma/js/jquery/jquery.qtip-1.0.0-rc3.js?ts=1356015781","code":"304","size":"-","referer":"http://192.168.26.143/pma/","agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.17"}
FluentdのMogoDBプラグインインストール
# /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mongo Fetching: fluent-plugin-mongo-0.6.13.gem (100%) Successfully installed fluent-plugin-mongo-0.6.13 1 gem installed Installing ri documentation for fluent-plugin-mongo-0.6.13... Installing RDoc documentation for fluent-plugin-mongo-0.6.13...
設定ファイル修正
# cp /etc/td-agent/td-agent.conf /etc/td-agent/td-agent.conf.file # vi /etc/td-agent/td-agent.conf
<source> type tail format apache path /var/log/httpd/access_log tag mongo.apache </source> <match mongo.**> type mongo # DB名「apache」のコレクション「access」に保存 database apache collection access # MongoDB接続先とポート host localhost port 27017 #インターバル flush_interval 10s </match>
Fluentdスタート
# /etc/init.d/td-agent restart Shutting down td-agent: [ OK ] Starting td-agent: [ OK ]
MongoDBにログが登録されたことを確認
# mongo
MongoDB shell version: 2.2.2
connecting to: test
> show dbs
apache 0.203125GB
local (empty)
> use apache
switched to db apache
> show collections
access
system.indexes
> db.access.findOne()
{
"_id" : ObjectId("510b0daeddfe4129a7000001"),
"host" : "192.168.26.1",
"user" : "-",
"method" : "GET",
"path" : "/pma/",
"code" : "200",
"size" : "6789",
"referer" : "-",
"agent" : "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.17",
"time" : ISODate("2013-02-01T00:34:50Z")
}
リモート環境にFluentdをインストールして、設定ファイル編集
# vi /etc/td-agent/td-agent.conf
<source>
type tail
format apache
path /var/log/httpd/access_log
tag mongo.apache
</source>
<match mongo.apache>
type forward
<server>
host 192.168.26.143
port 24224
</server>
flush_interval 10s
</match>
リモート環境でFluentdスタート
# /etc/init.d/td-agent start Starting td-agent: [ OK ]
ローカル環境にFluentdをインストールして、設定ファイル編集
# vi /etc/td-agent/td-agent.conf
+) <source> +) type forward +) port 24224 +) </source>
ローカル環境でFluentdリスタート
# /etc/init.d/td-agent restart Shutting down td-agent: [ OK ] Starting td-agent: [ OK ]
以下のフォーマットの場合のfluentdのformat定義
LogFormat "%h %A %t \"%m %U %H\" %>s %b \"%q\" %{OPENSOCIAL_VIEWER_ID}n \"%{Referer}i\" \"%{User-Agent}i\"" combined
<source> type tail format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<received_byte>[^ ]*) (?<sent_byte>[^ ]*) (?<response_time_micro_sec>[^ ]*))?$/ time_format %d/%b/%Y:%H:%M:%S %z path /var/log/httpd/www.example.jp/access_log/access_log pos_file /var/log/td-agent/pos/apache.access.api.pos tag apache.access.api </source> <source> type tail format /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\] (?<message>.*)$/ time_format %b %d %H:%M:%S %Y path /var/log/httpd/www.example.jp/error_log/error_log pos_file /var/log/td-agent/pos/apache.error.api.pos tag apache.error.api </source>
とりあえずログに残す場合は、以下の定義で。
format /^(?<message>.*)$/
一つの入力を複数の出力に渡す
<match mongo.apache.access>
type copy
# MongoDB
<store>
type mongo
database apache
collection accesslog
host localhost
port 20000
flush_interval 10s
</store>
# to Manage
<store>
type forward
host 192.168.0.1
port 24224
flush_interval 10s
</store>
</match>
Fluentdのregexpのテストが可能