By Lennart Koopmann, Sofware Engineer at Xing, developer of greylog2
I missed the parenthesis in the title (they are not mine, but come directly from Arrrr schedule), and they were there for a good reason: Lennart Koopman talk about logging, without going in the specifics of where they come from. He (his session) is more about what we do with them. His presentation is based around a "maturity scale of log management" established by Raffael Marty. It's interesting enough for me to repost it there:
- 0 : Do not collect logs at all
- 1 : Collect logs, from HTTP server
- 2 : Use logs for forensics / troubleshooting
- 3 : Save searches in log (for example, a typical grep command that is re-used)
- 4 : Share log searches with others
- 5 : Publish reporting (we have that many errors of type X each day...)
- 6 : Alerting (send mail or other way when something goes wrong)
- 7 : Collect more logs (collect from various sources)
- 8 : Draw correlation between different sources (analyse a "spike" in db, http, web server)
- 9 : Visual analysis (provide useful visualisations on the logs)
I think this is definitively something to keep, for the same reason that unit testing is not just about having test, good logging does not stop at having logs (my current company is very good at generating/collecting logs, and very bad at using them - which defeats the whole purpose). The levels are not so much about the tools that they are about the people using them.
Next part is about tooling, with Splunk as the 1000-pound gorilla: does everything that someone can dream of, commercial and expensive (Lennart made allusion about the cost skyrocketing for large installations). I did use Splunk on several occasions, and was pretty impressed with it, so if you can afford it, be sure to take a look at it. Well, if you can afford it, and if you use it.
Lennart himself is working on Graylog2: an open source alternative to Splunk, with less functionalities, but that might be sufficient in your usecase, using Rails and MongoDB. Graylog2 play nicely with another open source package, Logstash. Seems that Graylog2 is better at analysis, but that Logstash has an edge for large log-base, so people use Graylog2 for short term (day/week logs), and move them to Logstash regularly. Both project are evolving, so be sure to take a look on them once a while.
Lennart finished his speech by talking about the (very short term) future of Graylog2, and touted usage of Elastic Search as the "next big thing" for Graylog2, greatly improving its search capabilities. I did not know about ElasticSearch, so I did my homework during the Q/A. The project looked interesting and "proved" a point that I made some weeks ago: it may play nicely with rails, but ElasticSearch, as every search library it seems, is based around Lucene. Seems it's just too good to be avoided.
All said, a nice session, short but focused, with a speaker more interested to give you ideas / open your array of possible than to "sell" his product, which is quite nice. Once more, I'll remember the log scale the next time I'm around a big project.