Friday, April 20, 2012

The worst way to have access to logs

Do you have to access logs for your job? What the worst scenario you can imagine to make it hard to get to those logs and work through them? Let's try this:
  • Logs are on a windows box
  • To get to the log machine, you have to remote desktop to box A
  • But the logs aren't on box A, the logs are on a share via box B,C and D
  • You can't RDP to box B, C and D, so you have to map the drives on box A
  • Why do I need to map the drive? Because it's Windows and there are no Unix tools available
  • Cygwin is on box A not the others, so when I map the drive, I can access them via Cygwin
  • Usually just mapping the drive sucks for speed, because the I/O is via the mapped network connection, so this forces me to copy the logs to from B,C and D to A
  • Once moved, I can now grep and open logs in vi for search
Why is this bad? When an error occurs that we want to catch before the logs roll in live production, we need to get to the logs quickly and move them. This process above takes like 15 minutes even to begin looking at the logs. 

Why aren't the logs archived? They are, but I dont have access to that box either.

To top all this off, lets add a site to site VPN tunnel to slow the connection down even more.

How can this be better?

  • Give me access directly to the box with the logs
  • Install SSH on the box so I dont need a windowing system to get on (I don't need a host Unix system or variant, just some SSH action)
  • Aggregate the logs to a single location real time (using Apache Flume as an example)
  • Setup some log monitoring that automatically detects errors, fires off alerts with the relevant log details
  • Etcetera, etcetera, etcetera 

1 comment:

Jon said...

That's why we created a web-based logviewer.

Share on Twitter