Who needs an API or an RSS feed or any feed when you have proper logs? aka The Log: What every software engineer should know about real-time data's unifying abstraction
Pontifications
- Who needs an API or an RSS feed or any feed when you have proper machine readable logs? aka The Log: What every software engineer should know about real-time data’s unifying abstraction
QUOTE
One of the most useful things I learned in all this was that many of the things we were building had a very simple concept at their heart: the log. Sometimes called write-ahead logs or commit logs or transaction logs, logs have been around almost as long as computers and are at the heart of many distributed data systems and real-time application architectures.
You can't fully understand databases, NoSQL stores, key value stores, replication, paxos, hadoop, version control, or almost any software system without understanding logs; and yet, most software engineers are not familiar with them. I'd like to change that. In this post, I'll walk you through everything you need to know about logs, including what is log and how to use logs for data integration, real time processing, and system building.
END QUOTE