In my not so humble opinion generating a graph like this from a database shouldn't require a 70 line ruby script!

If you know how to run MongoDB and ruby on your computer or your server, then backupPublicPhotoMetadataByDateAndUser.rb is a simple Ruby script to backup your Flickr metadata for yours (or anybody else's) Flickr public photos for a given time period. Simple (it doesn't replicate flickr's richness nearly as comprehensively as Aaron's parallel flickr) but effective because then you can with very very simple programming use the MongoDB query language to do useful Flickr queries such as:

1. give me all the photos in gastown

2. give me the square 75x75url of all photos in Vancouver tagged "yellow"

3. give me all the photos geo-tagged in the city of Vancouver taken in 2011 during the morning rush hour (e.g. if you wanted to do Eric Fisher-like  maps)

HOW TO RUN IT:

1. Start up MongoDB and create some indexes: /Applications/mongodb-osx-x86_64-2.0.2/bin/mongo
   
1. > db.photos.ensureIndex({id:1});
   > db.photos.ensureIndex({datetaken:1});
   > db.photos.ensureIndex({dateupload:1});
   > db.photos.ensureIndex({lastupdate:1});
2. Set Environment variables to point to your MongoDB and your flickr api key and the userid e.g.
1. export MONGO_PORT=27017
2. export MONGO_HOST=127.0.0.1
3. export FLICKR_DB=rtflickr # your flickr database name in Mongo DB
4. export FLICKR_USER=yourfunnyFlickruserIdendinging@01
5. create a flickr.conf file like this:
1. Roland-Tanglaos-MacBook-Pro-2:mongoflickrbackup rolandtanglao$ cat flickr.conf
   api_key = a3b9eeatbeefdecafbad
3. specify time and date to backup e.g. to backup your 2009 public photos:
1. ./backupPublicPhotoMetadataByDateAndUser.rb 2009 1 1 2009 12 31  >backup.2009.stdout.txt 2>backup.2009.stderr.txt

mongohq  + GS API +  mongoPoll.rb on my VPS + emailReport2.rb + cron = awesome twice daily Thunderbird support report with trending tags, topics, anti-virus, ISPs and mail providers on posterous and in my inbox. Code is open source. Pull requests welcome!

Bugs

1. posting to posterous via email or the API results in truncated posts (I guess that's what happens when you have 2500 line long posts :-)! )

features that i intend to add:

1. trending Thunderbird add-ons
2. trending nouns that aren't: ISPs, Mail Providers, anti-virus, tags or add-ons
3. remove polling, use Web Hooks instead

don't know why time machine restore didn't fully restore my Mac OS X Lion ruby install but who cares :-) ! And don't know how I figured this out. Other than looking at  the yakshaving to get ruby working with mongodb on my VPS.

Hstore as expained in Ecstortive:The key value store everyone ignored (Postgresql) is an interesting non NO SQL way to store key value pairs. I wonder if it's faster than doing the same thing in Redis or MongoDB. My uninformed guess is yes :-) !

Presentation created for June 30, 2011 Vancouver Ruby / Rails meetup (flickr set, slideshare, pdf)

See my Better Support Living through Software for the full details

And just for fun here are the flickr pics of the slides:

I. Better Living

II. Just Say No

III.We Can Do This

IV. Support Comes to You!

V. Fork this!

Manual Support Forum browsing is soooo 1990s and a waste of support folks' time aka my Get Satisfaction Thunderbird API ruby and Mongo DB side project finally has born fruit :-) ! Seriously if you are manually searching your support forums and don't have customized support querying and reporting  so vital, can't miss stuff comes to your support contributors via email (or some other semi-automated method like RSS or whatever) then you are doing support wrong.

I wrote a Ruby script to read via Get Satisfaction pseudo-REST API all the Thunderbird Support Topics into a MongoDB and wrote a ruby script to generate  a custom email support report  so cool, relevant, urgent stuff comes to me instead of me having to surf the forums.

Not only that I wrote a brute force search engine that allows searching the fulltext of all 40000 Get Satisfaction topics and tags via regexes.

Even cooler: Anybody who uses Get Satisfaction for support can use my code!!!!!!! All you have to do is:

1. change "mozilla_messaging" to your Get Satisfaction company name
2. change "mozilla_thunderbird" to your product name
3. configure a gmail account config file called email.conf with 3 lines:
1. from_address
2. to_address
3. p for your password

Finally, here's the README which serves as the documentation

(since this is a side project, the README is shall we say lacking :-); I hope to somehow get this into the SUMO project so that all support contributors not just me can use it; to this end I am writing a spec for this, fingers crossed)

Roland's utilities for Get Satisfaction Metrics and other GS fun stuff

• hardcoded for now for the Thunderbird product i.e. getsatisfaction.com/mozilla_message/products/thunderbird
• easily hackable for any other product
• forgive the barebones documentation for this project :-) !
• HELP WANTED: would love for a real developer to fix this code and enhance it
• email rtanglao AT mozilla.com if you are interested

Prerequisites:

• requires a MongoDB running on local host with a database called "gs"
• and with collection in the gs database called "topics"

Usage:

Update the Mongo - ALWAYS do this first before running anything else!

./mongoUpdateSlurpGS.rb 2011 6 16 2011 6 19 2>mongoslurp.16-19june2011.818pm.stderr.txt & 

Daily email metrics (I run this twice a day, typically 9a.m. and 4p.m. Pacific)

./emailDailyMetrics.rb & 

Weekly Metrics to CSV files and stdout (run once a week to generate support metrics for the Thunderbird Project Call)

./weeklyMetrics.rb 2011 6 13 2011 6 19 2>13-19june2011.stderr.txt 1>13-19june2011.stdout.txt 

Search the Thunderbird Get Satisfaction MongoDB for regexes in the fulltext (title+content+replies) & tags

e.g. for Thunderbird 5 here's a sample search ("-r" is a comma separated list of regexes to search the fulltext for and "-s" is a comma separated list of regexes to search tags for)

./bruteforceSearch.rb 2011 6 1 2011 6 30 2>19june2011.tb5b1.bfs.stderr.1020pm.txt 1>19june2011.tb5b1.bfs.stdout.1020pm.txt -r tb5, "tb 5","thunderbird5","thunderbird 5",beta -s "tb 5",tb5,thunderbird5,"thunderbird 5","beta feedback",50,tb5,"tb 5",beta 

I have a mongodb  database created by my mongoUpdateSlurpGS.rb Ruby script with Thunderbird Get Satisfaction support topics from July 20, 2009 until the present (i.e. roughly 40000 topics) and I would like to know for a given time period how many support topics were created or modified. To calculate this, I wrote a second ruby script called alltopicsCreatedOrUpdated.rb (and the code is embedded below). My question is: is there really a need for a ruby script here.? Could the nested for "each" loop in alltopicsCreatedOrUpdated.rb  be replaced by a couple of mongodb queries using the mongodb command line interface?

Got some data that you want to persist and manipulate but you are not sure what to do with it i.e. you are in exploratory or "side project" mode? Then MongoDB is a fine choice in 2011. Easy to get the data in and out and then when you have figured out what you want to do when you grow up :-) you can store it in a "real" database as well e.g. PostgreSQL. e.g. If you want to get your data out of Get Satisfaction or any other system with an API for backup and other purposes, check out my "hacked in an hour or two but I think it works", mongoUpdateSlurpGS ruby script. I used it to slurp in 20000 Get Satisfaction Thunderbird Topics and their associated replies and tags. If you want to do the same for your Get Satisfaction community, it's trivial to do: all you need is about 1 hour of a ruby (or other language,the code is pretty elementary) developer's time and some time to run the script.