My previous post was unclear. I'll try again. It shouldn't take a 70 line Ruby script or indeed procedural programming to generate a CSV file from a MongoDB database. So Lazy web :-) what's the best Best MongoDB to CSV tool to generate CSV files for graphing using one of the JS toolkits, Excel, etc so I don't have to write a 70 line Ruby script like getOneDayAnswerRate.rb?
In my not so humble opinion generating a graph like this from a database shouldn't require a 70 line ruby script!
mongohq + GS API + mongoPoll.rb on my VPS + emailReport2.rb + cron = awesome twice daily Thunderbird support report
mongohq + GS API + mongoPoll.rb on my VPS + emailReport2.rb + cron = awesome twice daily Thunderbird support report with trending tags, topics, anti-virus, ISPs and mail providers on posterous and in my inbox. Code is open source. Pull requests welcome!
- posting to posterous via email or the API results in truncated posts (I guess that's what happens when you have 2500 line long posts :-)! )
features that i intend to add:
- trending Thunderbird add-ons
- trending nouns that aren't: ISPs, Mail Providers, anti-virus, tags or add-ons
- remove polling, use Web Hooks instead
don't know why time machine restore didn't fully restore my Mac OS X Lion ruby install but who cares :-) ! And don't know how I figured this out. Other than looking at the yakshaving to get ruby working with mongodb on my VPS.
And just for fun here are the flickr pics of the slides:
I. Better Living
II. Just Say No
III.We Can Do This
IV. Support Comes to You!
V. Fork this!
Improve my MongoDB query to get the number of Thunderbird support topics created or replied to within a given time period please
I have a mongodb database created by my mongoUpdateSlurpGS.rb Ruby script with Thunderbird Get Satisfaction support topics from July 20, 2009 until the present (i.e. roughly 40000 topics) and I would like to know for a given time period how many support topics were created or modified. To calculate this, I wrote a second ruby script called alltopicsCreatedOrUpdated.rb (and the code is embedded below). My question is: is there really a need for a ruby script here.? Could the nested for "each" loop in alltopicsCreatedOrUpdated.rb be replaced by a couple of mongodb queries using the mongodb command line interface?
Since the Get Satisfaction developer docs are shall we say in need of some love, here's an ruby irb session (exercise for the reader, convert to a python or lisp REPL session :-) !) that illustrates how to find all Get Satisfaction topics tagged Android (to play along and find the get satisfaction topics tagged "android" for your company, make sure you have ruby installed on Linux or Mac OS X along with the json ruby gem and substitute your company name for "mozilla_messaging"):
UPDATE November 2011:
The code has been dramatically re-factored so i deleted it from the gist. Please use the revised code at:
For my technical support lead job at Mozilla Messaging, I continue to try to find the best way to sort and sift and help out folks who need support and identify the hot issues. We currently use Get Satisfaction for Thunderbird support and I subscribe via email to everything but it's really difficult to read through the over 200 emails a day that Get Satisfaction sends
So I developed over the last 9 months a series of Ruby scripts to see the most active support topics (getactive.rb), the contributors who are actually being marked by the community of users as solving a support topic (getcontributors.rb), who's doing the most replies (gethappykludged.rb) and pick out some random support topics (getrandom.rb)
Up until last week I have been using these scripts on a manual basis but not as consistently as I would like since it's pain to run all four manually.
To make a single email report that ties together everything in one nice summary, last week I finally took 3 hours and wrote emailReport.rb (code after the jump). It's crude but it works and is a useful report that I run every day now for the previous day (e.g. on Monday morning, I run it for Sunday and Saturday). It runs all the above mentioned scripts and then emails a summary email:
Here's an example of how you invoke it:
(for August 2, 2010)
./emailReport.rb 2010 8 2 2010 8 2 2>emailreport.20100802.20100802.stderr 1>emailreport.2010082002.20100802.stdout
And here's a sample email:
From: firstname.lastname@example.org To: email@example.com subject: MoMo Support Report FROM: 2010.8.2 TO: 2010.8.2 Date: Mon, 02 Aug 2010 23:31:16 -0700
Get Satisfaction Top 5 active: 5,http://getsatisfaction.com/mozilla_messaging/topics/how_do_i_export_folders_as_mbox_files_thanks_chris 5,http://getsatisfaction.com/mozilla_messaging/topics/migration_of_thunderbird_data_from_old_to_new_computer 4,http://getsatisfaction.com/mozilla_messaging/topics/delivering_mail_98_thunderbird_problem 4,http://getsatisfaction.com/mozilla_messaging/topics/from_thunderbird_to_outlook_how 4,http://getsatisfaction.com/mozilla_messaging/topics/cannot_start_thunderbird_already_runnning Get Satisfaction Contributors: Top 10 Get Satisfaction Repliers: 2, cam_1319937 3, simonpaquet 3, ankitgupta1980 3, davro 3, pat_drummond 3, kent_james 4, bhuvnesh_thakar 5, wayne_mery 9, archaeopteryx 30, michael_a_pasek 5 Random Get Satisfaction Topics: http://getsatisfaction.com/mozilla_messaging/topics/crashing_after_start_up http://getsatisfaction.com/mozilla_messaging/topics/problems_editing_an_email_to_be_forwarded http://getsatisfaction.com/mozilla_messaging/topics/dmg_file_whats_this http://getsatisfaction.com/mozilla_messaging/topics/want_to_use_thunderbird_for_mass_mailing http://getsatisfaction.com/mozilla_messaging/topics/pasting_images_takes_minutes
What I did well
- It works and when it doesn't I can check stderr for diagnostics and usually quickly figure out what went wrong (Get Satisfaction has a known bug on topics that have URLS that begin with numbers and also JSON can't cope when folks paste binary data into support topics; fortunately both cases are rare -:-) !)
What could be improved
- The JSON API is called on the same data four times, once for each script (i.e. I don't store any of the GS data) so it takes 5-10 minutes to run emailReport.rb!
- It would be more efficient to just read the data into a database, e.g. couchdb and then have the four scripts read from the database
- Badly need to way to extract the GS conversations I am involved in and highlight the top 5
- Although a separate can of worms :-), I would love to have a really great search engine that works on the GS support data; unfortunately the GS management view search doesn't work so well.
I'd love some help :-) (what's in it for you! Open Source Whuffie, fame and adventure :-) !!!) in the following areas:
- Put the GS support data into a JSON friendly database like CouchDB
The code has been dramatically re-factored so i deleted it from the gist. Please use the revised code at:
After some digging and research around the web, my ideal mobile mad scientist programming language would:
- have the 2D and 3D graphic manipulation power of Processing, Nodebox and Shoes
- be cross platform mac, windows, linux, maemo on mobile, iPhone, android
- be 'web native' i.e. REST, JSON, XML and all the other web API stuff built in and not bolted on like it is Processing, trying to use the flickr api from Processing is shall we say kludge-o-rama (awesome code from bryan chung but indicative of the unnecessary struggle one is forced to engage with in Processing and other non web native languages)
- not use a Java-like syntax, death to curly braces and wasted semi-colons
- be dynamic, death to the Java/C++ cargo cult of typing for no reason
- be easily adaptable to new APIs and new sensors through the ability to create a domain specific language and/or easy to use and beautiful foreign function interface
- be open source, sorry but for my mobile art, i can't use programming environments and languages that are not open source
- support the REAL loop, I don't want to spawn threads for the sake of questionable 'concurrency' like I am forced to with OSGI and the Bug Labs Bug
IF I were an idealist that pretty much rules out everything :-)
Fortunately I am a pragamatist. So I will continue my experiments in:
- Nodebox & Python on the Mac
- Cocoa Touch and Objective C on the iPhone
What about Processing? Sorry can't handle the Java syntax and the pain of doing XML and JSON and REST programming and the kludge-o-matic way to access Java libraries. processing.js? too early and too much impedance mismatch to use all the lovely JS libraries out there. And Shoes is promising especially if it were improved so you could easily use normal Ruby gems but given its current "hibernation" "post-Why" not sure it will continue to be improved.
What should I use on Maemo if/when I get an N900? Ruby plus SWIG or some such foreign function kludge er interface :-) to access the sensor APIs which I assume are only available in C and C++ ?
What should I use on Android if/when I get an Android device?
What should I use on Windows? Not that I really care :-) But it would be lovely to have Windows people join in my fun without having to do anyting. Eines Tages!
Somehow I think the "mainstream" world is moving towards my ideal solution and the mainstream solution for what I want will look more like processing.js and ruby-processing or smalltalk i.e. scratch then it will look like Processing, Nodebox or CocoaTouch
What I am looking for in my ongoing mobile art experiments is REAL!
- Read the sensors (GPS, accelerometer, compass, etc)
- Evaluate the data from the sensors
- Art - Make some art (sound, graphic, image, etc) and display and store it on the mobile
- Loop - Back to step 1
And I want to do it in a dynamic environment that doesn't force me to do yak shaving like spawning separate threads for each of the sensors or other such needless complexity that's not needed by my artistic algorithms. Nor the slings and arrows of outrageous certificates or certifications or developer programs or DRM malarkey :-) !
After reading the N900 technical reviews from the Maemo summit, it appears that the unlocked version of the N900 is the closest current device that could do this:
- It's Linux so assuming the sensor APIs are available from a standard Linux C (i.e. not using some non standard craziness like Carbide C++ for Symbian) library, Python, Ruby, name your favourite dynamic language here, etc (C++ and Java are just not malleable and easily hackable enough, sorry!) bindings could be (and probably are or are in the process of being) built to those C libraries
- No need for developer fees or ridiculous certificates
- It has all the sensors I want for my current experiments
- It's NOT mass market, but it's mass market enough for hackers (unlike Bug Labs Bug which I love but already probably has less than 1/10 the amount of developers working on it as the N900).
Am I right? Time for me to watch the blogs for signs of the N900 and Maemo 5 and 6 making dynamic languages first class citizens unlike on Symbian where S60 Python was far too many steps behind Carbide C++ (and time for to save up for the N910 since the N900 will probably be crippled in some significant way as all 1st gen Nokia devices are e.g. N95-1 not having enough memory)
Here's how I did it (forgive my bad code, it was just a fun 1st project and of course you could do this in QuickTime Pro with no programming but that wouldn't be as much "fun" :-) !):
Dir.chdir("/a directory with 1796 skyte photos")
skyte_pics = Dir.glob("*.jpg")
size = skyte_pics.size
@skyte = image skyte_pics
animate(200) do |frame|
close if frame > size
@skyte.path = skyte_pics[frame]