If you've ever learned to play an instrument, at some point an instructor will give you a sheet of music that you've never seen before and ask you to play it.  It is a very objective way to assess one's skill level and the act of playing unseen music is called sight reading.

Wikipedia on Sight Reading

I've often wanted to use this technique to quantify a candidate's ability to analyze data.  We often hire people and won't know their real skill level until weeks or months into the position.  I'm toying with asking candidates to analyze a dataset.  My feeling is that people who have very good skills at exploratory data analysis make the best analysts/statsticians/data scientists.  

Has anyone done this successfully in an interview setting?

Would you be turned off to doing it as a candidate?

-John