We conducted assessment of our simulation with 4 students at College Track, an after school college-preparatory program in East Palo Alto.

Our assessment consisted of a pre-test questionnaire that asks participants for demographical information, followed by a quiz about basic financial concepts (true/false, multiple choice, and open-ended questions). We then used a contextual inquiry protocol and asked participants to play our simulation while thinking out loud and narrating their thought processes. Upon completing the simulation, participants were asked follow up questions regarding what they thought of the game in regards to the content, aesthetics, personal relevance to their lives and usability. Participants were also asked to re-visit the questions they answered incorrectly in the pre-test to see if they got those questions correct this time around. Participants were compensated with Jamba Juice gift cards. Participants were also asked to sign a consent form letting them know that anything they tell us will remain anonymous.

See below for copies of our assessment protocol and consent form:

Our prototype simulation was successful overall in education kids about financial concepts. Many kids did not know the difference between a want vs. a need, but after playing our simulation they were able to better grasp the difference. Kids also felt the game was relevant to their lives in that the financial decisions they encountered in the game were similar to those they have to deal with in real life. From our usability testing, some minor interface issues were raised, including improving the audio quality, making it more explicit who the voices of the characters in the game are, and lengthy instructional videos with too much audio that need to be accompanied with more visuals. Kids commented that they feel that their friends would also be interested in playing the game and they thought our simulation provided them with useful knowledge that they should have.

We tested our board game with high schoolers from the K-12 lab at the Hasso Plattner Institute of Design at Stanford. Participants were asked to think out loud while playing the board game and sharing their thoughts on what works, what didn't work, their questions, and suggestions for improvement.

The Board game assessment consisted of  the evaluation of the budget sheet at the end of the game and a conjunction of observational methods through the game. We were interested in finding out how well  users adopted and used their budget sheet and the interactions among players as they went through making their financial decisions and challenges in the game. At the beginning of the game each player was given a budget sheet that should be completed as they make and record their financial decisions. The goal for the budget sheet, which was the main part of the assessment, was to learn if users have actual knowledge of how to fill in the budget sheet and how to track their expenses and savings in the game. The observational method we used was a naturalistic method, where we took notes of the players interactions without interrupting the flow of the game. 

 Some of the findings and suggestions for improvements included:

• Set the context of the game. Participants were confused about how the financial setting of the game -- they did not know how much items in the game would cost and thus did not know how much they should budget.
• Lack of control of events. Since the game was based on chance (by rolling a die), the events of the board game occur randomly, and sometimes participants are not able to accomplish certain tasks (like buying a prom dress if they never land in the mall)
• Change the structure of the board game. Instead of going in a loop, perhaps try starting in the middle of the board, and then choosing a direction to go. This would resemble real life more because people usually choose where they want to go.
  
• Emphasize the use of the budget sheet. Participants found it easy and engaging to use the budget sheet but found integration of the budget sheet into the game confusing.
  
• Make a facilitator guide.  Since the board game was made to reinforce the concepts taught in the simulation, it is important for kids to be wary of this. It would be more helpful if the board game came with a facilitator/teacher guide to remind and guide kids about the concepts they learned in the simulation. This is especially important since we plan to use our board game in after school programs.
  
• Kids liked the interactivity of the game. They enjoyed rolling the die to move to places, and the bus passes worked well.
• Integrate more technology (sensors). This would help provide more feedback on decisions made during the board game. The integration of sensors would also better simulate the "randomness" of life (i.e., sensors would light up to indicate a sale or discount at a store when the player lands at the mall).