Learning Problem Theory Design Process Our Solution Assessment References

Assessment & User Testing

Methodology

Participants:Eight low-income high school students who want to go to college

Location:College Track in East Palo Alto, CA

Procedure:

  1. When students entered the room, they were asked to first complete a demographic survey so we could identify their grade level and how much they had already investigated topics related to college finance.
  2. Upon completing the survey, students were given a list of six topics to investigate. Half of the students used CAIA, while the other half used the most popular college finance website, the Federal Student Aid website. The topics they were asked to investigate were:
    • How you receive financial aid
    • How often you need to apply for financial aid
    • When you have to start repaying your loans
    • Whether your financial package stays the same for all four years of college
    • Who is responsible for repaying your student loans
    • The deadline for submitting your FAFSA
  3. Students were given approximately 15 minutes (or less if they finished early) to use the system they were given to investigate the above topics. When they finished, students were given eight open-ended questions to respond to, but were told they could leave them blank if they did not know the answer:
    • How is financial need calculated?
    • What form do you need to fill out to apply for financial aid?
    • What is the grace period for student loans?
    • How often do you have to submit a FAFSA?
    • Will your student loans be the same amount every year you are in college?
    • When is your first loan payment due?
    • Who is responsible for repaying your students loans?
    • If you were applying for financial aid in the 2008-2009 school year, what is the last day that you can submit your FAFSA?
  4. After completing the assessment portion of the process, we sat informally with each student to get a better idea of what they did and did not like about the system they used. Feedback from this interview portion served to inform the design of our final product.

Results & Conclusions

The results of our assessment can be seen on the chart to the right. Students who used CAIA averaged over twice as many correct answers on the assessment questions, which shows us that there is increased knowledge acquisition with our system. Another interesting point that can be taken away from the assessment is the number of responses that students left blank. Students who used the Federal Student Aid website left over three times as many responses blank as students who used CAIA. This is significant in getting at a less tangible, but very important, measure, which is students having confidence that the knowledge that they are receiving is correct and accurate.

Based on our observations of students during our assessment and feedback during the interview sessions as well as from colleagues, we made adjustments to our original prototype including:

  • Adding captions so students can follow along with CAIA while she is talking
  • Adding a notes section to keep track of information that can be more difficult for students to memorize
  • Changing CAIA's appearance to more closely resemble our target demographic