Home
Teaching and Learning
Assessment

Assessment Model
Focus Group
Assessment Catalog
Research

Directorate for Education
U.S. Air Force Academy
Suite 4K25
2354 Fairchild Dr.
USAF Academy, CO
80840-6210

 

DSN: 333-2740
(719) 333-2740
Assessment FAQ

Frequently Asked Questions about Focus Groups for Course and Program Evaluation:

  1. What is a focus group?
  2. What information about the course can be provided by a focus group?
  3. What should I do to prepare for a focus group?
  4. What should I do after the focus group?
  5. Why bother having a focus group when tests and quizzes can tell me all about cadet performance?
  6. Why bother having a focus group when I can get results from the USAFA End-of-Course Critiques?
  7. How do I find time for a focus group?
  8. Can the results of a focus group tell me anything useful, or is it just a forum for listening to cadets whine?
  9. How have departments, course directors and instructors used the information from focus groups to advance their assessment portfolios and curriculum revision process?
  10. What kind of information does a focus group provide me with?
  11. What are some examples of questions used in focus group sessions?
  12. Where will the focus group be held?
  13. What information will CEE provide me with once the focus group has been held?
  14. How many cadets should participate in the focus group?
  15. Do cadets want to participate in yet another assessment forum such as a focus group?
  16. Are cadets concerned about what they say during the focus group?
    1. What is a focus group?

      A focus group is an anonymous interview with a group of cadets to gather detailed feedback about a program, course, or a particular element of a course that provides richer, deeper data than can be collected by other assessment tools. Usually, a maximum of 15 cadets are asked to participate in one focus group session that lasts approximately 50 minutes.

      An experienced facilitator will collect data anonymously from cadets during the session and compile the information for your use in assessing the course or program. Typically, there are three activities completed during the focus group interview: an index card activity, used to assess the cadets’ overall impression of the course; a set of open-ended questions which are contained as a PowerPoint presentation for open discussion; and a roundtable activity where small groups of cadets come to consensus on a few overarching questions.

      Finally, focus group interviews are designed to be as flexible as possible, to ensure that you get the information you want.

      Top of Page

      What information about the course can be provided by a focus group?

      A particular focus group interview is designed by a team of at least two people: the faculty member(s) requesting the focus group and a member of the CEE staff. CEE’s role is to help the faculty member prepare focus group questions to meet the purpose of the particular focus group. CEE then prepares the materials to be used during the actual focus group interview session.

      CEE has prepared a list of questions that have been used in past focus group interviews. This list of questions can serve as a guide to help determine questions a faculty member might wish to include. Typically, most focus group interviews are designed to assess how cadets feel about different elements of the course, such as grading, instructional techniques, most and least valuable lessons, strengths and weaknesses, etc. Focus groups can also be used to collect baseline data for an established course, help evaluate the effectiveness of an new or experimental course, or evaluate the usefulness of a particular tool or teaching method.

      A focus group interview is not designed to assess the instructor per se, but frequently comments will be made about an instructor or instructors. Cadets are advised of the purpose of the interview at the very start of the session and are reminded that the purpose is not to assess the instructor, but rather the course content or program and design.

      Top of Page

      What should I do to prepare for a focus group?

      In general, the preparation and planning for a focus group begins with the desire of a faculty member to get feedback. Next, a meeting between the faculty member(s) and a member of the CEE staff starts the ball rolling. During this meeting, the specific goals of the intent of the focus group are discussed and suggestions are made to assist the faculty members(s) in developing a set of questions that might meet those goals. Times, dates and other logistics are also discussed.

      Following this initial meeting, CEE will coordinate with the faculty member(s) to help assemble a set of questions and will develop a set of PowerPoint slides. Also during the pre-focus group preparation, CEE will assist the faculty in selecting cadets from the course, and discuss other logistics, such as providing snacks or making other special arrangements.

      Top of Page

      What do I do after the focus group?

      Cadets who participate in the focus group are told that their feedback is valuable and that their comments will be carefully considered by faculty members and course directors. Therefore, it is very important that the faculty member requesting the focus group "close the assessment loop" by spending a few minutes of class time to thank the cadets for their participation and discuss some of the findings. Cadets need to know that their comments are valuable and will be considered for future revisions to the course or program whenever possible.

      It is also important that the data collected be integrated into the total program assessment effort for the department, program, or course. An added benefit of conducting focus groups as part of an on-going assessment plan is that longitudinal data can be used to identify trends or patterns. Most accreditation boards are interested in seeing a curriculum revision process that shows the collection and utilization of assessment data, and highlights the impact this data has had on the program revision process.

      Top of Page

      Why bother having a focus group when tests and quizzes can tell me all about cadet performance?

      Any good assessment plan includes data collection from a variety of sources. Tests and quizzes allow the assessment of academic "outcomes." Focus groups allow you to assess how the course or program "process" contributes to meeting the academic outcomes. Results of a focus group session will provide excellent qualitative data in addition to quantitative achievement data that you normally collect throughout the semester.

      Top of Page

      Why bother having a focus group when I can get results from the USAFA End-of-Course Critiques?

      Any good assessment plan includes data collection from a variety of sources. The USAFA end-of-course critique provides valuable data for the assessment of instructors and courses. Focus groups provide richer, deeper data than course critiques because they are conducted in groups, they employ "open ended" questions, and the facilitator has the option of asking additional questions to clarify cadet responses. Results of a focus group session will provide excellent qualitative data in addition to quantitative course critique data that you normally collect at the end of the semester.

      Top of Page

      How do I find time for a focus group?

      Deciding early on in the semester to set aside 50 minutes of time to hold a focus group is the best option. Ideally, focus groups should be held 4 to 6 weeks before the end of the semester, after the completion of a major block of instruction, project, or graded event. If you can’t allocate any of your regularly scheduled lessons, you may opt to ask for a "Scheduling Committee Action" so that cadets can be formally excused from after-class activities to participate in a focus group session. CEE can show you the paperwork for this option.

      Top of Page

      Can the results of a focus group tell me anything useful, or is it just a forum for listening to cadets whine?

      Qualitative data is very valuable if you are interested in getting detailed information about cadet attitudes toward their learning environment. Cadets greatly appreciate the opportunity to participate in these focus groups for the most part, as they feel the feedback they provide can help faculty recognize both the strong and weak points of the course or program through their viewpoint. Cadets also want to have some ownership in their learning and this activity gives them an opportunity to express their thoughts and ideas without concern for reprimand. The focus group facilitators will do their best to keep cadets on topic and will seek constructive criticism. Other cadet participants typically discourage other cadets from grandstanding and whining during the focus group session.

      Top of Page

      How have departments, course directors and instructors used the information from focus groups to advance their assessment portfolios and curriculum revision process?

      Since instructors compile their own unique set of questions a focus group provides an excellent set of qualitative data that can be combined with other assessment data (both quantitative and qualitative and process and product) to get a broader view of the effectiveness of the course or program. The data is useful to individual instructors, course directors, and department leaders who are responsible for revising a course or program or long-term program analysis as part of a department’s assessment process. It is important to "close the loop" with focus group participants by discussing revisions and changes that resulted from focus group data.

      Top of Page

      What kind of information does a focus group provide me with?

      Focus group questions are specifically designed to determine cadet attitudes are toward certain elements of a course or program. To give you a better idea of what types of questions are asked, refer to the list of potential focus group questions. After the focus group, CEE will provide a report which contains the following:

          Cadet satisfaction with the course depicted by a bar graph showing course satisfaction ratings (on a 1 to 5 scale) and descriptive words or phrases supplied by the cadets.

          A complete word-processed transcript of the responses of cadets to all of the open-ended questions asked during the focus group.

          Prioritized lists of the strengths and weaknesses of the course, which were collected during a small group activity.

      Top of Page

      What are some examples of questions used in focus group sessions?

      The open-ended questions presented to the cadets during the focus group are designed to elicit "deep" feedback on the issues identified by the instructor or course director. For example, many instructors are interested in cadet attitudes toward the content of the course, or the evaluation methods used. Also, many instructors want to know what cadets feel are the most important or significant things they’ve learned in the course or how the cadets perceive their learning in relation to stated course goals, objectives or skills.

      Typical fifty-minute focus groups include 7 to 10 open-ended items. A compilation of many of the specific questions instructors have asked during past focus groups can be found in the "Focus Group Question List," which is updated each semester. The questions are organized into four major categories:

      General Course Assessment

      Teaching and Advising

      Evaluation

      Course/Program Specific

      We recommend that you have topical areas in mind before you refer to the list.

      Top of Page

      Where will the focus group be held?

      Because each focus group is recorded on audiotape so that a transcript can be made, it is necessary to conduct the interview in a room that is designed for this purpose whenever possible. Our teleconferencing center (room 4H45) is the best choice for this as all of the equipment for presentation and audio recording are already in place. If necessary, CEE can come to a remote location or classroom to conduct the focus group, using portable equipment.

      Top of Page

      What information will CEE provide me with once the focus group has been held?

      Following the actual focus group, CEE will prepare a report containing all of the data collected and will meet with the instructor or course director to "debrief" the results. The report will contain a summary of all of the activities that comprised the focus group. The report will include a bar graph summary of the index card activity, depicting the distribution of cadet satisfaction ratings along with the words or phrases the cadets used to describe the course. The report also includes a summary of the group activity (roundtable) that helps to highlight the "common threads" or "themes" identified by cadets as relative strengths and weaknesses of the course. The report also includes a complete word-processed transcript of the responses of all cadets to all of the open-ended questions.

      This report becomes the property of the requesting party. It may be shared with other parties only if the requesting party agrees to such sharing.

      Top of Page

      How many cadets should participate in the focus group?

      The ideal number is 15 randomly selected cadets. This is because any groups larger than this make it elicit responses for all of the open-ended questions. Fifteen is also an ideal number for the small group activity as we can create three groups of five cadets.

      Top of Page

      Do cadets want to participate in yet another assessment forum such as a focus group?

      A focus group provides an opportunity for cadets to interact with one another and is viewed by cadets as quite different than completing a written survey. Our experience has shown that a forum that promotes open dialogue is enjoyed by cadets and provides useful feedback to instructors and course directors.

      Top of Page

      Are cadets concerned about what they say during the focus group?

      Cadets are informed at the very beginning of the focus group that their responses will remain anonymous and non-attributable. Cadets are each assigned a number at the beginning of the focus group and instructed to use their number when responding to open-ended questions. They are also informed that a complete written transcript will be made available to the department, but that the transcript will contain numbers, but no names. Our experience shows that cadets can be counted on to provide honest, candid responses.

      Top of Page

      What advantages do focus groups have over surveys?

      Because the focus group is designed to be an interactive open-dialogue activity, a much more in-depth collection of assessment data is possible through this technique than could be garnered by traditional paper surveys. Interaction and thoughtful discussion, as well as a small group activity, allow for a more comprehensive collection of assessment data. The facilitator not only keeps the focus group on track, but also probes for deeper explanations and examples. The focus group allows for deeper exploration on specific topics selected by the instructor or course director.

      Top of Page

      1. GeneRal Course Assessment

      How could the course be improved?

      What is the most important thing you learned in this course? (or strength)

      Is there anything else you would like your instructor(s) to know which we haven’t covered?

      What were your expectations of this course? Were your expectations met?

      What troubles you most about this course? (or weakness)

      How does this course give you a better appreciation of the field of [course/program dependent]? Please explain.

      What is the most significant thing you have learned from this course?

      Are there any topics that should be omitted from the course? Conversely, are there topics in the course which you feel should be covered in greater depth?

      Does this course help you to better understand and appreciate the role of [subject] in American society and in the military?

      What are the most important outcomes (themes) of the course?

      If you could offer one piece of advice to students who will enroll in [course name], what would it be?

      Did this course help you as a [program] major?

      Please comment on the course textbook.

      What foundational "elements" do you feel would have been helpful background or prerequisite knowledge for the content of this course?

      Are there topics presently covered in the course that you feel should be omitted?

      Are there topics that should be added to better serve the overall purpose of the course?

      Are there topics presently covered in the course that you feel should be more covered in detail?

      Does this course give you a better appreciation or motivation for _____ ? If so, how?

      What [program] skills are important to you? Has [course name] helped you with them?

          What do you see as the course objectives or intended outcomes of the course?

      Do you think [course name] contributes to you becoming a better officer? In what ways?

      These [list of course objectives] are the objectives that were set out by the instructors for this course. Are these objectives being met?

      What skills have you obtained through this course? [Students are provided with a list of course skills.]

      What skills do you feel weak in? [Students are provided with a list of course skills.]

      What [program] skills are important to you? What skills has the course helped you with?

      How do you view the significance or importance of the subject material of the course?

      What do you feel is the most important topic you’ve learned in this course?

      Do you feel this course has improved your ability to frame and resolve ill-defined problems?

      As a second lieutenant, do you believe [course name] will help you be proficient in your job?

      What aspects of this course did you find most valuable?

      What aspects of this course did you find the least valuable? (Please add any suggestions for improvement you might have.)

      Do you think this course contributes to your becoming a better student? Please explain.

      Did the course affect your thinking about other subjects you’re studying at USAFA? How so?

      Did the course affect your thinking about life? How so?

      What do you want your instructor or the course director to know right now about this course?

      If this course is offered again, would you recommend it to others? Why or why not?

      2. Teaching/Advising

      How do the teaching methods used in the course help you learn?

      In what ways did your instructor contribute to your experience in this course?

      Please share some examples which teachers in other courses have used which you feel either enhanced or detracted from your learning or enjoyment.

      Which teaching methods used in the course helped you learn?

      Which activities made this class more enjoyable while still allowing content coverage? Please give specific examples.

      What could your instructor(s) have done to better facilitate your learning of the material?

      What do teachers in other courses do that significantly enhance your learning?

      How did the use of [course specific] exercises and current events emphasize real world experiences? What changes would you make?

      What has helped you the most in learning [course name] material?

      Describe your relationship with your advisor. What did you like best? What did you like the least?

      Was the [name of class method being evaluated] helpful in learning the material? Please explain.

      What could strengthen [name of class method being evaluated]?

      3. Evaluation

      Do the evaluation methods used in the course allow you to demonstrate your level of mastery of the course material and concepts?

      What changes in the evaluation methods would you recommend?

      Do the evaluation methods used in this course allow you to demonstrate what you know about the subject matter?

      Do the evaluation methods used in the course allow you to demonstrate your level of mastery of the subject matter?

      Did you understand the grading system for the _______ portion of the course? Was it fair? How would you change it? Please provide examples.

      How do the evaluation methods used in the course (e.g., quizzes, homework, exams, etc.) allow you to demonstrate your level of mastery of the course material and concepts? What changes would you make?

      How did the evaluation methods used in the course allow you to demonstrate your level of mastery of the course material and concepts? Please explain.

      Did the GRs allow you to demonstrate how much you knew about the course concepts?

      Did the value of the [name of evaluation method] increase or decrease as the semester progressed?

      Please add any other comments you might have about:

      Teaching (advising)

      Evaluation

      Assignments

      Classroom activities

      4. Course/Program Specific

      Why did you major in [program]?

      What were your expectations of the [program] major? Were your expectations met?

      Which class in the major did you like best? Why?

      Which class in the major did you dislike? Why?

      Which class was the most difficult? Why?

      Which class was the easiest? Why?

      Do you have a better appreciation for the field of [program]? If so, how? Please explain.

      How "user friendly" was the _________ environment?

      Please comment on the logistics of using the course software …. installing, running, printing, etc.

      Has this software tool helped you in your ability to solve complex engineering problems? Why or why not?

      How did the [specific course component] help you learn basic programming concepts?

      What difficulties did you have with the [specific course component]?

      Please comment on the _______ software.

      Did you use the course website? If so, what did you use it for? If not, why didn’t you use it?

      What recommendations would you make regarding the course website?

      What had you heard about this class beforehand? Were the rumors true?

      How does this course give you a better appreciation of the field of ___________ ?

      Was the __________ a useful tool for learning and applying the course material?

      Which of the case studies were most useful to your understanding of the material? Which were least useful?

      Are there other case studies which you feel may be useful (e.g. based on other battles, missions, etc.) for this course?

      How did the guest speakers enhance or detract from this course and your understanding of the material?

      What do you feel a course web page should contain in order to improve your learning?

      After experiencing _______, do you feel you have a better understanding of the operational Air Force?

      What do you think you gained by working with the mentors during this course?

      After experiencing this course, are you more or less interested in the _________ career field? Why?

      Several instructors used web pages as a means to communicate with cadets. Were these effective and useful? Would you make any changes?


Point of Contact:

Director of Assessment
If you have any questions or comments concerning the contents of this web-site, please call 719-333-7990.

 

 

 
U.S. Air Force Academy, USAFA, CO 80840, (719) 333-1110 DSN: 333-1110, Updated: 25 Apr 14
Privacy & Security Notice   |   External Link Disclaimer   |   Contact Us / Submit Feedback / Webmaster | Accessibility/Section 508