What is a focus group?
A focus group is an anonymous interview
with a group of cadets to gather detailed feedback about a
program, course, or a particular element of a course that
provides richer, deeper data than can be collected by other
assessment tools. Usually, a maximum of 15 cadets are asked
to participate in one focus group session that lasts
approximately 50 minutes.
An experienced facilitator will collect
data anonymously from cadets during the session and compile
the information for your use in assessing the course or
program. Typically, there are three activities completed
during the focus group interview: an index card activity,
used to assess the cadets’ overall impression of the course;
a set of open-ended questions which are contained as a
PowerPoint presentation for open discussion; and a
roundtable activity where small groups of cadets come to
consensus on a few overarching questions.
Finally, focus group interviews are
designed to be as flexible as possible, to ensure that you
get the information you want.
Top of Page
What information about the
course can be provided by a focus group?
A particular focus group interview is
designed by a team of at least two people: the faculty
member(s) requesting the focus group and a member of the CEE
staff. CEE’s role is to help the faculty member prepare
focus group questions to meet the purpose of the particular
focus group. CEE then prepares the materials to be used
during the actual focus group interview session.
CEE has prepared a list of questions that
have been used in past focus group interviews. This list of
questions can serve as a guide to help determine questions a
faculty member might wish to include. Typically, most focus
group interviews are designed to assess how cadets feel
about different elements of the course, such as grading,
instructional techniques, most and least valuable lessons,
strengths and weaknesses, etc. Focus groups can also be used
to collect baseline data for an established course, help
evaluate the effectiveness of an new or experimental course,
or evaluate the usefulness of a particular tool or teaching
method.
A focus group interview is not designed to
assess the instructor per se, but frequently comments will
be made about an instructor or instructors. Cadets are
advised of the purpose of the interview at the very start of
the session and are reminded that the purpose is not to
assess the instructor, but rather the course content or
program and design.
Top of Page
What should I do to prepare
for a focus group?
In general, the preparation and planning
for a focus group begins with the desire of a faculty member
to get feedback. Next, a meeting between the faculty
member(s) and a member of the CEE staff starts the ball
rolling. During this meeting, the specific goals of the
intent of the focus group are discussed and suggestions are
made to assist the faculty members(s) in developing a set of
questions that might meet those goals. Times, dates and
other logistics are also discussed.
Following this initial meeting, CEE will
coordinate with the faculty member(s) to help assemble a set
of questions and will develop a set of PowerPoint slides.
Also during the pre-focus group preparation, CEE will assist
the faculty in selecting cadets from the course, and discuss
other logistics, such as providing snacks or making other
special arrangements.
Top of Page
What do I do after the focus
group?
Cadets who participate in the focus group
are told that their feedback is valuable and that their
comments will be carefully considered by faculty members and
course directors. Therefore, it is very important that the
faculty member requesting the focus group "close the
assessment loop" by spending a few minutes of class time to
thank the cadets for their participation and discuss some of
the findings. Cadets need to know that their comments are
valuable and will be considered for future revisions to the
course or program whenever possible.
It is also important that the data
collected be integrated into the total program assessment
effort for the department, program, or course. An added
benefit of conducting focus groups as part of an on-going
assessment plan is that longitudinal data can be used to
identify trends or patterns. Most accreditation boards are
interested in seeing a curriculum revision process that
shows the collection and utilization of assessment data, and
highlights the impact this data has had on the program
revision process.
Top of Page
Why bother having a focus
group when tests and quizzes can tell me all about cadet
performance?
Any good assessment plan includes data
collection from a variety of sources. Tests and quizzes
allow the assessment of academic "outcomes." Focus groups
allow you to assess how the course or program "process"
contributes to meeting the academic outcomes. Results of a
focus group session will provide excellent qualitative data
in addition to quantitative achievement data that you
normally collect throughout the semester.
Top of Page
Why bother having a focus
group when I can get results from the USAFA End-of-Course
Critiques?
Any good assessment plan includes data
collection from a variety of sources. The USAFA
end-of-course critique provides valuable data for the
assessment of instructors and courses. Focus groups provide
richer, deeper data than course critiques because they are
conducted in groups, they employ "open ended" questions, and
the facilitator has the option of asking additional
questions to clarify cadet responses. Results of a focus
group session will provide excellent qualitative data in
addition to quantitative course critique data that you
normally collect at the end of the semester.
Top of Page
How do I find time
for a focus group?
Deciding early on in the semester to set
aside 50 minutes of time to hold a focus group is the best
option. Ideally, focus groups should be held 4 to 6 weeks
before the end of the semester, after the completion of a
major block of instruction, project, or graded event. If you
can’t allocate any of your regularly scheduled lessons, you
may opt to ask for a "Scheduling Committee Action" so that
cadets can be formally excused from after-class activities
to participate in a focus group session. CEE can show you
the paperwork for this option.
Top of Page
Can the results of a focus
group tell me anything useful, or is it just a forum for
listening to cadets whine?
Qualitative data is very valuable if you
are interested in getting detailed information about cadet
attitudes toward their learning environment. Cadets greatly
appreciate the opportunity to participate in these focus
groups for the most part, as they feel the feedback they
provide can help faculty recognize both the strong and weak
points of the course or program through their viewpoint.
Cadets also want to have some ownership in their learning
and this activity gives them an opportunity to express their
thoughts and ideas without concern for reprimand. The focus
group facilitators will do their best to keep cadets on
topic and will seek constructive criticism. Other cadet
participants typically discourage other cadets from
grandstanding and whining during the focus group session.
Top of Page
How have departments, course
directors and instructors used the information from focus
groups to advance their assessment portfolios and curriculum
revision process?
Since instructors compile their own unique
set of questions a focus group provides an excellent set of
qualitative data that can be combined with other assessment
data (both quantitative and qualitative and process and
product) to get a broader view of the effectiveness of the
course or program. The data is useful to individual
instructors, course directors, and department leaders who
are responsible for revising a course or program or
long-term program analysis as part of a department’s
assessment process. It is important to "close the loop" with
focus group participants by discussing revisions and changes
that resulted from focus group data.
Top of Page
What kind of information
does a focus group provide me with?
Focus group questions are specifically
designed to determine cadet attitudes are toward certain
elements of a course or program. To give you a better idea
of what types of questions are asked, refer to the list of
potential focus group questions. After the focus group, CEE
will provide a report which contains the following:
Cadet satisfaction with the course
depicted by a bar graph showing course satisfaction
ratings (on a 1 to 5 scale) and descriptive words or
phrases supplied by the cadets.
A complete word-processed
transcript of the responses of cadets to all of the
open-ended questions asked during the focus group.
Prioritized lists of the strengths
and weaknesses of the course, which were collected
during a small group activity.
Top of Page
What are some examples of
questions used in focus group sessions?
The open-ended questions presented to the
cadets during the focus group are designed to elicit "deep"
feedback on the issues identified by the instructor or
course director. For example, many instructors are
interested in cadet attitudes toward the content of the
course, or the evaluation methods used. Also, many
instructors want to know what cadets feel are the most
important or significant things they’ve learned in the
course or how the cadets perceive their learning in relation
to stated course goals, objectives or skills.
Typical fifty-minute focus groups include
7 to 10 open-ended items. A compilation of many of the
specific questions instructors have asked during past focus
groups can be found in the "Focus Group Question List,"
which is updated each semester. The questions are organized
into four major categories:
General Course Assessment
Teaching and Advising
Evaluation
Course/Program Specific
We recommend that you have topical areas
in mind before you refer to the list.
Top of Page
Where will the focus group
be held?
Because each focus group is recorded on
audiotape so that a transcript can be made, it is necessary
to conduct the interview in a room that is designed for this
purpose whenever possible. Our teleconferencing center (room
4H45) is the best choice for this as all of the equipment
for presentation and audio recording are already in place.
If necessary, CEE can come to a remote location or classroom
to conduct the focus group, using portable equipment.
Top of Page
What information will CEE
provide me with once the focus group has been held?
Following the actual focus group, CEE will
prepare a report containing all of the data collected and
will meet with the instructor or course director to
"debrief" the results. The report will contain a summary of
all of the activities that comprised the focus group. The
report will include a bar graph summary of the index card
activity, depicting the distribution of cadet satisfaction
ratings along with the words or phrases the cadets used to
describe the course. The report also includes a summary of
the group activity (roundtable) that helps to highlight the
"common threads" or "themes" identified by cadets as
relative strengths and weaknesses of the course. The report
also includes a complete word-processed transcript of the
responses of all cadets to all of the open-ended questions.
This report becomes the property of the
requesting party. It may be shared with other parties only
if the requesting party agrees to such sharing.
Top of Page
How many cadets should
participate in the focus group?
The ideal number is 15 randomly selected
cadets. This is because any groups larger than this make it
elicit responses for all of the open-ended questions.
Fifteen is also an ideal number for the small group activity
as we can create three groups of five cadets.
Top of Page
Do cadets want to
participate in yet another assessment forum such as a focus
group?
A focus group provides an opportunity for
cadets to interact with one another and is viewed by cadets
as quite different than completing a written survey. Our
experience has shown that a forum that promotes open
dialogue is enjoyed by cadets and provides useful feedback
to instructors and course directors.
Top of Page
Are cadets concerned about
what they say during the focus group?
Cadets are informed at the very beginning
of the focus group that their responses will remain
anonymous and non-attributable. Cadets are each assigned a
number at the beginning of the focus group and instructed to
use their number when responding to open-ended questions.
They are also informed that a complete written transcript
will be made available to the department, but that the
transcript will contain numbers, but no names. Our
experience shows that cadets can be counted on to provide
honest, candid responses.
Top of Page
What advantages do focus groups have over
surveys?
Because the focus group is designed to be an interactive
open-dialogue activity, a much more in-depth collection of
assessment data is possible through this technique than
could be garnered by traditional paper surveys. Interaction
and thoughtful discussion, as well as a small group
activity, allow for a more comprehensive collection of
assessment data. The facilitator not only keeps the focus
group on track, but also probes for deeper explanations and
examples. The focus group allows for deeper exploration on
specific topics selected by the instructor or course
director.
Top of Page
1. GeneRal Course Assessment
How could the course be improved?
What is the most important thing you learned in this
course? (or strength)
Is there anything else you would like your instructor(s)
to know which we haven’t covered?
What were your expectations of this course? Were your
expectations met?
What troubles you most about this course? (or weakness)
How does this course give you a better appreciation of
the field of [course/program dependent]? Please explain.
What is the most significant thing you have learned from
this course?
Are there any topics that should be omitted from the
course? Conversely, are there topics in the course which you
feel should be covered in greater depth?
Does this course help you to better understand and
appreciate the role of [subject] in American society and in
the military?
What are the most important outcomes (themes) of the
course?
If you could offer one piece of advice to students who
will enroll in [course name], what would it be?
Did this course help you as a [program] major?
Please comment on the course textbook.
What foundational "elements" do you feel would have been
helpful background or prerequisite knowledge for the content
of this course?
Are there topics presently covered in the course that you
feel should be omitted?
Are there topics that should be added to better serve the
overall purpose of the course?
Are there topics presently covered in the course that you
feel should be more covered in detail?
Does this course give you a better appreciation or
motivation for _____ ? If so, how?
What [program] skills are important to you? Has [course
name] helped you with them?
Do you think [course name] contributes to you becoming a
better officer? In what ways?
These [list of course objectives] are the objectives that
were set out by the instructors for this course. Are these
objectives being met?
What skills have you obtained through this course?
[Students are provided with a list of course skills.]
What skills do you feel weak in? [Students are provided
with a list of course skills.]
What [program] skills are important to you? What skills
has the course helped you with?
How do you view the significance or importance of the
subject material of the course?
What do you feel is the most important topic you’ve
learned in this course?
Do you feel this course has improved your ability to
frame and resolve ill-defined problems?
As a second lieutenant, do you believe [course name] will
help you be proficient in your job?
What aspects of this course did you find most valuable?
What aspects of this course did you find the least
valuable? (Please add any suggestions for improvement you
might have.)
Do you think this course contributes to your becoming a
better student? Please explain.
Did the course affect your thinking about other subjects
you’re studying at USAFA? How so?
Did the course affect your thinking about life? How so?
What do you want your instructor or the course director
to know right now about this course?
If this course is offered again, would you recommend it
to others? Why or why not?
2. Teaching/Advising
How do the teaching methods used in the course help you
learn?
In what ways did your instructor contribute to your
experience in this course?
Please share some examples which teachers in other
courses have used which you feel either enhanced or
detracted from your learning or enjoyment.
Which teaching methods used in the course helped you
learn?
Which activities made this class more enjoyable while
still allowing content coverage? Please give specific
examples.
What could your instructor(s) have done to better
facilitate your learning of the material?
What do teachers in other courses do that significantly
enhance your learning?
How did the use of [course specific] exercises and
current events emphasize real world experiences? What
changes would you make?
What has helped you the most in learning [course name]
material?
Describe your relationship with your advisor. What did
you like best? What did you like the least?
Was the [name of class method being evaluated] helpful in
learning the material? Please explain.
What could strengthen [name of class method being
evaluated]?
3. Evaluation
Do the evaluation methods used in the course allow you to
demonstrate your level of mastery of the course material and
concepts?
What changes in the evaluation methods would you
recommend?
Do the evaluation methods used in this course allow you
to demonstrate what you know about the subject matter?
Do the evaluation methods used in the course allow you to
demonstrate your level of mastery of the subject matter?
Did you understand the grading system for the _______
portion of the course? Was it fair? How would you change it?
Please provide examples.
How do the evaluation methods used in the course (e.g.,
quizzes, homework, exams, etc.) allow you to demonstrate
your level of mastery of the course material and concepts?
What changes would you make?
How did the evaluation methods used in the course allow
you to demonstrate your level of mastery of the course
material and concepts? Please explain.
Did the GRs allow you to demonstrate how much you knew
about the course concepts?
Did the value of the [name of evaluation method] increase
or decrease as the semester progressed?
Please add any other comments you might have about:
Teaching (advising)
Evaluation
Assignments
Classroom activities
4. Course/Program Specific
Why did you major in [program]?
What were your expectations of the [program] major? Were
your expectations met?
Which class in the major did you like best? Why?
Which class in the major did you dislike? Why?
Which class was the most difficult? Why?
Which class was the easiest? Why?
Do you have a better appreciation for the field of
[program]? If so, how? Please explain.
How "user friendly" was the _________ environment?
Please comment on the logistics of using the course
software …. installing, running, printing, etc.
Has this software tool helped you in your ability to
solve complex engineering problems? Why or why not?
How did the [specific course component] help you learn
basic programming concepts?
What difficulties did you have with the [specific course
component]?
Please comment on the _______ software.
Did you use the course website? If so, what did you use
it for? If not, why didn’t you use it?
What recommendations would you make regarding the course
website?
What had you heard about this class beforehand? Were the
rumors true?
How does this course give you a better appreciation of
the field of ___________ ?
Was the __________ a useful tool for learning and
applying the course material?
Which of the case studies were most useful to your
understanding of the material? Which were least useful?
Are there other case studies which you feel may be useful
(e.g. based on other battles, missions, etc.) for this
course?
How did the guest speakers enhance or detract from this
course and your understanding of the material?
What do you feel a course web page should contain in
order to improve your learning?
After experiencing _______, do you feel you have a better
understanding of the operational Air Force?
What do you think you gained by working with the mentors
during this course?
After experiencing this course, are you more or less
interested in the _________ career field? Why?
Several instructors used web pages as a means to
communicate with cadets. Were these effective and useful?
Would you make any changes?
