Exercise Design and Evaluation

Exercise Design and Evaluation of Corporate Emergency Management and Response Capabilities

“Without a sound understanding of the organization’s core capabilities and critical tasks, sound evaluation may not be completed” (DHS, 2013).

On a regular basis, Emergency Solutions International (ESI) is asked to assist firms to design exercises and evaluate the preparedness and response capabilities of their executive teams and organization, to manage challenging crisis.

Setting the Stage

One of the greatest challenges when evaluating an exercise is to have participants perform as they would in an actual event.  Anxiety at the individual or group level can cause poor performance and thus, inhibit accurate measurement of capabilities.   The facilitator must conduct the exercise in a manner that ensures each participant understands that the exercises are practice evaluations, not personal tests.

Scope

When defining the scope of an exercise, the following must considered:

1. Type of exercise (Discussion, Table-top, Functional, Full Scale, etc.)

2. Level of Participant Capabilities

3. Duration

4. Parameters for engagement (usually determined from previous engagements)

Exercise Objectives

Objectives will be based upon existing organizational core capabilities.  Objectives must be succinct and understood by all participants.  The SMART model outlined below may serve as a guide in the creation of exercise objectives.

exercise design

 

It is critical that the exercise objectives align with the planning documents.

Screen Shot 2016-06-23 at 2.41.59 PM

Realistic and appropriate objectives are set with the client, including both strategic and tactical objectives. Strategic objectives are long-term organizational goals to help guide decision-making, while tactical objectives are operational short-term goals,  set with strategic objectives in mind.  Following exercise objectives setting, common stakeholder and agency-specific critical tasks are identified and linked to the objectives, as are the interdependent core capabilities. Core Capabilities are defined by the critical tasks that must be completed by individuals or groups within the crisis organization.  It is the critical tasks which become the subject of the evaluator’s attention.  Often, firms wish to improve upon their existing core capabilities.  As a result, Target Capabilities are written,  and the associated critical tasks are identified and validated as part of the exercise process. Together, these linked objectives, and measures and metrics prompt the creation of objective-testing injects that simulate a worst-case crisis scenario that the organization may face.

Exercise Design Components

Exercise Design Components, or Injects, are fictitious events used to create the structure of the exercise and provide the basis for evaluating the core capabilities as identified in the plan.  Injects are grouped to form a storyline that is a reasonable worst-case scenario.  Injects are written by the exercise designer and documented in a Master Inject Log (MIL).  The MIL lists each event in chronological order, and also outlines the event details, time of occurrence, and the expected action or outcome.

Simulated Real-Life Challenges For Training and Exercising
ESI challenges your organization with real-life situations, enhanced through the use of dynamic simulation injects to enrich the learning experience. Custom and interactive visuals help increase the realism of the exercise while keeping participants interested, excited to learn, and eager to practice the organization’s plans, policies, and procedures in response to an incident.
 

 

Exercise Design and Evaluation Process

Screen Shot 2016-06-23 at 2.44.02 PM

 

Gap Identification

During or after a well-run exercise, participants should be in a position (if they are honest) to identify the gaps in planning documents or the execution of critical tasks.  Planning documents guide critical tasks supporting core capabilities.  Participants get this chance to be honest during a “hotwash,” which follows quickly after the exercise.  The hotwash is a facilitated session where the evaluator keeps notes on the strengths and areas for improvement or refinement.  A more formal debrief is held with the evaluators and, many times, senior staff.

During ESI facilitated exercises, an Issues Board is created whereby participants chart their concerns about gaps as the exercise proceeds.  Often times, participants will say “I’ve always worried about this gap…”

 

Improvement Planning

Capability gaps become the output of the exercise evaluation process.  An Options Analysis will aid in the selection of the proper solutions to address the gaps.  Action must be taken to remedy the identified gaps in procedure.  There must be a communication process to let the participants understand the remediation process.  Plans are modified, budgets are set, and responsibility delegated.  When appropriate, training is provided and, at the next exercise, the newly filled gaps become the focused priority.

 

References:

Federal Emergency Management Agency U.S. Department of Homeland Security. 2013. Homeland Security Exercise and Evaluation Program (HSEEP). Retrieved from http://www.fema.gov/media-library-data/20130726-1914-25045-8890/hseep_apr13_.pdf.