Performance in Non-Face-to-Face Collaborative Information Environments

Abstract : Using technology to obtain and process information requires training not only in human-computer interaction but also in human-human-computer (collaborative) interaction. Warfighters must not only develop their own situational awareness (SA), they must understand each others' SA (Pew, 1995). This common ground is what each collaboration participant assumes about the others to ensure effective interactions (Ross, 2003; Wellons, 1993). Communication is key. Collaborators must coordinate and share information. Collaboration influences military operations at all levels. Technical interoperability is not enough to produce the synchronization required.

Twenty-eight pairs of college students worked together in pairs over a computer network to locate scud missile launchers in the computer-based game SCUDHunt. The students operated from different rooms in a research suite; they were physically isolated from each other. Each partner had a different array of intelligence gathering assets, with different capabilities and reliabilities, to employ to locate the launchers. After completing a demographics questionnaire, each isolated team member was taught the rules of the game and the characteristics of their assets. After the first training module, the team member completed a paper and pencil test on the training material and was given immediate test feedback. Each team member then received a second training module. For half the teams, this was a repeat of the first module in which each team member learned about the assets he/she would manage in the game; the other half of the teams learned about their partner's assets during this second training module. These conditions were called Own and All respectively. Again, each team member was given a test on the training material just presented and given immediate feedback. Teams then practiced the game for one turn. Experimenters answered any questions regarding operation of the game or the assets. Teams then played two 5-turn games of SCUDHunt selecting the three most likely launcher locations at the end of each turn. For one game each team communicated via voice over headphones (Oral) and used text chat (Chat) via the computer for the other game. After each game, partners completed a task load index. The number of launchers located on each turn was recorded (quality score) as well as the number of map grid locations selected in common by each partner (shared situational awareness).
v Findings: Method of communication had no significant (p>.05) effect upon either the quality or the shared situational awareness scores. The training condition did influence performance with the All teams locating significantly more launchers in game 2 than Own teams which had a significantly higher level of shared situational awareness. The Own teams also reported a higher workload after game 1. Analysis of the text messages indicated greater success in locating launchers in those teams with the highest frequency of messages in the communication categories regarding: game situation, player status and non-task related/social interactions.
These results indicate that performance on collaborative tasks benefits when collaborating participants are cross-trained to have a broader, system-wide view of the entire situation. Additionally, this cross-training reduces the initial workload for the cross-trained teams. This research also indicated identifiable communication patterns related to higher performance. Understanding the types of communication that develop more accurate situational awareness provides insights into how to train Soldiers to communicate effectively in non-face-to-face environments.

Utilization and Dissemination of Findings:
These findings can be used by Army trainers and training developers when designing or conducting training for collaborative network centric distal operations. Emphasis must be placed on the value of cross-training and effective communication among the collaborative team.

CONTENTS (continued)
List of Tables

INTRODUCTION
Today's military operates in a network-enabled environment where technological advances exploit gathering and sharing information to building shared-situational awareness. These technologies have transformed the nature of traditional collaborative work (Easley, Devaraj, and Crant, 2003). The research in this report helps us understand what is different in these network-enabled environments. Sonnenwald & Pierce (1998) observed that using technology to obtain and process information requires training not only in human-computer interaction but also in human-human-computer (collaborative) interaction. Warfighters must not only develop their own situational awareness (SA), they must understand each others' SA (Pew, 1995). This common ground is what each collaboration participant assumes about the others to ensure effective interactions (Ross, 2003;Wellons, 1993). Communication is key. Collaborators must coordinate and share information (Hevel, 2002;Salas, Prince, Baker, and Shrestha, 1995;Sonnenwald & Pierce, 1998). Griffin and Reid caution, Warfare is not 'network centric'. It is either 'people centric', or it has not centre at all. (Griffin & Reid, unpublished manuscript).
Collaboration influences military operations at all levels. Technical interoperability is not enough to produce the synchronization required. Major General Gordon C. Nash, Commander, Joint Warfighting Center, Director, Joint Training J7 states: .Collaboration capabilities can affect all aspects of joint operations. … These capabilities are important to Department of Defense efforts to transform the way we plan and execute joint operations. To accomplish this transformational task, we must improve collaboration among combatant commands, Services, agencies, and multinational partners. An environment of collaboration can enable and integrate such a cooperative effort among these organizations and help the joint force achieve decision superiority.
Information sharing while working towards a common goal is a key aspect of collaboration (Alberts 2003). To gain insights into issues involving collaboration and training, the U.S. Army Research Institute for the Behavioral and Social Sciences (USARI) conducted observations and interviews of U.S. Army personnel who operated digital systems incorporated into their units (Schaab & Dressel, 2003;Schaab, Dressel, & Hayes, 2005). It soon became clear that classroom training on how to use digital systems was not enough. Soldiers understood that job success requires an understanding of how their system interacts with other systems. Developing both a clear sense of how to collaborate with people operating those other systems and an appreciation of how important collaboration is in achieving and maintaining situational understanding requires experience in multiple training exercises that incorporate a variety of scenarios. In one command center, Soldiers actually place two different systems side-by-side and cross-train each other to promote collaboration. They comprehended the need to understand the interrelationship between their roles. Such opportunities to foster mutual understanding become more difficult, of course, when members are dispersed. Alberts (2003) cautions that, without appropriate training and practice, the network-centric environment might actually increase the fog of war rather than provide superior situational understanding. To insure the latter result and avoid the former, the training side of the military, in particular, needs to understand the dynamics of this new environment, where warfighters increasingly interact with their peers and leaders digitally.
To improve the warfighter's ability to perform effectively in this information-rich and changing environment, the military needs to train personnel at all levels to do essential coordination with others across horizontal and vertical networks. The current research examines characteristics of network-collaborative environments (e.g., personnel geographically dispersed, different knowledge sets, unacquainted) and how this environment impacts performance. A second phase of this research will examine different techniques to train participants to collaborate.

RESEARCH CONCEPTS
Researchers first selected a game that both allowed performance, using different communication modes and knowledge sets, to be examined and the collection of communication exchanges. Descriptions of the practical features of the game and several relevant underlying research concepts follow.

Research Venue
SCUDHunt, a game developed by Thought link, INC, was selected for this research on collaboration because it provided a simplified model of the interplay of shared awareness and communication, while permitting independent manipulation of variables thought to affect them. SCUDHunt requires participants to (1) collaborate from distributed locations and (2) share unique information from their intelligence assets for optimal game performance. The goal of the game is simple, to locate three SCUD missile launchers on a map. The game requires geographically-dispersed players to collaborate while executing digital tasks in order to achieve a shared goal.

Cognitive Task Analysis
The research included a cognitive task analysis of this game to identify critical points where collaboration would be beneficial (Ross, 2003). In general, players need to communicate planning strategies and to share gathered information in order to perform effectively. The collaboration areas identified were: Coordinating deployment: Players discuss where best to place their assets on the map grid, with the goals of (1) maximizing coverage of the area remaining to be searched, and (2) using certain assets to verify the results of earlier searches; Interpreting results: Players discuss the reliability of reports from different intelligence-gathering assets, leading to a determination of the likelihood that a SCUD launcher is at any particular location. This involves interpretation of results from the current turn, as well as integration of findings from previous searches.

Common Knowledge
This situation of networked individuals who have shared goals but unique roles and responsibilities raises a new question for research on collaboration at a distance: To what extent does knowledge about a partner's role influence an individual's performance effectiveness? Knowledge about others' roles, responsibilities, and job requirements has been termed "interpositional knowledge." One effective training strategy for increasing interpositional knowledge among team members is cross-training ).
Volpe, Cannon-Bowers, Salas, and Spector (1996) defined cross-training as a strategy where "each team member is trained on the tasks, duties, and responsibilities of his or her fellow team members" (p. 87). This involves having team members understand and sometimes practice each other's skills. The Volpe et al. (1996) initial research on cross-training, as well as an extension (Cannon-Bowers, , showed that those team members who received cross-training were better able to anticipate each other's needs, shared more information, and were more successful in task performance. Additional research has found that cross-training and a common understanding of roles contributes to shared mental model development, effective communication, and improved coordination (McCann, Baranski, Thompson, & Pigeau, 2000;Marks, Sabella, Burke, & Zaccaro, 2002).
Similarly, U.S. Army trainers found that Soldiers needed an understanding of how their digital system interacted and complemented other Army systems (Schaab & Dressel, 2003;Schaab, Dressel & Hayes, 2005). Computer systems training has been modified to include an introductory module on the unique contributions of each system and how they work together to achieve optimal situational understanding.
To investigate such issues, cross-training (called the All condition) versus intensive training in one role (called the Own condition) was included as a variable in this research. Players received either a double dose of training on the tasks they would perform (deployment and interpretation of their own assets) or a single dose of training on both their tasks and on the tasks of their partner; this is referred to as the All training condition.

Communication Patterns and Mode
Recent research has shown that communication content (Urban, Weaver, Bowers, & Rhodenizer, 1996) can influence team coordination and performance. To investigate these relationships, researchers explored how the type of communication during a collaboration effort was associated with success. A two tier communication categorization was used to analyze text chat communication between players (see Appendix A). The first tier consisted of 7 types of communication actions while the second tier considered the game related object of the first tier action. For example, a message could be categorized as a request for information (action) regarding asset capability (object of action). More specifically, consider the communication, "How reliable is your satellite in detecting a SCUD?" First tier (action) is a request for information and the second tier (object of the action) is asset capability.
In addition, communication mode was manipulated. All pairs wore headsets that allowed oral communication during one of the two games they played; during the other game, participants communicated by sending typed messages via an on-screen "chat" box. For a random half of the pairs, the "chat" game came first. The data analyzed included measures of the types and frequency of communication between participants in the "chat" game and game performance.

METHOD Participants
Fifty-six undergraduate students, 31 females and 25 males, received course credit for three hours of participation. Participants ranged in age from 18 to 35 years, with the overwhelming majority between 18 to 25 years. Five participants had military experience. Participants were paired for the research. Forty-two individuals forming 21 pairs had never met. The remaining participants were acquaintances or friends.

Instruments
Questionnaire: At the beginning of the session, participants completed a questionnaire that requested demographic information (e.g., gender, age, and military experience) and computer experience. Workload: The NASA Task Load Index (TLX) (Hart & Staveland, 1988) is a multidimensional rating tool that provides workload scores based on six subscales: Mental Demands, Physical Demands, Temporal Demands, Own Performance, Effort, and Frustration. A definition of each subscale is provided in Appendix B. The NASA TLX was administered to each participant after completing the first and second game of SCUDHunt.
Game Description: The SCUDHunt game presents players with the mission of determining where -on a five-by-five grid board representing the map of a hostile country -the launchers for SCUD missiles are located. Participants are told that there are three SCUD launchers, each in a different fixed location among the 25 squares on the board. On each of five turns, participants deploy intelligence-gathering assets (for example, a reconnaissance satellite or a team of Navy Seals), receive reports from those assets, and create a "strike plan" (to be sent to their fictional commander) indicating their independent best guesses based on all the information received both from that turn and previous turns as to the SCUD launcher locations. They are told that the final strike plans -after the fifth turn -will be used by their commander to direct an attack on the SCUD launchers, and they are given the results of this final strike plan in terms of which bombed location held a now-destroyed launcher. Participants control either air or ground assets (see Figure 1) with each asset having unique capabilities and returning intelligence reports of different reliabilities. For example, eyes on target reports from Human Intelligence assets would be more reliable than reports generated from satellites where sensors must interpret images from great distances. This game then is a representation of a situation in which Soldiers would use digital systems to execute tasks requiring collaboration for successful performance. Measures generated as the game is played are 1) the number of SCUD launcher locations identified, called quality score, 2) the degree to which the two participants on a team chose the same grid squares or location in their independent strike plans, called shared-situational awareness, 3) the number of text chat communications taking place, 4) the categorized content of the text chat, 5) measures of subjective workload reported on the NASA TLX, and 6) responses to questionnaire items on demographics and computer experiences.

Air Control Ground Control
Design. Three primary independent variables for this experiment were training condition, mode of communication and communication patterns. All versus Own training conditions involved training on the characteristic of the information-gathering assets used in the SCUDHunt game. Every participant received, as their first training module, an explanation of the characteristics of the assets they would be controlling. Half of the pairs (the Own condition) received a second exposure to the same asset training; the other half (the All condition) received training in which each participant learned the characteristics of the assets to be controlled by that participant's partner.

Mode of communication (Oral or Chat) was analyzed to determine if communication mode influenced game performance.
Communication patterns, as described in Appendix A, were also evaluated to determine if the frequency of certain types of communication were associated with higher quality scores and increased shared-situational understanding.
Procedure. Upon arrival at the laboratory, individual participants were ushered into separate rooms where they read and signed a standard consent form describing the experiment and their rights as participants. This was followed by each participant completing a questionnaire on demographics and their experience with computers and computer games. Researchers then explained that the experiment involved participants playing a computer game with a partner who was located in another room.
Several computer-based training modules were then presented on 1) the overall aspects of playing the SCUDHunt game and 2) the characteristics of the informationgathering assets used in playing the game. Following each training module, participants took paper and pencil quizzes on the material just presented and were given immediate corrective feedback, if necessary, to ensure that they understood how to play the game and the capabilities of their assets. After this training, the pair played a one-turn practice game, to ensure that the mechanics of playing the game were understood. After the experimenters answered any question the participants might have, the pair played two complete five-turn games of SCUDHunt. During these games, data were automatically collected on 1) the messages participants sent to each other, 2) the degree to which grid squares chosen as targets in the "strike plans" (submitted at the end of each turn) were identical for the two members of the pair, and 3) the number of those chosen target squares that actually contained missile launchers.

Performance by Mode of Communication
All participants played the game in each of the two communication modes (Oral or Chat) with the order of presentation balanced across pairs. Analyses of variance revealed no significant effect (p>.05) of mode of communication upon either the number of Scud launcher identified (the quality score) or the concordance of best guesses of the partners (shared situational awareness).

Performance by Training Condition
Participants were trained either on both their tasks and their partner's tasks (All condition) or they were trained twice on their own tasks (Own condition).
During the first of the two games played, no differences in performance were seen between those trained in the All or the Own conditions in either quality score or shared-situational awareness. In the second game, participants in the All condition located significantly more SCUD launchers than did those in the Own condition (see Figure 2), but those in the Own condition had significantly higher levels of sharedsituational awareness (see Figure 3). This means that participants who were crosstrained in both their role and in their partner's role were more successful in locating SCUD launchers. Participants trained solely in their role achieved higher levels of agreement with their partner on where they thought the SCUD launchers were located, but were wrong more frequently than those who were cross-trained.  Figure 3. Shared-situational awareness for participants trained in the Own versus the All condition. Participants in the Own reported more SCUD launcher location in common in Game 2 than did those in the All condition (F (1, 54) = 4.49, p<.05). No locations in common =0; All 3 locations in common=2 Workload.
Participants completed the NASA TLX after game 1 and game 2. During game 1, those in the Own condition reported higher levels of time demand (F (1, 53) = 11.162, p < .05) and frustration (F (1, 53) = 6.275, p <.05) than those in the All condition (see Figure 4). No differences in workload were reported during Game 2. This suggests that participants in the Own condition initially experienced higher levels of workload than those trained in the All condition.  Figure 4. Comparison of workload in the Own and All conditions. After Game 1, participants in the Own condition reported significant higher levels of Time Demand and Frustration than those in the All condition.

Age, Computer Experience, and Gender Differences
Fifty-one of the 56 participants (91 percent) were 25 years old or younger with none of the participants over age 25 reporting "a lot" of experience with either computer/video games or web-based gaming. Although differences in gaming experience were reported, no significant differences in performance between "older" and younger players were found. Comparisons between older and younger participants are tenuous because these groups are dissimilar in size and the "older" group was between 26-35 years of age.
Relationships between computer experience in gaming and gender further our understanding of how these variables interact. Most participants used computers for email, searching the internet and instant messaging (see Figure 5). Thirteen of 56 participants (23 percent) reported "a lot" of experience with computer/video games, while only 6 of the 56 (11 percent) had "a lot" of experience with web-based gaming. A significant correlation was found (r= -.264, p<.05) between experience with web-based gaming and the number of SCUD launchers located in the first game played. Participants with more web-based gaming experience actually performed worse during the first game. This difference was not seen in the second game. It is possible that prior gaming experience initially resulted in negative transfer to the new game. Additional research would be needed to understand this relationship more fully. No other significant correlations were found between any of the types of computer experience reported and performance. Significant gender differences were found, with males reporting more experience in the categories of web-based gaming (Ҳ 2 .1, 3 = 6.47) and computer/video games (Ҳ 2 .05, 3 = 15.37) than did females (see Tables 1 and 2) Despite these significant differences in gaming experience, no difference in performance, either in level of shared-situational awareness or number of SCUD launchers located, was found between males and females. Although males engage in more computer gaming activities than females, these experiences did not lead to better performance on this game.
Additionally, when looking at males only, no differences in game performances were found as a function of relative gaming experience. These findings imply that regardless of prior gaming experience, people can benefit from training that incorporates serious gaming activities. Also, no significant differences were found in quality scores or shared-situational awareness based on computer experience, military background, or acquaintance with partner.

Communication Patterns
Players with a higher frequency of communications in the categories of game situation, player status, and non-task related/social, identified a significantly greater number of SCUD launchers (Game Situation F (3, 51) = 4.78, p<.05; Player Status (F (3, 51) = 6.43, p<.05; Non-task Related/Social F (3, 51) = 8.166, p<.05). Examples of these types of communication patterns follow.

Game Situation:
The specific game context the players have created by their play of the current game.
"Not much going on in E2." or "Grid square C3 looks like a possible target." Interestingly, the second highest number of communications in each of these three categories occurred when only one Scud launcher was located. This difference was significant only for the Game Situation category (see Figure 6

CONCLUSIONS
These results demonstrate that collaborative tasks benefit when collaborating participants are cross trained to have a broader, system-wide view of the entire situation. Participants trained on both their role and their partner's role (All condition) were more successful in locating SCUD launcher locations and, in Game 1, did so with lower levels of Time Demand and Frustration than found in those trained solely on their role (Own condition). In addition, while those in the Own condition identified more suspected SCUD launcher locations in common (shared-situational awareness) than those in the All condition, these concurrences were more likely to be incorrect locations.
Training Soldiers of the 21 st century to effectively communicate non-face-to-face in order to develop accurate shared-situational awareness is essential in our transforming military. This research implies that Soldiers achieve higher levels of accurate situational awareness when they understand each other's role. Without this shared understanding, Soldiers may agree on what is taking place, but this agreement is more likely to be incorrect.
Participants with high levels of gaming experience performed similarly to those with little or no experience. Males and females performed equally well despite males reporting significantly more experience with gaming than did females. When looking at males only, those with high levels of gaming experience did not perform any differently than those with little or no experience.
Gaming, which was used in this research, is being investigated as a training tool because it may motivate trainees to train more frequently and for longer periods. Although additional research is needed to understand how to maximize the training benefits of gaming, the current research suggests that learning benefits of serious gaming may apply to both gamers and non-gamers.
Identifiable communication patterns are related to performance. Better performers communicated more often. Communication categories related to higher performance include player status, game situation, and non-game related social.
Understanding the types of communication that develop more accurate situational awareness provides insights into how to train Soldiers to communicate effectively in non-face-to-face environments. Good performers keep their partner informed of their status and the current situation. Interestingly, better performers engaged in higher level of non-game related social interaction. Perhaps these social communications aid in building understanding and trust.
Strike plan refers to the plan of selecting which grid squares are most likely to contain targets as a function of previous searches.
-Where are you thinking about striking?
Game rules/operation refer to the general rules which govern the play of the game -This next turn is our last.
-It's only the last strike that matters.
Note: A communication may contain several components each of which needs to be coded; however, each component will have only one code.