Frequently
Asked Questions

Team Qualification

13. Q: When applying, it says list team members and affiliations, we anticipate adding more people with time. Can I change the number of team members after applying?

    A: Teams will have the opportunity to add members at later, provided they meet the qualifications listed in Terms and Conditions.

14. Q: Is it possible to include a high school student as a team member for the competition? Additionally, can they use this competition project to also serve as their science fair project?

    A: Yes. As stated in the Terms and Conditions:

    “The DARPA Triage Challenge is open to individuals and team members of all nationalities and of all ages, with the following exceptions: All participants under 18 years of age require parental or guardian authorization.”

    For more specifics on who may participate and requirements to participate please refer to the Terms and Conditions.

    While the algorithms and the process to develop them is open to present at a science fair, the actual raw data is protected:

    “Competitors acknowledge DARPA’s mission-requirement and intent to safeguard privacy and civil liberties, and that sensitive or identifying data (including personally identifiable information (PII) or protected health information (PHI)) is not relevant to the DARPA Triage Challenge activities and that DARPA-provided datasets supporting those activities have been intentionally de-identified to ensure—to the greatest extent practicable—that there is no reasonable basis to believe that the data could be used to trace a specific identity or present a risk of harm to any individual. Accordingly, the DARPA Triage Challenge competitors agree they will not intentionally attempt to download, re-identify, share, or re-use DARPA-provided data.”

RITMO

15. Q: What types of interactions do DARPA Triage Challenge competitors have with RITMO performers? Can a DARPA Triage Challenge competitor suggest what new data should be collected?

    A: There will be multiple opportunities for DARPA Triage Challenge competitors to engage with RITMO performers. These interactions will provide critical insights for model development and interpretation based on RITMO performer’s expertise in the patients, physiology, practices, and sensors reflected in their data. Additionally, feedback to RITMO performers could potentially guide adaptive changes to RITMO approaches. However, DARPA Triage Challenge competitors should not plan on their suggestions being incorporated into RITMO plans.

16. Q: Is it too late to apply to RITMO?

    A: Yes, source selection for RITMO has concluded.

17. Q: Who is responsible for cleaning the data, organizing it into a usable format?

    A: Data will be organized and checked for imputing and linkage errors. However, an important part of the Challenge is handling the inherently messy aspects of physiological data recorded in real or simulated trauma settings. Competitors should use methods that provide real-time decision support based on the imperfect inputs expected in military and civilian mass casualty settings.

Primary Triage

18. Q: Is the focus going to be mainly military? Or civilian?

    A: The goal of DARPA Triage Challenge is to address the widespread need for improved triage. There will be both military and civilian scenarios.

19. Q: Can we use outside Gov’t equipment (e.g., furnished by a finished program)?

    A: There is nothing explicit in a challenge that would prevent this. Rules for use of equipment acquired for other contracts would depend on that contract.

20. Q: For the Primary Triage tracks, do you anticipate an API / standard output separate from the UI for scoring? Like a JSON file with location coordinates and assessment details? An API we have to call when detections are made?

    A: DARPA expects to publish an API for interfacing with the DARPA reporting and scoring servers.

21. Q: For the Primary Triage Real-World Challenge event, can you please clarify how the actors / patients will provide accurate signatures (e.g., thermal, breathing rate, pulse, visual paleness, etc.) associated with their simulated injury storyline for the sensors to detect and the algorithms to process against.

    A: DARPA expects to make the real-world competition event scenarios as realistic as possible using both well trained actors and high-fidelity manikins, which will create many opportunities for collection of relevant data through responsiveness, movement (walking, crawling, etc.), posture, anatomy and visible injury patterns, chest motion (e.g., reflecting respiratory or cardiac activity), thermal or other features. Actors will have monitors attached to them to provide accurate ground truth measures.

Algorithm

22. Q: Which algorithmic approach is more likely to succeed?

    A: DARPA will not comment on specific approaches or technologies outside of the formal feedback to proposal abstract submissions. Proposed algorithms must fit within the scope and spirit of the DARPA Triage Challenge Program.

23. Q: How big are the training datasets for Primary Triage?

    A: Few hundred GB for TATRC(real-world video) and ARA (virtual world).

24. Q: Will camera metadata and camera info will be given for the TATRC training data?

    A: Yes.

Sensors

25. Q: Do we need to use commercially available sensors?

    A: As the primary focus of the DARPA Triage Challenge is the development of physiological signatures and analytical algorithms, not hardware, proposers are encouraged to use COTS, or near-COTS sensors, and robotic platforms. However, if a proposing team has an aforementioned component under development that could be modified within the duration of the program and used in the challenge events, it can be included in the proposed approach.

26. Q: How much of our effort can be devoted to sensor development?

    A: The challenge is focused on signature identification not sensor development. While we understand there may need to be some sensor development, the program has a tight timeline. If proposing sensor development, teams are encouraged to include milestones on the path to fielding the sensors for data collection as well as risk mitigation steps.

27. Q: The Primary Competition of the DARPA Triage Challenge seems to be focused on only non-contact sensors. Would a contact sensor approach be completely outside the scope of the challenge?

    A: Developing contact sensors is out of scope for the DARPA Triage Challenge.

28. Q: Will video training data provided by TATRC be available to all tracks?

    A: Yes.

Communication Systems

29. Q: Will DARPA provide any infrastructure for competitors?

    A: DARPA is considering the possibility of providing access to a communications network during the competition events. Prospective teams should include a description of their networking approach and specifications in their proposals.

30. Q: What are the limitations on communication systems? Are there specific frequency bands or protocols that we should adhere to, or are we free to choose our communication technology?

    A: We are currently planning to allow teams to select their own communication technology with the cavate that they many need to shift channels to deconflict and they will not be allowed to use open LTE or 5G. The selected solution must be NDAA 2023 Section 817(a) - compliant. The signals must be encrypted. We may be providing a closed WIFI network for teams that choose to use WIFI.

GPS Usage

31. Q: Is GPS allowed? Do teams need to design systems for GPS-denied navigation as well?

    A: GPS will be allowed in year 1. In future years courses may include indoor components which would degrade GPS in these conditions teams will have access to an IPS solution, still to be determined.

Area Coverage and Time Frame

32. Q: How big are course areas that needs to be covered during the competition and what is the time frame in which teams are expected to complete this task?

    A: The current information can be found in the BAA. Current guidance is that a course would be 100ft x 100ft x 100ft or less and the time to complete the course would be between 15 and 60 minutes depending on the course. This information will be updated with the release of the rules document expected in December.

33. Q: Regarding visual occlusions, will casualties be accessible or completely buried?

    A: Full occlusion is possible in later challenge years. For year one, casualties will be assessable from multiple angles.

34. Q: Are the values of the features on the manikins changing with time? Do we need to align our system’s time with challenge’s time?

    A: There are time-variable physiology engines in the manikins, vitals will trend with scenario settings (casualty cards). Timestamp on the casualty report will be evaluated for accuracy and timestamp on when the report was received will be evaluated for golden window bonus points (see Rules).

Casualty Detection

35. Q: For Primary triage real-world platforms, can teams interact with people who are injured (e.g., talk with them, ask them to move)?

    A: Yes, non-contact interaction is a possible approach.

36. Q: What conditions are you looking for in primary triage? What about alive or dead?

    A: The DARPA Triage Challenge seeks to address scenarios that are relevant to a military or civilian mass casualty incident, and is concerned primarily with identifying casualties who urgently require life-saving treatment by responders (e.g., life-threatening hemorrhage and airway injuries). A variety of conditions are of interest, including those assessed as part of standard military and civilian triage protocols. We are not asking to report on whether the casualty is alive.

37. Q: Why are you not looking at localization and rate of detection in year 1?

    A: The goal in year 1 is to focus on challenges for physiological sensing.

38. Q: How do we reconcile manikin physiology (ie – breathing detected but HR=0)?

    A: Within the realm of what’s feasible – manikins are a little removed from reality. Explicit identification of manikin from humans is not required.

39. Q: How is the mannequins HR being generated?

    A: There’s no actual heartbeat for the manikins. Respiration rate is mimicked by expansion movement. Manikin HR will not be detectable by standoff. We expect only actors and virtual humans will have HR.

40. Q: What is the number of casualties we are expected to detect during the competition?

    A: The number of casualties will increase each year. Expect to start with approximately 10-12 per course in year 1.

41. Q: The initial triage seems to assume static patients, should we expect this?

    A: For Challenge 1, the casualties will not be ambulatory.

Data Display

42. Q: Are teams required to display the data collected over a mobile ground control station (GCS) or a stationary one?

    A: The GCS may be either mobile or stationary. Data collected will be sent over a wired network to the evaluation team.

Human Interface and GUI

43. Q: Could you clarify human elements of decisions are being made during the triage challenge.

    A: For the Systems Competition, systems should be autonomous for sensing and analysis. There should be no human input. Team members are present for safety and oversight to ensure systems/transmissions are working as intended.

44. Q: What are the requirements for the graphical user interface (GUI)? What are some desired functionalities?

    A: While a human interface is required, it will not be part of the scoring at events. However, there will be stakeholders at events that will have opportunities to look at the GUI and potentially give feedback. Usability of the GUI may affect the likelihood of future investment in the system by stakeholders after the DARPA Triage Challenge ends.

45. Q: Is BATDOK a requirement for the user interface portion of the challenge?

    A: It is not required but would be considered useful. If not, that's another consideration to add to future challenge iterations.

    A: It is not a requirement but would be considered a useful feature. Other considerations are easy interpretability of the data for emergency medical personnel and providing scope on the scale of the mass casualty scene.

46. Q: Is the person at the base station making decisions for the primary competition a member of the competition team or from DARPA?

    A: At the team base station there will be a team member who monitors data transmission but not interacting with the autonomous processes.

Virtual Platform

47. Q: What will the virtual track of the Primary triage consist of? Will it be another data driven event or will there be some simulation of the collection of data (e.g., robotic autonomy, etc.)?

    A: DARPA is funding the development of a virtual world to support the Primary Triage Virtual track. There will be simulation of the collection of data. It is anticipated that the complexity of the world and scenarios of the world will develop over the course of the challenge.

48. Q: What is the difference between virtual humans and avatars?

    A: An avatar is a very human looking 3d mesh. A virtual Human is an Avatar with physiology added to it.

49. Q: How do we account for the differences between virtual humans and actors/manikins?

    A: There will be limits to what eh virtual casualties can do. We at ARA want to hear which signals you are looking for so that we can research ways that we may be able to represent them.

50. Q: Can we interact with the virtual humans to uncover / move rubble?

    A: No, similar to the Systems tracks the goal is for teams to do evaluations as standoff.

51. Q: How realistic is the “simulation framework’ presented by ARA? While the visuals are impressive, have they performed any validation?

    A: We are adding information to these MetaHuman meshes. A good example is how we take data from BioGears then animate nodes like Subsurface Scatter to mimic heart beats per minute. These slight changes can be detected by rPPG.

Secondary Triage

52. Q: Will DARPA provide any infrastructure for competitors?

    A: DARPA expects to provide cloud computing resources for Secondary Triage Competitors.

53. Q: Do you anticipate there would be a subset of LSIs that are rare in the RITMO data, or that are administered with little or no previous clinical data for the patient?

    A: Yes, challenge scoring will consider the frequency of specific LSIs in the data and availability of preceding data for predictive model development.

54. Q: For Secondary Triage – we currently have an algorithm that works really well at predicting LSIs but it requires a specific data type or sensor, how do we apply if we don’t know if that data type is available?

    A: The goal of DARPA Triage Challenge secondary triage challenge is to promote the discovery of novel signatures that enable algorithms to predict LSIs using a variety of different data sources. We expect successful approaches will leverage multiple data sources from a variety of sensors and will be robust to drop outs of any given data source.

55. Q: Is there a potential that manually recorded data may not sync up well with automatically recorded data?

    A: Yes, there is a potential for this to happen. Algorithm development should consider the data-collection challenges in trauma settings.

56. Q: Will competitors in the Secondary Triage Challenge be given data specifications for the modeling/training data that will be used—e.g., data format, sampling rates and possible context?

    A: Specific data types for secondary triage are currently being selected. Details on the types of data we expect to have can be found in the RITMO BAA (HR001122S0043) More details on the data specifications provided will be released prior to kick-off.

57. Q: For secondary triage, will the metric of predicting LSIs administered consider the fact that in the real-world data, some LSIs may not have been administered when they were needed?

    A: We will not determine case-by-case whether LSI should have been implemented in the real-world data. Useful models should take this and other real-world considerations into account.

58. Q: Each unit of blood transferred – does it count as separate LSI?

    A: Yes, patients will have variable response to it. Patients that may require 2-3 units of blood is a different problem.

59. Q: Does the rate of transfusion differ depending on location/theatre?

    A: Yes. During helicopter transport, EMT only carry two units of blood.

60. Q: Is LSI event documentation in real time? Video recorded? Expo facto? What’s the error window in reporting?

    A: These are real life resuscitations; documentation is secondary to saving lives. Algorithm will need to accommodate sometimes 20-30 LSIs stacked on top of each other if they are documented at the same time. That’s why we have included additional data such as video recording.

61. Q: Are we getting FAST, eFAST, do we need to ID those types of interventions?

    A: Those systems are not included in the dataset.

62. Q: While working with the RITMO data, what do we do if we see Protected Health Information (PHI)?

    A: Please contact the DARPA team and we will remove it.

63. Q: For the RITMO data, will we have example file or readme?

    A: We will provide a data dictionary.

64. Q: To interpret blood pressure data, do we have information on the patient’s pose?

    A: Patients are all initially supine upon arriving at the trauma bay, after a first 5 min assessment, patients typically sit up, injury allowing. Reality is there’s no way to know their position when the data was taken.

AWS

65. Q: What’s the storage limit per team?

    A: Not determined at this point.

66. Q: Can RITMO data be downloaded from the AWS workspace?

    A: You may use a GitHub repo to access the data. One cannot move data out of AWS environment. APL will set up extraction buckets to be scanned and checked before export.

67. Q: For Sagemaker, what are the compute resources we have?

    A: We limit instances you can spin up. Team members can share the same instance.

Scoring Metrics

68. Q: Can we suggest modifications to the metrics?

    A: Please send your suggestions to the TriageChallenge@darpa.mil mailbox.

Primary Triage

69. Q: How is each casualty scored if we are not asked to localize them?

    A: Each casualty will have a QR code located near it to disambiguate between casualties. Please see rules for more details.

70. Q: The reports are automated reports? No human is involved in generating them? Will reports be submitted in real time?

    A: Yes, reports need to be automated. Operators cannot alternate automatically generated reports. Please see the Rules document for reporting and time-stamp details.

71. Q: For systems track, what is the verbal score? Will the casualty be making noise or talking? Should we prompt first or will they make sound or speak themselves?

    A: The casualties will be making noise eg breathing, moaning, gasping, the will not be likely to talk without prompting.

72. Q: What is the motor alertness score?

    A: Casualties will be making some movement but can be prompted to do additional movement (no walking in year 1).

73. Q: Can we play audio to the casualty or verbally interact?

    A: Verbal communication are allowed, however, the Q&A process. For the virtual competition, we can program slurring, speech, or series of standard questions for the virtual humans with different versions of answers.

74. Q: What are some example cues for actors on courses?

    A: Responsive, less responsive, trying to walk. We may ask them to slow their breathing or hold their breath for a brief period.

75. Q: What is the emphasis on false alarms? What if we see blood on the arms but the injury was actually on the chest?

    A: All actors and manikins will be clothed. Our approach is to develop casualty reports for defensible categories so there’s unambiguous ground truth.

76. Q: For the virtual competition, can you make more reports than the number of casualties?

    A: Yes, we will count the last report as your final report for each casualty.

Secondary Triage

HSR

77. Q: Given the rule that Self-Funded teams are prohibited from the collection or use of any other human subject data as part of their involvement in the DARPA Triage Challenge, are we allowed to bring previously developed algorithms that were trained on human subject data that fell under an IRB protocol?

    A: You may bring algorithms developed under approved IRB protocols prior to the challenge.

78. Q: Per the NIH Investigator Manual for Human Subjects Research: "IRB approval is not needed for secondary research when biospecimens and/or data are fully de-identified/anonymized by removing all identifiers and re-coding or disposing of the code key AND no one collaborating will have any way to re-identify the materials. Can we propose use of these datasets in our technical approach?

    A: As all such datasets would require IRB and HRPO review to determine that they are indeed fully deidentified, you would not be able to bring in deidentified datasets. DARPA will provide access to MIMIC III/IV datasets as additional training data.

79. Q: The Secondary Triage Challenge involves human medical data from the RITMO program; do we need a human subject research (HSR) plan?

    A: No, the medical data will be collected by RITMO performers for the purpose of supporting this Challenge. RITMO data will be deidentified prior to release to DARPA Triage Challenge competitors.

80. Q: In the Secondary triage competition, is all data provided by DARPA or we will have the chance to do some testing for data generation?

    A: Self-funded teams are prohibited from the collection or use of any other human subject data as part of their involvement in the DARPA Triage Challenge because DARPA HSR supervision is not feasible for teams not under DARPA contract.

81. Q: For Systems competition, there’s IRB from the DARPA’s side. Will we get to see the protocol? Do we need to keep data offline that we collect during the workshops?

    A: Yes, we will provide the IRB protocol. For data collected during workshops, you can’t share that data. TATRC will amend the protocol if competitors have sensors outside the scope of their IRB.

82. Q: If we have public, deidentified patient dataset out local IRB determined not HSR, can we use it?

    A: Please send us the protocol, IRB determination to get HRPO approval for use. Until HRPO approval has been granted, you cannot use the dataset for the DARPA Triage Challenge.

83. Q: Can we use HSR data we collected before for training?

    A: No. All HSR datasets used must go through HRPO approval.

84. Q: Can we use algorithms from previous work trained on other data?

    A: Yes, you can use existing algorithms as a starting point.