Search and rescue robotic system evaluation is sub-categoried into three tasks:
* Victim Search Task
* Victim Rescue Task
* Search and Rescue Task
Victim Search Task
Objective
The victim search task aims at evaluation of the system's capability of finding victims under the debris. Any set of equipments (robot, remote controlled devices, etc) are acceptable. Special categories dedicated for specific aspects of technologies may be created, such as a full autonomous system category.
Field Set Up
A simulated desaster site is created with debris as close as possible within logistical and other practical constraints, as well as to meet evaluation purpose. The size of disaster site for the in-door evaluation may be 5m by 5m. Numbers of debris are randomly set and the simulated victim is located randomly. Victims are simulated by specially designed dolls or objects which may emit sound, body heat, CO2, or other possible cue that actual victims may generate.
Evaluation
Performance of each system is evaluated on several criteria:
* Weighted time average for all victiums found. This corresponds to expected number of victims saved over all victims actually under the debris.
* Recover and Accuracy Rate
o Recover Rate: How many victims does the system identified out of all victims under the debris. If the system found 6 victims out of 10 victims actually under the debris, the system has 60% recovery rate.
o Accuracy: How many identification were correct out of all identifications made by the system. If the system spotted 8 places as victims, and actually only 4 of them are correct, accuracy is 50%.
* Operational Loading. Numbers of operations that has to made by each human operator. This measure evaluate how many equipments can be operated at the time of disaster where human resource is bottleneck.
Overall evaluation function will be provided shortly to linealize the evaluation criteira above.
Victim Rescue Task
The victim rescue task evaluates how quickly the system can pull out victims under the debris without hurting the victim itself. The location of the victim is assumed to be identified, and this benchmark evalute mostly on capability of the robot to safely remove debris and move victims out from the debris, to finally bring him/her to the first aid station.
Field Set Up
Compatible with search task. The in-door field is set up within 5 m by 5m field with debris and simujlated victims. Simulated victims are dolls with special pressure sensors that measures possible damanges imposed on their bodyt during the rescue task.
Evaluation
* Total time to took to rescue all victims
* Total damage imposed on victims
* Expected number of life saved
* Operation Load. Total number of operations made by each human operator during the rescue operation.
Search and Rescue Task
The goal os the search and rescue task is to provide a comprehensive task for search and rescue operation. Basically this is a comobiunation of two tasks mentioned above.
Field Set Up
A simulated desaster site is created with debris as close as possible within logistical and other practical constraints, as well as to meet evaluation purpose. The size of disaster site for the in-door evaluation may be 5m by 5m. Numbers of debris are randomly set and the simulated victim is located randomly. Victims are simulated by specially designed dolls or objects which may emit sound, body heat, CO2, or other possible cue that actual victims may generate. It is also equipped with pressure sensors to measure possible damages imposed to the victim during the operation.
Evaluation
Performance of each system is evaluated on several criteria:
* Weighted time average for all victiums found and rescued. This corresponds to expected number of victims saved over all victims actually under the debris.
* Coverage: How many victims does the system identified out of all victims under the debris. If the system discovered 6 victims out of 10 victims actually under the debris, the system has 60% coverage rate.
* Accuracy: How many identification were correct out of all identifications made by the system. If the system spotted 8 places as victims, and actually only 4 of them are correct, accuracy is 50%.
* Rescue Ratio: How many victims discovered were actually saved? If the system found 10 victims, but are only able to save 5 of them, the rescue ratio is 50%.
* Effective Rescue Ratio: How many victims out of all victims under the debris, the system finally saved?
* Operational Loading: Numbers of operations that has to made by each human operator. This measure evaluate how many equipments can be operated at the time of disaster where human resource is bottleneck.