Digging into Your Projects

Digging Into Your Projects: How to Ask the Right Question of what needs to be done and what needs to be learned

Description: 

There have been several ways in which humans have carried out Search and Rescue (SAR) operations. Conventionally it has been with manned aviation, or with foot searchers, or some combination thereof. More recently with the advancement in Unmanned Aerial Systems (UAS) and the associated technology, UAS platforms have been utilized in SAR operations, both because of the reduced operating cost, and because of the sensors these platforms can carry.

One main difficulty with UAS operations in SAR missions is the volume of data that can be collected with UAS platforms. This requires time to sift through, and even still, there needs a better way to sift through the images than to have humans do the work, because humans will miss something.

Loc8 is a desktop application that processes images by searching for individual, or groupings of specific colors. With Loc8, a person can prepare a large amount of images and assign the program certain colors for it to look for, and as the program processes these images, it pops up with matches, allowing a person to verify whether or not the match was a false-positive, or if the match was something that needs more attention.

This software is not perfect, and we plan to use this software and address some challenges associated with it, and compare the use of this software to conventional forms of SAR.

Throughout these missions, our group will be mainly utilizing a DJI Mavic 2 Pro for the data collection; however, an M600 might be used when evaluating image resolution differences as well as thermal imagery capabilities.

For all questions and challenges comparing time savings, time that it took for the entire UAS mission to be flown will be accounted for when necessary.

Questions and Challenges:

The first question we want to focus on answering is time savings. We want to compare how much time is saved using Loc8 as opposed to foot searches, or sifting through images manually and identifying targets. We plan to follow through with a SAR mission using various techniques. 

1. Foot-search

1. We plan to have manned foot-searches and attempt to find all the targets on the ground. This technique is similar to what you may have seen on television shows, where a group of people search a woods for a missing person. This operation will be timed, and we will evaluate afterward overall time and how many targets were successfully found.
1. UAS mission with human searcher

 .              We plan to fly a UAS mission to capture aerial images of the target area. These will be sifted through manually by a singular person and timed, and we will evaluate afterward overall time and how many targets were successfully found.

3. UAS mission with multiple human searchers

 .              Identical to number 2, except multiple people will be sifting the images so that there is more eyes looking.

4. UAS mission with Loc8 software

 .              We will use the images from the UAS mission in 2 and process them using Loc8. We will evaluate overall processing time, and how many targets were successfully found, as well as how many false-positives the software produced. Processing settings and colors used will be recorded as well.

These attempts should give us an overall answer to the time differences associated with each method, as well as challenges and benefits to each method. These attempts will also let us become familiar with the software, and what it can and cannot do, and some challenges associated with the software. Some of these challenges will be addressed in our second question/challenge.

The second question is secondary using the Loc8 software, and that is how much accuracy changes with various setting changes.

1. We want to look at how the Loc8 software processes images of different file types, and the benefits and drawbacks of various compressions. An example would be the difference in target identification and false-positives between a .jpg and a raw image, and what effects pixel blending from image compression has on end results.
2. We want to look at how the angle of the sensor affects the quality of the dataset, for example, using an oblique angle such as 60 degrees instead of nadir, and the benefits and drawbacks when it comes to different topography, such as forest, fields, water, or urban environments.
3. We want to compare how the Min Pixels Per Cluster setting in Loc8 affects the turnout of targets vs false-positives, and whether a high or lower Min Pixels Per Cluster produces better results.
4. We want to compare how the quantity of color values affects matches and false-positives.
5. We want to compare how the resolution of the imagery affects matches and false-positives.
6. We want to compare how the color of the clothing in various topographical areas affects the output results or false-positives from Loc8

The third question/challenge has to do with the capabilities of the UAS platform as well as the Loc8 software.

1. We want to determine if it is possible to transmit images from the UAS to a ground station immediately after image capture.
2. If number 1 is possible, we want to identify how Loc8 processes images from a folder, and whether images can be added after processing has begun.

This would help us determine if we could process images in real-time as they are being taken, and if so, how much benefit and time savings could be produced.

The fourth question or challenge we want to address relates to thermal imagery. If we can use thermal imagery and identify a body heat signature, we know thermal imagery could be effectively used in SAR.

1. We want to develop a method to assign standard color values to heat signatures, which would let us use that dataset in Loc8 and scan for a specific color value, which in turn would be associated with a specific heat signature.

This could be useful in various topographical areas, or if the color of the clothing of an individual is unknown.

The last challenge we want to look at is the difference in effectiveness of Loc8 in various topographical areas.

1. We want to fly multiple areas, such as a forest, field, water surface, and urban environment, and document challenges associated with each flying site.
2. We also want to fly multiple times of the day and cloud cover and evaluate how the overall lighting affects the results.

Key Deliverables:

This project will have some valuable deliverables associated with each of the questions we hope to answer, or challenges we plan to address.

For the time effectiveness of SAR methods, we plan to create a time table that depicts the total times it took to locate targets, as well as a total effectiveness, or how many targets were successfully identified. This will include images that were processed through Loc8 and had successful matches, or images with false-positives that we will use as examples.

The accuracy question will include image results from Loc8 including but not limiting to: total processing time, number of accurate matches to the target, and number of false-positives.

If a solution is found to address the real-time transmission idea/challenge, documentation will be provided in the form of a video showing the method in action, as well as overall processing time or time comparison to the other methods.

Deliverables associated with thermal imagery will include how the images were processed or converted to assign color values, as well as how it is standardized between datasets. Further deliverables will be determined after the first part is successfully completed, but we anticipate having matches to heat signatures as well as false-positives as examples, or if it is not possible, we will depict what challenges inhibited us from successfully processing the thermal imagery.

Lastly, we plan to create overall orthomosaics of each flying site--forest, field, water, and urban--and highlighting the location of each target, or we will at least have overall aerial views of each flying site with depictions of where each target was placed. We will then have processed images either correctly identifying these targets, as well as highlighting what false-positives the software identified, and challenges associated with each flying site that were discovered.

Research/ What is out there right now?

There are many organizations attempting to implement a UAV based SAR protocol, there are many variables that need to be addressed before we will be able to successfully implement UAVs into SAR missions. First and foremost, we must understand how a normal SAR mission without UAVs would be carried out. The following figures show how a typical SAR mission will be carried out, provided by the National Search and Rescue Committee. 

Figure 1: Pre incident procedures 

Figure 2: Procedures at the scene of the incident 

Figure 3: Post Incident Procedures

With knowledge of typical SAR protocols and procedures, we can now implement the use of UAS to test how normal protocols and procedures match up with the usage of an unmanned aerial vehicle. From our research into published academic papers on the usage of UAS in SAR missions, most seem to be in the early stages of development and are not ready to implement into a real life scenario, First to Deploy by Gene Robinson is the most in depth procedure out there right now. Most of these papers have also came to many of the same conclusions such as what aircraft platform should be used, as well as encountering many of the same hurdles such as how to get the exact GPS location of the object as well as how best to identify the object. The National Defense University of Malaysia created useful flow charts to display their procedures when conducting UAS SAR operations, pictured in the figures below. 

 Figure 4: SAR Flow chart provided by National Defense University of Malaysia 

Figure 5: Determining Location Flowchart provided by National Defense University of Malaysia

We can test how other procedures will match up against our procedure and the usage of the Loc8 software. Another major hurdle that has been encountered by others and by us is the software producing false positives and false negatives. Below is an image provided by the 

University of Trento Italy of a true negative image (top left), a true positive image (top right), false positive image (bottom left) and a false negative image (bottom right). 

Figure 6: Object Identification  

In order to be successful we will need to address all of these variables and find solutions that will work quickly and effectively.