Birdsnap: It's a Plover, it's a Turnstone, it's Killdeer!

As a part of Ben Shneiderman's course on How to do Great Research, graduate students are writing short articles on research that inspires them.

“The sharp thrill of seeing them [killdeer birds] reminded me of childhood happiness, gifts under the Christmas tree, perhaps, a kind of euphoria we adults manage to shut out most of the time. This is why I bird‐watch, to recapture what it's like to live in this moment, right now." — Lynn Thomson, Birding with Yeats: A Memoir

David Jacobs, professor of computer science at UMD, and his colleague Peter Belhumeur professor of computer science at Columbia University, have developed a new app that can recognize the species of a bird. All bird watching enthusiasts have to do is to upload an image of the bird and the app will tell them   everything   they   need   to    know
about that bird. In order to achieve this Birdsnap uses the same techniques that are used in facial recognition applications.

The app recognizes a bird by finding 17 different body parts of the bird (e.g., beak, tail, etc) then considers the relative location of these features and compares it to a database of 49,829 images of 500 of the most common species of North American birds. The likelihood of each species is calculated and the most likely ones are found.

Here are Sample images from the Birdsnap dataset, with bounding boxes and feature annotations. As can be seen in these images, body parts of different species have different relative distances. This difference is used heavily in the recognition process.

Many modern cameras embed the time and location of capture in the image files they produce which Birdsnap can use to refine its matching. Biological categories in particular often have a well‐studied geographic distribution, and it is wasteful not to use this information. For migratory animals, the distribution depends on time as well as location.

The importance of capturing location and time is visible in this image. Fixed‐time slices of spatio‐temporal prior show the Barn Swallow arriving from South America during its spring migration (above) and established in its summer grounds (below). Brighter regions indicate higher likelihood of a sighting.

With this feature, Birdsnap can make a more accurate decision when trying to find the most likely species. For example, if the image recognition tells us that a bird is either a Barn Swallow or something else, and we also know that the image was taken in March, in one of the northern states, we can say with some confidence that the bird is not a Barn Swallow. This features allows us to filter inaccurate results from the set of likely species.

Even using all of these features to help classify the species of a bird there are still many birds who are physically very similar so the relative distances cannot help with recognition, who have similar habitats or migrational patterns. For these situations, the app has the ability to instruct the user on how to distinguish visually similar species. After the algorithm finds the most likely species, users have to decide which one is the correct species. The instructions provided by app, make it very easy for the user to recognize the correct species.

So, before you head out bird watching make sure you download the free Birdsnap app from the App Store and make sure you have your phone with you. Birdsnap will be there to help you!

References:

1. Berg, T., Liu, J., Lee, S. W., Alexander, M. L., Jacobs, D. W., & Belhumeur, P. N. (2014, June). Birdsnap: Large‐scale fine‐grained visual categorization of birds. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on (pp. 2019‐2026). IEEE.

Contact Information:

Josh Reese (jreese [-at-] cs [dot] umd [dot] edu)
Parsa Saadatpanah (parsa [-at-] cs [dot] umd [dot] edu) Sunandita Patra (patras [-at-] cs [dot] umd [dot] edu) Melika