ANV052 Leveraging Smart Video Search

ANV052 Leveraging Smart Video Search


Application Note: Video – AN052

Leveraging Smart Video Search for Efficient Investigations

2023-05-16 Revision 1.0


Target Audience


This Application Note is intended for all End Users of Brivo Eagle Eye Video who have a need to search historical video and want to better understand how Smart Video Search works, how it can save time when there is a need to scrub video, how to investigate when needed, or to find something or someone across multiple cameras and/or locations. A working knowledge of Brivo Eagle Eye Video is recommended before learning more about Smart Video Search.


Introduction


Smart Video Search is a ground-breaking AI-powered solution that allows users to search for people, vehicles, or objects and immediately find the desired video across cameras at all sites - saving time and money. Users can search for people, vehicles, or objects using natural language search, similar to how people search Google or other search engines.


With Smart Video Search, you can search for the following;

  • People

    • Upper body clothing color

    • Lower body clothing color

    • Gender classification

  • Vehicles

    • Vehicle classification (e.g., car, truck, bus, motorcycle)

    • Vehicle make (e.g., Ford, Chevrolet, Fiat, Tesla)

    • Vehicle color

  • Objects

    • Backpacks

    • Handbags

    • Suitcases

    • Bicycles



Background


Historically, when an investigation of recorded video was required, this was an incredibly manual process that often required a person to scroll through hours of video footage, searching for anomalies or a singular event. As the technology has matured, tools have been implemented by VMS providers to help increase efficiency, but the process still requires lots of time spent scrubbing the video.


Eagle Eye’s Smart Video Search allows you to search for video in the same manner that has become commonplace through the widespread adoption of search engines, e.g., Google. If you know that you are looking for a black Honda, you can search ‘black Honda car’ and get a complete list of all “matches” across all cameras for the selected time period. To avoid the omission of video clips that might be matches, Brivo Eagle Eye Video utilizes a wide search criteria - this results in not only exact matches being displayed but also returns any footage that was detected as a possible match. This is done so that you can be confident that you are seeing all possible matches and that you don’t miss anything.


Using AI-powered Smart Video Search capabilities is a massive leap forward for the industry, and it brings a fundamental shift in how users will interact with their VMS in the future. Eagle Eye Networks will continue training these AI Models so that you continue to see improvements, additional search capabilities, and improved accuracy.


Three Common Use Cases for Smart Video Search


  1. You know exactly what you are looking for or have a good idea of who or what you’re looking for

  2. You know something happened in an approximate time period (overnight or within the last couple of days, as an example), but you don’t know all of the details and are looking to pinpoint the event/activity you are investigating

  3. You are performing proactive video verification to ensure that there were no anomalies during the time period you need to investigate


Architecture of Smart Video Search






The diagram above provides an overview of the architecture of Smart Video Search. This architecture enables the addition of intelligence to any supported ONVIF camera. When motion is detected, the Bridge/CMVR sends key images and the associated video to the Brivo Eagle Eye cloud for

real-time processing. Multiple AI models running in the cloud extract information from these key images and tag the videos. When search criteria are entered, Brivo Eagle Eye Video searches the metadata and displays all matching key images.




Key images received are passed through multiple AI models for inference. For example, when a person is detected, a cropped image of the person is passed to an additional AI model to retrieve clothing color and another AI model to retrieve re-ID. Similarly, after detecting a vehicle, the crop of the vehicle is passed to another AI model to find vehicle classification or vehicle make.


Recommended Bridge/CMVR and Camera Settings


Smart Video Search doesn’t require any additional hardware and works with all Eagle Eye Networks Bridges and CMVRs. Key images are processed by Eagle Eye Networks’ data centers, so make sure that the Bridge/CMVR is not configured to use “Minimum Bandwidth Mode”. This mode uses the least amount of bandwidth possible and overrides any preview transmit settings of the cameras and puts the Bridge into “on-demand” only mode, which is not ideal unless absolutely required. See below for an example of Bridge Settings.


The recommended resolution for the preview video is 640x360 (this size depends on the aspect ratio; 640x480 is another option). The goal of selecting the resolution is to provide just enough pixels within the region of interest to detect a person or vehicle with high accuracy.


When motion is detected, the key image sent to the Eagle Eye cloud will use the preview video resolution that was defined within camera settings. As a reminder, go to the Camera Settings tab -> Resolution -> Preview Video Resolution and choose 640x360 or higher. On a motion event, a thumbnail of this resolution will be sent to the cloud as a key image. Note that higher resolution improves the accuracy of detection. The trade-off is increased bandwidth so keep in mind that as resolution and accuracy increase, so does the use of bandwidth.


The Preview Video -> transmit mode should be set to “always” as shown in the image below. Do not select the “on demand” mode as shown below..



Motion Settings

As noted earlier, Smart Video Search uses motion to trigger key images (or thumbnails) to tag the video. Because of this, it is imperative to set the correct motion settings on cameras. It is also recommended that you create as many regions as necessary and name those regions appropriately. Well-thought-out region names can help group or filter search results and should be taken advantage of where possible.


For example, if there are 10 cameras located around the perimeter of a facility, a region named “fence” can be created for all cameras covering the fence. Now, an additional search filter of “fence” can be used, and search results will now give an overview of all events around the fence by listing all cameras with the associated tag. See application note AN007: Best Practices for Naming Cameras and Accounts Within the Eagle Eye VMS, to learn more.


In the example below, all cameras that are near elevators have been assigned the tag of “Elevator.” You can choose to filter by the “Elevators” tag to review all events associated with those tagged cameras.


In the example below, cars parked in the parking zone may be limiting the investigation capability when the user is interested in searching for a vehicle that entered/exited the facility. Hence, three regions of interest, such as “Parking-Zone-Left,” “Parking-Zone-Right,” and “Lane” would make sense and were used below. Filtering by the region “Lane” will only yield results of vehicles entering/exiting.

Example: Utilize ‘region names’ to help find vehicles that have entered/exited the “Lane” region.


Motion sensitivity and timestamps

If a camera is triggering too many motion events, you can check and adjust the motion sensitivity and size of objects from the Motion tab within Camera Settings as shown below. Do not put the timestamp on the camera stream. If there is no timestamp, go to the Motion tab to make adjustments. For example, if a TV, banner, or statue is triggering false alarms, the area can be masked by creating a new region and then setting the motion for that region to 0.

Example Image: A large TV is masked with Motion Sensitivity set to 0



Density Map

The Density Map helps you quickly key in on any video anomalies detected. The lighter the color square, the less activity there is for that camera. No color/gray scale indicates that there was no activity during the time selected for the search.


In the example below, the Density Map is showing periods of time where no key images were identified and various shades of blue. The dark blue squares in succession on the right half of the image indicate periods of high activity. Hovering over one of the squares will show you a pop-up text indicating the time block and number of key images tagged. Click on the image and it will outline the square, update metrics, and take you directly to the key images for further investigation.

Example Image: A selected tile (square) showing the time block and number of key image


Navigation Options


Two key navigation options are available from the default view:


  1. Click on the tiles → click on the left or right navigation buttons to review images. Once an image of interest is identified, simply click that image to review the metadata that was extracted. Click on the person/vehicle or the metadata to search for matching metadata across all cameras.



  1. Click on a specific camera → click on a time period and view all images matching the search criteria in that period as shown below.


Investigation Scenarios

There are many types of investigative scenarios that can occur. The Eagle Eye Networks Smart Video Search is built to drive efficiency and reduce the time it takes scrubbing video in all the following scenarios:


  • You know exactly what you are looking for or have some idea

  • You know something happened in an approximate time period (overnight, or last couple days, etc.) but don’t know all of the details and need to pinpoint the event/activity you are investigating

  • You are performing a proactive verification to ensure that there were no anomalies during the time period you need to investigate


If you know what you are looking for, you can type the description directly into the search bar. When the time bar has been expanded, you will notice that the days in which the search terms are matched on the timeline are underlined and highlighted as shown in the image below. Time periods with matching events are then highlighted within the density map. Hover over the density map and click on one of the tiles to review images from that time period if required. To view video, simply click on the history browser icon.


You may be receiving hundreds of visitors per day. If needed and in response to a security incident, you want to be able to find the person/vehicle/object as quickly as possible. Type in a description such as person/vehicle or click on the person/vehicle button. Then, choose the time period to narrow down to last night (12 hour or 4 hour windows for example).



Review key images on the tile or simply click on the camera name to drill into a specific camera. There is also an option to filter a camera from the camera drop down menu and there are additional filters that can be applied like using tags or a region of interest, if needed.



Once the person/vehicle of interest is identified and you would like to expand the search to understand more about the person/vehicle’s movements across the facility, to see who else they interacted with, or to find out what other places they visited, etc., you can further expand your search. This capability is enabled by default with the Standard Edition and is called, “Metadata Search.”


This is a search based on the description of a person or vehicle. For example, a “Man in white shirt with backpack” or a “Red Tesla.” Once a person of interest is identified, simply click on the person or the metadata listed below the image to broaden the search and look for similar metadata across the site.



Incident Explorer

For those utilizing the Enterprise or Professional Editions of Brivo Eagle Eye Video, you can further broaden your search and avoid clutter through the use of the incident explorer. The incident explorer works in conjunction with the “Locations” functionality. Once the Incident Explorer is opened, you will notice that it has pre-created a story of what was happening, with the option to explore the event on a timeline. Explore the incident further by selecting the event tiles in the grid of nearby times.



Re-Id Search and Personal Data

Re-identification is a non-invasive way to search for similar-looking people across multiple cameras and locations in the Professional and Enterprise Editions of Brivo Eagle Eye Video. Standard Edition users can still search across multiple cameras using metadata (but not the re-ID) as described above. Additionally, all Editions can also search a single camera spanning multiple time periods.


The re-identification algorithm provides a signature vector for a person. This data is stored in the cloud but does not contain Personally identifiable information (PII) or any Personal Data (PD). When a search for a person is performed, Brivo Eagle Eye Video looks for similar signature vectors to identify them across time and across multiple cameras and locations.


Within the Incident Explorer, highlight a person and click on the “Search” icon to initiate a re-ID search as shown below.

Example Image: Click on the Search icon to initiate a re-I search

See the example image below for an explanation of the Incident Explorer tab and to better understand included functionality, as well as where to go to initiate the re-ID search (by clicking on the search icon).



Searching for anomalies

Security operators who use the VMS on a daily basis are able to spot anomalies quite easily using the density map. The light blue tiles surrounded by an outline on the density map identify when there was something that took place that might be worth exploring further. This could be a suspicious person, unexpected motion, or the presence of a search term. Similarly, a non-colored tile might indicate the absence of anomalies and help save you time by not requiring you to review those time periods.


For a given search term, the density map allows users to focus on abnormal activities in a given time period. See the images below highlighting a few examples of this.


Example Image: Identify if workers arrived on-site and as scheduled. Arrivals, departures and absence periods can be gleaned from a single place.

Example Image: Identify unexpected activity or people in areas that may have had vandalism or where no activity was expected.

Example Image: Identify unusual absences. Look for absences in the Density Map (non-colored squares) to determine if someone is not where they should be.