AI video analytics software turns camera feeds into structured events: a person entered a restricted zone, a forklift came too close to a worker, a license plate passed a gate, a shelf went empty. Building it well is less about choosing a clever model and more about camera placement, testing on your own footage, controlling false alarms, and handling privacy law.
This guide explains the pipeline, the capabilities that are mature versus still shaky, the hardware and cost drivers, and how to judge a vendor or an in-house plan.
The pipeline, end to end
Almost every video analytics system follows the same flow, whether it runs on a box in a warehouse or in a cloud region.
- Ingest. Pull streams from IP cameras, typically over RTSP, with discovery and configuration through ONVIF on compliant cameras. Existing video management systems (VMS) can often forward streams or events instead.
- Decode and sample. Decode H.264 or H.265 video, ideally on hardware decoders, and sample frames at the rate the task needs. Counting people at a door might need 5 frames per second; reading plates on a fast road needs more.
- Inference. Run detection, segmentation, classification, OCR or pose models on each sampled frame or region.
- Tracking. Link detections across frames into tracks so the system knows it is the same object over time. Most useful rules (dwell time, line crossing, direction) depend on tracking, not single-frame detection.
- Rules and events. Apply zones, lines, schedules and thresholds to tracks to produce events: "vehicle stopped in no-parking zone for more than 120 seconds".
- Delivery. Push events with a short clip or snapshot to operators, a VMS, a ticketing system or an API, and store metadata for search and reporting.
Core capabilities and how mature they are
| Capability | Typical uses | Maturity | Main pitfalls |
|---|---|---|---|
| Object detection (people, vehicles, PPE, packages) | Intrusion, occupancy, safety compliance | Mature for common classes | Small or distant objects, unusual angles, night and rain |
| Multi-object tracking | Counting, dwell time, line crossing, heatmaps | Mature in moderate crowds | ID switches in dense crowds and occlusion |
| License plate recognition (ANPR/LPR) | Gates, parking, tolling, logistics yards | Mature with proper cameras | Needs dedicated angle, shutter speed and IR; regional plate formats |
| OCR on containers, labels, meters | Logistics, utilities, manufacturing | Good on controlled views | Glare, dirt, motion blur |
| Logo and product recognition | Shelf monitoring, sponsorship measurement | Good with custom training | New packaging requires retraining |
| Defect inspection | Production-line quality control | Strong in fixed setups | Rare defect types have few training examples |
| Pose and fall detection | Care homes, worker safety | Moderate | False alarms from sitting, bending, partial views |
| Anomaly detection | "Something unusual happened" | Inconsistent | Hard to define normal; noisy alerts |
| Natural-language video search with vision-language models | "Find the red van near dock 3 yesterday" | Improving quickly | Compute cost; needs verification by a person |
| Facial recognition | Access control with consent | Technically capable | Heavily regulated; bias and accuracy concerns; often prohibited or high-risk |
Vision-language models changed the field in the last couple of years. Instead of training a detector for every new question, teams can index footage with embeddings or captions and search it in plain language, or ask a model to describe a clip. That is powerful for investigations and audit, but it is slower and costlier per frame than a purpose-built detector, so production systems typically use small, fast detectors for real-time alerts and reserve larger models for search and review.
Edge, cloud, or hybrid
Where inference runs is the biggest architectural decision.
- Edge (on the camera or a GPU box on site): low latency, low bandwidth, and video can stay on the premises, which helps with privacy. You manage hardware, updates and failures across sites.
- Cloud: easy to scale and update, good for heavy models and cross-site search. Uploading many continuous HD streams is expensive in bandwidth and raises data-transfer questions.
- Hybrid: the common answer. Detect and track at the edge, send only events, clips and metadata to the cloud for dashboards, search and model improvement.
Some quick arithmetic shows why bandwidth matters. A 1080p H.264 stream at an assumed 4 Mbps is about 0.5 MB per second, roughly 1.8 GB per hour or 43 GB per camera per day. Fifty cameras streaming continuously to the cloud is over 2 TB a day before you analyze anything. Sending only events cuts that by orders of magnitude.
Compute scales with streams × analyzed frames per second × model size. Analyzing at 5 fps instead of 30 fps cuts inference work about sixfold, and cropping to regions of interest cuts it further. Benchmark your chosen models on your target hardware early; vendor throughput claims rarely match your camera mix.
Camera-heavy deployments often sit alongside other sensors; the guides to industrial IoT systems and smart pole infrastructure cover the device side, and fleet management covers in-vehicle cameras and telematics.
Accuracy: test on your own footage
Published benchmark scores tell you little about your loading dock at 6 a.m. in fog. What matters:
- Collect a test set from your cameras across day, night, weather, seasons and busy periods. Label the events you care about.
- Measure at the event level, not the frame level. Operators care whether a real intrusion triggered an alert (recall) and how many alerts were false (precision, or false alarms per camera per day).
- Tune per camera. Zones, thresholds and minimum durations often matter more than the model.
- Watch for alert fatigue. A system that cries wolf ten times a shift gets muted. Many deployments do better with fewer, higher-confidence alerts plus searchable metadata for everything else.
- Re-test after changes to cameras, lighting, layouts or models. Drift is gradual and easy to miss.
Camera placement deserves real attention: height, angle, resolution on target, lighting and shutter speed. Many "AI problems" are solved by moving or replacing a camera.
Privacy, biometrics and the law
Video of identifiable people is personal data under GDPR and similar laws, and biometric data used to identify people is a special category with stricter conditions. Practical consequences:
- EU AI Act. Several practices have been prohibited since February 2025, including untargeted scraping of facial images to build recognition databases, emotion recognition in workplaces and schools (with narrow medical and safety exceptions), and, with tightly limited exceptions, real-time remote biometric identification in publicly accessible spaces for law enforcement. Many other biometric identification systems are classed as high-risk, with documentation, testing and oversight duties. The text is in Regulation (EU) 2024/1689.
- US state laws. Illinois' Biometric Information Privacy Act requires informed written consent before collecting biometric identifiers and has produced significant litigation; other states have their own rules.
- Design for minimization. Prefer analytics that count or detect without identifying people, blur faces in stored footage where possible, keep retention short, and restrict who can search video.
- Signage and notices are required in many jurisdictions; a data protection impact assessment is often expected for large-scale monitoring.
Evidence integrity is a separate concern. If clips may be used in disputes, record hashes of exported footage and keep an access log. Some teams anchor those hashes on a ledger for tamper evidence; the guide to blockchain for cybersecurity discusses where that helps and where an ordinary signed audit log is enough.
Build vs. buy
Off-the-shelf analytics, either built into modern cameras and VMS platforms or sold as add-ons, cover common needs like intrusion, line crossing, counting and plate reading. Buy when your use case is one of those. Build or customize when you need detection of objects specific to your business (a particular part, a safety device, a product), integration with operational systems, or analytics across sites that no single product supports. A middle path is a commercial platform with custom models plugged in.
For a broader framework on scoping AI projects and vetting teams, see the guide to choosing an AI development partner.
Cost and timeline drivers
- Number of cameras and sites, and how many need upgrades to be usable.
- Custom classes. Each new object type needs collected and labeled examples, commonly thousands of images across conditions.
- Hardware. Edge GPU boxes per site, or cloud GPU hours, plus bandwidth.
- Integration with the VMS, access control, ERP or incident tools.
- Compliance work: impact assessments, retention controls, audit logging.
As a reasoned range: a pilot using existing detectors on a handful of cameras with a basic dashboard is often a small team for six to ten weeks. A multi-site rollout with custom models, edge fleet management and integrations is commonly a six-to-twelve-month program.
Questions to ask a vendor
- Will you run a pilot on our cameras and report precision, recall and false alarms per camera per day?
- What runs at the edge, what leaves the site, and where is footage stored?
- How are models updated across sites, and how do you roll back a bad update?
- Which features process biometric data, and can they be disabled entirely?
- Who owns the labeled data and custom models we pay to create?
- What camera specifications do you require for each analytic?
Frequently asked questions
Can AI video analytics work with my existing cameras?
Often yes, if the cameras provide RTSP streams at reasonable resolution. Some analytics, especially plate reading and anything at long range or at night, need specific camera models, angles and lighting, so expect a camera survey before committing.
How accurate is AI video analytics?
It depends heavily on the task and conditions. Detecting people and vehicles in a well-lit, well-placed view is very reliable; spotting rare events in crowded or poorly lit scenes is much harder. The only meaningful answer comes from testing on your own footage.
Is facial recognition legal for my business?
It depends on jurisdiction and purpose. It is prohibited or heavily restricted in some contexts in the EU and requires consent in places like Illinois. Many deployments avoid identification entirely and use non-biometric analytics instead.
Should processing happen on the edge or in the cloud?
Real-time alerts and privacy-sensitive sites usually favor edge processing. Cross-site search, heavy models and analytics dashboards favor the cloud. Most production systems combine both.
How do we reduce false alarms?
Tune zones, schedules and minimum durations per camera, require a track to persist before alerting, improve camera placement and lighting, and retrain on the false positives you collect. Measuring false alarms per camera per day makes progress visible.
What about vision-language models for video search?
They make it possible to search footage in plain language and summarize clips without training a detector for each question. They cost more to run per frame, so they suit investigation and review rather than continuous real-time alerting on every stream.