"The OpenCV Open Source Computer Vision Library: What’s New and What’s Coming?," a Presentation from the OpenCV Foundation

Copyright © 2016 OpenCV.ai 1
Gary Bradski
May 3, 2016
The OpenCV Open Source Computer Vision
Library: What’s New and What’s Coming?

OpenCV Overview

OpenCV at Glance
What The most popular computer vision library: http://opencv.org
License BSD
Supported Languages C/C++, Java, Python
Size >950K lines of code
SourceForge statistics ~13 M downloads
Github statistics >7500 forks, 4100 patches merged during 6 years (~2.5 patches per
working day), ~100 patches pending
Accelerated with SSE, AVX, NEON, IPP, OpenCL, CUDA, parallel_for, OpenCV HAL
The actual versions: 2.4.12 (2015 Jul); 3.1 (2015 Dec)
Upcoming releases: 2.4.13 (2016 Apr); 3.2 (2016 May)
Supports:
“A free and open computer vision library
to accelerate visual intelligence.”

Recent Statistics
~280K downloads/month
~ 13M downloads
World wide
Rated highly
NOTE:
This is only for source
forge. Many more
downloads come from
Git and many more
come on Unix distros.
OSs

OpenCV Areas Supported
Image Processing Transforms Fitting Optical Flow
Tracking
Segmentation
Deep Learning
Object recognition
Machine learning
Calibration Features
VSLAM
Depth, Pose
Normals, Planes,
3D Features
Computational
Photography
HighGUI:
I/O, Interface
CORE:
Data structures,
Matrix math,
Exceptions etc

OpenCV is in:
• Stanley (winner of the DARPA grand challenge)
• Streetview
• Google Maps
• Google self driving cars
Real World Use — Self Driving Cars

Real World Use — Robot Manipulation
• https://www.youtube.com/watch?v=IXxU4sL6hq0

• Academic and Industry Research
• Security systems
• Image/video search and retrieval
• Structure from motion in movies and robotics
• SLAM – mobile navigation for rolling robots and drones
• Machine vision factory production inspection systems
• Safety monitoring (Dam sites, mines, swimming pools)
• Microfluidics, biomedical
• Robotic bin picking
• Computational photography
• Face recog., face overlay.
Where Else is OpenCV Used

• Funding:
• Intel, Google, Magic Leap, Itseez
• Development Leads:
• Vadim Pisarevsky (lead)
• Itseez (development, maintenance)
• Gary Bradski (Director)
• Ethan Rublee (CEO, OpenCV.ai)
• Grace Vesom (Planning)
• Vincent Rabaud (Code quality_
• Stefano Fabri (Advisor)
• Adrian Kaehler (Documentation)
Who is Involved
• Interns:
• 13 to 15 interns supported/year
• Community:
• Over 500 pull requests
(contributions) since 3.0
(June’15)

What’s New in OpenCV

• Mostly compatible with
OpenCV 2.x;
• OpenCV 1.x C API is
deprecated and partially
removed
• Adds opencv_contrib
OpenCV 3. at a Glance
Why 3.0?: Brought 2.x to full C++ API; accellerated performance
(with T-API, IPP, NEON); modular structure + enables user
contributions, user commits (opencv_contrib) to keep up with state of
the art in computer vision.…
Aug’14 Nov’14 Apr’15 Jun’15
3.0 alpha 3.0 beta 3.0rc 3.0 3.1
Q4’15 • Highlights:
–even more modular and extendible
–very stable API tailored for a long-
term support
–enhanced out-of-box performance –
Hardware Acceleration Layer:
– IPP, OpenCL(T-API) and NEON
–lot’s of new functionality!

New: Opencv_contrib — User Content/State of the Art
• The “main” opencv repository
includes mature algorithms and is
fully supported
• A separate contribution repository
is for new CV algorithms that
people want to share:
http://github.com/itseez/opencv_
contrib
• Patches to the contrib repository
are tested as well by our buildbot
to ensure integrity! Deep Neural nets, Improved calibration,
Computational photography, Text detection,
Object Tracking, Python, Java, Matlab bindings
etc.

Recent Contributions 2013
https://www.youtube.com/watch?v=_TTtN4frMEA

https://www.youtube.com/watch?v=3f76HCHJJRA

https://www.youtube.com/watch?v=OUbUFn71S4s

Transparent API (T-API) for GPU acceleration:
•single API entry for each function/algorithm – no specialized
cv::Canny, ocl::Canny, gpu::Canny etc.
• uses dynamically loaded OpenCL runtime if available; otherwise
falls back to CPU code. Dispatching is at runtime, no
recompilation needed!
•~100 functions are optimized
Highlight: HAL — Hardware Acceleration Layer

Highlight: Hardware Acceleration. Khronos
OpenCV was one of the
key contributors to the
new Khronos accelerated
vision API: OpenVX
This is now a (small) part
of the OpenCV HAL
(Hardware Acceleration
Layer).

Highlight: Itseez Corp: ACV –ARM Acceleration Layer
http://itseez.com/products/accelerated-cv/

• Much more flexible calibration, AR, VR, stereo
• Each corner has it’s own key code.
• Allows calibration of complex arrangements of cameras.
Highlights: Better Calibration

Deep Neural Networks
• Train a network on any library
• Run it embedded via OpenCV
• No external dependencies
Highlight: Deep Learning (dnn in opencv_contrib)
http://dreamscopeapp.com

Highlight: Deep Learning (OpenCV 3.3)
• https://www.youtube.com/watch?v=BlxlzVn-HsY
Planned (GSoC 2016 and beyond):
• Adds support for other dnns:
TensorFlow, tiny-cnn etc.
• Improves performance, compared
w/other dnns:
https://github.com/soumith/convne
t-benchmarks
• Does object detection, saliency,
semantic segmentation etc. using
DNNs

• Computational photography
• Xphoto, ximgproc
• AR/VR … SLAM
• Ccalib, stereo, tracking, optflow, dnn, xfeatures2d
• Autonomous or assisted driving
• Tracking, optflow, dnn, xfeatures2d
• Security systems
• Dnn, tracking, dpm (deformable parts model)
• Mobile
• Text (detection, reading), dnn (recognition), face detection/recognition
• Drones
• Optflow, dnn, tracking, stereo, xfeatures2d
Examples of opencv_contrib use

OpenCV Future/OpenCV.ai

Embedded computers and Cameras
have become cheap and powerful.
OpenCV Future
• OpenCV will focus on:
• Embedded – smart cameras
• Light fields
• OpenCV.ai profit corporation
• That will support OpenCV.org (OpenCV remains open and free)

• OpenCV.ai SDK and Linux/Android Distro
Create a Smart Camera Platform, HW+SW
OpenCV.ai framework layer
HAL, RT, Config, plugins, IPC, RGl,
Cluster compute
Closed Applications/Licensing:
SLAM, Lightfield, face, text…
“Algorithm app store”
X86 debian
Linux
debian.
ARM
Android
Hardware Spec with manufacturing
partner
OPenCV, DNN, OpenGL,
OpenCL, Eigen
Networking
Supported SOC
Imagers
Sync
Networking
GPIO
Cloud
Synchronization and Phasing Layer SW+HW

Light Fields
• Light fields give many capabilities:
• Large depth of field
We can calibrate arrays in two
shots:
Light Field
Best regular
camera
Input
From Stanford Tech Report:
CTSR 2005-02
Imaging is 2D
Depth sense is 3D
Light field is 4D

Light Fields — Many Capabilities
• Depth
• Surface Normals
• Deblurring
• Lighting Direction
• Glint Separation
• Depth of Field
From: “Light Field D-blurring
for Robotics Applications”
Input
De-blurred
High Performance Imaging Through Minimization-
based Optimal Occlusion via Energy Minimization-
Based Camera Selection Optimal Camera Selection
• Hyper resolution in time
• Hyper resolution in space
• High dynamic range
• Occluding edges from parallax
• Variable zoom
• SFM
Light fields turn the camera into a software object
while the hardware exists to sample light.

OpenCV.ai will Build a Full Processing Stack to
Support Light Fields
Light field
Sensor
Light Field
Geometric
Processing
Light Field
Features
Real Time Control
light
Light fields
Deep nets
Classification
Localization
Depth
Pose
Flow
Device Bus HALLocalAPI
App
App
Drives GPIO Sensors PLCs
Dev Tools
Apps
Users
Cloud
Opencv Sensor DeviceOpencv
Platform
RemoteAPI

Light fields are a paradigm shift from the pinhole
camera. With the right HW and SW end-to-end stack,
they have wide applicability in:
Light Field Applications
Drone
scanning &
navigation
Robot
Manipulation
Depth
Estimation
Change
Detection
3D Displays SLAM/robot
navigation
Specularities View
generation
Cinematography Odometry Transparency Deep
Learning
VR/AR Recognition Reconstruction Segmentation
Bold is first targets

• Founders:
• Ethan Rublee, CEO;
• Gary Bradski, Founder, Funder and Board
Member
• Teams forming in US and China
OpenCV.ai
• Develop HW+SW spec for smart
cameras.
• Partner for HW here and in China.
• Parametric light field infrastructure.
• Sell prosumer, robotic and drone light
field Cameras!
Phase 1 Phase 2 Phase 3
Sync board
Camera board
100X raspberry
PI + camera
OpenCV.ai
Compute board
100x array
w/Sync
Kickstarter
preorder and
fulfillment
Yes, OpenCV.ai is hiring!

• OpenCV main site:
– http://opencv.org/
• Developer site:
– https://github.com/itseez/opencv/wiki
• Download packages:
– http://opencv.org/downloads.html
• Documentation:
– http://docs.opencv.org/
– Books: http://opencv.org/books.html
• Building the code instructions:
• http://docs.opencv.org/2.4/doc/tutorials/introduction/table_of_content
_introduction/table_of_content_introduction.html
Resources
OpenCV.ai website:
http://www.opencv.ai
(not up yet)
In print this summe
Order:
http://shop.oreilly.
765.do

• Talmudic tale:
• Very old man working in a field planting fruit trees.
• Q: “Why do you plant trees you may never harvest?”
• A: “When I came into the world, it was not bare of trees. Someone
planted for me and I will plant for someone who comes after.”
• The nature of OpenCV has always been to give more than it takes.
It’s mission is still to revolutionize vision.
Why the Open Source…

QUESTIONS?
garybradski@gmail.com

"The OpenCV Open Source Computer Vision Library: What’s New and What’s Coming?," a Presentation from the OpenCV Foundation

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to "The OpenCV Open Source Computer Vision Library: What’s New and What’s Coming?," a Presentation from the OpenCV Foundation (20)

More from Edge AI and Vision Alliance (20)

Recently uploaded (20)

"The OpenCV Open Source Computer Vision Library: What’s New and What’s Coming?," a Presentation from the OpenCV Foundation