2009

OpenCV Meeting notes for 2009 year

2009-01-06

Agenda

Update
VTK?
Finding planes with Radu’s code?

Minutes

Russian holidays
- Vadim working on using Lapack+BLAS in OpenCV
  - Using ATLAS & MKL
See about getting some of Radu Ruso’s mesh routines in OpenCV
Want to move C-descriptor into OpenCV
- What’s needed to make a paper out of it
3D reconstruction
- Good camera
- Textured light
- Rotating platform and rotate object captured frame by frame
- Robot can scan object
Structure from motion
- What is JD working on with 3D reconstruction
“OpenCV” entry in manufacturing robots

Action Items

Gary

Talk to JD about code
Record minutes of tomorrows discussion with the Vision workbench team
Talk to Calendar about producing a Star-C-descriptor paper

Vadim

Compare OpenCV star detector with ROS star detector

2009-01-13

Updates
Interface issues (what next)
Pattern Recognition
3D functionality

Minutes

Update Gary:

Set up stereo capture node and will collect 100’s of outlets and doorknobs
Collecting 3D object database of Ikea objects
Writing an opencv tutorial for Stanford class, will post that

Update Vadim:

Continued work on BLAS and LAPACK interface
- Eigen vectors and values,
- SVD,
- cvsolve,
- cvinvert and
- qr decomposition
- Accuracy is great and performance is great
  - 500×500 matrix is 2.5x faster
  - Small matrixes, 10×10, the new implementation is slower
    - Need to add branches in this case
- Netlib.org Jack Dungar’s implementation, puts out clapack with same interface
  - Eigen2 lib, small matrices are faster than lapack, but algorithms aren’t as well developed as lapack
multiwin replaced using wImage (code from Google)
This week is after Russian new year vacation.
- Anatoli working on Octrees for Spin images
CVS becomes obsolete this week

Update Victor

Was sick with mono for 3 weeks
Outlets
- Filter keypoints using random forest
  - Train image passes around keypoints
  - 2 class filter for inside/outside outlets, works 90%
  - 70% recognition on hard datasets with textured walls and different lighting condition
  - Low false positive rate
    - Need to turn down false positive rate to gain low false negative (missed detection) rate and then apply stronger models to the detections
  - Needs to wait to final camera setup
- Need datasets for US outlets for keypoint detection
  - Adapt hole searching code for US outlets
Doorknobs
- Dark round objects using series of thresholds and connected components
- RCG collected doorknobs together with chessboards.
  - Perspective corrected
  - Trained adaboost
  - Same for test set
    - Results weren’t that good — 40% detection with 15% false alarm
    - Gary suspects making many different templates to match — one each 10 degrees.
Random forest improvements
- Extreme random trees code implemented
  - Bugs fixed while in hospital
  - Compare with same datasets measure test errors.
  - Errors with noise variables
  - Variable importance — correction noise variables
  - If improvements, possible paper

Videri camera is supported in OpenCV

Nearest Neighbor

cvFeatureTree, already in OpenCV
David Lowes FLANN — test against cvFeatureTree()

Action Items

Gary

Get doorknob and outlet datasets to Victor
Get outlets from Milestone dataset to Victor
Get cannon outlet sites to Argus or Willow sites
Get Willow doorknobs to Victor

Vadim

Send SVN instructions for Window to Gary

Victor

Test random trees against random forest

2009-01-20

Agenda

Update:

Milestone2
- Outlets up close
- => Victor to Doorknobs?
  - 3D box with chessboard
Planning for Summer release, what should be in it

Minutes

Gary
Close range outlet detection to subpixel

Vadim

LAPACK is now included into OpenCV tree.
cvSVD, cvEigenVV, cvDet, cvSolve and cvInvert have been rewritten using LAPACK;
cvInvert and cvSolve have got an additional method CV_CHOLESKY, cvSolve also got additional method CV_QR.
Anatoly has implemented a draft version of octree for computing spin images. The existing implementation of octree did not contain the key algorithm for retrieving all the points in the cloud that belong to the given sphere and it was difficult to integrate it, so the new class has been written. Anatoly compared performance of this new code with OBB tree from VTK, which has been used by the existing spin image code and found that the performance of the two is very close now:

Hope to speed up spin feature by factor of 10
Future to speed up Radu’s histogram of normals

Next release

C++ interface with templates
- 30-50% done
- Templates will be optional
- More compatible with STL
Much better python interface
- Mark submitted a patch for python wrapper and use with numpy
3D features/related algorithm
- Spin images — goal is 10x speedup.
- Plane fitting from Radu
Alpha and/or 4 channel support
2D features
- C-descriptor (Calandor descriptor)
- Maybe Boundary fragments
Stitching/VSLAM
- Move in the V-SLAM stitching into OpenCV (?, more probably next …)
- Place localization/recognition as an app or example code for C-Descriptors
Optical flow
- Dense
- Rigid
- Affine
- Intern from Michael Black
Machine learning library
- Extremely Randomized Trees
- Nistor Tree
- Convolutional NN
- Additions to RF
- Extend cascade to other features beyond Haar

PRIORITIES

C++ interface
Python
3D Features (for Obj rec)
2D Features (for Obj rec)

(no order)

MLL
Alpha / multi-channel
Optical flow improvements
Stitching

Victor
Extremely Randomized Trees (ET) algorithm shows comparable performance with published in the original Geurts paper on 4 datasets while on 2 datasets (spambase and vehicle) it demonstrates larger errors (see the table below). We are still investigating this issue, the current main hypothesis is that there is a bug that plays a significant role only on these two datasets. Variable importance charts built for Random Forest (RF) and ET models are similar.

Name	RF MLL	ET MLL	ET Geurts
waveform	17.1	16.06	16.61
vehicle	21.88	47.29	26
spambase	5.41	22.68	4.17
isolet	6.61	10.52	7.61
twonorm7400	4.523944	3.745071	3.53	\| ringnorm7400	6.164789	3.138028	3.27

There is a bug on Vehicle and Spambase. UPDATE: This bug is fixed.

The application for automatic tracking of planar objects has been tested on windows and linux with usb camera and video.

A method for learning an RF model of an outlet invariant to affine transformations has been implemented. An RF is learned on patches of frontal view outlets that have been distorted with random affine transforms (following Lepetit’s paper). Each outlet candidate gets a weight (in [0,1]) from this model and low weighted samples are filtered out. Attached is a precision-recall curve of this algorithm obtained by varying the weights threshold. It shows the performance of the model (recall = 1 corresponds to no filtering). The most of false positives are gathered on few images with rich texture.

Have outlets detectors in 2D
- Single image
- At least 50×50 ROI
- Based on finding dark round objects (European) and dark round (American)
- 2 stage RF (European):
  - First stage dark small round holes (where metal prongs goes in)
    - But many false positives, filtered by RF
    - Look for pairs of holes
  - Second stage does the neighborhood of the found holes look like an outlet
- Need more American data and can train on that
Keep going — use this as one method of attack.

3D box relative to chessboard. The RCG (Alexi (tracking, doorknob) Maria (RF, ET))
- Today he’s training adaboost on doorknobs
- Error control for planar object position
  - Calibration

Action Items

Victor

Send images of current outlet detector
Train on Willow images to be sent

Gary

Send images of outlets to Victor (today)
Try to get several hundred
Get doorknob imagery to Victor
Take 3D model of doorknob
CC Vadim with Ara
CC to Jean-Bouguet about 4 channel
Talk with Michael about what needs to be done to get C-descriptors in OpenCV
Michael Black’s intern
Talk with Kurt about moving VSLAM stitching

Vadim

Spin images
- Preliminary study is quite fast, could be optmized more
- Most of the time in spin image is not in octree, but in the spin image
Send me OpenCV goals spreadsheet
Put up initial next release goals on OpenCV wiki

2009-01-27

Agenda

Updates

Minutes

For OpenCV Release

Company Retreat
Outlets
- Bottlenecks
  - Find outlets to sub-mm when robot is “in position”
    - Must detect that nothing is plugged into the outlet
  - Visual servoing after robot touches the wall with the plug
  - Finding the plug orientation
    - Finding a dropped plug(?)
- Other things which are “good” but not critical for milestone 2
  - Detecting Willow Garage outlets from a distance with and without stereo and/or laser data
  - Detecting outlets in general from a distance with and without stereo or laser data
Method for first bottleneck (find plugs to sub-mm accuracy)
Using a calibrated camera with rectified images:
1. Segment the outlet “ovals”, find 4 centroids
2. Compute homography to canonical view and remap
3. Match templates to find holes and more exact center of outlets
4. Re-project more exact center found above into the image to find floating point centers
5. Use these reprojected floating point centers to again compute a homography to the canonical view and remap
6. Use templates to find the exact location of outlet holes. This should now be sub-mm precision
7. Using the homography and measured outlet, compute the X,Y,Z location of the outlet holes
Arrange for common repository

Vadim

Lapack debugged — now using clapack
- Simple small matrix operations still use native OpenCV code
- Try fast lapack algorithms like ATLAS
- Make use of MKL if available
New memory allocator — fast for small matrixies A*B+C
- ~5x faster than malloc and free
Spin images
- 20x speed up on spin images
- Compare against Point Feature Histogram PFH
- Ready in 2-3 more days
- Can be further optimized by factor of 2x ~ 1 week
- 20 seconds on 200K points scene of a conference room using 40cm radius
Stereo
- Textured areas have problems, either use textured light or something like Kolmogorov
- Stereo SSE-2 optimization uses only gcc 4.2 or higher
- Chessboard detector is breaking on high angle views
  - High angles are not so useful for calibration, but good for plane finding
    - Moderate priority
- Optimizing Kolmogorov
  - Sparse graph cuts — might not work, but idea is to soft fix points that are unlikely to change
  - Run as background project for now
Next week
- C-descriptor
- C++ image classes
- cvBoostPatch() from Steve Gould and from Ian Goodfellow

C descriptor

In …/ros/ros-pkg/visino/calonder_descriptor (it should be named C-descriptor for “Classifier Descriptor”)
I spoke with Michael Calonder about getting his descriptor (which we use in visual odometry and place recognition) into OpenCV. He’s going to write up a short description of how it works. It is already fairly optimized using 16 bit vectors and SSE — it can do a 512×512 image in about 5ms (depending on complexity of course).

They’ve been using the “FAST” detector which has a GPL license. We might consider doing our own FAST detector. But, it can use any intrest point detector, it’s mainly a really good descriptor. At the end of training he used Matlab to compress the vectors using PCA — we’d have to replace that step with clapack.

Place Recognition/Nistor/Kmeans

In ../ros/ros-pkg/vision/place_recognition there are several things of interest:

Improved kmeans code (src/kmeans.cpp)
Nistor vocabulary tree which we can add to MLL (src/vocabulary_tree.cpp)
Place recognition (src/place_recognizer.cpp). This could be sample code or a really useful app that uses (is an example of use of) the C-descriptor above, along with the vocabulary tree and kmeans.

Action Items

Gary

Determine time for outlet meetings with Patrick
Data with plugs already in outlet
Get data to Victor of plug images
Put Victor in touch with Michael Calonder
Put Vadim in touch with Michael about C-descriptor
Find out from Kurt status or priority of stereo speedups
Talk to Scott about getting Vadim SSH access to machines in Willow
Sandbox in ROS for things like outlet detection experimentation.

Vadim

Stereo OpenCV vs Kurt algorithm comparison for speed on a known good machine.

2009-02-03

Agenda

Outlets will be moved to a meeting directly after the OpenCV meeting, so we will no longer discuss outlet specific issues here but instead focus on OpenCV issues.

Updates
- Vadim
- Victor
- Gary
Progress towards Summer release
- OpenCV200906
- Memory limitations in machine learning — design matrix limits
- Error handling???
PnP problem approaches

Minutes

Vadim

Anatoly and me: finished optimization of OctTree & Spinimage code. OpenMP and SSE2 gave ~2x speedup each. In result, the running time of one of our benchmarks dropped from (ROS was 80 secs), new c code is 880ms and now downto ~230ms with SSE and OpenMP. That is, together with ~20x speedup we achieved with new C code, the total speedup is over 75×. The code has been prepared for integration to OpenCV.
The 2 optimizations of the boosting code, proposed by Stephen Gould and Ian Goodfellow, have been integrated to MLL. No feedback from the guys yet.
Studied the C-descriptor source code the relevant papers by Michael Calonder and Vincent Lepetit. The algorithm is pretty clear now and the preparation to integration into OpenCV has been started.
~30% of the new matrix class integration has been done. This appeared to be significantly bigger task than I expected.
Received the adaptive skin detector code from Farhad Dadgostar; reviewed the code and recieved the corrected variant; The code will be integrated this week.

Victor

Outlet detection: calibration, template matching and hole detection have been combined in a single application. The method has been adjusted for another camera (narrow lens resulting in higher resolution outlets). A method for finding a tuple of 4 outlets has been improved for the case of large viewing angle. Recalculation to 3D coordinates in cm scale has been implemented and tested. If an outlet is found correctly the error of hole detection is less than 0.5mm for all cases but one.
Extreme trees (Maria): the code has been refactored, all ET functionality has been encapsulated into a separate class inherited from CvRTrees. The performance of RF and ET has been compared. For very deep trees (depth 20) the performance is comparable, while for more shallow trees (less than 10) ET is faster at least 2×.
- Has natural parallelism, look at it.
- Parallize all boosted and random prediction trees.
- Possible paper in statistical convergence to correct answer
3D object tracking (Alexey): the method for tracking 3D will be based on 5 points set by user on two different frames. Stereo pair calibration has been implemented.
- Right now we are trying to catch a bug in calculating 3D from disparity. Alexi out so this will wait.

Other: Alexey is on vacation Feb2-8, back in the office Feb 9.

Gary

Boosting

Will try to see if we’re still artificially limited by memory size

OpenCV errors

New implmentation moves completely to exceptions
Should work with debugger now

OpenCV website

For demos etc

Action Items

NEW

Gary

Email Steve Gould about boosting memory limits with Victor and Vadim
~~Email Ian Goodfellow about opencv error catching with Vadim.~~

Vadim

Check ML design matrix

Victor

FROM LAST TIME

Gary

Determine time for outlet meetings with Patrick
- 10am (9pm NN) on Tuesday
Data with plugs already in outlet
Get data to Victor of plug images
Put Victor in touch with Michael Calonder
Put Vadim in touch with Michael about C-descriptor
(?) Find out from Kurt status or priority of stereo speedups
(?) Talk to Scott about getting Vadim SSH access to machines in Willow
- I talked, but no action yet.
Sandbox in ROS for things like outlet detection experimentation.
- Just use sourceforge and ROS can pull it in if you build say the outlet node.

Vadim

Stereo OpenCV vs Kurt algorithm comparison for speed on a known good machine.

2009-02-10

Agenda

Quick updates:

Gary

Helped get Mech Turk interface into Willow Garage (thanks to Alexander Sorokin and Scott Hassan’s 2 daze of hacking last week). View some results (click on the numbers under “completed hits”).
- We can use this for massive data sets and even say sponsor people to create better face recognizers etc.

Vadim

Matrix interface
80% of cxcore is ported to the new C++ interface
- Automatic garbage collection (reference counter based)
- Can use with STL vector of images and matrixes
- Have template interface matrix(i,j) does the right thing
  - Thread safety coming, can use atomic operations, efficient on Intel and AMD archs
IPP will now be installed/built at configure instead of dynamic
- Using cmake, if IPP is detected the build flag will be on by default, user can explicitly turn it off (same way as ffmpg libs etc)
Decision tree memory optimization, needs about 2 weeks (Maria knows the code a bit, but Vadim expert)
- Needed to move cascade recognizer to general decision trees, beyond Haar features
- Think about collaborating with someone to train good working models for the release
Spin images are done but waiting to complete the C++ integration before integration (probably by Friday)
Focus results, … lower priority

Victor

Extreme Trees
- Categorical support is added
- Don’t need sorting of variables, so lower memory requirements
- Can handle 100K samples of 1000 features, Extreme trees worked OK
- Code is in SVN
- Recognition performance is comparable to RF but faster
  - Inherently more parallel
  - Try on outlets?
3D object tracking (Alexander was on vacation)
- Estimate pose of box in space from motion and knowlege of box
- Function for disparity to 3D has bug
- 2D version … finished … but special case of 3D, integrate

OpenCV release tracking

Go over the development plans

Minutes

Tracking the summer release:

General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - … see above
- Better python
  - Vadim, Alex …
  - C++ will take another month but can improve python in parallel
- 2D features C-descriptor, HoG Boundary Fragments …
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
  - Boundary fragments … working on focus.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
  - SPIN almost done
  - Radu is organizing his code into a library … maybe we can just lift this
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
  - will ask Jean-Yves for pointers
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documenation
- .
High GUI
- .

Action Items

NEW

Meeting next week will be on Wednesday

Gary

Send Vadim MSER code
Ask Jean-Yves for priority list of functions to work with mask and alpha channels
Ask Jean-Yves about image stitching support/libraries/functionality
Ask Radu about his library progress.
Send email reminder for Wednesday meeting next week.

Vadim

Adaptive skin tracker app/test
Finish cxcore rewrite, make sure all the tests pass, update the documentation.
Start cv module revision.
Integrate spin images.

Victor

FROM LAST TIME

Gary

Email Steve Gould about boosting memory limits with Victor and Vadim
~~Email Ian Goodfellow about opencv error catching with Vadim.~~

Vadim

Check ML design matrix

Victor

2009-02-18

Agenda

Accomplishment Tracking

.

Chessboard issues

Detection failures
2 functions? One chessboards for calibration, another chessboards for pose?

3D capture, discuss implementation of

http://www.robots.ox.ac.uk/~awf/turntable/index.html

cxcore re-write, cv-rewrite

Spin, skin

Release tracking.

Minutes

Tracking the summer release:

General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - … see below
- Better python
  - Vadim, Alex …
  - C++ will take another month but can improve python in parallel
- 2D features C-descriptor, HoG Boundary Fragments …
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
  - Boundary fragments … working on focus.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
  - SPIN almost done
  - Radu is organizing his code into a library … maybe we can just lift this
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
  - will ask Jean-Yves for pointers
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- .

New Interface

cxcore revision is complete (except for XML/YAML input/output functions). All the algorithmic tests for cxcore and cv pass, all the samples run well.
- As a small illustration of the new API capabilities, here is a some code that I used to debug SVD:


// matrix A comes in
SVD svd(A);
int n = A.rows;

double a = norm(svd.u*svd.u.t() - Mat::eye(n,n), CV_C);
double b = norm(svd.vt*svd.vt.t() - Mat::eye(n,n), CV_C);
int nz = countNonZero(svd.w >= 0);
double c = norm(svd.u*Mat::diag(svd.w)*svd.vt - Mat::eye(n,n), CV_C);
CV_Assert( a < 1e-5 && b < 1e-5 && c < 1e-5 && nz == n );

Can template this
- Can create image with up to 64 channels
- Some functions limit to 4 channels
- Splitting channels is easy
  - Work with IPP, for cache, want to extract channel

General Issues:

Chessboard

cvFindCornersSubpix fails with large search window, diverges.
- Ordering of chessboard corners and splitting
- Will lookat
Initial location of corners can be wrong
- Extreme angles can create problems for morphological operations
  - vertical and horizontal operations can make distance between corners smaller than within squares

3D

http://www.robots.ox.ac.uk/~awf/turntable/index.html
Skin detection should work using highGUI camera

Spin images, trying on Ikea

3D tracking (Victor, Alexi) (3D box)

3d positions of markers using Chessboard
- Currently unstable depending on where the user clicks
- Somewhere in disparity to 3D transformation
- Generating tests on synthetic data by tomorrow.

Memory use with MLL

Shrank memory use several times
Debugging

Action Items

NEW

ALL

Send in list of what’s been accomplished so far in OpenCV since Willow started supporting development.

Gary

Look into sending prosilica and other cameras to Argus
Send full data link with mask images for Spin image work
Get Zisserman 3D paper to Vadim (on ACM)

Vadim

Look at better calibration and pose object
- Equal or better in accuracy and stability
Check what you think 3D scanning would take

Victor

FROM LAST TIME

Meeting next week will be on Wednesday

Gary

Send Vadim MSER code
[but he didn’t answer yet] Ask Jean-Yves for priority list of functions to work with mask and alpha channels
[same] Ask Jean-Yves about image stitching support/libraries/functionality

Ask Radu about his library progress. Radu:



We took a few steps today to unify the point cloud -related libraries in ROS under one umbrella. The result is that

there is now a point_cloud_mapping package which contains the following things:

- cloud_geometry: areas, distances, intersections, 3D normals/features estimation, norms, projections, statistics,

transformations, etc;

- cloud_io: a library for Input/Output PCD (Point Cloud Data) processing;

- cloud_kdtree: a library for nD spatial decomposition and nearest neighbors searches for PointCloud messages using k-D

trees;

- cloud_octree: a library for 3D spatial decomposition of PointCloud messages using octree data structures;

- sample_consensus: a library for SAmple Consensus (SAC) like methods (RANSAC, MSAC, MLESAC, etc,) and geometric shape

models;

There are also 2 basic nodes included with the package:

- cloud_downsampler_node: a node which uniformly downsamples a 3D point cloud

- normal_estimation_node: a node which estimates local surface properties at each point such as surface normals,

curvature changes, moment invariants, etc.

~~Send email reminder for Wednesday meeting next week.~~

Vadim

Adaptive skin tracker app/test
Finish cxcore rewrite, make sure all the tests pass, update the documentation.
Start cv module revision.
Integrate spin images.

Victor

2009-02-24

Agenda

Accomplishments
Interface update
Chessboard improvements
3D Box relative to chessboard
3D Object Capture paper/analysis
Spin images
Summer release tracking

Minutes

Tracking the summer release:

General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python
  - Vadim, Alex …
  - C++ will take another month but can improve python in parallel
- 2D features C-descriptor, HoG Boundary Fragments …
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Boundary fragments … working on focus.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
  - SPIN almost done
  - Radu is organizing his code into a library … maybe we can just lift this
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
  - will ask Jean-Yves for pointers
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
  - Generalize cascade
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- .

OTHER

New Interface

Whole new interface for cxcore is finsished
- Moving to cv
- Examples: Update samples. .c code will be old interface, .cpp will be new interface
- Output is simplified by automatically correct type
- Start effort to doxygen (lower background priority)

Chessboard code that

Vadim has released a patch that seems to fix all known problems. Congrats!
High angle (60 degrees) boards still cannot be done
Detect lines, automatically detect board size, perhaps remap image to frontal view.
- Sub-pixel finding is now put inside the algorithm rather than after.

3D box

3D object tracking: the code is in debugging phase. The calibration of a stereo pair is still unstable, producing large reprojection errors too often.
Alexi not much time to work on in
Make relative to chessboard

3D Object Capture

Bundle adjustment missing from OpenCV
Mesh reconstruction, not in OpenCV
Might take a month if we can’t get the above code
- http://www.3dsom.com/index.html seems to use the same technique.

Spin images

Not incorporated yet, code is ready, just time to integrate
- Test code is ready
- 1 or 2 days

Extremely random trees

Put a callback in the splitting function

MSER

Alex has MSER code

Action Items

NEW

Gary

Ask Kurt about where Bundle adjustment is … if it is
~~Send Victor Florian’s paper~~

Vadim

Doxygen documentation as background, lower priority task.

Victor

FROM LAST TIME

ALL

~~Send in list of what’s been accomplished so far in OpenCV since Willow started supporting development.~~

Gary

Look into sending prosilica and other cameras to Argus
~~Send full data link with mask images for Spin image work~~
~~Get Zisserman 3D paper to Vadim (on ACM)~~

Alexi

Finish up 3D box

Vadim

Look at better calibration and pose object
- Equal or better in accuracy and stability
Check what you think 3D scanning would take

Victor

2009-02-24

Agenda

Accomplishments
Interface update
Chessboard improvements
3D Box relative to chessboard
3D Object Capture paper/analysis
Spin images
Summer release tracking

Minutes

Tracking the summer release:

General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python
  - Vadim, Alex …
  - C++ will take another month but can improve python in parallel
- 2D features C-descriptor, HoG Boundary Fragments …
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Boundary fragments … working on focus.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
  - SPIN almost done
  - Radu is organizing his code into a library … maybe we can just lift this
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
  - will ask Jean-Yves for pointers
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
  - Generalize cascade
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- .

OTHER

New Interface

Whole new interface for cxcore is finsished
- Moving to cv
- Examples: Update samples. .c code will be old interface, .cpp will be new interface
- Output is simplified by automatically correct type
- Start effort to doxygen (lower background priority)

Chessboard code that

Vadim has released a patch that seems to fix all known problems. Congrats!
High angle (60 degrees) boards still cannot be done
Detect lines, automatically detect board size, perhaps remap image to frontal view.
- Sub-pixel finding is now put inside the algorithm rather than after.

3D box

3D object tracking: the code is in debugging phase. The calibration of a stereo pair is still unstable, producing large reprojection errors too often.
Alexi not much time to work on in
Make relative to chessboard

3D Object Capture

Bundle adjustment missing from OpenCV
Mesh reconstruction, not in OpenCV
Might take a month if we can’t get the above code
- http://www.3dsom.com/index.html seems to use the same technique.

Spin images

Not incorporated yet, code is ready, just time to integrate
- Test code is ready
- 1 or 2 days

Extremely random trees

Put a callback in the splitting function

MSER

Alex has MSER code

Action Items

NEW

Gary

Ask Kurt about where Bundle adjustment is … if it is
~~Send Victor Florian’s paper~~

Vadim

Doxygen documentation as background, lower priority task.

Victor

FROM LAST TIME

ALL

~~Send in list of what’s been accomplished so far in OpenCV since Willow started supporting development.~~

Gary

Look into sending prosilica and other cameras to Argus
~~Send full data link with mask images for Spin image work~~
~~Get Zisserman 3D paper to Vadim (on ACM)~~

Alexi

Finish up 3D box

Vadim

Look at better calibration and pose object
- Equal or better in accuracy and stability
Check what you think 3D scanning would take

Victor

2009-03-03

Agenda

SURF and OpenCV Announce
Marius Muja is at Willow for 6 months
- FLANN incorporation
New interface update
3D Box — it slips
3D Model
MSER

Minutes

Tracking the summer release:

General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python
  - Vadim, Alex …
  - C++ will take another month but can improve python in parallel
- 2D features C-descriptor, HoG Boundary Fragments …
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Boundary fragments … working on focus.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
  - SPIN almost done
  - Radu is organizing his code into a library … maybe we can just lift this
- 3D capture http://www.3dsom.com/index.html
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
  - will ask Jean-Yves for pointers
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
  - Generalize cascade
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- .

OTHER

OpenCV Announce …

SURF is in #1 in change log
Post change log to wiki
OpenCV Weekly …

FLANN

Some timing results at bottom
Code is in good shape
- 3D party directoy can be put there
- For mac, can be put in system path
- Linux can select
In CV module, can create a wrapper for CvMat interface to FLANN
- Make test, so can use included version of FLANN
- First looks to internal unless user specifies external
  - Put flag to download external
  - We keep reasonably up to date with external

Interface update

cxcore in SVN
- Mac and Windows,
- Linux: gcc caused changes due to template handling differences
convolution in cv
- Then geometrical transforms
- Optical flow
- Add new alpha blending function with separate transparency for each pixel
- Further functions can be lower priority

3D box

Box slips up (pretty sure)
Image is rectified
- If camera is close, distortion can be large
- Tomorrow will ship new video with box being tracked
Could be chessboard slippage ?

3D Object Capture

Bundle adjustment missing from OpenCV
Mesh reconstruction, not in OpenCV
Might take a month if we can’t get the above code
- http://www.3dsom.com/index.html seems to use the same technique.
Initialization of extrinsics (pose) is hardest part of the code
Different from general SFM or stereo
smart inversion of J^t*J
- pass to SVD (might be 10x slower but easier)

MSER

Gary sent
- Victor looking at right now,
  - Alex doing his own version faster at lower priority ~ couple of weeks

SURF

detection and descriptor united
- Should be split
- “Find_obj.cpp” is demo code in samples/c

Vadim:

cxcore has been built with GCC 4.0 and GCC 4.2 on MacOSX. Quite a few changes have been made because GCC handles some template stuff differently and usually catches more bugs. Another portion of changes were made to make all the tests pass. This tested version of cxcore has been committed to SVN repository.
The work on revising cv has been started, new interface for filtering functions (linear separable and non-separable, Laplacian, median, morphogical operations) is ~90% ready (the code was written and now debugged).
fixed 1 minor bug in boosting (reported by Ian Goodfellow) and 2 bugs in camera calibration (reported by Sergey).
Anatoly started the work on benchmarking various fast n-dim search algorithms for quality & speed. For now we have 3 algorithms in OpenCV: kdtree, spilltree, LSH (locally-sensitive hash), and there is also FLANN. We could not run LSH properly yet, but here are some first results on synthetic uniformly-distributed data in n-dim cubes:
- tAvgDist is the average (over the test set) distance to the nearest point, as returned by various alorithms (DumbBruteForce is the naive algorithm), the closer to the naive algorithm, the better.
- M1 is average over the test set of (min_dist/average_dist), the smaller the better.
- M2 is the number of times the returned point was closer than MAX, 0)
- and M3 is average of (average_dist – min_dist)/stddev(distance). More than 3 should good enough from statistical point of view, but the larger – the better.
These preliminary results show that kd-tree is the fastest method and generally not that bad, FLANN is as accurate as naive algorithm, yet significantly faster (but significantly slower than kd-tree, we wonder, whether it can be tuned to be less accurate but faster)
spill tree is only good for low-dimensional space, otherwise brute force is much faster.
LSH is yet to be tested.

Action Items

NEW

Gary

Make change log wiki page, and send to Vadim
Make OpenCV Montly Newsletter and send to Vadim
Ask J-Y again about mask image uses/image stitching
Find JD’s code and send to Vadim
~~Send email to Radu to point to mesh code/techniques~~
Calendor to Andrew Ng

Vadim

Put up change log wiki page and monthly page after Gary set’s up

Victor

Do new video of box

FROM LAST TIME

Gary

Ask Kurt about where Bundle adjustment is … if it is
- Umm, JD Chen (no longer with Willow) supposedly did this, but for stereo code, sent him an email
~~Send Victor Florian’s paper~~
- http://www.robots.ox.ac.uk/~vgg/publications/papers/schroff08.pdf
- http://www.robots.ox.ac.uk/~vgg/publications/papers/schroff06.pdf

Vadim

Doxygen documentation as background, lower priority task.

Victor

2009-03-10

Agenda

Summer of Code

New interface status

FLANN Benchmarks

Status of Extremely Random Trees — is it in

MSER?

3D Box

3D capture

Bundle adjustment missing from OpenCV
Mesh reconstruction, not in OpenCV
Might take a month if we can’t get the above code
- http://www.3dsom.com/index.html seems to use the same technique.

Minutes

Tracking the summer release:

General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python
  - Vadim, Alex …
  - C++ will take another month but can improve python in parallel
- 2D features C-descriptor, HoG Boundary Fragments …
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Boundary fragments … working on focus.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
  - SPIN almost done
  - Radu is organizing his code into a library … maybe we can just lift this
- 3D capture http://www.3dsom.com/index.html
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
  - will ask Jean-Yves for pointers
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- .

OTHER

Summer of code

We already have not just a cascade, but a cascade-tree
Gradient boosting trees(?) — project?
Optimize general boosting, not haar
Have Mechanical Turk
Image stitching
HighGUI

New interface

Filtering functions in cv all pass test
- SSE optimizations added for most important filtering functions
Image warping next
Usergroup glitch (old macro) posted to SVM

FLANN

Investigation is continuing by Anatoly
- With new FLANN #2, the performance gap is smaller
  - Accuracy is the same
  - kd-tree in OpenCV is very fast, but not so accurate
  - FLANN, accuracy is high, but speed is slower.
    - We made it faster, with same accuracy, but don’t know how to make it less accurate for more speed.

Extremely Random Trees

Is in. Needs sample code

MSER

Got it, stripped out other parts and compiled it.
Alexander working on it.

3D Box

There was a bug, found it, corrected it.
Sent email

REPORTS

Vadim

Debugged the new filtering functions (called by cvSmooth, cvFilter2D, cvErode, cvDilate, cvSobel). Now all the tests pass. Also, added SSE2 optimization to some important functions, like morphology and separable filters (like Sobel and Gaussian), so they should be much faster even if IPP is not available.
Started revision of image warping functions.
Anatoly continued investigation of various n-dim search algorithms. In the latest version of benchmark and with FLANN #2 the FLANN part runs faster, but still there is gap in speed-accuracy tradeoff. Also, the LSH part still does not give adequate results to be included into the benchmark. These 2 items are being investigated. Anatoly also ported the benchmark to Linux and sent it to Marius.

Victor

See outlet detection for main report
3D tracking: several bugs have been corrected, the new video is at ftp://argus-cv.dnsalias.org/3DTrackDemo/logitec_demo.avi, login willow, password awgarage.

Action Items

NEW

Gary

Put up GSofC projects
Job application page
Put up application
Ask Marius to interact with Anatoly on FLANN

Vadim

Put up GSofC projects

Victor

Put up GSofC projects

FROM LAST TIME

Gary

Make change log wiki page, and send to Vadim
Make OpenCV Montly Newsletter and send to Vadim
Ask J-Y again about mask image uses/image stitching
Find JD’s code and send to Vadim
~~Send email to Radu to point to mesh code/techniques~~
Calendor to Andrew Ng

Vadim

Put up change log wiki page and monthly page after Gary set’s up

Victor

Do new video of box

2009-03-17

Agenda

[No meeting next week, I won’t be in]

Google Summer of Code

HOG and SPIN

3D rotation + Translation of stereo images

Updates

New interface

Monthly announce, Change Logs

FLANN Benchmarks

3D Box

3D capture

Minutes

OpenCV Summer Release

General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python
  - Vadim, Alex …
  - C++ will take another month but can improve python in parallel
- 2D features C-descriptor, HoG Boundary Fragments …
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Boundary fragments … working on focus.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
  - SPIN ~~almost~~ done
  - Radu is organizing his code into a library … maybe we can just lift this
- 3D capture http://www.3dsom.com/index.html
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
  - will ask Jean-Yves for pointers
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- .

OTHER

GSoC

Already received an application, but Google decides who’s in on the 18th
I’ve been working on some stereo data collection for people detection
- Also for object detection too

HOG

2D feature

SPIN

Code and
Demo reads vrml
Constructs oct tree C++
Has compute normal points separate functions (takes oct tree)
Then computes spin image

3D ROTATION TO NEW IMAGE

3d perspective transform
perspective projection
for each point, find generalized interpolation
- Computer weighted sum of K nearest points
- To search in projected view, use quad tree or regular grid

Vadim update

Made another few corrections in filtering functions to make the API a bit more convenient. Updated filtering-depdendent functions (that used not cvSobel, cvFilter2D etc., but the internal API): SURF, Snakes, CornerMinEigenVal etc. Now the tests for these functions pass too. Here are some performance numbers (on Core2Duo #6GHz laptop, assuming that performance of C code has not been changed much comparing to the old version of OpenCV, this is how much faster the summer version will be comparing to #0 and #1pre).
Started the work on updating image warping functions. cvResize & cvWarpAffine have been rewritten and optimized using SSE2 intrinsics. The code is now being debugged.

FLANN:

Anatoly finished benchmarking FLANN and OpenCV’s KD-tree.

Below is the table (M1 and M3 mean the same thing as in previous report:
- M1 – ratio of the distance to the found neighbor to the average distance to points in DB. The smaller – the better.
- M3 – normalized to standard deviation difference between average distance and best distance. >= 2-2.5 is already very good. The larger – the better.

Conclusion
- Current conclusion is that OpenCV’s kd-tree is comparable to FLANN in both speed and accuracy (by varying parameters we can get either better accuracy or better performance).
- Anatoly now started working on HoG and the framework for model matching using spin-images.

3D Box

Sent video of tracking another box
Put in local SVN

Application area

Under trunk put OpenCV Apps

3D Reconstruction

Nothing

Action Items

NEW

Gary

Talk to Radu about 3D rotation of points and create an image
Gary to send Anatoly stereo data for trying HOG and SPIN
Send gaussian curvature paper
Kurt to send paper on fast plane fit
Histogram of Gradient link

Vadim

Anatoly — prioritize HOG, SPIN
- Fast plane fit

Victor

Set up the first OpenCV appliation: 3D box
- Video of it working
- Raw video that spits out marked object

FROM LAST TIME

Gary

Put up GSofC projects
Job application page
Put up application
Ask Marius to interact with Anatoly on FLANN

Vadim

Put up GSofC projects

Victor

Put up GSofC projects

2009-03-31

Agenda

Interface progress

Python machine learning

Updates for

Accomplishments
Monthly announce, Change Logs
Contributors credit page?

Spin and HOG progress

How was HOG done?

Trying 3D capture code

Point cloud library

Application areas

New HighGUI

Improving the machine learning interface(?)

There is also the issue of a new classifier design for robotics that needs the following capabilities:
- Needs to do one shot learning (but can refine later)
- Needs to be able to combine separately learned objects in a sub-linear way
- Needs the ability to express some structure in the data, particularly spatial or geometric structure.

Minutes

OpenCV Summer Release

General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python
  - Vadim, Alex …
- 2D features C-descriptor, HoG Boundary Fragments …
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Boundary fragments … working on focus.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
  - SPIN ~~almost~~ done
  - Radu is organizing his code into a library … maybe we can just lift this
- 3D capture http://www.3dsom.com/index.html
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
  - will ask Jean-Yves for pointers
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- .

OTHER

Vadim:

Revised, converted to the new API and tested:
- all the image warping functions (Resize, WarpAffine, WarpPerspective, Remap & Undistort), with SSE2 optimization the performance improves up to 50-100%.
- integral images
- fixed-level threshold and adaptive threshold + SSE2 optimization
- functions for background statistics accumulation
- For the previously added median blur function added SSE2 optimization for 3×3 and 5×5 kernels, up to x10 speedup.
Fixed haartraining app. problem (reported by Caroline Pantofaru) by reverting couple of files to the versions from OpenCV #0.
Modified spin images to use the new C++ matrix class. The updated code is in SVN.
Integrated MSER code by Liu Liu. For now the C interface is retained.
Anatoly implemented function for extracting HoG features based on the contributed by Alberto Orlandi code. At new version is also ~50% faster. Some further optimization is possible. The code is in SVN now.
Started working with the dataset of IKEA objects. HoG features did not provide any reasonable output (e.g. comparing to SURF), because the object outlines are very simple and HoG seems to require a classifier on top of it. Now trying to run spin images.

Next for API is optical flow
- Pyramid and adding
- Dense optical flow)
- High Gui
MLL
- Extremely random trees Importance Variables
- Input interface: to csv and loaded with standard function in SVN tomorrow

ROS stable this week.
- latest_tested_snapshot is the SVN flag for ROS

Stereo data

On mechanical turk
Radu, has developed a plane finder interface

Nearest neighbor

.

Spin features

Tried on synthetic data and vmrl files
Missing elements vs Andrew Johnson features
- Using geometric constraints if occluded object
API 3D point cloud from oct tree
- Pass set of points into spin function
- List of 3D locations to compute spin images at
- Computes normals
- Pass pts and normals to a function that computes spin images
  - Support is the radius of included points
Memory and time hogs — hard to optimize

HOG

Anatoli tried HOG in a “Haar style” manner
HOG features compared in matching set of points

Lighting

Want to start getting in specularity and lighting direction detectors
Train by by rendering

Stanford feature

Sub patches from mean

3D capture code

Start Anatoly on 3D capture

Action Items

NEW

Gary

Vadim

Check kdd trees on small vectors

Victor

FROM LAST TIME

Gary

Talk to Radu about 3D rotation of points and create an image
Gary to send Anatoly stereo data for trying HOG and SPIN
- Look in
- http://vault.willowgarage.com/wgdata1/vol1/IKea_Related/Class
- For file Objects_kjlKJL.tar.gz (0.5G)
Send gaussian curvature paper
Kurt to send paper on fast plane fit
Histogram of Gradient link

Vadim

Anatoly — prioritize HOG, SPIN
- Fast plane fit

Victor

Set up the first OpenCV appliation: 3D box
- Video of it working
- Raw video that spits out marked object

2009-04-07

Agenda

We need to update:
- Accomplishments
- Monthly announce
- Change Logs Can we get new functions listed here?
- Contributors credit page?
- Update what people are working on.
Vadim
- Stereo block match at two scales. Progress?
- kd_tree tests for small vectors?
- New interface progress
- 3D Capture — software in? Testing?
- Python interface based on numpy matrices?
- Optical flow code?
Victor
- Code for 3D box tracking
- Chamfer matching
  - 2D
  - 3D?
- Machine learning reading csv
  - Other MLL improvements

Minutes

Victor

3D box code (put into OpenCV application section)
Chamfer matching
- Used in outlet matching — works well for orange matching
  - Many false positives in high texture
  - Works worse on white on white because edges are unreliable
  - Useful for European outlets
  - Edge template better, but depends on edge fidelity
MLL
- a major commit to opencv svn.
  - The new memory-efficient structure for tree-based classifiers has been implemented. The new memory limitations are still under investigation (because of the recent changes in cxcore).
  - About 2X saving for old version of OpenCV, looking at new version
  - 10,000 samples, 1000 variables.
  - Doing new tests (Maria)

Application section

Same svn link, but put opencv_extra
For example https://opencvlibrary.svn.sourceforge.net/svnroot/opencvlibrary/trunk/opencv_extra/3d/createsamples3d/

Vadim

Stereo starting tommorrow
kd-tree on small vectors — will send table

Activity results:

Converted to the new API and optimized (by factor of 2) pyramid functions.
Integrated Fuzzy MeanShift Tracker by Farhad Dadgostar (the same person who contributed adaptive skin detector).
Integrated Farnerback dense optical flow.
Anatoly started investigation of 3DSom software for 3D reconstruction. A brief report has been sent.

Plans for this week:

Purchase some lighting setup for better background isolation. We have found some good possible setups in the internet already.
Finish optical flow intergration (other dense optical flow algorithm and better sparse optical flow).
Implement adaptive-window variant of stereo correspondence algorithm
Issues: we still have not received the license update from 3DSom.

Python

Can create numpy array and call OpenCV function
C-type tool in more use
Supports overload

Optical Flow

Forward back algorithm
Have another dense algorithm almost ready, will integrate
Improve sparse optical flow with color and image gradients
About 1 week more work (delayed for stereo)

Contours and Object detection

Need C++ interface
Contours
- 2 week working time (data structure issues)
Object detection
- Probably 1 week

HighGUI and MLL

Can be converted in interface
HighGUI fast
MLL ~ 1 month

External code, improvement in

The details of the implementation can be found here:

http://www.springerlink.com/content/p32p311820r63k68/

and one of the related videos also is here:

http://www.youtube.com/watch?v=gVPCrQBKKqE&feature=channel_page

OpenCV Summer Release

General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python
  - Vadim, Alex …
- 2D features C-descriptor, HoG Boundary Fragments …
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Boundary fragments … working on focus.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
  - SPIN
  - Radu is organizing his code into a library … maybe we can just lift this
- 3D capture http://www.3dsom.com/index.html
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- .

OTHER

Action Items

NEW

Gary

Write back up note for 3D capture software

Vadim

Send small vector results for kd-tree
Stereo block matching improvements
Assess opencv use with numpy

Victor

FROM LAST TIME

Gary

Vadim

Check kdd trees on small vectors

Victor

Send link for application section

2009-04-14

Agenda

OpenCV Summer Release

Feature freeze end of May, June for test and doc.
General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python
  - Vadim, Alex …
- 2D features C-descriptor, HoG Boundary Fragments …
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Boundary fragments … working on focus.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
  - SPIN
  - Radu is organizing his code into a library … maybe we can just lift this
- 3D capture http://www.3dsom.com/index.html
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- .

Minutes

Vadim

Converted image I/O to the new API, added extra parameters to cvSaveImage and its new counterpart that can regulate jpeg image quality, PNG compression.
Added in-memory image I/O: cvDecodeImage & cvEncodeImage. For jpeg’s and PNG’s a special implementation has been done. For other formats a temporary file is used.
Later on, the temporary files can be eliminated.
Played a bit with block correlation stereo correspondence.
Currently, the following scheme is used:
1. first, the disparity map is computed with large block size = bigWSZ (e.g. 21×21).
2. then, the disparity map is computed with smaller block size smallWSZ (like 11×11) and if at some pixel the new correspondence value C is within certain [A, B] range, where A is minimum and B is maximum, computed over bigWSZ x bigWSZ neighborhood.
  the results on a couple of stereo pairs are attached.
Continued the work on porting optical flow to the new API and improving it. Unfortunately, the newest BBW code seem to be left at Intel, need to reproduce it from scratch. For now concentrated on improving the sparse optical flow.
Purchased small lighting system (cube + 3 lamps) for the further experiments with 3D capturing.
Anatoly is on vacation this week.

Victor

Another version of chamfer matching with templates represented as a binary image (as opposed to vector CvSeq representation) has been implemented. (Victor)
MLL: data loader has been implemented. Now one can load a dataset, learn a classifier and calculate test error with 3 lines of code!
CvMLData, new class. And in sample session TreeEngine.cpp for examples of loading and training.
- Split into train and test, feature selection, missing values … all inside this class
- Works for any tree based classifier (ExtremelyRandom forest)
Will update latest_tested_snapshot flag today.
New in memory I/O (with jpg or png) is implemented
- Returns byte vector, template vector class (opencv version — has reference counting etc).
Support James Bowman on python interface
Benchmarks for kd-trees.
- 3D space, FLANN is much faster
- When dimensionality grows, difference is zero or OpenCV trees become faster
  - Might be memory allocation overhead
2 possible Experts for stereo
Bought lighting for 3D, segmentation works well
- Need the 3D license

Action Items

NEW

Gary

Write about 3D license.
Grep through ROS/vision for usage of OpenCV

Vadim

Send the experimental stereo code to Kurt
Send Gary email when the latest_tested_snapshot flag is moved
Need to check kd tree benchmarks
Vadim communicate May feature freeze for OpenCV and June for test and doc

Victor

Send link to application section

We need to update:

Accomplishments
Monthly announce
Change Logs Can we get new functions listed here?
Contributors credit page?
OpenCV_Team: Update what people are working on.

Kurt

Send better explanation for small-large/inner-outer windows

FROM LAST TIME

Gary

Write back up note for 3D capture software

Vadim

Send small vector results for kd-tree
Stereo block matching improvements
Assess opencv use with numpy

Victor

Link to applications’ section

2009-04-21

Agenda

Improving stereo progress report
Possible to get Navneet Dalal’s original HOG code http://pascal.inrialpes.fr/soft/olt/
New API update and time-lines
Python interface with James Bowman
Remember to touch and check-in the OpenCV ROS Makefile when you update latest_tested_snapshot SVN flag.
Machine learning
- Nister trees
- Plans for a robot vision architecture
  - Can we minimize up the memory footprint in nearest neighbor type apps?
Speed up to Chamfer matching
(Gary) Exploring new 3D feature that approximates surface normals
Fast segmentation

OpenCV Summer Release

Feature freeze end of May, June for test and doc.
General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python In progress thanks to James Bowman’
  - Vadim, Alex …
- 2D features C-descriptor,
  - HoG ‘’’(status)‘’’
    - Boundary Fragments … In progress thanks to Marius Muja
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation, normals
  - SPIN
  - Radu is organizing his code into a library … maybe we can just lift this
- 3D capture http://www.3dsom.com/index.html
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
- Optical flow
  - Move code derived form former work in OpenCV
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - More convenient training via a data class
  - Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
  - Regression tests … maybe general cross validation
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- .

Minutes

Vadim

Image encoding/decoding has been further improved: for bmp, ppm and tiff the codecs do not use temporary files. Performance of PNG encoder has been improved by factor of 5 by tuning the compression parameters passed to libpng.
latest_tested_snapshot tag has been finally updated. Prior to this, OpenCV has been built and tested on MacOSX, 64-bit Linux and Windows. The update includes all the recently added functionality, including slightly improved block matching-based stereo correspondence.
added new C++ classes for multi-dimensional matrixes, cv::MatND and cv::SparseMat and the derived template classes with convenient operators for element access. The classes will be used to port histogram processing to C++. For cv::MatND the set of basic operations (arithmetics, statistics etc.) has been added too, 45 functions in total. The functions just process the n-D matrices plane by plane using the corresponding functions operating on cv::Mat’s.
About 70% done with new API
- Histograms, contours, polar, obj detection maybe (probably a thin wrapper since extending the object detectors)
Conversion of MLL would be next
High GUI should be easy to convert
Anatoly is back from vacation, we started design and implementation of 3D reconstruction code. For feature matching we are going to use SURF with outlier filtering based on epipolar geometry constraits. For the optimization part the existing Levenberg-Marquardt solver from OpenCV.
Continued work on improving sparse optical flow algorithm and porting it to C++ interface. The draft version is ready. Testing is to be done this week.
Stereo sample code is with Kurt, haven’t done anything until hear back.
- Small windows did improve a bit

Coming:

Improved Haar training
3D reconstruction

Victor

Maria looking at Adaboost cascade detector trying to make it work with arbitrary features instead of just Haar wavelets.
- This code is separate from MLL classifiers and specialized to cascades so we can’t just move general boosting in.

Python

wrapper code for py→c can maybe extend for py→C++ for James python implementation

MLL for robots. Want:

Combine separate learning in way that grows sublinearly in recognition time.
- Willing to use more memory for this
For features. Needs to deal with
- Missing features
- Newly added features
Want some form of memory optimization that can be done offline.

Action Items

NEW

Gary

Adaboost with cascade comparison? Papers?
Send HOG code to Vadim
3D capture license …

Vadim

Look at moving Navneet Dalal’s original HOG code http://pascal.inrialpes.fr/soft/olt/ into OpenCV replacing its data structures and learning with OpenCV and MLL structures and functions. Maria?

Victor

We need to update:
- Accomplishments
- Monthly announce
- Change Logs Can we get new functions listed here?
- Contributors credit page?
- OpenCV_Team: Update what people are working on.

FROM LAST TIME

Gary

Write about 3D license.
Grep through ROS/vision for usage of OpenCV

Vadim

Send the experimental stereo code to Kurt
Send Gary email when the latest_tested_snapshot flag is moved
Need to check kd tree benchmarks
Vadim communicate May feature freeze for OpenCV and June for test and doc

Victor

Send link to application section

2009-04-27

Agenda

New API progress
Python progress
Optical Flow progress
Fast Graph-Cut based Segmentation — Konstantin Mirin might do this, we’ll see.
- Based on A Multilevel Banded Graph Cuts Method for Fast Image Segmentation
  - step.polymtl.ca/~rv101/bandedgraphcuts.pdf
MLL
- Arbitrary feature cascade (Adaboost cascade detector trying to make it work with arbitrary features)
- HOG application
- parts/whole object recognition
Utility function (?) Specify translation and angle change from face on normal and return equivalent homography matrix.
Maruis is fairly close to being able to contribute fast Chamfer matching with orientation.
- Another function is simply robust orientation from noisy line segments.

OpenCV Summer Release

Feature freeze end of May, June for test and doc.
General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python In progress thanks to James Bowman’
  - Vadim, Alex …
- 2D features C-descriptor,
  - HoG ‘’’(status)‘’’
    - Boundary Fragments … In progress thanks to Marius Muja
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation, normals
  - SPIN
  - Radu is organizing his code into a library … maybe we can just lift this
- 3D capture http://www.3dsom.com/index.html
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
- Optical flow
  - Move code derived form former work in OpenCV
  - Variational optical flow (maybe)
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - More convenient training via a data class
  - Regression tests … maybe general cross validation
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- In memory .jpg, .png

Minutes

Vadim:

Added new-style histogram functions. Besides the use of the new C++ multidimensional matrix classes, the functions can now compute histograms or back projections directly from color images, there is no need to split an image into separate color planes.
Finished improved sparse optical flow. The new function can use color information and spatial image derivatives for better accuracy. Rewritten cvCalcOpticalFlowBM after James’ bug report. The new implementation can still be improved a lot, e.g. some modern algorithm used in video codecs can be implemented.
Fixed several bugs in the new code: bug in Farnerback’s optical flow with non-squared images (patch by Ballard Stonewall), bug in reshape (by asmwarrior), bugs in cvCalcOpticalFlowBM and cvGetAffineTransform (by James Bowman),
bug in morphology (by Marius). Another bug in highgui about capturing rectified stereo from Videre camera has been discovered but not fixed yet.
Anatoly continued the work on feature registration part for 3D reconstruction project. We used the recently purchased lighting setup to capture some objects with nicely separated background, and find the corresponding feature points. We tried SURF with different parameters and cvGoodFeaturesToTrack + cvCalcAffineFlowPyrLK. When for every pair of views for which we found many enough corresponding pairs of features we estimated fundamental matrix and used it to filter outliers. Unfortunately, we found that normally only about 10 pairs satisfy the epipolar constraints and that’s for a good reason – SURF often failed to find the correct corresponding points; affine flow’s “hit ratio” was higher, but the number of features was less than in case of SURF, so the total number of good matches was about the same. May be we should take SURF keypoints and try affine flow on it, or try another descriptor, or preprocess images somehow to remove noisy features. The experiments continue.

Plans for this week:

Finish C++ interface for cv module (contours, color conversion; object probably, camera calibration and object detection too)
Update latest_tested_snapshot tag
Study the HOG people detector
Advance in feature matching for 3D reconstruction.

API

Contours
About a dozen more, so 1 week for cv to be finished
- Then to decide on MLL change over.
  - Convert decision tree methods, K nearest too

HOG

Paper

Cascade generalization

Maria custom class for data retrieval
Idea was to extract previous re-write of the tree code
- Now its done via call-backs
  - Allows running in or out of memory
  - Issue of how to do from a user stand-point. Positive samples are obvious, negatives are harder since we have to provide filtered (surviving) negative samples to later stages.
For now, want to just rebuild haar training with this. 2 weeks: Implement and debug. Then 1 week/feature.
Training data
- INRIA dataset on line
  - Convert

Optical flow

BM fixed and done
Sparse optical flow can work with color now
Lost variational optical flow (Horn Schunk with local property)

Action Items

NEW

Gary

Talk to Kurt about expanding the team for stereo
Ask James about python progress
3D capture license …

Vadim

HOG image code
- Look at moving Navneet Dalal’s original HOG code http://pascal.inrialpes.fr/soft/olt/ into OpenCV replacing its data structures and learning with OpenCV and MLL structures and functions. Maria?

Victor

All:

Read the ICCV 2009 paper
We need to update:
- Accomplishments
- Monthly announce
- Change Logs Can we get new functions listed here?
- Contributors credit page?
- OpenCV_Team: Update what people are working on.

FROM LAST TIME

Gary

~~Adaboost with cascade comparison? Papers?~~
Send HOG code to Vadim
3D capture license …

Vadim

Victor

2009-05-05

Agenda

HOG code?
OpenCV Python. Now in ROS, how about into OpenCV?
Morphology bugs (James Bowmen reported)?
Is there news on dense optical flow? Get Michael Black’s code?
C++ conversion of cv?
- contours, color conversion; object probably, camera calibration and object detection
3D feature matching/3D reconstruction progress?
Parts based model from outlets into OpenCV?
Cascade generalization progress
3D features for recognition (Victor)?
Normals etc for 3D point clouds in grids (as derived from dense stereo) (Gary)
Fast Chamfer matching with orientation into OpenCV (Marius)?

OpenCV Summer Release

Feature freeze end of May, June for test and doc.
General Structure:
- Decisions on library structure (Google’s request, separate packages as is …)
  - Provide way to build OpenCV statically, then size varies with binary size
New Features
- C++ interface
  - cxcore
  - cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
- Better python In progress thanks to James Bowman’
  - Vadim, Alex …
- 2D features C-descriptor,
  - HoG ‘’’(status)‘’’
    - Boundary Fragments … In progress thanks to Marius Muja
  - C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
    - No re-implementation yet. Code based on randomized tree, but specialized version.
  - Get contributions of 2D features externally
  - MSER … (Gary has code)
  - HoG
  - Star is already in and in for ROS build
- Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
  - Optical flow
- 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation, normals
  - SPIN
  - Radu is organizing his code into a library … maybe we can just lift this
- 3D capture http://www.3dsom.com/index.html
- VSLAM
  - Wait on what Kurt and James get together
- Image stitching
- Optical flow
  - Move code derived form former work in OpenCV
  - Variational optical flow (maybe)
- MLL
  - Extreme trees are coming
  - Reduce memory footprint
  - More convenient training via a data class
  - Regression tests … maybe general cross validation
- Misc: Focus detector, adaptive skin tracking
  - As discussed above
Optimizations
- .
Documentation
- .
High GUI
- In memory .jpg, .png

Minutes

Gary

Working on 3D features


int computeNormals(const CvMat *xyzd, const CvPoint &p, vector<triangle_offsets> &v, float sqr_root_max_radius, float max_depth_difference,
vector<float> &normal_pt3d, vector<vector<float> > &normals, vector<vector<float> > &normal_angles);

vector findMedianND(vector<vector > &v, int &median_index);

vector centerOfMassND(vector<vector > &v, vector *b = NULL);

Vadim

fixed a few bugs in the new histogram functions; still camshiftdemo and chessboarddetector fail despite that histogram tests pass; so need to explore it.
started porting contours and comp. geomery functions to the C++ API. In the new API a contour is represented as Vector or Vector, so it’s much easier to operate with.
studied HOG training and detection code by Navneet Dalal. Here are my findings:
1. There are precompiled binaries are available for Mac and Linux; the source code is also avaialble and it builds fine (on MacOSX at least).
  All the training data for people detection is also available. The training procedure runs fine on this training data.
  The code depends on Boost, Blitz++ and Imlib2 and includes a small subset of Vigra library.
  Also, the training pipeline uses modified SVMLight (the modification was done to make SVMLight work with huge amount of data,
  e.g. authors mention #7Gb buffers necessary for training). SVM training is run as a separate executable from within the shell script.
2. The training is done as following:
  - HOG descriptors are computed for every positive sample and randomly selected windows from negative images. The default window size for person detection is 64×128. This is done by dump_rhog utility.
  - Then the dump file is converted via dump4svmlearn to the format acceptable by svm_learn from SVMLight
  - After this initial model is trained, it is passed to another utility “classify_rhog” that is run in a special mode. Normally, the utility is used to just use the trained model to do batch detection and optional visualization of the results. But in this mode it runs exhaustive search for “hard” negative samples, i.e. the false positives that the initial SVM model gave.
  - After that all the initially used positive and negative samples, and also as many “hard” negative samples as fit into the memory are passed (via dump_rhog and dump4svmlearn) o svm_learn that produces the final model.
  - Linear SVM is used by default and seems to be the only fully supported SVM type supported by the detection part. Authors say, they tried RDF kernel as well, but the increased training and detection time did not worth the small decrease of false positive scores. Thus, the result of all the training procedure is a single floating point vector M (With the default parameters it has 3781 elements). And the basic “binary” detection rule, once HOG descriptor d is computed, is “M^t*[d, 1] > c ? true : false”, where c is some parameter, that we set depending on the hit-rate/false positive-rate balance that we want to achieve.
  - On top of the binary detector, there is mean-shift filtering procedure that is used to group the found regions (and may be throw away some, depending on the computed confidence measures)
  - HOG computing in the PhD thesis and in the code is quite different from the code contributed to OpenCV:
    - the original HOG supports color images, OpenCV’s version does not
    - the original code can do the optional gamma normalization (turned on by default)
    - the original code applies Gaussian weighting function to the gradient values
    - the original code uses tri-linear interpolation when updating histograms to avoid/reduce aliasing effect.
    - the original code uses a bit more tricky histogram normalization (so called L2-hysteresis normalization, borrowed from D. Lowe paper on SIFT)
  - The whole code is quite difficult to read because of using multiple abstraction layers and many advanced C++ concepts. I suggest the following:
    - the training code is quite sophisticated, has many options and pretty stable. It can take quite a bit of time to reimplement it (although, if we only implement a subset of possible options, e.g. only the default mode, that may be faster to implement). But as the training is not an online procedure demanding realtime performance, we can reuse the existing training code as is, at least for now.
    - we can implement conversion utility from svm_learn output to an XML file in OpenCV-compatible format, that will contain both the decision boundary vector (M) and the HOG parameters.
    - the HOG descriptor in OpenCV can be extended to the original variant in a week (the default mode only for now). It will certainly be much, much faster than the original. That is, in a week we can have some fast people detector that we can evaluate and decide whether we should put extra efforts to porting the training application.

Plans for next week:

fix remaining bugs in histograms
finally update latest_tested_snapshot
extend HOG descriptor in OpenCV and try to make a sample for the people detector
stereo experiments

HOG

Triggs code modifies SVM light to work with large data ~ pseudo perceptron mode
Detection is dot-product
Can make detection 10x or more faster
- Keep original training for now, don’t understand
Speed up detector and allow specific window location search mode

Python for OpenCV

Separate sourceforge project
- 30 tests
would like more tests
- If finds bug, but them in test
Wiki isn’t up to date
People use ctypes
- Point to James on wiki
Right now, we have the swig interface

Documentation

html is most up to date
James to look into

Morphology bug

fixed the image shift bug
Will update latest tested shapshot

Dense flow

Tomnorrow will try to get dense optical flow

C++

This week will be finished

3D Reconstruction

Resume on May 11 when Anatoly gets back

Cascade generalization (Maria)

Rewriting haar training application
Extend decision tree
- Support data that will not fit in memory using callbacks
Finish this week … will send estimate on finishing cascade

Outlets

One way descriptor Fua et al
- Include into OpenCV
Perona, constellation model

Action Items

NEW

Gary

Look at license for HOG and see about use of code
Pointer to new python interface

Vadim

Look at license
Send update esitmate for finishing cascade

Victor

James

Look into cocumentation solutions

All:

Read the ICCV 2009 paper
We need to update:
- Accomplishments
- Monthly announce
- Change Logs Can we get new functions listed here?
- Contributors credit page?
- OpenCV_Team: Update what people are working on.

FROM LAST TIME

Gary

Talk to Kurt about expanding the team for stereo (not done)
Ask James about python progress.
3D capture license … (darn)

Vadim

HOG image code
- Look at moving Navneet Dalal’s original HOG code http://pascal.inrialpes.fr/soft/olt/ into OpenCV replacing its data structures and learning with OpenCV and MLL structures and functions. Maria?

Victor

One way descriptor into OpenCV
- Haven’t

All:

Read the ICCV 2009 paper
We need to update:
- Accomplishments
- Monthly announce
- Change Logs Can we get new functions listed here?
- Contributors credit page?
- OpenCV_Team: Update what people are working on.

2009-05-26

Agenda

Update HOG implementation
- Flexible Cascade
Update Python
- What’s the coverage? doc plans? examples?
Code contribution — official plan
- Test, docs etc
Update Doc
Summer Release
- Code freeze
- Need to catch up test/regression code
- Need to switch to and revamp the documentation

OpenCV Summer Release

Feature freeze end of May, June for test and doc.
General Structure:
- :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- C++ interface
  - {OK} cxcore
  - cv — start with filtering functions, resize, optical flow, object detection
- Better python interface
  - ‘’In progress thanks to’’ (’’’*James Bowman*’’’)
- 2D features
  - {OK} HOG
    - Boundary Fragments … In progress thanks to Marius Muja
  - C-descriptor … (Vadim)
    - :\ ? See below for LePetit style randomized tree.
  - {OK} MSER
  - {OK} Star
- {X} Alpha channel, alternatives to contours RLE
  - {X} New alpha transparency (per pixel) added
  - {X} Introduce functions with advance ways of blending
  - {X} Mask channels
- 3D features
  - Plane fit (Gary from Radu’s code)
  - LePetit style one way feature (Victor)
  - {X} PFH, FPFH from Radu,
  - {X} 3D Delaunay,
  - {X} 3D convex hull,
  - {X} mesh generation,
  - normals in grid (Gary)
  - normal based 3D patch (Gary)
  - {OK} SPIN
  - {X} 3D capture http://www.3dsom.com/index.html
  - {X} VSLAM
  - {X} Image stitching
- Optical flow
  - {OK} Dense (Farnerback)
  - {X} Variational optical flow
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - Nister trees (Patrick)
  - FLANN or appropriate parts (Marius)
  - :\ ? LePetit style random trees
Misc:
- {X} Focus detector,
- {OK} adaptive skin tracking
Optimizations
- Various functions, ongoing …
Documentation
- Latex
High GUI
- In memory .jpg, .png
- {X} move to C++ interface
Testing functions
- Regression tests for all functions
Define user contribution process:

Minutes

Vadim Update:

HOG has been opimized by factor of ~4. Instead of #8sec it now processes person_191 image from INRIA database in less than 0.5sec. The HOG has been integrated into the library, demo sample opencv/samples/c/peopledetect.cpp has been added.
The taged branch, latest_tested_snapshot, has been updated to include various bug fixes and the HOG detector.
9 bugs have been fixed: 2 bugs in drawing functions (reported by James and others), 2 bugs in cvFindFundamentalMat (one of them was reported by Anatoly), bug in cvSolve (unnecessary assert()), bug in cvConvertMaps (reported by Caroline), 3 bugs in build scripts. All the fixes are put into the tested branch as well.
Maria finished rewriting the haartraining application and now debugging it.
Anatoly continues work on the image registration part for 3D reconstruction. Several problems (including bugs in cvFindFundamentalMat) have been fixed, but the results are not satisfactory yet to move further.

HOG

In SVN and latest_stable
Needs commit access to ROS package
Training is original still
2.5×4 = 10 faster (took 4 seconds, latest version takes 0.5)

Cascade

Maria finished more flexible but
- right now Haar only right now
Debugging
- override features method
- General boosting code will end up somewhat different
- Darrius (wife Alexander Sorokin) balancing between tree depth and gradient boosting
  - Open code, will check out

Python

~300 functions
No machine learning
No cvaux
- Skipped tricky functions
- Many bugs
Samples are being done
Return’s output parameters
- new is much more “pythonic”
Python bindings (gen.py)
- Not using ctypes
- Can be adapted to new C++?
  - Can make that switch (in James’ spare time)

Documentation

C, C++, Python switchable.
Just place holders
Standardize to LaTex
Wiki pointer for documentation bugs

cvaux

SPIN
Background
and older stuff like HMM faces

Code coverage

gcov for coverage
- bullseye was used at Intel
MLL covered for decision trees
New functionality SPIN, HOG, Stereo NOT covered
Covered is simple image processing, Matrix functions
Complex functions — how to implement a test

Summer

OUT:
- Image stitching
- SLAM
- Bundle adjustment
  - Estimation of 3D coords
  - Bundle adjustment
- Alpha blending
- Focus detector
- UI enhancements in HighGUI
- PFH will wait for Radu to be here
IN
- 3D Capture
  - Image registration, 2D projection to get tracks for SFM
  - MV Geometry pipeline
    - Estimation of 3D coords
- Randomized trees
  - Random forest
  - Extremely Randomized Trees
  - c-descriptors, one way trees
- Gunnar Farneback dense flow is in
- Grid based normals
  - Feature
- Skin detector
- SPIN
- Codebook
- HighGUI
  - In memory jpg
  - Stereo camera support (with optional rectification, check with point gray)
- Nister tree across
- *Marius*’s library

Action Items

Vadim

Figure out code contribution process
Put up link for bugs and documentation bugs

Gary

3D features
Get Vadim write access to ROS packages
Ask Alexander about Darrius’s gradient boosting code
Figure out code contribution process (namespace, datastructures (use OpenCV)
Put up link for bugs and documentation bugs

Victor

One way descriptor

Marius

Fast Chamfer with orientation

Patrick

Nister trees in

All:

We need to update:
- Accomplishments
- Monthly announce
- Change Logs Can we get new functions listed here?
- Contributors credit page?
- OpenCV_Team: Update what people are working on.

2009-06-02

Agenda

Orientation filters for features from SIFT (“featureOrientation2D()”)
updates
- 3D feature
- boundary
- Vadim
- Docs
Code freeze and general status
Code contribution

OpenCV Summer Release

Feature freeze end of May, June for test and doc.
General Structure:
- :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- C++ interface
  - {OK} cxcore
  - cv — start with filtering functions, resize, optical flow, object detection

Better python interface
- ‘’In progress thanks to’’ (’’’*James Bowman*’’’)

2D features
- {OK} HOG
  - Boundary Fragments … In progress thanks to Marius Muja
- C-descriptor … (Vadim)
  - :\ ? See below for LePetit style randomized tree.
- {OK} MSER
- {OK} Star
- Self-similar detector
{X} Alpha channel, alternatives to contours RLE
- {X} New alpha transparency (per pixel) added
- {X} Introduce functions with advance ways of blending
- {X} Mask channels

3D features
- Plane fit (Gary from Radu’s code)
- LePetit style one way feature (Victor) (to Anatoly)
- {X} PFH, FPFH from Radu,
- {X} 3D Delaunay,
- {X} 3D convex hull,
- {X} mesh generation,
- normals in grid (Gary)
- normal based 3D patch (Gary)
- {OK} SPIN
- {X} 3D capture http://www.3dsom.com/index.html
- {X} VSLAM
- {X} Image stitching

Optical flow
- {OK} Dense (Farnerback)
- {X} Variational optical flow

MLL
- {OK} Extreme trees
- {OK} Reduce memory footprint
- {OK} More convenient training via a data class
- Nister trees (Patrick)
- FLANN or appropriate parts (Marius)
- :\ ? LePetit style random trees
- Generalized (boosted) cascade (generalizing Haar cascade)

Misc:
- {X} Focus detector,
- {OK} adaptive skin tracking

Optimizations
- Various functions, ongoing …

Documentation
- Latex

High GUI
- In memory .jpg, .png
- {X} move to C++ interface

Testing functions
- Regression tests for all functions
Define user contribution process:

Minutes

Put in feature orientation
- Use SURF or SIFT like technique as a general convenience to find gradient orientations at points in an image
Rainer has contributed an implementation of self-similar features for OpenCV

Vadim

Resumed the work on integrating C-descriptor into OpenCV. Specifications have been prepared, ~30% of the conversion work has been done (data structures design, outline of the internal functions etc.)
Anatoly continues working on the image registration part of 3D rec. package. The picture with the latest results is attached. Here we use one-way descriptor, received from Victor. The results are notably better than when using SURF descriptor, but there are not many matches. Hopefully, a better keypoint detector could help here. {{"correspondence.png}}":images/correspondence.png}}
Maria pretty much finished debugging haartraining application. Now it’s quite stable and achieves the target hit rate and false alarm rate on our training dataset. Maria is now adapting the face detection example to use the newly trained cascade (as the data structure is different).

Rainer Lienhart

self-similarity feature for the summer release of opencv. The documentation should mention the following two papers:

Shechtman, E., Irani, M.: Matching local self-similarities across images and videos. CVPR, (2007)
Eva Horster, Thomas Greif, Rainer Lienhart, Malcolm Slaney. Comparing Local Feature Descriptors in pLSA-Based Image Models. 30th Annual Symposium of the German Association for Pattern Recognition (DAGM) 2008, Munich, Germany, June 2008.

I also put the PDFs of these two papers into the top-level directory for reference.

At the moment the self-similarity features are not rotation-invariant. This could be changed in future if needed.

Everything in “src/” should be compiled and linked. It only needs opencv as an external dependency. I used Eclipse to compile everything, so you might want to use the project file.

I uncommented all IPP call and replaced them with slow c-loops. If you you have IPP it is easy to change back.

I also put some documentation of the various data types into the main directory. It’s called MMC Library Reference Manual.

Action Items

Gary

Find tools for code contribution

LAST TIME

Vadim

Figure out code contribution process
Put up link for bugs and documentation bugs

Gary

3D features
Get Vadim write access to ROS packages
Ask Alexander about Darrius’s gradient boosting code
Figure out code contribution process (namespace, datastructures (use OpenCV)
Put up link for bugs and documentation bugs

Victor

One way descriptor

Marius

Fast Chamfer with orientation

Patrick

Nister trees in

All:

We need to update:
- Accomplishments
- Monthly announce
- Change Logs Can we get new functions listed here?
- Contributors credit page?
- OpenCV_Team: Update what people are working on.

2009-06-09

Agenda

Phase space contributing Sergey’s Bundle adjustment
Self similarity feature
James’s document conversion to Latex→Sphinx→html
Unit testing code (gtest?)
3D registration, Haar training
Documentation
- http://opencv.willowgarage.com/documentation/index.html
Python
HOG/Cascade

OpenCV Summer Release

Feature freeze end of May, June for test and doc.
General Structure:
- :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- C++ interface
  - {OK} cxcore
  - cv — start with filtering functions, resize, optical flow, object detection

Better python interface
- In progress thanks to James Bowman

2D features
- {OK} HOG
  - Boundary Fragments … In progress thanks to Marius Muja
- C-descriptor … (Vadim)
  - :\ ? See below for LePetit style randomized tree.
- {OK} MSER
- {OK} Star
- Self-similar detector
{X} Alpha channel, alternatives to contours RLE
- {X} New alpha transparency (per pixel) added
- {X} Introduce functions with advance ways of blending
- {X} Mask channels

3D features
- Bundle Adjustment (probable contribution by Phase Space)
- Plane fit (Gary from Radu’s code)
- LePetit style one way feature (Victor) (to Anatoly)
- {X} PFH, FPFH from Radu,
- {X} 3D Delaunay,
- {X} 3D convex hull,
- {X} mesh generation,
- normals in grid (Gary)
- normal based 3D patch (Gary)
- {OK} SPIN
- {X} 3D capture http://www.3dsom.com/index.html
- {X} VSLAM
- {X} Image stitching

Optical flow
- {OK} Dense (Farnerback)
- {X} Variational optical flow

MLL
- {OK} Extreme trees
- {OK} Reduce memory footprint
- {OK} More convenient training via a data class
- Nister trees (Patrick)
- FLANN or appropriate parts (Marius)
- :\ ? LePetit style random trees
- Generalized (boosted) cascade (generalizing Haar cascade)

Misc:
- {X} Focus detector,
- {OK} adaptive skin tracking

Optimizations
- Various functions, ongoing …

Documentation
- Latex

High GUI
- In memory .jpg, .png
- {X} move to C++ interface

Testing functions
- Regression tests for all functions
Define user contribution process:

Minutes

Phase Space, via Sergey and Xavier DeLacour
- Are probably going to contribute bundle adjustment to OpenCV
- Xavier also contributed Octave wrappers, KD Trees
Stereo demo. Updated.
New interface almost finished for C++ interface
Self similarity … not looked at yet
- Not rotation invarient
For 2D rotation invariance
- Transform each patch using random rotations
- Compute matches for each
- Choose best
Documentation http://opencv.willowgarage.com/documentation/index.html
- Located in doc2’
- move to doc
- Need to update the Wiki
- Can turn on document build in cmake configure stage
Unit testing
- Current OpenCV system cxts module
  - Cxcore (lots) 50-75%
  - Cv (lots) 30%
  - Mll (little) ?
  - C++ interface (all of Cxcore, half of Cv) 3/4 coverage
- Not covered
  - New functions
  - C++ interface
- Python tests 300
  - Test files about 100
Python
- Bugs that are bottleneck
HOG/People
- Pass a mask to the detect function valid top left corners
New Cascade
- Have been trying to reproduce old cascade using ~small database
- Now are trying the whole face database
  - This week ~ might be done
- Current haar training uses whole dataset, can we do online training

Vadim

Nearly finished converting cv module to C++ interface. The converted functions are:
Canny, cvtColor, motion templates, CAMShift + Meanshift, Kalman filter, floodfill, watershed, inpainting, distance transform, Hough transform (circle & line detection), moments, findContours, contour processing functions (like approxPoly, convexHull, contourArea etc.), Haar object detection, block matching stereo, feature detectors (Star, SURF, MSER, GoodFeaturesToTrack), 3D geometry (findHomography, findFundamentalMat, solvePnP …) and calibration functions.
There remaning non-yet converted functions in cv are:
Planar subdivisions (it’s unclear what to do with the data types)
cvCalcEMD2, cvPyrSegmentation, cvPyrMeanShiftSegmentation, cvCalcOpticalFlow{LK|BM|HS} – old optical flow algorithms,
ConDensation algorithm (probably, need to be moved to cvaux or extended), cvSnakeImage,
kd-tree (need to revise the implementation), POSIT, graph cuts-based stereo (needs some optimization)
+ about a half-dozen of other small functions.
Except for kd-tree, EMD and maybe couple of other functions, the rest will probably be converted after the release.
Anatoly works on integration of the one-way descriptor to OpenCV. 1 serious bug has been found and fix is sent to Victor.
Maria wrote a converter from the new cascade XML representation to the old one, supported by OpenCV functions. Several bugs have been fixed in haartraining that keeps getting better.

Plans:

finish integration of C-descriptor
convert kd-trees to the new interface or take it from FLANN.
finalize the new haartraining and replace the old version with the new one (there is a risk that we can not stabilize it enough for the release, probably, move it to the next after 2.0 update)
figure out if MLL classes can be quickly adjusted to be compatible/easier to use with the new C++ data structures.

Victor

On vacation

Action Items

Vadim

Look at priority 6 python bugs
- Will look at today

James

Merge doc2 to doc
In generated html
- Document bug button(s)
  - OR: Other ways of getting feedback such as a comment sections
- link back to OpenCV Wiki
- cmake to build

Caroline

Email mask interface considerations

Done

Not done:

3D features
Nistor Tree
Figure out code contribution process
Documentation bug link
Fast Chamfer with orientation

2009-06-16

Agenda

Ongoing status
New docs up
Bundle adjustment from phase space
Contribution template discussion
3D scan progress
Where is HOG training code
Some post release thoughts —
- more concentation on 2D and 3D features, object recognition
- Ability to run in cloud

OpenCV Summer Release

Feature freeze end of May, June for test and doc.
General Structure:
- :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- C++ interface
  - {OK} cxcore
  - cv — start with filtering functions, resize, optical flow, object detection

Better python interface
- In progress thanks to James Bowman

2D features
- {OK} HOG
  - Boundary Fragments … In progress thanks to Marius Muja
- C-descriptor … Vadim
  - :\ ? See below for LePetit style randomized tree.
- {OK} MSER
- {OK} Star
- Self-similar detector
{X} Alpha channel, alternatives to contours RLE
- {X} New alpha transparency (per pixel) added
- {X} Introduce functions with advance ways of blending
- {X} Mask channels

3D features
- Bundle Adjustment (probable contribution by Phase Space)
- Plane fit (Gary from Radu’s code)
- LePetit style one way feature (Victor) (to Anatoly)
- {X} PFH, FPFH from Radu,
- {X} 3D Delaunay,
- {X} 3D convex hull,
- {X} mesh generation,
- normals in grid (Gary)
- normal based 3D patch (Gary)
- {OK} SPIN
- {X} 3D capture http://www.3dsom.com/index.html
- {X} VSLAM
- {X} Image stitching

Optical flow
- {OK} Dense (Farnerback)
- {X} Variational optical flow

MLL
- {OK} Extreme trees
- {OK} Reduce memory footprint
- {OK} More convenient training via a data class
- Nister trees (Patrick)
- FLANN or appropriate parts (Marius)
- :\ ? LePetit style random trees
- Generalized (boosted) cascade (generalizing Haar cascade)

Misc:
- {X} Focus detector,
- {OK} adaptive skin tracking

Optimizations
- Various functions, ongoing …

Documentation
- Latex

High GUI
- In memory .jpg, .png
- {X} move to C++ interface

Testing functions
- Regression tests for all functions
Define user contribution process:

Minutes

Trying to push documentation bugs to bug reporting system
CvMat Mat Documentation
unit tests
- Use old cxcoretest, cvtest,
- Put new tests in python (?)
  - Contact James
Haar training seems to be what it was
- Finalize to release
- Add new features
  - Implement new class from base class
  - Compute feature on image (call back)
  - Implement read write functions: 2 phase: boosted emsembles, then feature descriptions
  - Enumerate the features. Current Haar sets rect size and then enumerates all possible features
    - Example should go into documention
- but probably not prior to release
Need to slip code release due to major items 2 weeks
- The remaining things to be done (all of the items below can be done in parallel, but I have to spend some time on each of the items):
  - kdTree cleanup (now it does not pass the tests) or possible FLANN integration
  - debugging C-descriptor (1-2 days)
  - integration of one-way descriptor. Integration of the fixes from Victor (~1 week)
  - code review of the rewritten haartraining, ingtegration (~1 week)
  - bundle adjustment integration (1 week at least, may be more)
  - Nister trees integration (1 week at least)
3D scan code
- Stuck at registration part: Need to collect many correspondences per view
  - Trying different descriptors — one way descriptor seems better than SIFT and SURF for this
  - Need good, dense keypoint detectors
On way descriptor speed up delay a week for object detection.
Spin image code sent, best way to match … maybe kdtrees
HOG training code
- Put OpenCV extra
  - svn opencv.lib …/trunk-opencv-extra
- Default coefficients are built in for pedestrian detection
Website for user contribution
- Need voting site

FUTURE, POST RELEASE

Simple interface for object learning model
- Optimize parameters by cross validation
- Compare by ROC curve
  - False alarm vs False detect automated
  - prediction chart
  - variable importance
- Cross validation
- R interface (?) …
  - See if R interface was done Vasili (in outsourcing company) (done for PNL?)
- “HighGUI for ML”
  - Brings up interactive window
    - Visualize model
    - test error
    - model complexity
    - In python (?)
More features

Vadim:

C-Descriptor is 99% finished (the absent load/save functions will be added today). The code will be sent to Michael Calonder, who kindly agreed to review it.
- The feature training part can do either random projection-based (as in the original code) or PCA-based (as Michael adviced me to do) compression of the output classifier.
kMeans initialization has been changed (from random labels to random centers) for much better convergence properties. That fixed OpenCV SF bug #2804614.
OpenCV has been successfully built on Win64 (except for ffopencv module that requires precompiled 64-bit ffmpeg libs on Windows). Many warnings have been fixed; probabilistic Hough transform has been modified to work around internal MS compiler error. All the tests except for DFT, DCT and addWeighted passed.
Another bug in GEMM has been fixed (reported by Anatoly)
2 priority-6 bugs reported by James Bowman have been fixed.
Most of the highgui API (with the notable exception of video I/O functonality) has been converted to C++.
- 1 week, might be lower priority
Maria finished debugging haartraining. On our face database the new version is capable of producing cascade classifiers comparable with the original cascades.
Simple version of one-way descriptor compatible with the new OpenCV C++ data structures has been implemented by Anatoly (~80% complete). It is derived from the *Victor*’s code, but much shorter than the original because it’s lower-level (does not include any I/O or dataset management) and all the memory management is done automatically by C++. Another bug (basically the same bug with patch blurring) has been found and reported to Victor.

OpenCV release status:

I’m afraid, we need to postpone the code freeze by 1-2 more weeks.

Action Items

Gary

Contact James about unit tests in python, CC Vadim
Ask Kurt / James about Bundle adjustment in ROS (J.D Chen’s code)
Ask Scott about voting site for user contribution

Vadim

Talk to Sergey about bundle adjustment on his side

From Last Time

Vadim

Look at priority 6 python bugs
- Will look at today

James

Merge doc2 to doc
In generated html
- Document bug button(s)
  - OR: Other ways of getting feedback such as a comment sections
- link back to OpenCV Wiki
- cmake to build

Caroline

Email mask interface considerations

Done

Not done:

3D features
Nistor Tree
Figure out code contribution process
Documentation bug link
Fast Chamfer with orientation

2009-06-23

Agenda

Status, go through summer release list below
Documentation, still no feedback button.
Bundle adjustment status?
HOG training
One way descriptor — Victor
User contribution
Post release …
- Browser connections?
- Simple mesh routines
- 3D fitting
- Of course — object recognition: features, routines

OpenCV Summer Release

Feature freeze end of May, June for test and doc.
General Structure:
- :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- C++ interface
  - {OK} cxcore
  - cv — start with filtering functions, resize, optical flow, object detection

Better python interface
- In progress thanks to (James Bowman)

2D features
- {OK} HOG
  - Boundary Fragments … In progress thanks to Marius Muja
- C-descriptor … (Vadim)
  - :\ ? See below for LePetit style randomized tree.
- {OK} MSER
- {OK} Star
- Self-similar detector
{X} Alpha channel, alternatives to contours RLE
- {X} New alpha transparency (per pixel) added
- {X} Introduce functions with advance ways of blending
- {X} Mask channels

3D features
- Bundle Adjustment (probable contribution by Phase Space)
- Plane fit (Gary from Radu’s code)
- LePetit style one way feature (Victor) (to Anatoly)
- {X} PFH, FPFH from Radu,
- {X} 3D Delaunay,
- {X} 3D convex hull,
- {X} mesh generation,
- normals in grid (Gary)
- normal based 3D patch (Gary)
- {OK} SPIN
- {X} 3D capture http://www.3dsom.com/index.html
- {X} VSLAM
- {X} Image stitching

Optical flow
- {OK} Dense (Farnerback)
- {X} Variational optical flow

MLL
- {OK} Extreme trees
- {OK} Reduce memory footprint
- {OK} More convenient training via a data class
- Nister trees (Patrick)
- FLANN or appropriate parts (Marius)
- :\ ? LePetit style random trees
- Generalized (boosted) cascade (generalizing Haar cascade)

Misc:
- {X} Focus detector,
- {OK} adaptive skin tracking

Optimizations
- Various functions, ongoing …

Documentation
- Latex

High GUI
- In memory .jpg, .png
- {X} move to C++ interface

Testing functions
- Regression tests for all functions
Define user contribution process:

Minutes

NEED TO MAKE SURE PEOPLE GET CREDIT — will put in contributors page in wiki
- http://opencvlibrary.svn.sourceforge.net/svnroot/opencvlibrary/trunk/opencv/THANKS

Bugs in C-descriptor, look into and compare against ROS implemenation
Implemented kd-tree based on Xavier code, see below
K-means+algorithms by Aurthur and Vassilvitskii have better method of choosing centers.
- First center at random
- Other centers from a distribution that biases towards distant centers
- thereafter regular kmeans
- Fixed reported bug — takes many less iterations to converge
  - Might be in FLANN
Integration of FLANN …
Gary introduced more direct link to bugs from wiki
Bundle adjustment API
- Sergey implementing functionality with agreed API
OpenCV works fine under Mac OS
HOG training code.
- Try out original training code
  - Original code might have bugs in it. Binaries work for Mac, but maybe not source code for Linux
  - They use linear SVM, say RBF works but slower
  - Maybe upload to opencv extras and point to it.
One way descriptor
- Not waiting on Victor for now, but code needs this week for Beta release
- One way descriptor compares PCA affine perturbed patches
- C-descriptors compares PCA signatures from random tree affine perturbed patches.
  - Once and for all alphabet
- One way descriptors can use as many features as want, can keep adding
3D capture, Anatoly is experimenting with keypoint detectors and descriptors. Tried FAST last week.
- Good features to track is faster than FAST and doesn’t seem worse
John Hsu is using 3D models in simulator for training and test — good way to go
- Have more and more capture techniques
- Silhouette capture
Still no decision about user contribution
Nistor trees

Vadim

C-Descriptor is complete, but the results are pretty bad (very few found corresponces are correct). I continue debugging the code.
New C++ version of kd-trees, based on Xavier Delacour code, has been created. It is ~20% more accurate (in some cases ~2x more accurate) than the previous version (because of better prunning of the tree branches) and yet significantly faster, especially the tree construction part and the query in case of low dimensionality. Currently it only supports L2 metrics, but L1 is quite easy to add.
Another impovement in KMeans: now it’s possible to use the kmeans++ algorithm by David Arthur and Sergei Vassilvitskii, which is theoretically and empirically much superior to the random center initialization.
Maria fixed couple more bugs in haartraining and started integration of LBP (local binary pattern) features, which, in one hand, is close enough to Haar features and, in the other hand, could be a good verification that haartraining is extensible.
Talked to Sergey and agreed on the bundle adjustment API.
Added precompiled header support for Xcode and GNU makefiles, which should reduce build time.

Plans for the week:

finish debugging cdescriptor.
integrate one-way descriptor.
replace the old haartraining with the new one.

Action Items

Gary

Ask about Nistor Trees
Have Marius contribute Chamfer+orientation
Talk to Marius about contributing FLANN
PUT IN CONTRIBUTORS DIRECTORY INTO WIKI
Talk with Ian Goodfellow about bugs in HOG training code
See if we can get Balan’s matlab code for symmetric object 3D capture

Vadim

Find out about user contribution website similar to Matlab style

PREVIOUSLY

Gary

Contact James about unit tests in python, CC Vadim
Ask Kurt / James about Bundle adjustment in ROS (J.D Chen’s code)
Ask Scott about voting site for user contribution

Vadim

Talk to Sergey about bundle adjustment on his side

Done

Not done:

3D features
Nistor Tree
Figure out code contribution process
Fast Chamfer with orientation

2009-07-02

Agenda

Status, code/freeze. go through summer release list below
- Beta release
FLANN from/by Marius
- *Marius*’s Chamfer matching
High GUI interface
New descriptors from bazAR tracking package
Bundle adjustment status?
HOG training
One way descriptor — Victor
User contribution
Post Release
- Alex Teichman’s feature framework
  - His “kernel based” boosting framework
- Optimized pipelines

OpenCV Summer Release

Feature freeze end of … sometime in July for test and doc.
Release in August, 2009

General Structure:
- :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI (except for video I/O
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - (./) Use this to expand unit test functions for OpenCV

2D features
- {OK} HOG
  - (./) Chamfer matching with orientation In progress thanks to Marius Muja
- (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
  - (./) LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
  - {OK} C-descriptor … (Vadim)
  - (./) LePetit one way descriptor (Victor) (to Anatoly)
- {OK} MSER
- {OK} Star
- (./) Self-similar detector
{X} Alpha channel, alternatives to contours RLE
- {X} New alpha transparency (per pixel) added
- {X} Introduce functions with advance ways of blending
- {X} Mask channels

3D features
- (./) Bundle Adjustment (probable contribution by Phase Space)
- {X} Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
- {X} PFH, FPFH from Radu,
- {X} 3D Delaunay,
- {X} 3D convex hull,
- {X} mesh generation,
- (./) normals in grid (Gary)
- (./) normal based 3D patch (Gary)
- {OK} SPIN
- {X} 3D capture http://www.3dsom.com/index.html
- {X} VSLAM
- {X} Image stitching

Optical flow
- {OK} Dense (Farnerback)
- {X} Variational optical flow

MLL
- {OK} Extreme trees
- {OK} Reduce memory footprint
- {OK} More convenient training via a data class
- (./) Nister trees (Patrick)
- (./) FLANN or appropriate parts (Marius)
- {X} LePetit style random trees
- {OK} Generalized (boosted) cascade (generalizing Haar cascade)

Misc:
- {X} Focus detector,
- {OK} adaptive skin tracking

Optimizations
- (./) Various functions, ongoing …

Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- (./) Proof read
- (./) Add content
- (./) Create C++ documentation
- (./) Create Python documenation

High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
- {X} move to using QT as the HighGUI interface look and feel

Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write

(./) Define user contribution process:

Minutes

Vadim

c-descriptor has been debugged (thanks to Rainer for the code review and useful recommendations on what could be wrong). It now shows comparable with SURF localization accuracy, provided that the object scale in the scene is specified more or less correctly. Much better robustness can be achieved by using Lepetit’s YAPE (Yet Another Point Extractor) detector – see item 2 below. Perhaps, it makes sense to make more universal class than just C-descriptor. The class can easily combine randomized trees-based classifier (as in the original paper by Lepetit from 2006), Michael Calonder’s c-descriptor, ferns-based (a.k.a. semi-naive Bayesian) classifier and the corresponding descriptor. The techniques used are very similar, and in the papers ferns are claimed to provide better accuracy than randomized trees.
We did some experiments with open-source (GPL) bazAR tracking package from cvlab.epfl.ch site and the results are very promising – the keypoint detector constantly finds the same points on the quite different views of the same object, i.e. the stability is great! Unfortunately, we could not make the multi-scale variant of the detector work, but the idea is clear, and, I think, we can do independent implementation pretty quickly.
Maria added LBP features to the new “traincascade” application (the substitute for haartraining). It worked well and now we are adding the corresponding detection function to OpenCV (for now it will be sort of reference code w/o any serious optimization).
Generation of normally distributed random numbers is now ~3x faster with Ziggurat algorithm (it can be useful for faster keypoint detection and classifier training using Lepetit method, as they add uniform or gaussian noise to the generated patches (and seem to prefer the gaussian noise))
Several bugs have been fixed: 1) build issues on Linux when using autotools, 2) precompiled headers did not work well on Linux, 3) crash in calcHist (reported by Alex Teichman), 4) crash in mltest, found and fixed by Maria

Discussion Minutes

Stable keypoint detection. Lepetit’s group
- Run keyboard detector, used pyramid, maximal Laplacian detection with edge suppression
  - Repeat procedure 100-1000 times using affine distortion (know where the correspondences are)
  - Find most stable keypoints that are stable to affine distortion
  - Train classifiers using forest or ferns
    - Can be GPU’d (Anatoly knows for rendering) OpenCL Intel/NVidia, CPI, AMD
    - Suitable for SSE for image warping
  - One way descriptor uses multiple points in kd-tree, others use patches in random forests/ferns
  - In bazAR there are several demos, keypoint detection and training in randomized trees
- We have c-descriptor code but bazAR performs better, maybe it will work the same with the better key points above
- Will finish and commit to SVN this week
- C descriptors are in

FLANN
- Marius have commit privilege
- In 3rd party directory
  - Documentation to be melded in …

Alex Tiechman sent in feature pipeline for architectural feedback
- We may consider GPU of high payoff features found with this

HighGUI
- Probably use QT (LGPL) ~GTK
- Do it after release

Bundle Adjustment
- Sergey said that it will be ready this Friday. Debugging
- Initialization might not be included, just pure optimization
  - Will include unit tests
- To run, you need to provide initial estimates for camera intrinsic and code
  - Documentation will describe
- For complete 3D pipeline we are missing
  - Initialization
  - Bundle adjustment
  - Verification and
  - Mesh reconstruction
  - Probably only bundle adjustment will go into Summer release. Anatoly working on
    - Waits on the robust keypoints from LePetit’s group
    - LePetit’s group uses larger solution ImageModeler (Autodesk product)

HOG
- Put the training code for now to OpenCV extra
- Coefficients for human detection are shipped

Python
- If interface is C++
- We can adjust James scripts to handle new functions
  - New functions create output images will be created or resized if haven’t been created before

For BETA, we want
- C-descriptor
- LePetit’s affine robust YAPE keypoint detector
- improved training cascade classifier (training cascade supporting haar and other featuers
- Self-similarity feature
- Brief documentation for C++ interface
- ~ FLANN Integration
- Bundle adjustment

For Release
- Make Python interface for C++ functionality
- FLANN
- Finish documentation
- Write python tests
- Chamfer matching
- One way descriptor

Action Items

Gary

Ask Marius about chamfer matching with orientation
- Documentation ? LaTex into OpenCV
Look into purchasing ImageModeler
Talk with James about interface to C++ code

PREVIOUSLY

Gary

{X} Ask about Nistor Trees
:\ Have Marius contribute Chamfer+orientation
(./) Talk to Marius about contributing FLANN
{X} PUT IN CONTRIBUTORS DIRECTORY INTO WIKI
:\ Talk with Ian Goodfellow about bugs in HOG training code
{X} See if we can get Balan’s matlab code for symmetric object 3D capture
{X} Ask Scott about voting site for user contribution

Vadim

Find out about user contribution website similar to Matlab style

Not done:

3D features
Nistor Tree
Figure out code contribution process
Fast Chamfer with orientation

2009-07-06

Agenda

Static build for OpenCV or other minimal footprint methods?
Cascade generalization Haar | LBP or ~~Haar & LBP~~ ?
- New facial features working with new cascade?
Bundle adjustment status
User contribution / website with voting
Documentation read-over.
- Grammar being corrected now, almost done
Victor: Status on outlet detectors

For BETA, we want
- {OK} C-descriptor
- {OK} LePetit’s affine robust YAPE keypoint detector
  - They use 1 way descptor, C-descriptor
- {OK} improved training cascade classifier (training cascade supporting haar and other featuers
- Self-similarity feature
- Brief documentation for C++ interface
- (./) FLANN Integration
- Bundle adjustment

For Release
- Make Python interface for C++ functionality
- FLANN
- Finish documentation
- Write python tests
- Chamfer matching
- One way descriptor

Nistor trees (ask Patrick)
Python interface for C++ API — what needs to be done (Ask James)
Status of ML documentation (ask James)
Chamfer matching code (ask Marius)
FLANN (Marius)
- Does Marius have commit privileges yet?

OpenCV Summer Release

Feature freeze end of … sometime in July for test and doc.
Release in August, 2009

General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI (except for video I/O
- Better python interface
  - {OK} In progress thanks to James Bowman
  - (./) Use this to expand unit test functions for OpenCV

2D features
- {OK} HOG
  - (./) Chamfer matching with orientation In progress thanks to Marius Muja
- (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
  - (./) LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
  - {OK} C-descriptor … (Vadim)
  - (./) LePetit one way descriptor (Victor) (to Anatoly)
- {OK} MSER
- {OK} Star
- (./) Self-similar detector
{X} Alpha channel, alternatives to contours RLE
- {X} New alpha transparency (per pixel) added
- {X} Introduce functions with advance ways of blending
- {X} Mask channels

3D features
- (./) Bundle Adjustment (probable contribution by Phase Space)
- {X} Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
- {X} PFH, FPFH from Radu,
- {X} 3D Delaunay,
- {X} 3D convex hull,
- {X} mesh generation,
- (./) normals in grid (Gary)
- (./) normal based 3D patch (Gary)
- {OK} SPIN
- {X} 3D capture http://www.3dsom.com/index.html
- {X} VSLAM
- {X} Image stitching

Optical flow
- {OK} Dense (Farnerback)
- {X} Variational optical flow

MLL
- {OK} Extreme trees
- {OK} Reduce memory footprint
- {OK} More convenient training via a data class
- (./) Nister trees (Patrick)
- (./) FLANN or appropriate parts (Marius)
- {X} LePetit style random trees
- {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)

Misc:
- {X} Focus detector,
- {OK} adaptive skin tracking

Optimizations
- (./) Various functions, ongoing …

Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- (./) Proof read
- (./) Add content
- (./) Create C++ documentation
- (./) Create Python documenation

High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
- {X} move to using QT as the HighGUI interface look and feel

Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write

(./) Define user contribution process:

Minutes

Static builds of the library will be done past Beta release. Need to see if it is easy.
Cannot mix features in one cascade
New Cascade application still supports old formats
Bundle adjustment …
- Will talk to Sergey next week.
  - Debugging the code, no estimate yet on completion
Still don’t know about a voting website for user contributed code. Ask at Argus
- Matlab exchange as “prototype”
- Need to develop a simple template contribution function that goes all the way through
_’FLANN_’
- Has public interface for the functionality — wrap those in OpenCV structures
  - Example that Vadim proposed
  - Write regression tests for it
  - Marius needs write permission.
    - Needs some feedback
    - Marius to send in sourceforge log in
      - Start today — probably this week done
Chamfer matching
- Currently a library in ROS, depends on OpenCV only
- Interface needs cleaning
  - Chamfer match over entire image
  - Or use list of starting points
  - Has orientation but maybe need orientation flag to turn it on and off
    - Marius put in orientation on or off flag
    - When ready, send email to Vadim
Testing c-descriptor with YAPE detector
One way descriptor, couple of days, but delayed to next week

Outlet detection
- Fast version of one way descriptor is written
  - Changing the interface
  - Talked to author Stefan Hinterstoisser
    - Stephen might share the code
  - 2 days for restructuring the interface
- Implemented Generalized Hough Transform
  - 2 stages, 1st to filter out outliers, 2nd to refine detection
  - 1 week
- Have a lot of data from forearm
  - Could generalize the orange outlet code
  - End of the week decide
- Move to test on forearm with one way.

Reports:

Vadim

Lepetit’s keypoint detector is finished, now it’s being tested together with the c-descriptor.
Maria finished the detection function for LBP features. This week the new traincascade application will be integrated to OpenCV.
Several build issues have been resolved (reported by JD, Oliver Sidla an other guys). Also, it’s now possible to include top-level OpenCV CMakeLists.txt into a bigger project – which is very convenient.
Added haar cascades for facial feature detection (contributed by Modesto Castrillon Santana).

Action Items

Gary

Ask Marius about chamfer matching with orientation
Talk with James about python interface to C++ code
Ask about Nistor Trees
Chamfer code
PUT IN CONTRIBUTORS DIRECTORY INTO WIKI
Ask Scott about voting site for user contribution
- Think of some template application as an example
Documentation “gaps” read over

Vadim

Find out about user contribution website similar to Matlab style

PREVIOUSLY

Gary

(./) Look into purchasing ImageModeler — trying to get download for 3dsom instead
{./} See if we can get Balan’s matlab code for symmetric object 3D capture

Vadim

:\ Find out about user contribution website similar to Matlab style

2009-07-14

Agenda

Release progress review
Contribution site … blah
Outlet summary
One way descriptor news?
- Stefan Hinterstoisser release code?
Bundle adjustment status
YAPE
Static builds?
3D feature status
- I have Matlab code for symmetric object model creation.
C-descriptor in?
FLANN and 2D Chamfer
’Post release’
- Flesh detection techniques

OpenCV Summer Release

Feature freeze end of … sometime in July for test and doc.
Release in August, 2009
General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI (except for video I/O
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - (./) Use this to expand unit test functions for OpenCV
- 2D features
  - {OK} HOG
    - (./) Chamfer matching with orientation In progress thanks to Marius Muja
  - (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
    - (./) LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
    - {OK} C-descriptor … (Vadim)
    - (./) LePetit one way descriptor (Victor) (to Anatoly)
  - {OK} MSER
  - {OK} Star
  - (./) Self-similar detector
- {X} Alpha channel, alternatives to contours RLE
  - {X} New alpha transparency (per pixel) added
  - {X} Introduce functions with advance ways of blending
  - {X} Mask channels
- 3D features
  - (./) Bundle Adjustment (probable contribution by Phase Space)
  - {X} Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
  - {X} PFH, FPFH from Radu,
  - {X} 3D Delaunay,
  - {X} 3D convex hull,
  - {X} mesh generation,
  - (./) normals in grid (Gary)
  - (./) normal based 3D patch (Gary)
  - {OK} SPIN
  - {X} 3D capture http://www.3dsom.com/index.html
  - {X} VSLAM
  - {X} Image stitching
- Optical flow
  - {OK} Dense (Farnerback)
  - {X} Variational optical flow
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - (./) Nister trees (Patrick)
  - (./) FLANN or appropriate parts (Marius)
  - {X} LePetit style random trees
  - {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
Misc:
- {X} Focus detector,
- {OK} adaptive skin tracking
Optimizations
- (./) Various functions, ongoing …
Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- (./) Proof read
- (./) Add content
- (./) Create C++ documentation
- (./) Create Python documenation
High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
- {X} move to using QT as the HighGUI interface look and feel
Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
(./) Define user contribution process:

Minutes

Planar detections

Use ferns.
- Entry is computed in large table. Set of responses of every binary weak classifiers give indices in a table
- Learns that patch and tracks it
- Can add c-descriptor functionality where we get descriptor key
YAPE keypoint is in separate class

Object detection

Cascasde needed a stand alone function for LBP
Adding new features — need to make it easier to add.
- Add data structure
- Load classifier from xml to memory
- Load features
- Call back that computes features for training and recognition
  - feature I/O
Maria working on avoiding code duplication as much as possible
Todo

self similar features from Rainer
Cascade from Maria
Bundle adjustment integration
Will work on this week
Look at static builds

Bundle adjustment

Sergey was supposed to finish this week

3D

Almost done with my “surface plugs”
Do as the first contribution example
Spin images (Anatoly can concentrate on this)
- Generation is done from point cloud to spin image
  - Parameter issues are fixed
- Can’t use very well since you can only do pairwise matching
- Need to filter geometrically
Anatoly tested 3dsom
- Need good lighting
- Good green screen
- Don’t have sophisticated background segmentation
- They use special pattern for camera pose ID
- Capture object from various views
- Pass view to objects
- tried on 2 or 3 objects and results were good with good lighting
- Practiced capture takes less than half an hour per model. Maybe can be done in 10 minutes.

Documentation

Python (James)
C++ interface
Collect ctypes in one place, hyper link to it

One way descriptor

Later this week
No promise about sharing code

Outlets

Tomorrow will turn over and focus on one way
Orange outlets are still being tested
Need to do for robot
- Now getting high percentage on some outlets >95%, low on others <95%
- Much larger scale change in forearm, influences one way descriptor matching
- Camera position estimation needs to be redone for lots of different outlets
  - 2×1 with same geometry
  - 2×2 with same geometry
  - Several views of outlets, assuming distance of holes are same, can use to refine scale/structure from motion
- With orange outlets we had only one
  - The robot should be able to launch specific training (single frontal view with holes marked)
    - Mech turk holes
- Hope by end of the week to have start of end to end

Toolkits

3D model/mesh capture
- symmetric objects
Feature framework Tiechman working on
Color space handling/invarient color
- Flesh detection
  - Histogram
  - Adaptive skin ?

Reports:

Victor

Outlet detection:

One way descriptor code has been revised. CvOneWayDescriptor code has been reviewed and is ready for submission to OpenCV. The helper class CvOneWayDescriptorBase that does actual work on comparing and classifying the descriptors is under revision now. The fast version of the feature generation has been implemented based on precomputed descriptors of pca components of features. The new code is faster than the previous one at least one order of magnitude (generating descriptors for 100 features takes several seconds instead of several minutes now).

The generalized hough transform for detecting outlets has been implemented and tested on the forearm data. Its integration into the outlet detection ROS package is on the way. The summary statistics for forearm data is given below.

Outlet Detection Forearm Camera
|Outlet_|Num Images_|Num Detected_|
|cracked |100 |79 |
|fake_wall_white |113 |111 |
|fake_wall_orange_2x2 |35 |35 |
|orange_2x1 |103 |88 |
|conference_white_2x2_extreme_lighting |67 |20 |
|hallwhite2×1 |103 |100 |
|white2×2 |101 |70 |

Spin images:

Anatoly has started improving spin images code. Random sampling of spin images, automatic generation of spin image parameters have been implemented. Spin image correspondence filtering based on geometric consistency is in progress.

Vadim

Finally, planar object detection is working! It involves Lepetit’s keypoint detector, fern-based point classifier (random trees can easily be added too), and RANSAC-based homography estimation. The results on the standard OpenCV sample is attached.
Maria has implemented a draft version of standalone OpenCV object detection function based on LBP features. Some refactoring of the training application was needed to extract the code for reading classifier and running it, so that OpenCV libs do not depend on the whole source of “traincascade”.
8 bugs have been fixed, including 5 reported via SF bug tracker.

Action Items

Gary

Foward mesh improvement paper

Vadim

Mail Michael how to turn Fern into descriptors (log vs regular output).

PREVIOUSLY

Gary

(./) Ask Marius about chamfer matching with orientation
(./) Talk with James about python interface to C++ code
(./) Ask about Nistor Trees
(./) Chamfer code
{X} PUT IN CONTRIBUTORS DIRECTORY INTO WIKI
{X} Ask Scott about voting site for user contribution
- Think of some template application as an example
{X} Documentation “gaps” read over
{./} See if we can get Balan’s matlab code for symmetric object 3D capture

Vadim

Find out about user contribution website similar to Matlab style

2009-07-21

Agenda

Release review
- Documentation lag, though Machine learning docs now up.
Vincent LePetit is working for 2 weeks at Willow. Questions for him on YAPE or other descriptors?
- Status of YAPE,
- {OK} c-descriptor now in
Self similar feature?
Bundle adjustment status?
Static builds
3D surface plugs by Thrusday.
- Spin image status?
3D capture status?
Alex Teichman’s “feature engine”?
Outlet detector to forearm status

. OpenCV Summer Release

Feature freeze end of … sometime in July for test and doc.
Release in August, 2009
General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI (except for video I/O
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - (./) Move into OpenCV and deprecate SWIG only version
  - (./) Use this to expand unit test functions for OpenCV
- 2D features
  - {OK} HOG
    - (./) Chamfer matching with orientation In progress thanks to Marius Muja
  - (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
    - (./) LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
    - {OK} C-descriptor … (Vadim)
    - (./) LePetit one way descriptor (Victor) (to Anatoly)
  - {OK} MSER
  - {OK} Star
  - {OK} Self-similar detector
- {X} Alpha channel, alternatives to contours RLE
  - {X} New alpha transparency (per pixel) added
  - {X} Introduce functions with advance ways of blending
  - {X} Mask channels
- 3D features
  - (./) Bundle Adjustment (probable contribution by Phase Space)
  - {X} Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
  - {X} PFH, FPFH from Radu,
  - {X} 3D Delaunay,
  - {X} 3D convex hull,
  - {X} mesh generation,
  - (./) normals in grid (Gary)
  - (./) normal based 3D patch (Gary)
  - {OK} SPIN
  - {X} 3D capture http://www.3dsom.com/index.html
  - {X} VSLAM
  - {X} Image stitching
- Optical flow
  - {OK} Dense (Farnerback)
  - {X} Variational optical flow
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - (./) Nister trees (Patrick)
  - (./) FLANN or appropriate parts (Marius)
  - {X} LePetit style random trees
  - {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
Misc:
- {X} Focus detector,
- {OK} adaptive skin tracking
Optimizations
- (./) Various functions, ongoing …
Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- {OK} Proof read
- {OK} Put up
  - {OK} cxcore
  - {OK} cv
  - {OK} MLL
  - {OK} Aux
  - {OK} HighGUI
- (./) Add content
- (./) Create C++ documentation
- {OK} Create Python documenation
High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
- {X} move to using QT as the HighGUI interface look and feel
Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
(./) Define user contribution process:

Minutes

Updating latest tested snapshot for ROS (cmake 2.4.6 or .7)

Tomorrow (Wed or Thrus)
Some versions of cmake don’t handle everything we need. Between 2.4-2.6 some version doesn’t make 2.6.0 — need to check in Ubuntu 8.10

3D capture

Just looked at the paper
But Anatoly busy with spin image

Spin image

Changing the inteface
Adding more features
Adding correspondence parts, filtering out wrong correspondences based on consistency
- Matching spin images 1 by 1 is too ambiguous
3D recognition project
Couple more weeks

User contribution

Asked local designer
- He’s asking around
- Waiting

Static builds

Not sure if before beta or after
- Needs testing
- cmake makes it easy, just put “static” instead of “shared”
  - But question of how to use them properly
  - Might build cmake option to build tests section statically
- After Beta

Signature based on Ferns questions

multiplying signatures (or summing their logarithms) tends to be better
- For ferns
For the signatures, better to do sums
- Michael Calonder used random trees
  - Computed PCA compressed probability signatures
  - Detection stage, these were summed
  - Recent paper showed multiplied these, PAMI 2009
- For signatures, use the sum of the probabilities
  - To compare, used simple NN search. K-d trees, Vocab trees
Point detector YAPE
- Vadim tried both versions
  - Ferns version is latest
  - Previous found more robust to affine
- compute LoG but diagonal edges can cause false features due to aliasing
- Tried FAST detector — was slower than expected
  - Compared Good features to track, SURF, found first version of YAPE
- Written from scratch version of older YAPE is in OpenCV
- Speed of YAPE vs Fast — haven’t measured yet, but think they are comparable, maybe YAPE is faster.

Vadim is done with his code, waiting for other adjustments

Waiting for Cascade
Waiting for Bundle adjustment

Bundle adjustment

Will talk to Sergey tomorrow with separate mail about it

One way descriptor

Basically runs
But need a more general class, right now its tied to outlets
By Friday

Cascade

Detection
- You write a small class that reads the xml feature during load time
Training
- Need to write xml features to file and
- enumerate all features
- After complete, the whole cascade is written to an xml file.
  - First part is indepenedent of features — the cascade and params
  - Then actual features
- For haar, 2 or 3 rectangles

Reports:

Marius has been working on

FLANN and
Chamfer
Neighther commited now

Vadim

The following has been done last week:

Self-similarity descriptor has been integrated to OpenCV (it’s in SVN trunk now)
More than 20 bug tracker records have been closed.
While not in the tracker, lot’s of fixes have been made in C++ sparse matrices (as Victor and Alexey are looking at making use of it).
The latest videoInput & ffmpeg 0.5.0 have been added to the repository; they are by highgui on Windows.
Maria finished implementation of standalone OpenCV C++ class for Haar features and started implementation of similar class for LBP features.
Anatoly continues work on spin image improvement. A whole big C++ class for a set of spin images representing a 3D model has been designed and implemented.

There is also helper cv::Mesh3D class that can only store points, octtree and normals, but could contain some triangulation information in the future.

Victor

The code for working with new outlet detection is in repository. In order to use it you have to define macro GHT and recompile outlet.dylib. The interface is the same, however you have to explicitly load a template for each outlet you are looking for. An example is in sample.cpp, line 75. You can run the code on forearm data by starting outletsample binary and passing 5 parameters to it (second to last being the path to the outlet template).
- Creating a template is getting a single image and then editing a YML file.
  - No instructions so far
  - Make and image
  - Crop it to only outlet
  - Label the holes
  - Copy the rest from existing templates
  - One way descriptor is many images affine transformed
    - 20 PCA components
    - Create affine versions
    - Recombined — 100M for this application (once and for all for any US type outlets)
      - User just has to create a template for each outlet
- Will make a simplified tool
- Add to Cmake.lists need to add _GHT there

I will send you now edited forearm images (collected by Patrick and myself and rearranged in convenient form).

I will also send you the outlet templates. This will be a zip file containing two folders — configs (containing the templates themselves) and descriptors (containing precomputed highres and lowres pca components as well as the precomputed one way descriptors of highres pca components). The templates are in configs folder, for instance, in order to detect fake wall orange 2×2 outlet, use configs/forearm/fake_wall_2x2orange. Templates reference descriptors with relative path so configs and descriptors folders should always be in the same folder.

The code works now 0.5-1.5sec (depending on an outlet) on my laptop. We are working to make it work faster and more accurately. We didn’t investigate what would happen if a template for one outlet is used for detecting another, but on earlier stage of this project I was experimenting with two different white 2×1 outlets and the matching results were bad so I think for now a specific template should be used for each outlet.

Outlet detection: the GHT code has been speeded up a factor of 6 by more efficient memory managing and replacing high-resolution hough histogram with a low-resolution followed by post-processing. The accuracy remains roughly the same, the processing time is 0.5-1.5s per frame. More speedups are on the way.

The code has been restructured to allow simpler use. The format of templates has been simplified.

Camera pose estimation for arbitrary outlets has been implemented.

Accuracy is >90%. Extreme lighting has 30% error, but in this set, one out of 4 outlets is very saturated, so perfect results are not possible.

Action Items

Gary

Ask Brian about storing large data (outlet specific data — image plus hole data and then PCA components (small) and descriptors (large 100M uncompressed)

Vadim

Send reminder email to web designer about user contribution site

Victor

Add one way descriptor.

PREVIOUSLY

Gary

(./) Foward mesh improvement paper
{X} PUT IN CONTRIBUTORS DIRECTORY INTO WIKI
{X} Ask Scott about voting site for user contribution
{X} Documentation “gaps” read over
Ask James about python documentation

Vadim

(./) Mail Michael how to turn Fern into descriptors (log vs regular output).
(./) Find out about user contribution website similar to Matlab style

2009-07-29

Agenda

Outlet
- data, where to store
- Plans — 8mm lens
- Arms
One way descriptor
User contribution
Daisy code is available
{OK} Self similar feature
Bundle adjustment
Spin images
Documentation
Merging python in
FLANN, Chamfer, Nister

Post release
- Static builds
- 3D Capture
- Alex’s feature engine

. OpenCV Summer Release

Feature freeze end of … sometime in July for test and doc.
Release in August, 2009
General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI (except for video I/O
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - (./) Move into OpenCV and deprecate SWIG only version
  - (./) Use this to expand unit test functions for OpenCV
- 2D features
  - {OK} HOG
    - (./) Chamfer matching with orientation In progress thanks to Marius Muja
  - (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
    - (./) LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
    - {OK} C-descriptor … (Vadim)
    - (./) LePetit one way descriptor (Victor) (to Anatoly)
  - {OK} MSER
  - {OK} Star
  - {OK} Self-similar detector
- {X} Alpha channel, alternatives to contours RLE
  - {X} New alpha transparency (per pixel) added
  - {X} Introduce functions with advance ways of blending
  - {X} Mask channels
- 3D features
  - (./) Bundle Adjustment (probable contribution by Phase Space)
  - {X} Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
  - {X} PFH, FPFH from Radu,
  - {X} 3D Delaunay,
  - {X} 3D convex hull,
  - {X} mesh generation,
  - (./) normals in grid (Gary)
  - (./) normal based 3D patch (Gary)
  - {OK} SPIN
  - {X} 3D capture http://www.3dsom.com/index.html
  - {X} VSLAM
  - {X} Image stitching
- Optical flow
  - {OK} Dense (Farnerback)
  - {X} Variational optical flow
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - (./) Nister trees (Patrick)
  - (./) FLANN or appropriate parts (Marius)
  - {X} LePetit style random trees
  - {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
Misc:
- {X} Focus detector,
- {OK} adaptive skin tracking
Optimizations
- (./) Various functions, ongoing …
Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- {OK} Proof read
- {OK} Put up
  - {OK} cxcore
  - {OK} cv
  - {OK} MLL
  - {OK} Aux
  - {OK} HighGUI
- (./) Add content
- (./) Create C++ documentation
- {OK} Create Python documenation
High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
- {X} move to using QT as the HighGUI interface look and feel
Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
(./) Define user contribution process:

Minutes

Outlet detectors
- Multi-scale outlet detector is getting ~100% for high scale variation dataset
- Specular dataset is only one that has lots of failures
- Wrote outlet data labeler
- Made template creation application
  - template creation
  - data set labeling
  - tester
- One way descriptor isn’t good for general outlet detection
  - Will test the planar detector
Spin images
- Generated a scene from a model and added noise
  - Tested spin image on this for geometric consistency and grouping of correspondences make sense
    - Artificial test worked fine when testing reprojection
    - But when used on real Ikea data (4 objects), but didn’t work as well
      - If radius is too large, clutter hurts, if too small, then not discriminative
      - But theoretically, occlusion shouldn’t be such a problem
      - Anatoly is working on 2 different paths: Why occlusion is hurting, and problem of small support.
- Have done most of Johnson’s pipeline except the mesh verification
Planar detector
- LePetit’s fern point classifier is now in
  - Multiplying probabilities give 90% accuracy, averaging gives much less 60%
- C-descriptor is based on this, but not implemented yet
- One way descriptor had poor performance
- Victor will test on latest data
- Will add c-descriptor
User contribution … designer is on vacation

Beta Release
- Train cascade on more general object detection
- Bundle adjustment, gave seminar today.
- Documentation, started
- Friday next week
Release
- No major functionality to add
- Python interface integrated
- Add more documentation
- Add more tests
- Additional 3 weeks

Reports:

Vadim

latest_tested_snapshot has been finally updated (for that OpenCV has been checked on 32-bit Vista (MSVC + GCC 4.3), 64-bit Ubuntu 9.04, 32-bit Ubuntu 8.04 and MacOSX 10.5. The snapshot includes working planar object detector based on the fern point classifier (it’s not the original c-descriptor, it will be added after beta), fixed HOG descriptor & detector, unified keypoint detectors (they all now output a vector of KeyPoint structure instances) and many bug fixes, new ones (more accurate cvLog(), corrected cvDrawContours etc.) and fixed earlier in trunk.
added gzip support in OpenCV XML-YAML I/O functions. Now you can simply do:

cvSave(“bigmatrix.xml.gz”, mybigmatrix); and similarly mybigmatrix = (CvMat*)cvLoad(“bigmatrix.xml.gz”);

cvOpenFileStorage() and all other I/O functions calling it (including save() & load() methods in MLL classes) support this new feature. This was primarily done to reduce disk footprint for the new fern-based planar object detector, since it can easily take >200MB of disk space. When compressed, such file takes just ~20-30MB. Further reduction of memory/disk footprint by factor of ~4 is possible and will be done after beta release.

fixed 3 bugs in highgui on Mac; now all the samples run fine.
started updating the documentation with the new material on C++ interface; integrated the docs build into OpenCV CMake build system.
received the final version of traincascade application + the detection code supporting LBP and Haar features. Integrating it now.

Victor

Outlet detection: a multiscale version of outlet detection has been implemented. Multiscale support has been added to one way descriptor — features are extracted now from several levels of image pyramid and descriptors are built for all of the features. The best match from any level determines a feature class (power hole/ground hole/background). Hough transform also operates on several levels of pyramid, scaling the template and matching it to each of pyramid levels, selecting the best solution by a score that depends on the number of features matched and reprojection error. As a result instead of 17 misdetections on white2x2 dataset (where scale variation is high — more than 2x) we get only 3 misdetections out of 105 images by using just 2 levels of a pyramid.
The application for automatic generation of outlet template has been created. It requires user to manually label the holes using a simple GUI. The same app has the functionality of automatically testing the performance of an outlet detector by comparing the results with the labels. All forearm data has been labeled and starting tomorrow we will use this app for automatically generating test statistics.
Spin images: geometric consistency filtering and grouping functionality has been tested. Affine transformation from scene to model using RANSAC has been implemented. It selects one group of correspondences by using a score that depends on the number of correspondences and reprojection error. Grouping is proved to give reasonable results for a scene that is obtained from a model by adding random noise to each 3D point. However we failed so far to build reasonable group of correspondences for Ikea object. If spin image support is small, we are getting too many correspondences because of symmetry and ambiguity of a local shape descriptor. If we increase spin image support, scene and model spin images differ a lot because part of the scene is occluded. We will investigate the small spin image support case and try to obtain a consistent group of correspondences.

Action Items

Gary

Send Victor wiki page documentation examples
Talk to Patrick, Marius about Nister trees
Talk to Marius about FLANN, Chamfer

Vadim

Send note to Marius about FLANN

PREVIOUSLY

Gary

Ask Brian about storing large data (outlet specific data — image plus hole data and then PCA components (small) and descriptors (large 100M uncompressed)

Vadim

Send reminder email to web designer about user contribution site

Victor

Add one way descriptor.

2009-08-04

Agenda

Outlet report
One way descriptor
User contribution
Daisy code is available, can try it
Bundle adjustment
Spin images
Documentation
Merging python in
{OK} FLANN, (./) Chamfer, ? Nister
3D feature report

Post release
- Static builds
- 3D Capture
- Alex’s feature engine

OpenCV Summer Release:

Dates:
- Feature freeze July 31
- Beta Release Aug 10.
- Release August 31, 2009
  *
- {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI
  - (./) I/O for sructures, histograms
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - (./) Move into OpenCV and deprecate SWIG only version Post Beta
  - (./) Use this to expand unit test functions for OpenCV Post Beta
- 2D features
  - {OK} HOG
    - (./) Chamfer matching with orientation In progress thanks to Marius Muja
  - (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
    - {OK} LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
    - (./) C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
    - {OK} Fern based planar patch detector from LePetit’s group
    - (./) LePetit one way descriptor (Victor) (by Aug 7)
  - {OK} MSER
  - {OK} Star
  - {OK} Self-similar detector
- 3D features
  - (./) Bundle Adjustment (probable contribution by Phase Space … Talk to Sergey)
  - (./) normals in grid (Gary)
  - (./) normal based 3D patch (Gary)
  - {OK} SPIN
- Optical flow
  - {OK} Dense (Farnerback)
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - (./) Nister trees (Patrick) Post Beta
  - (./) FLANN or appropriate parts (Marius) Post Beta
  - {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
Misc:
- {OK} adaptive skin tracking
Optimizations
- (./) Various functions, ongoing …
Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- {OK} Proof read
- {OK} Put up
  - {OK} cxcore
  - {OK} cv
  - {OK} MLL
  - {OK} Aux
  - {OK} HighGUI
- (./) Add content
- (./) Create C++ documentation
- {OK} Create Python documenation
High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
(./) Define user contribution process:

Put off to Future

2D features
- Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
  - Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
3D features
- Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
- PFH, FPFH from Radu,
- 3D Delaunay,
- 3D convex hull,
- mesh generation,
- mesh simplification/subsampling
- Extend to using 2D XYZA grid as well as oct/kd trees
3D capture
- Probably by using silhouettes and then texture for cavities (?)
  - Possibly with textures or phased light
- VSLAM
- Image stitching
Optical flow
- Variational optical flow
MLL
- Tiechman’s
  - Kernel based gentle boost
  - Feature selection engine
- Automatic parameter tuning
  - Cross-validation, ROC curve generator
High GUI
- move to using QT as the HighGUI interface look and feel
Misc:
- Focus detector,
Lighting/Specularity
- Look into:
  - Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
  - Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
  - 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
  - Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
Histograms
- Insert value function with interpolation
- Automatic bin size adjustment for recognition performance using a test set
OpenCV Ideas 2009

Minutes

One Way Descriptor

Extended to use keypoint class
- Had to ripple through this change
3 classes — writing example

Documentation

Need list of attributes for each function
Can this be done with semi-automatic script such as James

User contribution

Nothing yet on website

Daisy

Permission for BSD license

Bundle adjustment

Haven’t talked to Sergey
Supposedly finished
- He will be online from tomorrow
- False correspondences may or may not be handled
Use for STF on outlets from a robot

Chamfer, Nistor

Postpone until after Beta

Beta Release

Still need to integrate cascade training (today)
One way descriptor
End of this week

Final Release

Documentation
Python integration
Test functions
Bundle adjustment
+ 3 weeks

Histograms

Read Write
- C structures CvSave CvLoad
- New data structures, no I/O yet, after Beta
Input data
Better performance by interpolation between 2 neighbor bins linear interpolation

Reports:

Vadim

added C++ video capturing interface to highgui. Now the whole highgui is covered by the C++ API.
Maria and I integrated the latest object detection code to OpenCV, it is in SVN together with sample face detection cascade built on LBP features. During integration the interface has been simplified a lot. The training application will be committed today. The new code performance is on par with the prev. C implementation, but OpenMP threading has not been added yet.
wrote 8-page introduction section for the new chapter “C++ Interface” of opencv.pdf. Added description of Mat, Mat_, Rect, Point, Vector, Vec, MatND and SparseMat classes. The rest of cxcore reference is in progress.
made the newly added flann bindings by Marius compile on Windows and Mac.

Gary

3D surface plug is finished was not robust to stereo noise for similar shaped objects.
- As bag of features it gets 87% on 3 confusable objects
I am 70% done trying instead to use extend PFH to whole objects

Victor

Outlet detection: the most of the week has been spent to restructure the one way descriptor code in order to submit it to opencv. It took much more time than expected (4 days opposed to one). As a result we have three objects. CvOneWayDescriptor encapsulates functionality for individual descriptor. CvOneWayDescriptorBase is intended for storing a collection of descriptors, efficiently building them from precomputed PCA descriptors and finding the closest one. CvOneWayDescriptorObject encapsulates a collection of descriptors that have support for visual words. Multiscale support has been reimplemented for more elegant code. Now the code is ready for submission, every function is documented. The sample of usage is on the way.
3D descriptors: the dependence of the transformation from model to scene on the group of spin image correspondences has been implemented. It was found that the quality of the transformation does not correlate with the group size. The transformation that has lowest reprojection error is calculated on a group with 5 correspondences although there are groups with >100 correspondences. We came to the conclusion that we need interactive visualization in order to understand the matching process on Ikea dataset. Anatoly has now compiled ROS and rviz and is working to get spin image matching visualization there.

Other: Alexey is on vacations June 31-August 3rd.

Action Items

Gary

Talk to James about wiki additions to the manual.
Ask James about automating list of parameters
Ask Kurt about status of bundle adjustment
Talk to about C-descriptor

Vadim

Write Sergey a message about Bundle
- Uses generated datasets
- Ask about handling false correspondences
Update opencv monthly http://opencv.willowgarage.com/wiki/OpenCV%20Monthly

Victor

PREVIOUSLY

Gary

? Send Victor wiki page documentation examples
Talk to Patrick, Marius about Nister trees
{OK} Talk to Marius about FLANN, Chamfer
Provide histogram specs

Vadim

{OK} Send note to Marius about FLANN

2009-08-25

Agenda

Floating point Floodfill (already in)
Progress reports
Python
Outlet

OpenCV Summer Release:

Dates:
- Feature freeze July 31
- Beta Release Aug 10.
- Release August 31, 2009
Keys:
- {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI
  - (./) I/O for sructures, histograms
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - (./) Move into OpenCV and deprecate SWIG only version Post Beta
  - (./) Use this to expand unit test functions for OpenCV Post Beta
- 2D features
  - {OK} HOG
    - (./) Chamfer matching with orientation In progress thanks to Marius Muja
  - (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
    - {OK} LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
    - (./) C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
    - {OK} Fern based planar patch detector from LePetit’s group
    - (./) LePetit one way descriptor (Victor) (by Aug 7)
  - {OK} MSER
  - {OK} Star
  - {OK} Self-similar detector
- 3D features
  - (./) Bundle Adjustment (probable contribution by Phase Space … Talk to Sergey)
  - (./) normals in grid (Gary)
  - (./) normal based 3D patch (Gary)
  - {OK} SPIN
- Optical flow
  - {OK} Dense (Farnerback)
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - (./) Nister trees (Patrick) Post Beta
  - (./) FLANN or appropriate parts (Marius) Post Beta
  - {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
Misc:
- {OK} adaptive skin tracking
Optimizations
- (./) Various functions, ongoing …
Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- {OK} Proof read
- {OK} Put up
  - {OK} cxcore
  - {OK} cv
  - {OK} MLL
  - {OK} Aux
  - {OK} HighGUI
- (./) Add content
- (./) Create C++ documentation
- {OK} Create Python documenation
High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
(./) Define user contribution process:

Put off to Future

2D features
- Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
  - Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
3D features
- Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
- PFH, FPFH from Radu,
- 3D Delaunay,
- 3D convex hull,
- mesh generation,
- mesh simplification/subsampling
- Extend to using 2D XYZA grid as well as oct/kd trees
3D capture
- Probably by using silhouettes and then texture for cavities (?)
  - Possibly with textures or phased light
- VSLAM
- Image stitching
Optical flow
- Variational optical flow
MLL
- Tiechman’s
  - Kernel based gentle boost
  - Feature selection engine
- Automatic parameter tuning
  - Cross-validation, ROC curve generator
High GUI
- move to using QT as the HighGUI interface look and feel
Misc:
- Focus detector,
Lighting/Specularity
- Look into:
  - Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
  - Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
  - 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
  - Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
Histograms
- Insert value function with interpolation
- Automatic bin size adjustment for recognition performance using a test set
OpenCV Ideas 2009

Minutes

Schedule:

Beta release on August 31st
Gold release on September 21st

For Beta

Integrate bundle adjustment
Finish documentation C++
Document new CvAux functions

For Gold

Improve documentation
Integrate new Python interface
- James must look at the new C++ and how hard wrapppers would be for that
  - Can he redo what he did for the new interface
- Add machine learning library
Expand test code

Need Dedicated server

Quad core server
Allocate Anatoly
- for different platform
- for ROS

Spin Images

Work well on synthetic data
they are parameter sensitive for noisy data
WPFH may work better (Gary working on)
- Focus on

Vadim

Most time spend on C++ documentation
- cxcore done
- New cv module in process
Fixed some bugs
- SVM old bug fixed
- EM bug fixed Ira Shevtsova
- Anatoly reported reference cout bug … will be fixed involves thread safety for reference count
The threaded version of the new object detection methods was improved and committed to SVN (Maria Dimashova).
- We also developed the new internal interfaces for threading the decision tree training algorithm (optimal split selection), which is also the core of boosting and random trees training. Maria is now working on implementing it.
Bundle adjustment code is in from Sergey
- Sergey implemented test code for opencv test system

Victor:

Outlet detection: several problems with the one way descriptor matching have been corrected. A strong shadow cast on an outlet caused incorrect classification of outlet holes (ground/power mismatch). This was fixed by adding a variation in scale for matching. Also, patch resolution has been increased 2 times, from 24×24 up to 48×48. The resulting method works significantly better on orange2x1, solving the most of the shadow problems. The testing framework has been fully automated allowing to achieve statistics for all of the datasets. Below are the results for the older and newer methods.

Improved One-way Descriptor

fake_wall_2x1white	97/114
fake_wall_2x2orange	33/35
orange2x1	97/103
specular	25/72
white2x1hall	94/103
white2x2	100/107

Old One-way Descriptor

fake_wall_2x1white	96/114
fake_wall_2x2orange	34/35
orange2x1	89/103
specular	26/72
white2x1hall	97/103
white2x2	97/107

3D feature matching: VRML rendering for debugging matching results has been implemented. Several methods for limiting the class of the resulting affine transformations (model → scene) have been implemented. We have failed to get Johnson algorithm for spin image matching to work in generic setting. The main problems are shape ambiguity (model objects have axial symmetry) and occlusion (shape on the scene is not full, absence of the points influences spin images). Probably the usage of prior knowledge (planar geometry from the table, axial symmetry) as well as image feature matching will help to solve the practical problem.

= To Do =

Victor

Send in flow chart of outlet matching
- As done on milestone
- As done now

Gary

Ask about outlet plugging in plans
Ask clean workstation ROS
- Jaunty

2009-09-01

Agenda

Schedule
Test system
3D capture
Outlets

Schedule:

Beta release on ~~August 31st~~ Sept 4th due to Bundle and C++
Gold release on September 21st

For Beta

Bundle adjustment (failed)
Finish documentation C++
Failure on mingw
{OK} Simplify the OpenCV wiki homepage.

For Gold

Improve documentation
Integrate new Python interface
- James must look at the new C++ and how hard wrapppers would be for that
  - Can he redo what he did for the new interface
- Add machine learning library
Expand test code

OpenCV Summer Release:

Dates:
- Feature freeze July 31
- Beta Release Aug 10.
- Release August 31, 2009
Keys:
- {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI
  - (./) I/O for sructures, histograms
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - (./) Move into OpenCV and deprecate SWIG only version Post Beta
  - (./) Use this to expand unit test functions for OpenCV Post Beta
- 2D features
  - {OK} HOG
    - (./) Chamfer matching with orientation In progress thanks to Marius Muja
  - (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
    - {OK} LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
    - (./) C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
    - {OK} Fern based planar patch detector from LePetit’s group
    - (./) LePetit one way descriptor (Victor) (by Aug 7)
  - {OK} MSER
  - {OK} Star
  - {OK} Self-similar detector
- 3D features
  - (./) Bundle Adjustment (probable contribution by Phase Space … Talk to Sergey)
  - (./) normals in grid (Gary)
  - (./) normal based 3D patch (Gary)
  - {OK} SPIN
- Optical flow
  - {OK} Dense (Farnerback)
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - (./) Nister trees (Patrick) Post Beta
  - (./) FLANN or appropriate parts (Marius) Post Beta
  - {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
Misc:
- {OK} adaptive skin tracking
Optimizations
- (./) Various functions, ongoing …
Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- (./) Proof read — Needs complet re-do
- {OK} Put up
  - {OK} cxcore
  - {OK} cv
  - {OK} MLL
  - {OK} Aux
  - {OK} HighGUI
- (./) ’C++ Interface’
  - {OK} C++ cxcore
  - (./) cv (90%)
  - (./) Aux
  - (./) HighGUI
- (./) Add content
- (./) Create C++ documentation
- {OK} Create Python documenation
High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
(./) Define user contribution process:

Put off to Future

2D features
- Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
  - Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
3D features
- Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
- PFH, FPFH from Radu,
- 3D Delaunay,
- 3D convex hull,
- mesh generation,
- mesh simplification/subsampling
- Extend to using 2D XYZA grid as well as oct/kd trees
3D capture
- Probably by using silhouettes and then texture for cavities (?)
  - Possibly with textures or phased light
- VSLAM
- Image stitching
Optical flow
- Variational optical flow
MLL
- Tiechman’s
  - Kernel based gentle boost
  - Feature selection engine
- Automatic parameter tuning
  - Cross-validation, ROC curve generator
High GUI
- move to using QT as the HighGUI interface look and feel
Misc:
- Focus detector,
Lighting/Specularity
- Look into:
  - Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
  - Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
  - 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
  - Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
Histograms
- Insert value function with interpolation
- Automatic bin size adjustment for recognition performance using a test set
OpenCV Ideas 2009

Minutes

Reference counting and Python

Does it conflict?
Use numerical numpy array before calling OpenCV, OpenCV can point a header at it. This shuts off OpenCV reference counting.

Talked with NVidia about OpenCV support

He will discuss with NVidia management
There is also OpenCL Which abstracts “CUDA” code

New workstation

MacPro with 8 core, each at 2.8GHz, with lots of cache and memory
Can test multi-threaded scaling behaviour beyond dual core
Next year, Larrabee will be out (at least 16 cores, maybe 32 core version)

Currently, team members are on:

Vadim
- documentation et al
Victor
- outlet
Alexy
- outlet also, pose estimation, generalized Hough
Maria
- parallel MLL
Anatoly
- Spin images might switch to developing test system (unless we hire Andrew for 2 months)

Test system with ROS

2-4 weeks
- Test with ROS
- Making automatic tests
- Documentation builds
- Automate trunk→latest tested snapshot promotion

Changes in user group and main page

We (James, Gary) simplified the main wiki page to make the main things (docs, download and user group) easier to find
I made the messages on the Yahoo groups readable by anyone so that Google, Bing can index them.

One way descriptor

Does affine distortion and can use the match to estimate pose

Spin images aren’t very discriminative on stereo scans of glasses since the legs don’t show up

3Dsom

Vadim

The following has been done last week:

The operations on Mat::refcount made thread-safe; now they use atomic increment & decrement operations. That eliminated some crashes we get in some recently added multi-threaded code (like spin images and parallel cascade training).
Bundle adjustment has been integrated and put to SVN. Tests compile too, but they fail currently. Need to study that.
C++ Documentation for cv is nearly (~90%) complete. Documentation for cvaux and highgui has not been updated yet.
Maria implemented added OpenMP-parallelized code to the best split selection methods in MLL’s CvDTree (decision tree). So now we have parallel decision tree construction, boosting & random trees training, and also the parallel traincascade application.
- On the traincascade we get especially good results with OpenMP, about ~1.9 speedup on dual core machines.
When OpenCV is built with Mingw (gcc for windows), the current ML tests crash or freeze. It is likely because of recently added threading.
- Maria is now studying the problem.

Victor

Outlet detection: the detector based on one way descriptor was improved by tuning several parameters, the latest results are in the table below. Lower detection rate on orange2x1 is due to one way descriptor instability caused by strong shades, one way descriptor does not match correctly the points in specular because of extreme lighting.

Using Generalized Hough Voting



cracked (correct/total):              99/100     99.0%

fake_wall_2x1white (correct/total):  109/111     98.2%

fake_wall_2x2orange (correct/total):  34/34     100.0%

orange2x1 (correct/total):            98/103     95.2%

specular (correct/total):             24/56      42.9% (extreme lighting)

white2x1hall (correct/total):        101/103     98.1%

white2x2 (correct/total):            102/104     98.1%

Pose description obtained from one way descriptor was used to detect the outlet. A template was transformed according to a pose from each of the detected keypoints and template keypoints were matched with the keypoints from the test image. The pose with the largest match count was taken as the final answer result. The results are only slightly worse than hough transform:

Using the pose estimation from the one way descriptor



cracked (correct/total):             94/100

fake_wall_2x1white (correct/total): 110/111

fake_wall_2x2orange (correct/total): 34/34

orange2x1 (correct/total):           96/103

specular (correct/total):            22/56

white2x1hall (correct/total):       102/103

white2x2 (correct/total):            91/104

Speed of one way descriptor
- 0.5-2 seconds total detection time for 640×480 Can possibly improved, might be in memory allocation.
- Hough was sped up to ~0.1 secs so it is no longer the bottleneck
Foils for the orange outlet detector method and one way descriptor method have been prepared.
Spin image matching: several bugs in spin image filtering have been found. We have proved that in spite of the parameter tuning spin images do not produce matches that can be used to find a correct transform from a model to a scene. The reason is that the whole scene surface where spin images can be calculated is the same locally resulting in similar shape descriptors that are biased by unstable normals and noise in point cloud.

Action Items

Gary

Talk with James about use of numpy arrays and having OpenCV wrap them
Get 3D data to Victor

Vadim * Have Sergey look at bundle adjustment

Get out Beta

From Last time

Victor

~~Send in flow chart of outlet matching~~
- ~~As done on milestone~~
- ~~As done now~~

Gary

~~Ask about outlet plugging in plans~~
- Ask clean workstation ROS -
- -Jaunty -

2009-09-08

Agenda

Schedule
Bundle adjustment
Outlets

Schedule:

Beta release on ~~August 31st~~ ~~Sept 4th due to Bundle and C++~~ Sept 9th
Gold release on September 21st

For Beta

build the installation package on Windows – since we migrated from InnoSetup to NSIS + CMake, that requires some extra work on the CMake scripts
- We’ve moved to cpack (failed last night due to the python interface) # update the ChangeLog (now 50% done), the INSTALL document and doc/index.html.
repair the autotools + configure script (or put the note).
Tuesday Morning, Sept. 9th, 2009 Pacific Time

For Gold

Improve documentation
Integrate new Python interface
- James must look at the new C++ and how hard wrapppers would be for that
  - Can he redo what he did for the new interface
- Add machine learning library
Expand test code

OpenCV Summer Release:

Dates:
- Feature freeze July 31
- Beta Release Aug 10.
- Release August 31, 2009
Keys:
- {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI
  - (./) I/O for sructures, histograms
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - (./) Move into OpenCV and deprecate SWIG only version Post Beta
  - (./) Use this to expand unit test functions for OpenCV Post Beta
- 2D features
  - {OK} HOG
    - (./) Chamfer matching with orientation In progress thanks to Marius Muja
  - (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
    - {OK} LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
    - (./) C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
    - {OK} Fern based planar patch detector from LePetit’s group
    - (./) LePetit one way descriptor (Victor) (by Aug 7)
  - {OK} MSER
  - {OK} Star
  - {OK} Self-similar detector
- 3D features
  - (./) Bundle Adjustment (probable contribution by Phase Space … Talk to Sergey)
  - (./) normals in grid (Gary)
  - (./) normal based 3D patch (Gary)
  - {OK} SPIN
- Optical flow
  - {OK} Dense (Farnerback)
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - (./) Nister trees (Patrick) Post Beta
  - (./) FLANN or appropriate parts (Marius) Post Beta
  - {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
Misc:
- {OK} adaptive skin tracking
Optimizations
- (./) Various functions, ongoing …
Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- (./) Proof read — Needs complet re-do
- {OK} Put up
  - {OK} cxcore
  - {OK} cv
  - {OK} MLL
  - {OK} Aux
  - {OK} HighGUI
- (./) ’C++ Interface’
  - {OK} C++ cxcore
  - (./) cv (90%)
  - (./) Aux
  - (./) HighGUI
- (./) Add content
- (./) Create C++ documentation
- {OK} Create Python documenation
High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
(./) Define user contribution process:

Put off to Future

2D features
- Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
  - Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
3D features
- Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
- PFH, FPFH from Radu,
- 3D Delaunay,
- 3D convex hull,
- mesh generation,
- mesh simplification/subsampling
- Extend to using 2D XYZA grid as well as oct/kd trees
3D capture
- Probably by using silhouettes and then texture for cavities (?)
  - Possibly with textures or phased light
- VSLAM
- Image stitching
Optical flow
- Variational optical flow
MLL
- Tiechman’s
  - Kernel based gentle boost
  - Feature selection engine
- Automatic parameter tuning
  - Cross-validation, ROC curve generator
High GUI
- move to using QT as the HighGUI interface look and feel
Misc:
- Focus detector,
Lighting/Specularity
- Look into:
  - Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
  - Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
  - 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
  - Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
Histograms
- Insert value function with interpolation
- Automatic bin size adjustment for recognition performance using a test set
OpenCV Ideas 2009

Minutes

The current activities (TBD before beta):

build the installation package on Windows – since we migrated from InnoSetup to NSIS + CMake, that requires some extra work on the CMake scripts
- We’ve moved to cpack (failed last night due to the python interface)
update the ChangeLog (now 50% done), the INSTALL document and doc/index.html.
repair the autotools + configure script (or put the note).
Tuesday Morning, Sept. 9th, 2009 Pacific Time

Python

Currently requires manual work for each function, but as API stabilizes, this shouldn’t matter
Need to get a defined way to use the new C++ API in OpenCV

New machine

Got it, set it up
Anatoly is now working on the new build system.

Outlets

Goal: 2×1 white on white
- Either have template or send image
Slow accurate code is 1.5 sec/frame, fast is 0.5 seconds per frame.
- Use better detector first
No sub-pixel estimation
- Detects small, dark connected components. Very lighting dependent on direction of the light.
- Width of the slot hole 3-5 pixels
- But, once we have the holes and the template, we do a homography estimation
Input is an image of an outlet + outlet template
- Output gives you an array of structures containing coordinates of the holes in 2 and 3D plus hole contrast etc
Chessboard
- Victor sent plug.cpp

Outlets

Victor

Outlet detection: one way descriptor has been optimized. Algorithmic optimization and parameter tuning (reduction of low resolution pca from 100 to 20 components as well as reduction of the pose number from 500 to 100) resulted in >3x speedup (0.5s down from 1.5s per image). L1 norm did not give any speedup relative to L2. Accuracy dropped slightly, see the tables below. Now we are looking at alternative techniques that are faster than one way descriptor (ferns and binary matching — S.Taylor et al “Robust feature matching in 2.3 microseconds”, feature descriptors workshop @ CVPR’09). The outlet template generator has been integrated into a single application that can be used to mark the holes and calculate outlet physical dimensions.
Plug detection: short experiment with a small 4×3 chessboard showed that a simple findchessboard + findextrinsiccameraparams gives a stable 3D coordinates of the chessboard relative to the camera (wide-angle iSight from my MacBook Pro) when the chessboard is 10-20 cm away.

Outlet detection accuracy before optimisation (~1.5 sec/frame):



cracked (correct/total):             94/100

fake_wall_2x1white (correct/total): 110/111

fake_wall_2x2orange (correct/total): 34/34

orange2x1 (correct/total):           96/103

specular (correct/total):            22/56

white2x1hall (correct/total):       102/103

white2x2 (correct/total):            91/104

Outlet detection accuracy after optimisation (~0.5 sec/frame):



cracked (correct/total):             91/100

fake_wall_2x1white (correct/total): 110/111

fake_wall_2x2orange (correct/total): 33/34

orange2x1 (correct/total):           96/103

specular (correct/total):            23/56

white2x1hall (correct/total):       102/103

white2x2 (correct/total):            89/104

Outlet detection accuracy when GHT is used for outlet pose estimation instead of one way descriptor (GHT takes an additional 0.5-1s per frame)



cracked (correct/total):             99/100 | Average time: 1.386944

fake_wall_2x1white (correct/total): 109/111 | Average time: 1.005288

fake_wall_2x2orange (correct/total): 33/34  | Average time: 1.938700

orange2x1 (correct/total):           97/103 | Average time: 0.723746

specular (correct/total):            25/56  | Average time: 0.886223

white2x1hall (correct/total):       102/103 | Average time: 0.554561

white2x2 (correct/total):           103/104 | Average time: 1.654778

Vadim:

The whole documentation (cxcore, cv, highgui) is ready and is in SVN. cvaux and mll documentation updates to be done after beta release.
A serious bug in filtering functions has been fixed; about a dozen of other smaller (like inconsistent APIs, duplicating functions …) have been fixed during the documentation updates. Another bug in the boosting save/load functions, reported by Rainer, has been fixed.
The documentation has been moved from doc2 to doc; the old HTML reference has been removed.
Ran the tests on Windows, Linux (64-bit) and MacOSX, ran the samples. Everything works well.

Action Items

Gary

Ask James about automating python (how automatic)
- C++ for Python
Ask Patrick what he did with the chessboard and tip of plug
Get images to Victor

Vadim

Get out Beta
Ask Sergey to help make Bundle adjustment work (generate test data automatically)

Victor

Ask Sergey to help make Bundle adjustment work (generate test data automatically)

From Last time

Gary

{OK} Talk with James about use of numpy arrays and having OpenCV wrap them
{X} Get 3D data to Victor

Vadim * Have Sergey look at bundle adjustment

{X} Get out Beta

2009-09-15

Agenda

Release

What needs to be done for Gold release
- Realistic Schedule
Bundle adjustment
Test system/new machine
Go through list below
Update website, announce to the group …

Outlets

Outlet progress on the bad images
Results over all orange images?

Schedule:

Beta 1.2.0 is released! Sept 14th
Gold release on September 29st, announce at ICCV 2009

For Gold 2.0

Fix installation bugs
Documentation
- Fix formulas
- Fix links
- Improving references
Anatoly and Maria are creating and running tests of OpenCV in various condigurations, OS et
Bug tracker clean out

OpenCV Summer Release:

Dates:
- Feature freeze July 31
- Beta Released: Sept 14th.
- Gold Release Sept 31, 2009
Keys:
- {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI
  - {OK} I/O for sructures, histograms
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - {OK} Move into OpenCV and deprecate SWIG only version → selected in CMake either or both
  - {OK} Use this to expand unit test functions for OpenCV
- 2D features
  - {OK} HOG
    - {X} Chamfer matching with orientation In progress thanks to Marius Muja
  - (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
    - {OK} LePetit keypoint detector YAPE * (Yet Another Point Extractor). We have found this to be the most stable detector.
    - {X} C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
    - {OK} Fern based planar patch detector from LePetit’s group D
    - {OK} LePetit one way descriptor (Victor) D
  - {OK} MSER — list of contours … how to extract keypoint from this
  - {OK} Fast detector *
  - {OK} SURF * D
  - {OK} Star *
  - {OK} Self-similar detector
- 3D features
  - {OK} Bundle Adjustment (by Phase Space / Sergey)
  - {X} normals in grid (Gary)
  - {X} normal based 3D patch (Gary)
  - {OK} SPIN
- Optical flow
  - {OK} Dense (Farnerback)
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - {X} Nister trees (Patrick) Post Beta
  - {OK} FLANN or appropriate parts (Marius)
  - {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
Misc:
- {OK} adaptive skin tracking
Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- (./) Proof read — Needs complet re-do
- {OK} Put up
  - {OK} cxcore
  - {OK} cv
  - {OK} MLL
  - {OK} Aux
  - {OK} HighGUI
- (./) ’C++ Interface’
  - {OK} C++ cxcore
  - {OK} cv
  - (./) Aux nothing for new functions
  - {OK} HighGUI
  - (./) Needs improvement on class descriptions
- {OK} Create Python documenation
High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
- {X} visualizations, statistical data extensions
Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
- {OK} Bought new server for automated tests and builds
  - (./) creating auto test system → Usable results 3 weeks
(./) Define user contribution process:

Put off to Future

2D features
- Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
  - Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
3D features
- Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
- PFH, FPFH from Radu,
- 3D Delaunay,
- 3D convex hull,
- mesh generation,
- mesh simplification/subsampling
- Extend to using 2D XYZA grid as well as oct/kd trees
3D capture
- Probably by using silhouettes and then texture for cavities (?)
  - Possibly with textures or phased light
- VSLAM
- Image stitching
Optical flow
- Variational optical flow
MLL
- Tiechman’s
  - Kernel based gentle boost
  - Feature selection engine
- Automatic parameter tuning
  - Cross-validation, ROC curve generator
High GUI
- move to using QT as the HighGUI interface look and feel
Misc:
- Focus detector,
Lighting/Specularity
- Look into:
  - Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
  - Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
  - 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
  - Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
Histograms
- Insert value function with interpolation
- Automatic bin size adjustment for recognition performance using a test set
OpenCV Ideas 2009

Minutes

Documentation

Separate chapter for C++, not up yet
- Not complete, functions are described
- Classes are not described
Proposal
- Do we have 3 manuals? C++, Python, C
- By conditionals in the Latex source?
- Update doc continuously
Bug in Platex doesn’t seem to work.
- M4? But not on Windows
Talk about fix online

Bugs

> 100, need to clear out

C++ Python interface

Functions in C are still functions in C++, but inside namespace cv
- Python interface for these can look the same
- One important difference for new C++ interface, output is always Mat
  - Mat will be of same size as input of type specified
  - For python, people use PIL
    - All operators are methods of the image
    - Python people would be used to this
But there are quite a few real classes.
- MLL is an example
How to make Python and C++ intefaces more alike
- Python should wrap he C++
- ptrs to Mat are not referenced counted in C++
- Python keeps points to python and to C matrices
- All new functionality is in C++
  - dense optical flow
  - HOG
  - One way descriptor
- Vadim can take a C++ function, dense optical flow and see how this will work in Python

Versions

Beta is 1.2.0
Gold will be 2.0
Clear out old release

Bundle adjustment

Sergey is working on auto data generation for test
But the code should work
Documentation
- Needs to be added
- Needs sample function

Victor

Outlet detection: experiments on the new dataset showed sensitivity of one way descriptor to a change in intensity.
- Matching of low brightness features from the new dataset and older bright template does not work. If a template is chosen from the new dataset, matching is much better, resulting in higher detection (below).
Automatic Template Selection is Possible: Experiments with detecting an outlet without knowing the correct template have been performed. A detector was run with each template, one by one, until an outlet is detected. This algorithm resulted in correct detections for the most of 2×1 outlets and bad detection on 2×2 outlets. The next step here is to run the detector with all the templates and choose the detection that has the higher likelihood (this is on the way).
- This algorithm will allow us to automatically detect the correct template and then use it with no performance overhead.
Plug detection: chessboard detection does not work on the images from the new dataset — chessboard is too low resolution. However all black squares are detected with a hole detector (designed for outlet hole detection). So I am working now on a “low resolution chessboard detector”. The detection part is finished, ordering algorithm is on the way (standard ordering algorithm from cvFindChessboardCorners is hard to use due to differences in corners and black square centers).

New data with template taken from one of the images



fake_wall_2x1white_plug  (correct/total): 34/38 | Average time: 0.964066

fake_wall_2x2orange_plug (correct/total): 20/21 | Average time: 3.460710

Vadim:

The following has been done last week:

OpenCV 2.0 beta has been released.
- The source package for Linux/Mac/*BSD and the binary package for Windows are available at https://sourceforge.net/projects/opencvlibrary/files/
- The corresponding tag 2.0beta has been added to the SVN repository. # Two MacOSX-specific bugs have been fixed after Windows release and before the Linux one: altivec and STL conflict in cvSmooth source code and ffmpeg compilation (ported from ROS).

The current post-beta activities:

I have read a few reports about running the precompiled OpenCV binaries on machines without Visual Studio installed, because of missing runtime libraries. While the users already posted a workaround for this problem, I’m now fixing this problem in the packaging script.
The documentation needs some fixes (already found a few bugs in the formulae), there are missing descriptions for some new classes and functions, quite a few links do not work, and the bibliography is not properly formatted; working on that too.
Will study the new Python bindings (cvpy) and figure out how to extend it to the C++ classes from MLL, cv and cvaux.

Action Items

Gary

Ask Marius about Chamfer
Ask about FLANN docs
- There is sample code
~~Connect Victor and Jack~~

James

Work through some documentation examples for C++
Talk with Vadim about docs on line
Go through bugs, assign
Talk with Vadim about how to make Python and C++ interface more the same

Vadim

Talk to James about docs on line
Talk with James about how to make Python and C++ interface more the same

Victor

From Last time

Gary

-Ask James about automating python (how automatic) -
- C++ for Python ’MOVE OFF OF C’
~~Ask Patrick what he did with the chessboard and tip of plug~~
Get images to Victor

Vadim * ~~Get out Beta~~

Ask Sergey to help make Bundle adjustment work (generate test data automatically)

Victor

Ask Sergey to help make Bundle adjustment work (generate test data automatically)

2009-09-22

Agenda

Release

Bundle Adjustment
Contributor acknowledgement
Schedule for Gold release?
Python etc
Code change statistics
IPP/MKL
64 bit support

Outlets and Doorhandles

Results on new arm data

Schedule:

Beta 1.2.0 is released! Sept 14th
Gold release on September 29st, announce at ICCV 2009

For Gold 2.0

Fix installation bugs
Documentation
- Fix formulas
- Fix links
- Improving references
Anatoly and Maria are creating and running tests of OpenCV in various condigurations, OS et
Bug tracker clean out

OpenCV Summer Release:

Dates:
- {OK} Feature freeze July 31
- {OK} Beta Released: Sept 14th.
- Gold Release Sept 29, 2009
Keys:
- {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI
  - {OK} I/O for sructures, histograms
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - {OK} Move into OpenCV and deprecate SWIG only version → selected in CMake either or both
  - {OK} Use this to expand unit test functions for OpenCV
- 2D features
  - {OK} HOG
    - {X} Chamfer matching with orientation In progress thanks to Marius Muja
  - (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
    - {OK} LePetit keypoint detector YAPE * (Yet Another Point Extractor). We have found this to be the most stable detector.
    - {X} C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
    - {OK} Fern based planar patch detector from LePetit’s group D
    - {OK} LePetit one way descriptor (Victor) D
  - {OK} MSER — list of contours … how to extract keypoint from this
  - {OK} Fast detector *
  - {OK} SURF * D
  - {OK} Star *
  - {OK} Self-similar detector
- 3D features
  - {OK} Bundle Adjustment (by Phase Space / Sergey)
  - {X} normals in grid (Gary)
  - {X} normal based 3D patch (Gary)
  - {OK} SPIN
- Optical flow
  - {OK} Dense (Farnerback)
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - {X} Nister trees (Patrick) Post Beta
  - {OK} FLANN or appropriate parts (Marius)
  - {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
Misc:
- {OK} adaptive skin tracking
Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- (./) Proof read — Needs complet re-do
- {OK} Put up
  - {OK} cxcore
  - {OK} cv
  - {OK} MLL
  - {OK} Aux
  - {OK} HighGUI
- (./) ’C++ Interface’
  - {OK} C++ cxcore
  - {OK} cv
  - (./) Aux nothing for new functions
  - {OK} HighGUI
  - (./) Needs improvement on class descriptions
- {OK} Create Python documenation
High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
- {X} visualizations, statistical data extensions
Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
- {OK} Bought new server for automated tests and builds
  - (./) creating auto test system → Usable results 3 weeks
(./) Define user contribution process:

Put off to Future

2D features
- Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
  - Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
3D features
- Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
- PFH, FPFH from Radu,
- 3D Delaunay,
- 3D convex hull,
- mesh generation,
- mesh simplification/subsampling
- Extend to using 2D XYZA grid as well as oct/kd trees
3D capture
- Probably by using silhouettes and then texture for cavities (?)
  - Possibly with textures or phased light
- VSLAM
- Image stitching
Optical flow
- Variational optical flow
MLL
- Tiechman’s
  - Kernel based gentle boost
  - Feature selection engine
- Automatic parameter tuning
  - Cross-validation, ROC curve generator
High GUI
- move to using QT as the HighGUI interface look and feel
Misc:
- Focus detector,
Lighting/Specularity
- Look into:
  - Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
  - Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
  - 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
  - Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
Histograms
- Insert value function with interpolation
- Automatic bin size adjustment for recognition performance using a test set
OpenCV Ideas 2009

Minutes

Release Schedule

Vadim going over bug tracker
- Some are very old
- Some new, more critical ones.
  - HighGui can’t be built on snow leopard
    - Quick solution is to disable those features on snow leopard
    - 3 parts: {X} interface, {OK} image i/o libjpeg, {X} video i/o
      - Possibly use GTK and ffmpeg for work around
      - Longer term: Use

IPP, MKL support

MKL was used only for matrix multiplication
- For this release, we won’t restore
IPP is used different, before we detected IPP automatically. But now we use IPP like any other library.
- Detect libraries at compile time using cmake scripts
- Code is platform independent, but the cmake scripts support IPP only on windows
  - Need to extend to Mac and Linux
- Will be done for 2.0
- A lot of new code uses SSE in 2.0, so IPP is less important
  - FFT, polar, haar can still benefit cocoa

Matrix:

We aren’t going to use MKL, instead we’ll use the Free library called
Atlas
- Why not Eigen? Atlas is BLAS interface, Eigen is a new API with templates etc. Atlas is lower level, so its faster

Bundle adjustment

Sergey said that he ran the tests, but it didn’t pass :-(
Sergey is in Moscow and will be back on Thursday.

User contributors

There’s a thanks page

64 bit support

OpenCV runs on 64 bit machines
- It might not be stable over huge arrays since there have not been tests with this
- Need bug submits from people who do this

Vadim:

The following has been done last week:

The new C++ OpenCV documentation has been scanned another time, many formula and a few typos have been fixed. Bibliography has been fixed too.
The HTML documentation can now be generated again, added a workaround for the problem with plastex.
Checked the python bindings on Windows, they built well. Thanks to James for the quick fix.
IPP support has been partially restored (now it’s detected on Windows only, but it’s easy to add Linux & Mac support too). The functions area covered are:

DFT (IPP’s DFT is still significantly faster than OpenCV’s), color conversion and Haar object detection.

About 15 records in the bug tracker have been closed. Most of them are just obsolete, but some were real bug reports.
Maria wrote a test for about half of the classifiers in MLL, all the tree based test
- The test reads a database, trains, tests, saves the model, loads the model and retests. Scores should be the same. Rainer suggested this test as a sanity check.
Anatoly has made good progress in the build system. We can build in different configurations, with different compilers gcc 4.1 through 4.3
- Can enable or disable openmp and SSE
- Will extend to Windows and Mac 0S 10
- About 1/3 done
- Test with ROS

The current post-beta activities:

Continue to clean the bug tracker. Actually, out of ~140 bug tracker records approximately 50 or less are likely serious bugs that may affect many users. Most other bugs are obsolete, refer to the old-style Python bindings reports, or various feature requests, or very special issues with video capturing etc. So, it’s a good chance that the tracker will be cleaned by 2.0 release.
Documenting the new cvaux stuff.

Plugs

Victor

Outlet detection_: the one way descriptor is used in the ROS outlet node now. A lot of time has been spent on setting up the online testing system. Finally I stopped on using prosilica_camera/fake_poll_node that simulates a stream of images from a camera with a set of images from a given folder. Another package, image_view, is used to generate images from bag files. One way descriptor takes unexpectedly long time (1-4s per frame) to classify keypoints in test images from new bag files collected by Jack. This is due to a large number of false alarms from the hole detector. ldetector works much faster (0.01-0.18s per frame) but gives more errors on white outlets (74 correct detections out of 100 as opposed to 90 with a GHT detector). The next step is to speed up 1NN in one way descriptor by using search structures. Also the outlet node crashes when processing 3D data from the outlet detector. Clearly it has to be generalized for arbitrary outlets (right now it assumes 2×2).
Plug detection_: a new method for small plug detection has been implemented. The method uses hole detector to detect black squares and then groups them assuming an affine transformation from the chessboard model to a scene. It detects all chessboard in data collected by Gary, all chessboards on white and black outlet data collected by Jack and about 50% of chessboard from orange outlet data collected by Jack. The latter is because of false alarms from the hole detector. The algorithm can be made more robust to such cases. The method is relatively fast — ~0.1s per frame. cvFindChessboardCorners as well as cvFindChessboardCornersex do not detect chessboards on these images at all.

Action Items

Gary

~~set up Thanks wiki after the meeting~~
Send Victor Jack’s code
Some test images to Victor on outlets

James

Vadim

Move THANKS to wiki
Get SVN commit statistics

Victor

Implement detection and tracking
Testing plug detection with new chessboard detector
Make sure outlet node doesn’t crash
Generalize for arbitrary outlets
Test for robustness
Talk to Brian about where to put data for cmake to pull in for the outlet detection node.

From Last time

Gary

~~Ask Marius about Chamfer~~
~~Ask about FLANN docs~~
- There is sample code
~~Connect Victor and Jack~~

James

Work through some documentation examples for C++
Talk with Vadim about docs on line
Go through bugs, assign
Talk with Vadim about how to make Python and C++ interface more the same

Vadim * Talk to James about docs on line

Talk with James about how to make Python and C++ interface more the same

2009-09-28

Agenda

Release
- Announcable schedule for 2.0?
  - Documentation?
    - Especially the C++ and FLANN
  - Bugs
Test system
IPP link issues?
Calondar Features
Jitendra’s N-Cut code
Bundle Adjustment status
Code change statistics
Some chat about post-release plans
- Segmentation
- 3D model capture
- 3D support — working with Radu
NVidia, LRB optimizations
CVPR

Outlets and Doorhandles

Results on data that I sent right before ICCV

Schedule:

Beta 1.2.0 is released! Sept 14th
Gold release on September 29st, announce at ICCV 2009

For Gold 2.0

Fix installation bugs
Documentation
- Fix formulas
- Fix links
- Improving references
Anatoly and Maria are creating and running tests of OpenCV in various condigurations, OS et
Bug tracker clean out

OpenCV Summer Release:

Dates:
- {OK} Feature freeze July 31
- {OK} Beta Released: Sept 14th.
- Gold Release Sept 29, 2009
Keys:
- {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

General Structure:
- :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
New Features
- {OK} C++ interface
  - {OK} cxcore
  - {OK} cv
  - {OK} HighGUI
  - {OK} I/O for sructures, histograms
- Better python interface
  - {OK} In progress thanks to (James Bowman)
  - {OK} Move into OpenCV and deprecate SWIG only version → selected in CMake either or both
  - {OK} Use this to expand unit test functions for OpenCV
- 2D features
  - {OK} HOG
    - {X} Chamfer matching with orientation In progress thanks to Marius Muja
  - (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
    - {OK} LePetit keypoint detector YAPE * (Yet Another Point Extractor). We have found this to be the most stable detector.
    - {X} C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
    - {OK} Fern based planar patch detector from LePetit’s group D
    - {OK} LePetit one way descriptor (Victor) D
  - {OK} MSER — list of contours … how to extract keypoint from this
  - {OK} Fast detector *
  - {OK} SURF * D
  - {OK} Star *
  - {OK} Self-similar detector
- 3D features
  - {OK} Bundle Adjustment (by Phase Space / Sergey)
  - {X} normals in grid (Gary)
  - {X} normal based 3D patch (Gary)
  - {OK} SPIN
- Optical flow
  - {OK} Dense (Farnerback)
- MLL
  - {OK} Extreme trees
  - {OK} Reduce memory footprint
  - {OK} More convenient training via a data class
  - {X} Nister trees (Patrick) Post Beta
  - {OK} FLANN or appropriate parts (Marius)
  - {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
Misc:
- {OK} adaptive skin tracking
Documentation
- {OK} Move to one Latex source
  - {OK} Use to generate HTML, PDF
- {OK} Add search function
- {OK} Add user feedback/documentation bug system
- (./) Proof read — Needs complet re-do
- {OK} Put up
  - {OK} cxcore
  - {OK} cv
  - {OK} MLL
  - {OK} Aux
  - {OK} HighGUI
- (./) ’C++ Interface’
  - {OK} C++ cxcore
  - {OK} cv
  - (./) Aux nothing for new functions
  - {OK} HighGUI
  - (./) Needs improvement on class descriptions
- {OK} Create Python documenation
High GUI
- {OK} In memory .jpg, .png
- {OK} move to C++ interface
- {X} visualizations, statistical data extensions
Testing functions
- (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
- {OK} Bought new server for automated tests and builds
  - (./) creating auto test system → Usable results 3 weeks
(./) Define user contribution process:

Put off to Future

2D features
- Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
  - Alpha channel, alternatives to contours RLE
  - New alpha transparency (per pixel) added
  - Introduce functions with advance ways of blending
  - Mask channels
3D features
- Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
- PFH, FPFH from Radu,
- 3D Delaunay,
- 3D convex hull,
- mesh generation,
- mesh simplification/subsampling
- Extend to using 2D XYZA grid as well as oct/kd trees
3D capture
- Probably by using silhouettes and then texture for cavities (?)
  - Possibly with textures or phased light
- VSLAM
- Image stitching
Optical flow
- Variational optical flow
MLL
- Tiechman’s
  - Kernel based gentle boost
  - Feature selection engine
- Automatic parameter tuning
  - Cross-validation, ROC curve generator
High GUI
- move to using QT as the HighGUI interface look and feel
Misc:
- Focus detector,
Lighting/Specularity
- Look into:
  - Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
  - Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
  - 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
  - Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
Histograms
- Insert value function with interpolation
- Automatic bin size adjustment for recognition performance using a test set
OpenCV Ideas 2009

Minutes

release:

Windows binary doesn’t run on windows machine unless Visual studio is installed.
- Since they used manifest
- Mark might know how to fix
Windows should be out today
Linux out today or tomorrow.

Test build

Test build system is now working for Linux
Anatoly is working on the windows version running in Virtual box
Test data: Machine learning has more test code.
- Will probably use python

IPP link was broken in Beta, but now its fixed in Gold

CUDA or Larrabee. Instead of accelerating already fast functions. Would be better to do

Graph cut stereo
Dense optical flow
Presence of GPU or LRB can be detected at run time and then called

Segmentation algorithms

Watershed segmentation (in, and fast)
Mean Shift filter — produces posterization
N-Cut Malik said he would donate this code.
Felzenschwalb fast algorithm

Calondar’s algorithm

Currently, outputs non-compressed signatures
Michael said he’s willing to help

Bundle adjustment

Still the way it was. Have tests, have code, but the test do not pass
- Fix this after the release
Based on sparse Levenberg–Marquardt solver
- Need to be able to compute the reprojection errors
- and form the Jacobian

Code change statistics

From SVN

Documentation:

James rerun script for FLANN

CVPR tutorial?

OpenCV on Robots and/or robot simulators?

Vadim on vacation until Oct 1st – October 18

Future

Support for numpy
Support for the C++ interface
MLL

Outlet detection:

Trying to implement a system in ROS, bag file is played where a plug is inserted into an outlet. Calibrate so that distance is minimum.
Learning ROS technical details
- Plug detector uses the new chessboard detection algorithm
- Pose of plug looks reasonable
Outlet detector has a bunch of false detection on hard to observe images
- Right now, filter out cases by scale selection
- Orange outlets, detection is quite stable
- White outlets are hard due to low contrast
  - Classify features using 1-way descriptor on several scales
  - Then run generalized Hough transform over multiple scales against the template
    - When contrast is low, power hole can be classified as small ground hole due to bad scale
  - Solution is to search on scale first.
- Plug detector is working with the new algorithm

Vadim

OpenCV 2.0 for Windows and Linux/Mac has been released, announcement was posted to yahoogroups.

60 bug reports have been moved to pending state, mostly with the resolution “fixed”. Currently there are about 80 opened bugs.

vs.net2003 support added, fixed build on MacOSX 10.6 with default compiler (thanks to Mark Asbach for helping me)

good progress on the build system has been made. It builds opencv on 64-bit linux with 3 compilers, debug/release, sse on/off, openmp on/off. In each configuration the tests are executed. All the results can be viewed online, the link will be sent separately. Now Anatoly is working on adding in the windows part

Victor

Outlet detection: A filter for partially observed outlets has been implemented. With a forearm camera a partially observed outlet is a common situation so we have implemented a filter that reduced the amount of false detections. The outlet detection algorithm assumes that a part of an outlet tuple can be occluded and deals with these situations correctly the most of the time (e.g. results on specular dataset). However, if only 2 out of 4 outlets are visible, detection is imprecise, so we are filtering these cases out.
Outlet templates downloading has been automated. A tar.gz archive is downloaded and unpacked when outlet_detection package is built.
Plug detection: a new plug detection function has been plugged into the plug_node. Plug detection has been tested online using fake_poll_node, the plug is detected correctly the most of the time and its pose is reasonable. I am working now on a final test that will calculate the difference between outlet hole pose and plug tip pose. The success criterion is a small distance between the two when the plug is inserted into the outlet.
Gary, I’ve implemented automated download of outlet templates on the built stage of the outlet detection package. Right now the archive can be downloaded from http://itseez.com/data/outlet_templates_20090928.tar.gz. Could you please place it somewhere in ipr:/var/www/pr.willowgarage.com/data/outlet_detection?

Action Items

Gary

James

Rerun script so that FLANN documentation shows up

Vadim

Victor

From Last time

Gary

~~set up Thanks wiki after the meeting~~
Send Victor Jack’s code
~~Some test images to Victor on outlets~~

James

Vadim * Move THANKS to wiki

Get SVN commit statistics

Victor

Implement detection and tracking
Testing plug detection with new chessboard detector
Make sure outlet node doesn’t crash
Generalize for arbitrary outlets
Test for robustness
Talk to Brian about where to put data for cmake to pull in for the outlet detection node.

2009-10-06

Agenda

Vadim is out on post release vacation

Gary at ICCV and TUM on thesis committee

No meeting except for *Victor*’s progress update on outlet detection

Minutes

Victor

Outlet detection: a new algorithm for scale-space search has been implemented. Now all keypoints are classified (into ground holes/power holes/background) in the same scale, the same scale is applied to a template during GHT search. Iteration through scales is an outer cycle of the algorithm. As a result accuracy on orange2x2_gain64_exp066_30hz test dataset has grown from 15 correctly detected images to 32. Also, the algorithm is more stable to occlusions, allowing correct detections even with a plug in front of the outlet (see attached image).

Filtering of inconsistent outlet detection results is implemented. GHT returns very rough positions of the holes and we choose the closest detected hole for each of the returned ones. However the closest hole can be a misdetection (for instance, a black circle near the ground hole on 2×2 fake wall orange outlet). The current algorithm calculates an error between the detected holes and the corresponding holes of a template projected with an affine transform. If error is large enough, positions returned by hough are used.

Outlet node is tested together with plug node for outlet/plug pose consistency. After minor modification, rviz shows consistent behavior of 3D positions of the plug and the hole that the plug is inserted to. Right now this was tested on orange2x2_gain64_exp066_30hz only, plug node still has to be generalized for 2×1 outlets.

Plug detection: the new chessboard detection algorithm has been tested on outlet_fuzzy_camera dataset. 850 images have been sampled from the orange outlet bag file. A plug is detected on 560 of them.

calonder_descriptor ROS package has been ported to mac.

Action Items

Gary

James

Vadim

Victor

From Last time

Gary

James

Rerun script so that FLANN and C++ documentation shows up

Vadim

Victor

2009-10-13

Agenda

Minutes

_’Outlets_’

0.5 – 1 second for outlet detection
Plug pose is real time
- “out of the box” it worked 25% of the time on new plug (new forearm camera has a lot of noise).
Right now the outlet and plug nodes work with prosilica camera as a service — send a request and get a response back
- Need same mechanism with forearm camera

_’CVPR_’

Could the outlet detector be a paper?
Fusing edges and local descriptors features

Vadim

Out of town

Gary

I moved the outlet templates

Victor

Outlet detection: a node that allows testing outlet and plug detection has been implemented. It simulates prosilica_poll service from a set of jpg files, just like fake_cam_node in prosilica_camera package, but it issues each image exacly two times, making sure it received outlet and plug poses for each image. As a result we are able to calculate distance between outlet and plug images synchronously. The new node, fake_poll_node_sync, is placed into outlet_detection package.

Plug detection has been improved by recalibrating the plug on the new data (calibration finds the relative position between the chessboard and the tip of the plug).

Outlet detection has been speeded up from 3s per frame down to <1s per frame by selecting ROI based on the previous frame. Features outside of ROI are filtered out and they do not participate in one way descriptor classification. The latter is still the bottleneck of the method. We are experimenting with ferns (ldescriptor) and calonder descriptors. Ferns shows worse results compared to one way even after parameter tuning, results on calonder descriptor are on the way.

Anatoly

parallelized the buildbot for linux, now different configurations can be built in parallel. A lot of time was spent on Windows OS: cmake produces various errors when started from python (no problems in cmd shell). This is still in progress.

Maria

studied graph cuts and segmentation based on graph cuts as well as ways of reusing graph cuts implemented in opencv.

Action Items

Victor

Send examples of forearm camera before and after pictures showing noise.
Work on stability of plugs and outlets with new data
How to work with forearm camera nodes

Gary

More data from point of contact of plug with outlet
Arrange working visit from Victor
Renewal decision

James

Vadim

2009-10-21

Agenda

C++ Documentation
OpenCV Code Reivew
Outlets
Future direction brainstorm

Minutes

Documentation changes
- Make C++ primary
  - Keep C but reference the C++ for big descriptions
- Vadim will start with CV module and see how it goes. Probably 2 weeks
  - Most CV is done 90%
  - Whole cxcore
  - None of cvaux
  - All of HighGUI

Information on relative speed-ups for function optimisations

FUTURE DIRECTIONS BRAINSTORM

General Idea:
- Build a tool chain
  - Implement higher level functionalities
  - Find out why it “took so long”
  - Fix the gaps
  - Iterate to next higher level functionality
Python wrapper covering all of OpenCV
Get Calander descriptor in! Almost there.
Getting the C++ docs out
Higher level algorithms for complete solutions
Collage/stitching
Image stitching
Working with humans — face, hands, skins, finding in 3D (do with single camera)
- Face recognition
- Indoor people recognition
Time for C++ interface to be more consistent
- Point classes — some operators that are missing (can multiply but not divide)
OpenCV 2.0 “new features detectors and descriptors”
- OpenCV 3.0 “new object detectors and 3D functionality”
  - Face rec, people
  - Project with outlet detection
3D descriptors
Open alternative to SIFT
- Definitively have an alternative to SIFT/SURF for keypoints on tracking and on recognition scores
Completing Segmentation toolbox
Visual odemetry, visual slam (from Kurt and James?)
Performance — acceleration opportunities — GPUs via OpenCL
- Graph cut, fast stereo, fast HOG
Object recognition with sparse stereo data (Paper Silvio forwarded)
Collaboration with Bernt Scheile ?
- Model based vision
3D rendering for 2D and 3D visual training
3D model fits (Move 2D and 3D chamfer matching into OpenCV from Marius)
3D model capture
Lighting perception — specularities, direction of lighting, shape from shading and shape from texture
flexible objects
transparent objects
surface recognition (wet, wood, cement …)
Collaborate with Sid’s toolkit?

Papers to Consider

Object Recognition and full Pose Registration from a Single Image for Robotic Manipulation
- Collet, Berenson, Srinivasa, Ferguson
A Probabilistic Framework for 3D visual Object Representation
- Detry, Pugeault, Piater
Latent SVM
- Object Detection with Discriminatively Trained Prt Based Models
  - Felzenszwalb, Girshick, McAllester, Ramanan
    - Code: http://www.cs.uchicago.edu/~pff/latent
Appearence-based Keypoint Clustering
- Estrada, Fua, Lepetit, Susstrunk

Outlet Progress

Images are low contrast
Low exposure
Working to improve the results across conditions
Working on using lines instead of keypoints
- Show working on European outlets

Weekly reports

Victor

Outlet detection: images extracted from the latest dataset plug_touch_2009-10-17-21-21-33-topic (using image_proc) are considerably lower quality than the images from previous forearm datasets. Outlets are not detected robustly because of low contrast holes. Accuracy is about 50%, however many false alarms can be filtered out based on aspect ratio and scale. We did not do this filtering before because we meet such issues for the first time.
A new version of outlet detector where keypoint classification is done with calonder descriptor instead of one way descriptor is implemented. Classification accuracy depends on parameters of calonder descriptor, outlet detection results are comparable to one way descriptor on the older datasets (orange2x2_gain64_*). We expect calonder_descriptor to work much faster than one way descriptor.
Outlet node has been updated to be compatible with the latest ROS rc.
Plug detection: the new chessboard detection algorithm has been improved by a better corner ordering. Plug is robustly detected on the plug_touch_2009-10-17-21-21-33-topic dataset.

Vadim

5 bugs in OpenCV 2.0 have been fixed.
started collecting statistics and sample conversion to the new interface:
- Converted samples from the OpenCV book ch2 (100%) and ch3 (50%). The archive is attached.
- Here is the statistics collected:
  - the number of functions converted from C to C++: 401 (note that some functions, like cv*PCA*, or cv*SVD*, or cv*SURF* etc. have been converted to C++ classes, so the real number of the functions is slightly bigger, somewhere in the range 400-420)
- the number of the optimized functions:
  - ~25 (cxcore) + 33 (cv) + 2 big classes in cvaux (spin images & HOG).
  - Given that some of the functions (like cvFilter2D etc.) include many flavors, the real number of optimized internal functions is likely over 100.
  - Statistics for the year 2009:
    - SVN: read transactions >130K, write transactions ~760K, files modified ~3400.
    - bug tracker activity (number of records opened/closed): 359/318
    - number of downloads: 389000.
  - Statistics for the year 2008:
    - SVN: 6/50/450 (since the migration only happened in December); the statistics on CVS has been lost.
    - bug tracker: 87 opened/27 closed
    - number of downloads: 476000
Anatoly continues working on the build system. Last week he added
- full Windows and 32-bit Linux support (where the virtual Windows machine is loaded, the buildbot service is automatically executed, connected to the master and starts building and testing of OpenCV on Windows in different configurations).
- Because of various problems with Virtualbox and VMWare (low performance) it was decided to migrate to MacOSX as the host platform and use VMWare Fusion. The migration was successfully completed.
Maria is implementing GraphCut, >50% of the code is written.

Action Items

Gary

Prepare review foils
Set up a review date
Send a categorised brainstorm list

James

Vadim

Make standalone version of C++ docs
- Make depecation statement for the C docs and list of C++ only functions
Send foils for OpenCV review
Send before/after optimization information

Victor

Send additional foils for outlet if you think they will help

From Last time

Gary

James

Victor

2009-10-27

Agenda

Review
Brainstorm
Outlets

Minutes

Previous Statistics
Here is the statistics collected:

the number of functions converted from C to C++: 401 (note that some functions, like cv*PCA*, or cv*SVD*, or cv*SURF* etc. have been converted to C++ classes, so the real number of the functions is slightly bigger, somewhere in the range 400-420)

the number of the optimized functions:

~25 (cxcore) + 33 (cv) + 2 big classes in cvaux (spin images & HOG).
Given that some of the functions (like cvFilter2D etc.) include many flavors, the real number of optimized internal functions is likely over 100.
Statistics for the year 2009:
o SVN: read transactions >130K, write transactions ~760K, files modified ~3400.
o bug tracker activity (number of records opened/closed): 359/318
o number of downloads: 389000.
Statistics for the year 2008:
o SVN: 6/50/450 (since the migration only happened in December); the statistics on CVS has been lost.
o bug tracker: 87 opened/27 closed
o number of downloads: 476000

_’Review_’
- Start times
  - 2008, May Vadim
  - 2008, Dec Victor
    - 2009 later for rest of team
- Statistics
  - Increase in code/functions
    - MLL (Cascade, FLANN)
    - Specific functions
      - stereo, HOG, Self similar,
      - ’OpenCV 2.0 Change log’
  - Popularity
    - SVM (source forge.net opencv library (helpful reviews)
    - User group
    - Book
- Code examples
  - Internal OpenCV: code (how the C++ interface is done)
  - Use of OpenCV: Write same example use of OpenCV in
    - C and
    - C++
    - Python
- Demos
  - Plug and outlet
    - ’Plan A‘: Use Jack’s data
    - ’Plan B’: Set of files captured from BAG — stream from prosilica pose and outlets
    - ’Plan C‘: Make outlet with imagery that looks like Jack’s data
  - Examples of C vs C++
    - Examples of first 2 chapters
  - Show python
    - New wrappers are from James
    - Move to C++ — Vadim
    - Old style swig wrappers
    - Face detection (python)
  - C++
    - People detection for HOG
- Future
  - Make ~mobil eye/Felzenschwalb Machine set up — parts-whole meta-functionality
  - 3d object recognition package
  - NVidia
    - CUDA
    - OpenCL
  - Version 2 of book

Previous Brainstorm
```
			
```
- General Idea:
  o Build a tool chain
  + Implement higher level functionalities
  + Find out why it “took so long”
  + Fix the gaps
  + Iterate to next higher level functionality
- Python wrapper covering all of OpenCV
- Get Calander descriptor in! Almost there.
- Getting the C++ docs out
- Higher level algorithms for complete solutions
- Collage/stitching
- Image stitching
- Working with humans — face, hands, skins, finding in 3D (do with single camera)
  o Face recognition
  o Indoor people recognition
- Time for C++ interface to be more consistent
  o Point classes — some operators that are missing (can multiply but not divide)
- OpenCV 2.0 “new features detectors and descriptors”
  o OpenCV 3.0 “new object detectors and 3D functionality”
  + Face rec, people
  + Project with outlet detection
- 3D descriptors
- Open alternative to SIFT
  o Definitively have an alternative to SIFT/SURF for keypoints on tracking and on recognition scores
- Completing Segmentation toolbox
- Visual odemetry, visual slam (from Kurt and James?)
- Performance — acceleration opportunities — GPUs via OpenCL
  o Graph cut, fast stereo, fast HOG
- Object recognition with sparse stereo data (Paper Silvio forwarded)
- Collaboration with Bernt Scheile ?
  o Model based vision
- 3D rendering for 2D and 3D visual training
- 3D model fits (Move 2D and 3D chamfer matching into OpenCV from Marius)
- 3D model capture
- Lighting perception — specularities, direction of lighting, shape from shading and shape from texture
- flexible objects
- transparent objects
- surface recognition (wet, wood, cement …)
- Collaborate with Sid’s toolkit?

Papers to Consider

Object Recognition and full Pose Registration from a Single Image for Robotic Manipulation
o Collet, Berenson, Srinivasa, Ferguson
A Probabilistic Framework for 3D visual Object Representation
o Detry, Pugeault, Piater
Latent SVM
o Object Detection with Discriminatively Trained Prt Based Models
+ Felzenszwalb, Girshick, McAllester, Ramanan
Code: http://www.cs.uchicago.edu/~pff/latent
Appearence-based Keypoint Clustering
o Estrada, Fua, Lepetit, Susstrunk

Vadim

Over 10 bugs in OpenCV 2.0 have been fixed (http://sourceforge.net/tracker/?limit=75&func=&group_id=22870&atid=376677&assignee=&status=4&category=&artgroup=&keyword=&submitter=&artifact_id=&assignee=&status=4&category=&artgroup=&submitter=&keyword=&artifact_id=&submit=Filter + a few bugs not in the list). these are various fixes for build problems, test crashes, incorrect functions behaviour in some cases etc.
C doc has conditionals for Python
- One set of docs that is all conditionals
  - C++ is separate since Vadim didn’t want to break
    - Conditionals should go into the C++ docs
Graphcut draft version is ready, not working well yet (as usual), Maria is debugging it now
Build system on Windows has been debugged, MacOSX added.

Outlets

Victor

Outlet detection: GHT algorithm has been improved. The old algorithm allowed voting of several keypoints for the same position and the same pose. As a result we could get 12 votes for an outlet pose from 12 keypoints with the same coordinates. In practice we were getting lots of ambiguous poses. This is prohibited now, and the number of votes for a specific pose can’t be larger than the number of holes (12 for 2×2 outlet).
- Calonder descriptor has been integrated into outlet detection as an alternative for one way descriptor for faster classification.
  - The calonder results are worse but we didn’t finish tuning the parameters. The final test results are on the way.

The literature research on possible ways of simultaneous tracking of points and lines is done. There are two possible ways:
use edges for voting within generalized hough transform;
use geometric hashing for pairs of keypoints and edgels.

Current results
- cracked (correct/total): 100/100 | Average time: 0.622961
- fake_wall_2x1white (correct/total): 109/111 | Average time: 2.145840
- fake_wall_2x2orange (correct/total): 34/34 | Average time: 2.277736
- orange2x1 (correct/total): 99/103 | Average time: 0.535686
- specular (correct/total): 20/56 | Average time: 0.949764
- white2x1hall (correct/total): 103/103 | Average time: 0.489211
- white2x2 (correct/total): 102/104 | Average time: 1.026165

Action Items

Gary

James

Vadim

Send speedups

Victor

From Last time

Gary

Prepare review foils
(./) Set up a review date
Send a categorised brainstorm list

James

Vadim

Make standalone version of C++ docs
- Make depecation statement for the C docs and list of C++ only functions
Send foils for OpenCV review
Send before/after optimization information

Victor

Send additional foils for outlet if you think they will help

2009-10-29

Agenda

Documentation

Minutes

Large amount of documentation
Probably 2 weeks just to do anything to the whole corpus
Need to merge a single document with the APIs in
- Vadim: C document will not change much
  - Won’t change much
  - Main reference manual would be shorter with less conditionals
- James:
  - Uses C document a lot and there’s lots of document bugs there (bugs, broken links, inaccurate text, missing info)
  - Don’t want to freeze this as is
V: Put most of things to new document, old API could mostly link to the new document
J: That’s one way of having “one” document, but bad for users
V: Old style interface is getting obsolete
J: 3.0, C is deprecated
V: Old API will stay past 3
J: Was using cvCalibrateStereo
- A lot of this document is common to C, C++, Python
- meaning of enums etc. Everyone will want to know common
- value of the function description is >> call signature
V: function descriptions can stay, but formulas, descritions can
J: This will be a pain for users to keep multiple links
V: Documentation can get quite big
- Argus is switched to new API
- Would just do good documentation for new API
J: Then what’s the plan for C documentation?
- V: Function and parameter descriptions as is, rest point to new API in C++ reference
J: Most of the complexity is in the parameter descriptions. Where would these description reside
- V: feature not bug, because if C++ extends API, then it can change and C doesn’t change
- J: Bugs in documents as well as C
V: Most users will go to new API in 1-2 years

’Issue:’
- 2 docs (C++ and C) vs a merged one with conditionals
  - Advantage is simpler main doc
  - Disadvantage is maintain 2 version
Maintaining semantics
- New API may get extra parameters
But, lots of work needs to be done to improve the documentation
- So its not the case that the C document can be locked down
- Might be many 1000’s of changes to the documents
  - That gets hard with 2 documents
Main issue is duplication: something in 2 places is harder to maintain
- Wiki and HTML slowly drifted, want to avoid this drift again
- CxCore, CvRef — all in big files, maybe if these are split into smaller blocks
  - Camera calibration for example, the API is roughly the same between C and C++
  - Feature descriptors are only C++
- Maybe splitting the documents that are mostly different, but if only parameter differences, then makes sense for 1 function
- For example, HOG is just in C++
- Pulling new functions into C++/Python only files
V: CvAdd has large differences, allows arbitrary. C is overloaded. Default parameters
- In C++ can’t default references This causes differences. Python uses output tuples

Platex is quite fragile and so we need to make sure that documents

Decision

We should get someone
We should bring the Python and C++ together.
- Numerical python … use their numerical types so that we can make use of python’s optimization etc functionality
- Can replace a lot of cxcore
MLL (C and C++) can be done by Maria
Cluster docs into join and separate groups
Merge functions listed in *Vadim*’s email (functions with significant differences)
- This will set the style
- Someone else do this for the rest
Make sure the latex documents still builds

Action Items

Gary

Find someone who can do latex to merge and divide C and C++
- Produce a file with conditionals for all 3 target languages

James

Take functions Vadim listed (that have significant API differences C vs C++) to set the style of how to document

Vadim

Cluster the doc files into “Conditionals” and “Separate” files
Document the undocumented new functions

2009-11-02

Agenda

OpenCV Review

Code examples C vs C++
- Internal (before and after)
- External (such as writing a focus detector old-style, new-style
Demos
- Python person detector
- C++ … ? (dense optical flow)
- Outlet detector
Speed up statistics
Template compatible/STL/Boost
Testing

Minutes

Vadim

Completing work document on OpenCV members
Victor on himself and Alexi
Prepared spreadsheet on performance test results comparing 1.0 (IPP and without) to 2.0 (IPP and without)
- A few foils of summary results
C++ Test coverage cxcore 80%, cv 55%, cvaux 0%, MLL 30%
Python Test coverage
Template
- Internally OpenCV uses templates
- Template over cv::Mat
- Template is light
  - Can make a template cv::Mat with certain restrictions —
    - multi-channel OK
    - Every channel has the same type (8bit, integer, fp, 64) but no mix
    - OpenCV can take STL vectors of the above types

Demos

Person detector C++
Dense optical flow
Grabcut
- Uses EM
- Then graphcut
Outlet detection

2010

Improve documentation
- Help to merge conditionals
- Improve the content
Getting the test coverage up to 100% (probably mostly using python)
3D capture
- Robust segmentation
- 3D for silhouette
  - Gives you a point cloud
- Point cloud to mesh
- Use textured stereo and do a global fit
  - With and without calibration pattern
3D Model fitting
MLL
- Latent SVM,
- Better classification infrastructure
- Something like IDEAL, or interface to R, Python
  - Object detection framework (visualization)
    - Use package for CRF
- 2D and 3D features engines
Image collage for Google summer of code
Image stitching (subset of Visual Odemetry/VSLAM)
- Alternatives for SIFT/SURF
Tracking toolkit

GPU Acceleration NVidia

Super resolution from video (interframe interpolation)
Latent-SVM + HOG
Stereo and energy min stereo

Outlet Detection

Outlet detection: outlet node has been generalized to handle 2×1 outlets.
- Experiments with white2x1_gain64_exp033_30hz have shown that pose estimation is less stable compared to 2×2 outlets. Also,
- when a plug is inserted into an outlet, detection disappears because 50% of the structure is occluded.
- Outlet and plug nodes have been adapted to simplify their demonstration.
- A new version of outlet templates is prepared with several bug fixes.
Outlet detection has been improved by adding chessboard filter:
- features that belong to chessboard are not used for outlet detection and this increases detection rate for cases when a plug is close to an outlet.
The first version of edges tracking based on geometric hashing algorithm has been implemented and is being debugged.
- Interpretable results are expected on Th.

Action Items

Gary

(./) Ask James about Python test coverage
(./) Finish the ppt presentation
(./) Finalize *Victor*’s Outlet detection trip

Vadim

(./) Send speedups
(./) Example of use of STL in C++
(./) Send some person detector images
(./) Account for Argus times

Victor

(./) Get working example of outlet detection to Gary
(./) Work on high angle/low contrast outlets
(./) Account for time spent

2009-11-09

Agenda

Discuss next year’s organization and priorities:

The Willow OpenCV committee wants more visibility into who is doing what. More people should work in an SVN branch and then Vadim or other can merge to trunk.
- Victor notes that many of his and Alexey’s committes were in ROS around outlet detections
- I also think it would be good when people start working, they either log into our wiki (I could set up a monthly page) and just put a line about what they are doing: “optimizing GrabCut, fixing documentation for Calibrate, …”
  1. Maybe we should use Twitter for this? We could make a joint account and just twitter a line.
  2. James wants to know if someone is working on something he is using
- We can actually use our Hudson computer to run test code whenever there is a checkin and send email to the offenders. I think we definitely should do this (even if you also do it locally).
James and Brian want more effort on test, bug fix and documentation
- James questions whether Vadim should be the main bug fixer or test cases for older functions, that someone more Junior should do this full time, continuously.
- Steve: Once more visibility into who is doing what is established, It’s possible that maybe we should add low level people to code maintenance to complement the dual purpose of advanced code development.
- Brian said that, like ROS, the core, highly used functions should have complete code coverage and good documentation, but other functions can be less strict.
Brian and James are setting up Bullseye — a code coverage reporter. I think they will set it up to run once a day and you’ll be able to check code coverage statistics.
Going forward, the development methodology should be that no new function should be added without
1. documentation
2. unit tests
3. Unit tests can be in Python, it might better if they are in C++ though
4. Unit tests (doesn’t matter, C++ or C) can/will be set up run on every new checkin.
5. We’ll need in OpenCV’s case to set this for 32 and 64 bit systems as well as under different OS’s
Steve wants to get back to a release every 6 months even if it means limiting what goes in. He understands that there was a lot of re-org going on this year, but he wants to get to that. If 6 months just doesn’t make sense, then we can propose something different.
In light of the above, most current work (except for finishing off outlet) should go to test system and docs
- we’ll want to show good improvement in coverage over the next month. I can help some in this once I see how a test should be written in C++ or Python.
- We can consider hiring a person to get the docs merged into a single doc relatively quickly … I might do that on our side.
- I wonder if we should put in speed tests into each test so that we can just read off the optimization improvements?
Our areas for next year are still:
- Object Recognition and Pose Toolkit
- 3D Toolkit (Pretty much Radu will drive this … or not — off the list)
- 3D Model Capture
- Visual Odometry (James will drive this … or not — off the list)

Minutes

Moving SVN

How to do? Brian, Jeremeny, James or Scott?
- Looks like Brian will do this, just got an email

Going forward, every new function must have

Documentation in Latex
Test code

Bugs

130 open bugs, some duplicates
Before 2.0, tried to fix the most critical
- About 80 of the bugs were about SWIG which will be obsolete
- Some are old
Bugs are better fixed by the author of the function
- Low level bugs can be hard in someone else’s code

Test

This is good for less senior people
Keep people assigned to areas that are important and that they become expert in
Code coverage tool needs to be run while working on to make sure you hit the branches
Prioritize by most used functions

Documentation

Everyone at Argus can write and read English
Less senior people can also do this
Vadim needs to pre-cluster the docs into language specific and all language
- Then perhaps Anatoly can put in the latex conditionals
- Vadim to do the initial clustering
Anytime a test is done, docuementation should be clarified
Every function has to have I/O type specified
- In place operation or not
Prioritize by most used functions

Attaching Machine learning to an external toolkit (Sage, Orange, R?)

Fern as good as Calandor in accuracy
**

Put Calander in as first as an example of going forward.

With good docuementation
With test code
- Alexey has been working with this on the outlets

Vadim

2 new opencv Python demos (people detection, dense optical flow), 3 C++ demos (dense optical flow, grabcut, convex hull) have been added, people detection demo in C++ has been extended to work with a list of images.

opencv C++ test system has been extended to display random seed values right in the console, which will help to reproduce test crashes.
Accuracy of the functions’ speed estimations has been improved: on MacOSX mach_absolute_time is used; the new cv::getCPUTickCount() reads CPU timestamp on x86, x64 and ppc architectures. Median of the several measured function executions is selected.

GrabCut – Maria has rewrote OpenCV maxflow code in C++, where the edge weight can now have an arbitrary type. This new maxflow code is now used in GrabCut, so we have a complete implementation of the algorithm.

Started experiments with numpy. For now we do not subclass numpy’s nd-array (perhaps, we can do that later right in Python), but use it as-is.
- Numpy does not support multi-channel arrays directly (one can make an array with elements of custom type, but this is very inefficient, and incompatible with OpenCV). Instead, it is possible to emulate multi-channel arrays using 3-dimensional arrays, e.g.: 320×240 3-channel image can be represented by 240×320×3 array. The layout of such arrays will be absolutely compatible with OpenCV, and the element access will also be very convenient.
- Experimental implementation of cv.create (counterpart for cv::Mat::create, the key method in the C++ API) has been written.
The C++ OpenCV API style is when you pass the destination image into the function: cv::erode(src, dst);
The Python/numpy style is when you create the destination array and return it: dst = erode(src).
The proposed OpenCV Python API style is to add “dst” keyword with default “None” value (def erode(src, dst=None)):

real_dst = erode(src[, dst=…]) # usage

- If dst is passed and it is a matrix of the proper size and type, it is reused (no reallocation is happened)
- If dst is a matrix of different size or type, or it’s not specified, when the new output matrix is created and returned.
This way we can decrease the footprint, do in-place processing when needed, and in general provide
more convenient Python-style API for quick experiments.
- cv.add with the proposed style has been implemented: def add(src1, src2, dst=None, mask=None).

Victor

Outlet detection: the first version of affine-invariant joint detection of edges with keypoints has been implemented. The algorithm constructs affine bases spanning all possible triplets of two adjacent edge points and one keypoint for each pair of edge and keypoint.
- A 2D hash table (discretization of affine coordinates) contains links to corresponding model (pair of edge and keypoint) and basis if there is a point from the model that has coordinates in the corresponding region.
- Recognition phase consists of building a voting histogram of bases (counting votes for specific bases from each of the test edge+keypoint pairs) and running a verification stage (match error calculation) for each basis that receives sufficiently high amount of votes.
- Detection on orange2x2_gain64_exp066_30hz and white2x1_gain64_exp033_30hz sequences shows that oval edges around each outlet get consistently higher scores given a training edge from corresponding templates.
  - Keypoints in this algorithm are used only for constructing bases. We believe that after algorithmic optimization the algorithm will improve detection and pose estimation.

Outlet and plug detection demo preparation for OpenCV review.

Experiments with calonder descriptor instead of one way descriptor.
- Calonder descriptor shows lower accuracy than one way descriptor even after parameter tuning.
  - It often classifies outlet holes as background causing outlet misdetections.
- The next step in optimizing outlet detection is to implement kd-tree based NN search in one way descriptor.

Victor will be in the office on the 24th

Action Items

Gary

(./) Ask about how to upload SVN (Brian sent email, will do)
Ask about how Bullseye handles dynamically loaded code
Get a place for Victor in the Green room
Send Steve a note about contract renewal

Vadim

Pre-cluster the docs into language specific and all language
- Then perhaps Anatoly can put in the latex conditionals
Make sure we migrate SVN and work with Brian to get test code coverage up and running

Victor

Send node about contract renewal
Have Alexey prepare Calandor feature for OpenCV inclusion together with
- Documentation
- Test code
- Usage sample

From Last time

Gary

(./) Ask James about Python test coverage
(./) Finish the ppt presentation
(./) Finalize *Victor*’s Outlet detection trip

Vadim

(./) Send speedups
(./) Example of use of STL in C++
(./) Send some person detector images
(./) Account for Argus times

Victor

(./) Get working example of outlet detection to Gary
(./) Work on high angle/low contrast outlets
(./) Account for time spent

2009-11-17

Agenda

Code Coverage
Documentation advances
Code
- Grab cut
- Calandar descriptor
CVPR plans

Minutes

The way Brian set up code coverage mixes everything together, in particular cvaux (experimental code) so it doesn’t show progress
Get him to change the display

Code coverage

Probably 10% improvement to around 70% function coverage
Need to get Hudson to report by library categories

Documentation

Finished cxcore part of reference manual
- It’s now clustered into universal and particular language differences
- Conditional are in
- It’s not up yet
Need to finish this with cv and with highgui
- Can finish by next meeting so we get C++, then long merge to minimize duplication
mll — don’t know how to process it yet
- 2.0, no C++ documentation for MLL
- Also by next Tuesday

New MLL interface

For each train and predict, added new style matrices
Have to add these to the mll manual
C++ torch
Java Weka (is the most popular library right now)
R
Python?
Maybe the best way is to just write interface to spit out features on images setting locations/scales etc in different image locations
- Output into their input framework
  - Feature production toolkit
  - OpenCV XML to HDF, Matlab, R
Might think of putting algorithms that are
- Fast
- Scalable
  - Extremely random trees and/or FLANN
- Build features for this
  - Put together cluster here

GrabCut

In cv
No documentation
Doesn’t have test
Does has sample code

Focus detection

Function outputs Gaussian Pyramid
Turn into Laplacian
- Upscale and subtract

Image pyramids

Have the original
Can do with smooth and resize
- In IPP functions that combine both steps
  - smooth and decimate, only compute decimated pixels

Calandar Descriptor

Alexi could do it. Probably a week
- Remove boost dependencies
- Write test and sample (not in ROS)

CVPR

Tutorial deadline Dec 4th
- Probably suggest with ROS/robot perception, OpenCV C++ overview
Workshop deadline Nov 22nd
Demos deadline April 13
- We’ll do this for sure
- Try to get NVidia involved or doing their own

Vadim

Migrated to the new SVN at code.ros.org
GrabCut API has been simplified (reduced to a single function). The code is now in SVN [Maria]
Wrote several new tests for OpenCV in order to improve the test coverage:
- cxcore: 2 tests (RNG, XML&YAML I/O) [Vadim].
- cv: 2 tests (inpainting, watershed) [Anatoly]
- cvaux: 1 test (FAST corner detector) [Anatoly]
Fixed 4 bugs in RNG + 3 problems with compiling OpenCV in different configurations Vadim, Stonewall Ballard]
Started repairing the docs. The cxcore reference manual has been processed [Vadim]

Outlet Progress

Outlet detection: detection accuracy improved from 0% to ~25% on the latest dataset orange2x2_2009-10-30-12-53. The necessary step is gamma correction, the original images were taken with very high gamma and were over-saturated. Also, they differ from the datasets we had before by outlet scale and view angle. The current accuracy is still not high, but this is probably a matter of tuning the validation step following GHT. We will have an update of accuracy this week.
One way descriptor has been optimized by including FLANN search over poses as opposed to exhaustive search. Even when we iterate through all nodes (exact solution) we get a significant speedup of 2-3×.
- We should have automatic gama correction for more general purpose
- One way descritor is not robust to a change in gama.
  - Ferns are robust to this but ferns like texture
    - Try LBP ? Would take some time
  - Different matching engine
Original FERNS code has been tested together with opencv implementation on outlet feature classification problem. Both even after parameter tuning have shown results significantly worse than one way descriptor. We believe that this is because of a lack of texture around each feature.
Geometric hashing method has been tested. A single edge from a training image was used for building a model. During recognition step, each pair of keypoint and edge that are close to each other are tested by finding best bases in the hash table and verifying them. The verification step just calculates the distance assuming an affine transformation given by bases, without recomputing it directly from point correspondences. The results are illustrated on images below. Blue shows the matched edges extracted from test images by canny, green shows the matched model. The algorithm is robust on orange outlets where orange blob borders are extracted well, but also works on white dataset where edges are available only partly. The method runs about 10-20ms per frame. In order to bring it in production it is necessary to improve the verification step that will filter out false alarms, and fuse it with feature classification engine.

Victor incoming 23rd

Code Coverage

OpenCV Coverage on Hudson
Current Coverage:

Action Items

Gary

Send Victor link to Adaptive Regression Kernels
Talk with Blaise about gama other
Make sure Victor has building key
- Get Victor an account

Vadim

Contact Brian to change the reporting format

Victor

Calandar descriptor: Have Alexi get this in removing Boost dependency, adding documentation and test code

From Last time

Gary

(./) Ask about how to upload SVN (Brian sent email, will do)
(./) Ask about how Bullseye handles dynamically loaded code — Brian answered(?)
(./) Get a place for Victor in the Green room — 6 new machines with reserved place for Victor
(./) Send Steve a note about contract renewal

Vadim

Pre-cluster the docs into language specific and all language
- Then perhaps Anatoly can put in the latex conditionals
Make sure we migrate SVN and work with Brian to get test code coverage up and running

Victor

Send node about contract renewal
Have Alexey prepare Calandor feature for OpenCV inclusion together with
- Documentation
- Test code
- Usage sample

2009-11-23

Agenda

Getting off of source forge
- bugs
- release instructions
- SVN instructios
- Messages to the user group
Coverage Progress
Documentation update
Grab cut
Calandor Descriptor
Handling TIFF images
NVidia
CVPR

Minutes

Login to OpenCV wiki

Just gave Vadim access

Getting off of source forge

Put links on opencv wiki directly to the download, Vadim to figure out and put on opencv wiki

Message to user group

When oepncv wiki is finalized, send another message to the user group

Coverage

cxcore – +2% (to 74%), cv – +2% (to 55%), highgui – +10% (to 32%).
3 more big tests (for HOG, Face Detection and Video I/O) are in progress.
The current coverage chart is attached.

Documentation

Clustering is completed
- Files are all universal instead of cxcore.tex we have cxcore_array cvxore_xml etc
  - cv.tex => cv_calibration.tex
    - These have big conditionals for python, C and C++
      - This is not fine grain conditional, but everything is all in one place now even if the documentation sentences are duplicated 2x in one document
      - Now a dedicated editor can feather in fine grain conditionals but maybe wait until python is switched to C++
  - highgui.tex stays
- Next step is to fix how this appears online — do it this week
Attached is an example screen shot of the new pdf documentation

Background subtraction

In OpenCV
- cvaux
  - Codebook
  - universal interface cv_bgStatModel
    - adaptive background mixture with shadow detection gaussian mixture
    - histogram per pixel
- Use it to seed grab cut
- GrabCut is done with kmeans with postprocessing for speed in OpenCV
  - Might be robust to outliers

New functions

GrabCut entered with test and sample, no doc yet
- Document
Calandor function
- Alexy Integrating Calander
- Merge with planar patch detector
  - fern based and calaner are very similar
    - calander add prob dist, fern multiplied. Then use extremely random tree or fern based.
  - (Victor) Difference (calander extremely random and add, fern bayesian and multiply. Calander in ROS optimizes classification — not just probabilities but classification using random projection.
    - Suggest leave separate for now (choosing pairs of points is same, but the classification is different)
  - (Vadim) Thinks adaptation of calander to fern will be faster than keeping separate due to Boost lib dependencies.
    - Spent a bunch of time reshaping calander descriptor to be more like opencv style, interface and internal coding. Branch that didn’t use sse didn’t compile and had to re-write with the original code.
  - (Victor) Alexy is already studying Calander to remove Boost dependencies
    - Reshape code later (~2050)
    - Alexy is writing a sample for Calander
    - Should be checked against ROS output

C++ in PDF is reasonable form

TIFF image has tags
Adding alpha channel ~“photometric interpretation” RGB RGBA or something like mixed channel, write to TIFF
- normally interleaved
- OpenCV 3 to 4 channel, use mixChannels give 3 and 1 and copy to 4 channel

NVidia

Eugene at another company was talking about film demos (opencv running in heavy computer vision
- Potentially a collaboration? They give data and reference code, we optimize
(Gary) Push them along
- Edge blending example using OpenCV GPU to make seemless
  - 10Mpix rendered in real time using NVidia Cuda (Anatoly took 3 weeks, got 200x speed up)
Ideas
- Do real time stereo on 10Mpix ExMor chips?
- Block matching — has aperture problem, small has many false alarms even with high resolution due to image warping, large cannot catch fine details
  - Might limit the resolution of the disparity map
  - How big of an image can we do real time using Kolmogorov?
- Real time high resolution Latent-SVM (Hog pyramid)
  - Just speeding up HOG

Look into octangonal integal images (SRI technology — they didn’t patent)

CVPR

workshops are closed
tutorials are Dec 4 (propose a tutorial)
Demos April 23 (robot based demo)

Vadim

2 new tests have been added (by Anatoly Baksheev):

cv – grabcut
highgui – image I/O.

The code coverage was increased, respectively:

Documentation clustering has been completed. The produced PDF now consists of 3 parts: C API reference, C++ API and Python API. The table of content has been synchronized for all the 3 parts. The screenshot is attached.

Plans:

finish the in-progress tests,
implement test of stereo calibration & rectification
integrate calonder descriptor to OpenCV

Outlet Detection

Outlet detection: a new method for feature clustering based on geometric hashing has been implemented. This is an alternative to generalized hough transform that was shown to poorly perform for large viewangle (because larger viewangles cause larger object aspect ratios and this results in larger errors in calculating object center that is blurred between several bins). During the training phase GHash algorithm iterates through all bases built upon triplets of features detected on an outlet template and builds a hash consisting of bases referenced by corresponding affine coordinates of other features in the template. Online phase of GHash consists of iterating through a small subset of bases built on triplets of points from a test image, calculating votes for each of the bases from the hash entries and validating matches for bases with large number of votes. The method performs well on frontal images showing accuracy comparable to GHT, while being much better on orange2x2_2009-10-30-12-53 dataset (see the test results in the end of the message). The method works for 2×2 outlets, still has to be generalized for 2×1 outlets (this is a matter of making several parameters such as the minimum number of votes dependent on outlet template).

A bug causing occasional crashes has been fixed in kdtree version of one way descriptor.

Calonder descriptor port is in progress.

Best Regards, Victor

Action Items

Gary

Ask about allowing anonymous bug reporting in OpenCV Trac
- ROS has mirrors around the world, what to do about that.
- Put download links on OpenCV wiki?
Clarify team roles, make a case for 3&3
Check back with NVidia

Vadim

Figure out how and where we should put SVN and releases and make appropriate links on http://opencv.willowgarage.com

Victor

From Last time

Gary

(./) Send Victor link to Adaptive Regression Kernels
(./) Talk with Blaise about gama other
(./) Make sure Victor has building key
- (./) Get Victor an account

Vadim

(./) Contact Brian to change the reporting format

Victor

Calonder descriptor: Have Alexi get this in removing Boost dependency, adding documentation and test code

Alexey

Check that OpenCV Calonder output is the same as the ROS version

2009-12-01

Agenda

Code coverage and statistics
Documentation
- progress
- plans
Bug clearing
- Open all bugs (even instantly solved ones, even if it is just a one sentence note) so that it can be closed and thus recorded.
Show case new function
External contribution history
More frequent SVN checkins, more by the rest of the team
Case for 3&3 team at least:
- Need both to maintain/enhance the old code and advance the new code
- New things that we’d like to get accomplished
  - 3D Model Capture
  - 3D rigid object tracking
  - Replace SIFT
    - Update Moped with this
    - Update 3D textured model capture
  - Visual odemetry
    - Image stitching
  - Scalable learning and feature architecture
  - Gabor jets
  - Latent SVM and extentions
  - State of the art face recognition
  - State of the art indoor people detection.
  - Transparent and shiny item recognition
  - Surface perception
  - New features such as steerable kernels
Plugging in progress

Minutes

Future plans

Need really usable tookits
- Object category recognition toolkits
  - User has to write too much glue (Geometric hash, generalized HOG, selection …)
- Geometric tool kit
  - Should be able to know camera pose
  - project
  - pose
Use as “drivers” next year
Object recognition pipeline
Tracking pipeline
Geometric toolkit

Maintenance

Documentation
- Clustering made
- Online format is good now
- Main thing now is to add and improve documentation
  - MSER
  - HOG
  - Chessboard ordering
  - Face detector
  - LBP (now part of cascade)

Need to expose interfaces for
- LBP 9 sub-rectangle against central
- HOG

Calonder descriptor
- Testing it so that it re-produces original code’s results
  - Windows seems to have a bug since it diverges a lot

Bugs — what is outstanding, new and fixed

Make focus detection a “flag ship” contribution
- What’s the best way to contribute code so that its usable without formal contibution?
- Use several layers, use GIT
- CMake is easy to add new files
- Tests and docs is difficult
  - Need some unit test method such as pyunit
  - For documentation sphinx
For user contribution, copy ROS technique?
- Have a project creation script that
  - Creates a directory
  - Puts in a CMake template
  - Creates a Latex file
    - Maybe an image directory
  - Creates a test directory
  - Creates a test stub
    - Create a data directory that is only pulled in if you build it?
  - Creates a sample directory
  - Creates a sample stub
  - Creates src and include directory
  - A simple blacklist file would stop compilation if it fails
    - OS dependent black list
Existing contributions

Activities

several new tests have been added:
- highgui (video I/O, GUI functionality) [Anatoly]
- dense optical flow (is not fully complete yet) [Anatoly]
- face & people detection [Maria]
- KD-tree & FLANN [Maria]
- kmeans [Maria]
- calonder descriptor (not integrated yet) [Alexey]
These noticeably improved the function coverage, especially the highgui and cv.
online C++ documentation generation has been repaired, the table of content has been sync’ed with C & Python docs [Vadim]. Thanks to James, it’s now available online:
- http://opencv.willowgarage.com/documentation/cpp/index.html
2 bugs in highgui, 1 bug in cv and 1 bug in cxcore have been fixed [Vadim]

PLANS:

add the following tests : ml’s EM and kNearest
- stereo correspondence,
- feature detectors and descriptors (SURF, Star, MSER, YAPE …) & planar object detection
- background/foreground segmentation
add documentation for C++ cascade detector & hog detector.

Plug and Outlet

Outlet detection: position of the robot forearm for plugging has been ratified by Stu (a compromise between perception and control).
- Lots of experimental data have been collected.
- There are almost no false positives in outlet detection (about 1 in several hundred images), but detection stability can be improved.
- The algorithm works on geometric hashing that was adapted for 2×1 outlets. The experiments uncovered several bugs in geometric hashing that have been fixed.
The main obstacle for plugging in is high error in pose estimation (about 5-10mm, should go down to at least 2mm).
- Error is in both outlet and plug pose estimations, and has both constant and random parts.
A small utility for saving images from a given topic has been implemented.
- It is called extract_images and is a part of image_view package. It can be used to save images with a given frame rate from a bag file as well as directly from a robot.

Action Items

Gary

Create focus detector
Meet with Kurt about what he needs/wants
Work on “pipelines” schemes for next year as drivers

Vadim

Send list of major contributions
Close out bugs, list what’s next, migrate to Trac
Run code coverage once more before Friday. Make sure machine learning is there

Victor

Meet with Kurt about what he needs/wants

From Last time

Gary

Ask about allowing anonymous bug reporting in OpenCV Trac
- ROS has mirrors around the world, what to do about that.
- (./) (Vadim really) Put download links on OpenCV wiki?
(./) Clarify team roles, make a case for 3&3
(./) Check back with NVidia

Vadim

(./) Figure out how and where we should put SVN and releases and make appropriate links on http://opencv.willowgarage.com

Victor

2009-12-02

Agenda

Collection of mini-meetings
Discuss with Kurt what he’d like to see in OpenCV 2010
Discuss with James
Latest code coverage, some other statistics
Set up OpenCV twitter account

Minutes

OpenCV Twitter Account was set up

http://twitter.com/opencvlibrary

Ideas for further opencv development:

Bridge from opencv to eigen will be useful for people @ Willow (eigen is more convenient than lapack)
A generic interface for descriptors, for instance, fast search for descriptors in a specific region
Sparse bundle adjustment (needs [re]implementation?)
Nister trees (available for submission to opencv)
Point Cloud Library
Stereo (Stereo Processing by Semi-Global Matching and Mutual Information Hirschmuller, improving correlation-based algorithm with projected light). Might want to switch to stereo correlation in opencv
Turn opencv into ROS-like structure —
- have independent “filters” that can exchange messages.
- This will simplify programming — in some cases people will use a collection of these filters without actually writing and compiling code.
- However this might be very platform-dependent.

Future opencv pipelines to be attacked first:

Object recognition with pose estimation
Visual SLAM/Visual odometry (this is being rewritten for M3, is a candidate for inclusion into opencv with further development)
3D image libraries

List of Contributions

General 2000-2009:

Linux and MacOSX makefiles (pre CMake-era)
swig-based Python
camera and video support in highgui Linux & MacOSX (video4linux 1/2, ffmpeg, gstreamer, xine, quicktime), GUI on MacOSX (Carbon)
OpenEXR and jpeg 2000 support

2007:

SURF
O(1) median filter
trained haar-based cascades for fullbody detection, profile face detection
Octave bindings

2008:

floating-point bilateral filter
unicap support in highgui
Google’s C++ image wrappers
Matlab-like sort
kdtree
at least 2 official patches (submitted via SF tracker) have been applied

2009:

MSER
LSH (locally sensitive hash)
Spilltree
Self-similarity features
FAST corner detector
adaptive skin detector + fuzzy meanshift tracker
Entire, state of the art nearest neighbor library: FLANN
sparse bundle adjustment (Phase space)
faster boosted classifier prediction
Haar cascades for eye detection
Haar cascades for body parts and facial features detection
gstreamer writer + remote file support
draft version of OpenCV docs in RST format
at least 12 official patches (submitted via SF tracker) have been applied

Code Coverage

Action Items

Gary

(./) Put coverage status into a spreadsheet googledoc

Victor

(./) to talk to Kurt about status/improvements in stereo

Vadim / *Victor*_

(./) to be included in the discussion on keypoints detectors /descriptors → have a better/unified interface for both in opencv

2009-12-08

Agenda

Twitter needs to tweet more
- Every week, let’s say Tuesday, the “plans” should go there with names
- Then people should say what they are working on every day, what they plan to do
  - Should I have an account on the opencv twitter?
A function is complete when:
- It is coded (obviously)
- It has documentation
- It has good conditional test coverage
- It has sample code, which might be the same as test coverage if it is well explained, but might be better if it is kept very simple.
  - A non-author should try to use the function from the documentation
  - For priority Willow code, we might want a “user test” here.
We need to collect a priority list of functions that should be at full compliance
- I sent out a post to Willow
  - Functions Willow is using/wants to/plans to use (Calibration, Stereo, Homography, Affine, GrabCut, Calonder, FAST, background, things in support of visual odemetry … even Focus …)
- Core image processing functions (arithmetic, logical, matrix ops, matrix reference counting,
- Basic cv functions (Canny, Optical Flow, flood fill, convolution, edge detect, blur, threshold, morphology etc)
- Basic learning: (Cascade, Main techique in FLANN, K-NN, k-means)
- Others?
We will then make priority tiers and some basic time lines
All out focus this on the calibration tickets and documentation (about half the tickets might be documentation there?)
- Vadim, take what resources you need including myself and Victor if we can be useful in this.
Adding circular element calibration patterns (What Victor is working on here)
Need everyone’s email … so I can remind them if they don’t twitter enough. I will create an internal willow opencv mailing list so that willow people can mail the team directly.
NVidia statement of work
Some comments about future strategy

Minutes

People will tweet more often (1-2x/day at least) and tell them to use it informally … just what are you doing.
New functions should have a non-author use the documentation before the documentation is considered complete
Next release of code
- _’May_’ sometime (for new functions and hardened code)
  - Put out intermediate release in _’Feb_’ sometime that has mostly fixes, docs, tests
Priority functions
- Obvious simple ones such as
  - Image arithmetic, logic, creation/destruction
  - Cascade, Main techique in FLANN, K-NN, k-means
  - Canny, Optical Flow, flood fill, convolution, edge detect, blur, threshold, morphology etc
  - Willow ones: Calonder, one way, stereo, calibration …
  - Come up with candidate lists of priority functions ’by Monday’
Calibration:
- Victor to help with documentation
- Change function to allow control of principle and focal point
- Simple of way of getting min and max
  - 0: only good section, 1: all, ROI returned of entire and can pass values in between

Calibration pattern with circular elements

Victor wants to change template matching in OpenCV in general
- Feature detection and template matching as separate functions
Separate the topology mapper from the blob detector
- Make this modular
- Submit point matcher to OpenCV
- Separate topology matcher
- Generalized Hough transform
- Geometric hashing
Make a very stable detector
- Look at the SRI detector

Draft a research plan going forward after Calibration

NVidia

Can we get large speed up for HOG on GPU?
Stereo is a clear win on GPU however, that should probably be first
- Correlation based
- Stereo (Stereo Processing by Semi-Global Matching and Mutual Information Hirschmuller, improving correlation-based algorithm with projected light). Might want to switch to stereo correlation in opencv
Then put this together in an object segmentation/recognition framework to be inivestitgate

Report

several new tests have been written, several old tests have been extended:
new sparse optical flow tests, extended Farneback test [Anatoly]
new SURF & StarDetector tests – the repeatability is checked [Anatoly]

new tests for LSH (locally sensitive hash), PCA, reduce [Maria]
2 new MLL tests: knearest, EM [Maria]

new simple regression test of stereo calibration & rectification, based on samples/c/stereo_calib.cpp sample [Vadim]
extended tests to cover C++ API: Canny, floodfill, Rodrigues, template matching, color conversion [Vadim]

The current coverage chart is attached.

Fixed 6 bugs in the code and the documentation [Vadim]:
trac ## 7, 8, 11.
other bugs have been reported in opencvlibrary-devel or e-mailed to me (build on Win64, potential crash in SparseMat::convertTo, uninitialized memory read in bitwise_not).

Calonder descriptor is now in cvaux [Alexey]

Implemented draft version of the function cv::getOptimalNewCameraMatrix that should solve all the mentioned difficulties with OpenCV undistortion functions [Vadim]

The spec is:


Mat getOptimalNewCameraMatrix( const Mat& cameraMatrix, const Mat& distCoeffs,
const Mat& R, Size imgSize,
double minFScale, Size maxImgSize,
Size& newImgSize);

(A similar function will be added to C API as well)

The function takes the estimated camera matrix, distortion parameters and the optional rectification transform R, as well as the original image size. It computes the optimal new camera matrix and the new image size, so that the whole undistorted image fits the new image rectangle, or some part of it.
Some pictures from single-head camera are attached.
- wide_angle_calibration_orig.jpg – original frame from wide-angle camera (thanks to Victor for the image and the associated camera data)
- undistort_small.jpg – the image resolution is kept the same, and the new focal length is computed so that the image fits entirely.
- undistort_large.jpg – the image resolution is increased by 20% at max, while the new focal lengths fx, fy are chosen close to the original ones.
- any other reasonable variant is possible with the function too (e.g. keep the focal length intact, make the image as big as necessary to fit the undistortion result).
One stereo camera the function is being debugged.

PLANS:

continue to increase cxcore & cv coverage.
add *James*’ Python tests into the regression testing loop
close urgent other tickets at Trac and SF bug tracker.

Outlets

Outlet detection:
- pose estimation for outlet detection has been stabilized. The distance between the tip of the plug and the ground hole that the plug touches is within a range of 0.1-3mm.
- Outlet and plug nodes have been changed from working with prosilica poll service to forearm messages processing.
Texas parking:
- A color template detection method has been implemented for texas parking. It is based on the code that was used for outlet detection (hole detection + geometric hashing).
- A video from texas camera has been obtained by hacking a gstreamer plugin.
- Now working on a ros node that will actually drive texas to the parking.
Other: A lot of talking to opencv stakeholers, trying to boost the communication between opencv and willow teams.

Action Items

Gary

Vadim

Victor

From Last time

Gary

<:( Create focus detector
(./) Meet with Kurt about what he needs/wants
:\ Work on “pipelines” schemes for next year as drivers

Vadim

(./) Send list of major contributions
Close out bugs, list what’s next, migrate to Trac
(./) Run code coverage once more before Friday. Make sure machine learning is there

Victor

(./) Meet with Kurt about what he needs/wants

2009-12-15

Agenda

Coverage statistics
- We must generate a list of functions to be covered for next release
  - One algorithm for this is: All code used by Willow + any prominent bugs from the community
  - All new functions need: Conditional test coverage, docs, sample code.
- The calibration pipeline (mono, dual, rectification, alignment, correspondence, disparity and depth) is complex and
  - may need its own tutorial. I will try to help write some of this …
    - but where to put this — enlarged section of the manual?
  - Needs a more comprehensive testing plan
  - Calibration also should have more general pattern finders: Element matchers separate from their geometry … at least on the plane
Twittering — need to do more often, it’s just a snapshot of the state.
OpenCV expansion
- NVidia statement of work is out
- Possible to get Intel involvement for some matrix friendly operations that they want
Future research work (builds somewhat on the calibration genreralization)
- In talks with Radu and myself, will be tied closer to Willow deliverables
- Rigid, textured item recognition
  - Has connections with visual odometry

Minutes

Coverage

According to literature, 100% conditional coverage means that the actual code conditions are about 50% covered
- Not worth going to 100%, more like static in 90%s and Conditional > 70%
  - Especially, we’re not changing the API anymore, so …
Need to make a list of priority functions
- Functions that Willow uses
- Functions with bugs
- Chart this also
Bug reports on community wiki have to be re-directed to Trac page

Calibration

Needs sort of a tutorial
- Build around Stereo Calib sample
- Cases in Calibration
  - Very good lens
  - Birds eye
  - One shot
  - Other calibration advice
  - Maybe look through Jean-Yves Matlab tool box and take some of the same examples
  - Examples of varying how much of the image is in or out of the rectified
High distortion images need looking out
- Kurt/SRI had a square matcher that was more robust but slower … as an option?

Research plan 2010

Geometric arrangement of interest points
- Moped library kind of does this, but with SIFT and licensing problems
- We also want to handle: Mono, Stereo and deformable objects/cloth
- Look for low resolution detector/descriptors (?)
  - Daisy
  - High res … !LePetit
  - Calonder descriptor
    - The code is already in cvaux library. We are preparing some test code for it. We probably need to unify some of the framework (ferns, oneway,…) and then have a detectors/descriptors overall framework. We also want to add deformable region detectors.
    - Its in cvaux.hpp (in includes) Look for RTreeClassifier at the bottom.
    - It needs documentation for to be moved into cv.
10MPix stereo

Google Summer of Code

Focus around collage, stitching

Reports

Vadim

OpenCV twitter is now being updated regularly (http://www.twitter.com/opencvlibrary) [all]
stereo rectification & undistortion improved, extended cvStereoRectify, added cvGetOptimalNewCameraMatrix;
- it’s now possible to control the visible area after undistort/rectification;
- different branches of the “camera calibration” section in the ref.manual have been merged and the content was improved [Vadim]
several new tests have been written/improved:
- Calonder descriptor – new test [Alexey]
- estimateRigidTransform – new test [Anatoly]
- 3D affine rigid transform – new test [Anatoly]
- SURF & StarDetector – improved robustness of the test [Anatoly]
- big new regression test for elementary matrix operations (cxmat.hpp) [Anatoly]
- Farneback’s test – more branches tested [Anatoly]
- kdtree & flann – more branches tested [Maria]
- haar & lbp object detection – more branches tested [Maria]
A LOT of bugs have been fixed:
- trac: tickets #10, #12, #13, #29, #34, #36 [Vadim]
- SF: #2906973, #2909108 [Vadim]
- also insufficient accuracy in XML/YML in the case of some floating-point numbers [Vadim]
- 3 bugs in MLL and traincascade, reported by JIA PEI [Maria]
We now use *James*’ unit tests in Python in our regular testing James, Anatoly].
The current coverage chart is attached below

PLANS:

continue to increase cxcore & cv coverage.
close remaining major tickets in trac and SF
extend cxcore & cv documentation.

Victor

Texas is parking fairly reliably, at least in its current location
Software is committed to sandbox
Web button is made by Josh for Texas behavior

Pluggin in

We have reliable hole and chessboard detection from *Victor*’s code
- Except for a sensitivity to light/gamma correction setting.
Full development notes in PluggingIn

Action Items

Gary

(./) Re direct bug reports in wiki to the Trac page
- (./) Send out message to group about this

Vadim

Work on developing a test set list
Docs and tests
(new) We need to consider a move to Eigen
- It’s what Willow has standardized on
- Radu used this and beat SSE optimization by Kurt and Ethan. Eigen is well thought out in this way.

Victor

Research direction: “Textures Geometrically Arranged” or “TAG” toolbox
- We need to study Radu’s PCL (Point Cloud Library) and develop a compatible architecture for the geo-feature matching
  - For one, we need to have a nice framework to:
    - Find interest points (or do dense or sub-sampled-dense)
    - Compute descriptors there
  - We need to look at !LePetit and Fua’s work since we will also want to handle deformable later

From Last time

Gary

Vadim

Victor

2009-12-22

Agenda

Use of IPP in general and latest AVX extensions for IPP in particular
Plan of record
Geometery and Texture
Eigen
Docking and plugging in

Minutes

IPP
- No latest extensions
- Have to choose IPP at configure stage
- They should sponsor someone for their new chip and IPP since we’re flat out working on Willow and community issues.
Plan of record until Jan 15: https://docs.google.com/a/willowgarage.com/Doc?docid=0AZyRqEyzcAajZGZmanozNTZfMTJuZzR3em1keg&hl=en
- Make spreadsheet that tracks that
- Along with general tracking
Geometry and Texture
- For now, Victor is
  - looking into speeding up the chessboard finder when there is no chessboard in the image.
  - Also stabilizing the locations of the corners for sub-pixel accuracy (creates noticable differences in calibration results)
  - Looking into separating features and detectors … probably do as a separate function rather than trying to re-write the existing calibration.
- Have SIFT as a comparison
- Need unified interface with fast search
- Also edges to consider (Zisserman’s local shape descriptor) combined with affine invariant edge trackers
- Also deformable and flexible objects (look at !LePetit and Fua’s work).
Eigen, ROS has standardized around this, need to work with it seemlessly.
- Vadim has been considering, had a good idea of how to implement
  - Not so difficult to implement hopefully (passing Eigen matrices to any OpenCV functions).
  - Idea: Not to switch completely (drop OpenCV for Eigen) because OpenCV is more flexible.
    - I/O with Eigen should be seemless
Docking
- “Too much CPU”, but it doesn’t have to run all the time
- No feedback … can improve in many ways, need to know if someone is working on.
- ROS node is not up and running.
  - ssh to Texas and run launch file and press the button. It will park
    - Node should be a service

Action Items

Gary

Send Victor sample sequences of textured objects under known pan and tilt
Find out who can be the “consumer” to get docking working better.
- Talk with Josh Tyler

Vadim

Add calibration functions as extra line to spreadsheet

Victor

Speed up chessboard finder when no chessboard is in the scene
Stabilize sub-pixel corners
Find SIFT code that we had before so that we can use it for a comparison study

From Last time

Gary

(./) Re direct bug reports in wiki to the Trac page
- (./) Send out message to group about this

Vadim

(./) Work on developing a test set list
Docs and tests
(./) We need to consider a move to Eigen
- (./) It’s what Willow has standardized on
- (./) Radu used this and beat SSE optimization by Kurt and Ethan. Eigen is well thought out in this way.

Victor

(./) Research direction: “Textures Geometrically Arranged” or “TAG” toolbox
- We need to study Radu’s PCL (Point Cloud Library) and develop a compatible architecture for the geo-feature matching
  - For one, we need to have a nice framework to:
    - Find interest points (or do dense or sub-sampled-dense)
    - Compute descriptors there
  - We need to look at !LePetit and Fua’s work since we will also want to handle deformable later

2009-12-30

Agenda

Progress update on Plan of Record.
- Thanks Vadim for adding this category to the spreadsheet.
I like the twittering, more is better.
- Not accomplishments, anything you are currently doing: “Writing text for X, bored and has a bug, think I’ll play some tetras for awhile”.
Calibration progress
- Chessboard rendering
Geometery and Texture, would like to start this
- Separate detectors/descriptors and their geometry
- Edges, especially shape edges
- Flexible objects
- Nister trees
Plugging in (Probably nothing much until Jan).
Texas (Unknown status right now, could try to gather more data).

Minutes

Plan of record
Spreadsheet of calibration progress.
All on vacation until the 11th. Victor 3rd-7th, Jan 23?
cv:
- Proposal to use move some seldom used functionality into cvaux
- Move experimental functionality (new stuff) into cvcontrib
Segmentation GrabPoints
- Soft points needed — they are there, need to be exposed
Calibration finish point. Holidays push off 100% coverage point
- Improving accuracy — subpixel [Victor]
- Robustness of detection. No known examples right now
- Calculate the errors of calibration such as Jean-Yves does by Jan 12? [Victor]
- Want automatic chessboard rendering as a sample
  - [Vadim]: I asked Anatoly to prepare the sample (actually, to clone and modify calibration.cpp sample) that uses the chessboard generator.
Texture and Geometry
- Need architecture, Victor propose, look at Radu, get Patrick involved
- Want to extend to flexible objects
- Nister tree
  - ROS one tied to Calonder descriptors
  - Want it to be flexible
Plugging in wait until Jan
Texas —
- click and go there
- improve template recognition
!GraphCut based segmentation with fixation (similar to !GrabCut, but uses log-polar transformed space)
- Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
- Code (Malab and windows at least).
GSOC
- Port OpenCV to Android
- OpenCV in a cloud (Gphone image goes into a server and processing happens across a cloud)
- Image stitching
- Recognition of ground plane, place for augmented reality.
- Texture based object recognition — ease of use on the phone
  - Flexible objects

NEXT MEETING: JAN 12, 2010

Reports

Victor

Chessboard detector: a function сvCheckChessboard(…) that runs a fast test on whether there is a chessboard with a given number of corners in an input image.
- There is a new flag in cvFindChessboardCorners(…), {{{CV_CALIB_CB_FAST_CHECK}}},
  - if it is on, then this check is performed before the main algorithm and if a chessboard is not found, the function returns 0 instead of wasting 0.3-1s on doing the full search.
  - The check works correctly on all test images we have in the algorithmic test.
- A new algorithmic/timing test has been implemented for сvCheckChessboard(…).
- The running time for cvFindChessboard(…) in the case when there is no chessboard in the input image has dropped ’from 0.3-1s to 0.01-0.05 s per frame’.
  - This fixes the opencv bug report https://code.ros.org/trac/opencv/ticket/41#comment:3.
An algorithmic test on cvFindCornerSubpix(…) that uses artificial chessboard generator is on the way.

Vadim

This week was mainly spent in improving camera calibration functionality coverage:
- functions coverage: 65→76%
- conditional coverage: 64→67%
Here are the details:
- 7 new tests have been written,
- 4 existing tests have been extended:
  - created 2 huge tests of various inline operations on matrices (matrix expressions) and other data structures [Anatoly].
    - 370 out of 750 such functions are covered now.
    - 11 bugs have been found and reported
- implemented artificial chessboard generator for the new set of strict calibration tests [Anatoly]
- extended PCA test, 1 bug found [Maria]
- implemented regression test of drawing functions [Maria]
- implemented test for the new C++ calibration functions, cvCalibrationMatrixValues and cvStereoRectifyUncalibrated [Maria]
- implemented test for cvUndistortPoints, extended the test of cvUndistort [Alexey]
- extended cvGEMM, cvTransform tests [Vadim]
- 10 bugs have been fixed (input data corruption in cv::eigen, Trac ## 20, 35, 38, 39 (several small bugs), 43, 44, 46, 51, SF #2907097) Vadim, Maria]
- OpenCV test engine was improved [Vadim]:
  - now the detailed information about each failed test is printed right into the console
  - “-r” mode was added, in which the engine keeps running even if some of the tests crashed.
- ~30% of cv functions using CV_ERROR, BEGIN, END_, CVCALL etc.
  - (the old-style error processing) have been modified to use c++ exceptions (CV_Error(), CV_Assert()).
  - The subset includes all the calibration functionality, except for the chessboard finder (since it has been modified by Victor at the same time).
  - That is, now we can implement “bad arg” tests for the calibration functions.
The updated code coverage statistics is here

Vacation: it looks like there will be official holidays in Russia all-the-way to 11th of Jan (updated information will be sent tomorrow), so

Happy New Year to everyone! We are going to holidays vacation.

Action Items

Gary

Edit C++ interface tutorial
(./) Send out log polar “grabcut” paper

Vadim

Victor

Send texture and geometry API around
Subpixel accuracy improvements for calibration.
Adding calibration error estimates

From Last time

Gary

:\ Send Victor sample sequences of textured objects under known pan and tilt
- I’m coding up the pan tilt stage control this week
(./) Find out who can be the “consumer” to get docking working better. (Melonee)
- (./) Talk with Josh Tyler

Vadim

(./) Add calibration functions as extra line to spreadsheet

Victor

(./) Speed up chessboard finder when no chessboard is in the scene
(./) Stabilize sub-pixel corners
{i} Find SIFT code that we had before so that we can use it for a comparison study

2009

OpenCV Meeting notes for 2009 year

2009-01-06

2009-01-13

2009-01-20

2009-01-27

2009-02-03

2009-02-10

2009-02-18

2009-02-24

2009-02-24

2009-03-03

2009-03-10

2009-03-17

2009-03-31

2009-04-07

2009-04-14

2009-04-21

2009-04-27

2009-05-05

2009-05-26

2009-06-02

2009-06-09

2009-06-16

2009-06-23

2009-07-02

2009-07-06

2009-07-14

2009-07-21

2009-07-29

2009-08-04

2009-08-25

2009-09-01

2009-09-08

2009-09-15

2009-09-22

2009-09-28

2009-10-06

2009-10-13

2009-10-21

2009-10-27

2009-10-29

2009-11-02

2009-11-09

2009-11-17

2009-11-23

2009-12-01

2009-12-02

2009-12-08

2009-12-15

2009-12-22

2009-12-30

Clone this wiki locally