2012

OpenCV Meeting notes for 2012 year

Template is at https://github.com/opencv/opencv/wiki/Template
http://code.opencv.org/help/wiki_syntax
http://code.opencv.org/help/wiki_syntax.html

2012-01-10

Agenda

Updates
LSD
TOD
Plan for next release
Machine vision type algorithms
Docs
Transparency
Linemod
http://code.opencv.org/help/wiki_syntax
Minutes

Everyone is back from vacation, not much new from last time but going forward
Dense tracking
- Kurt sent paper on photometric tracking
- RGB+Depth. Can integrate into kinect fusion but has much 2D
- This might build on top of both 2D and 3D. Ecto?
How to get higher level vision functionality out that depends on other libraries?
- Use Troy’s build system (used in ROS, but doesn’t depend, just CMake)
  - Can make Debians
Debians of OpenCV
- Ethan wants to create a Debian release repository for OpenCV
  - Based on some stable branch of OpenCV
Version numbers in OpenCV
- Major.Minor.Patch
  - Major API changes. Every time it requires user code to recompile. Bug fix etc.
- OpenCV adds functions fast …
  - Discuss further by email
Current release
- Better documentation, examples etc
  - Better description of arugments
  - User guide
- New website//stack overflow (Sometime in Feb.)
Next release functionality will be driven by needs
- Will be for Khronos, Standards, Mobile and industrial
Khronos Jan 30 -

Action Items

Gary

Send note to mobile team, CC Victor
Arrange for Vadim’s stay.

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

Send out schizzle on version number system ideas

From Last Time

Gary

Transparent to Ilya
(./) Marc Levoy on computational photography
- We will discuss after he gets his toolset going and after we are finished with phase one of mobile

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-01-17

Agenda

New website
Getting started section
Transparency
Revamp machine learning interface?
Khronos

Minutes

Ready to move to mecurial
- mecurial because it is an easier transition for svn users and has real branches and it’s faster over https (at least it was)
- need to transfer the bug base
  - If code.ros.org uses sqlite for trac, we can
  - mysql or postgresql, we need a dump of
  - trac-admin
kinfu (our version of kinect fusion). It’s in PCL
- Refactoring. Looks fairly stable, Anatoly is supporting it right now
- Smooths away noise but is not metric. Data aggregation is what we really want.
- bilateral filtering doesn’t work for our subtle segmentation problems because the noise is on the level of the features
- But, if scene is ~planar, it will lose track if it doesn’t use texture also (photometric features)
- We can use a robot to move precisely, so we might only need 3-4 images.
  - Can we get a CPU version?
  - If we need standalone, it could be done quickly
- Mixing in photometric to kinfu, many ways of doing this, so in discussion
Visual odometry
- Halki’s code, but different than TUM’s code (dense photometric Steinbrucker et al) Maria looking at to solve the ambiguity of planar surfaces
New website
- 2 parts: User and Developer
- User part is moving slowly, web developers that were used are not very responsive
- Developer: moved to Mecurial. Trac port is waiting on code.ros.org. Need email notification ~ end of month.
  - Want inverted index from functionality to functions
  - line segments: canny to approx poly (contour.cpp example) stl vector of points
- Getting started area
Transparency
- Trying to simplify algorithm for fitting silhouette of object to the observed object. Have closed form algorithm that might get rid of gradient descent in rotation space
- 3D pose from silhouette is what we’ll get out. CAD model, project to plane, find rotation that explains

Victor

Transparent objects: Our algorithm have used SolvePnP to estimate a 3D pose from a 2D transformation but this approach is awkward. We have derived and implemented an analytical non-iterative estimation of a 3D pose under the assumption of weak perspective projection (valid for objects with small size comparable to the distance to the camera). The 3D pose exists and it is unique. Right now it is used as the initial guess in SolvePnP (however, this call to SolvePnP doesn’t affect final poses on our dataset). The difference in the final object poses estimated by the old and new algorithms has the order of 0.1 mm.

- Removed the dependency on VTK and pcl visualization in the default build and dependency on ROS completely.
- Updated the old code of the algorithm evaluation to use the new interfaces from the transparent_objects Ecto module.
- Updated default config files to use new Ecto API.
- Filtered cases when 2d-2d ICP converges to a degenerate transformation.
- Investigated two potential bugs in the final pose evaluation when results were counter-intuitive. [Ilya]

KinFu: We have studied the paper by Steinbrucker et al (TUM) on visual odometry from dense RGB images. The implementation of this approach as a standalone algorithm is in progress. Meanwhile there is an ongoing discussion on KinFu refactoring to enable modular modifications. [Maria]

Vadim

1. (Feature Detectors & Descriptors)
- Several bugs in the recently refactored SIFT code have been fixed
- SIFT-like keypoint detector can now be used in ORB.

2. (Tests)
- The new OpenCV team member, Alexander Reshetnikov, has implemented 2 new tests:
for cv::eigen() (Jacobi algorithm for finding eigenvalues and eigenvectors of a symmetrical matrix)
for cv::findHomography().
2 more tests are in progress.

3. (OpenCV repository)
- OpenCV repository has been migrated from SVN to Mercurial at the new OpenCV hosting. After some more testing we will start using it [Alexander Shishkov]

Issues:
Unfortunately, we have not got any reply from code.ros.org admins regarding the dump of OpenCV bug tracker database. Without it we can not move completely to the new bug tracker at code.opencv.org.

Anatoly (GPU)

Accomplishments & statuses:

1) PCL.

· Minor updates of kinfu (2 minor bugs, warning for users about Fermi GPU required, etc.)

· Mesh generation from TSDF volume:

o Learned different isosurface building methods (grid projection, variants of marching cubes). Decided to start with ‘Marching Cubes 15’ (simple, all tables are available, ambiguity is not critical) and implement it right on GPU. Finished Marching Cubes GPU code. Now debugging it.

2) OpenCV GPU

· Added docs for FAST_GPU and ORB_GPU.

· Updated GPU Demo Pack according to Joe’s comments (added needle map to BROX optical_flow demo, added ORB demo).

· Fixed bug with sub-matrix in some GPU functions (remap, resize)

· Started working on Optical Flow on GPU (it will be required by video stabilization)

3) Autocalibration

· Tweaked the autocalibration method parameters. As result, ‘Point-to-epipolar-line RMS error’ in fundamental matrix estimation was decreased from ~5 pixels to ~1 pixel.

· Found and fixed a few bugs in the bundle adjustment refinement step which led to failures when some of image pairs were matched poorly.

· Implemented auxiliary functionality drawing epilines to check whether they are actually close to keypoints. Outcome: they’re actually close.

· Created more new scenes using the LG camera, done more experiments. Here are the best results achieved:

Average intrinsics initial relative guess error (28 runs):
fx
fy
Ppx
ppy
0,175910054
0,119935433
0,15020694
0,134989868

Average intrinsics final error:
fx
fy
ppx
ppy
0,039920564
0,033327766
0,089192789
0,146436884

Average final rotation vector (pattern-based calibration result = [0.0007,0.0424,0.0008]):
rx
ry
rz
0,000509849
0,003470322
-0,002961118

Average final translation vector (pattern-based calibration result = [1.000000,-0.035135,-0.118919]):
tx
ty
tz
1
-0,01003881
-0,168400068

These results are quite good, but obtained by intensive parameters tweaking for the particular dataset.

Issues:
· Result of the autocalibration method depends very much on dataset, parameters, and initial guess.

· Even for the best dataset taken be the LG camera, fundamental matrix estimated isn’t close to one estimated using the pattern-based approach. In the same time, for VIDERE camera both fundamental matrices are close.

Plans:
· Continue with optical flow on GPU.

· Rig autocalibration plans to be discussed with Victor (no questions to Kurt now).

· Finish Polygonal mesh extraction from Kinect Fusion. Consider implementing other 3D features or another task (registration?)

Action Items

Gary

Get code.ros.org for a dump of the bug tracker database.

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

From Last Time

Gary

Transparent to Ilya
(./) Send note to mobile team, CC Victor
(./) Arrange for Vadim’s stay.

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

(./) Send out schizzle on version number system ideas

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-01-24

Agenda

Website
Release

Minutes

Vadim here Feb 3rd.
Wrote simplified proposed Khronos spec
Talk with Xilinx coming up: FPGA algorithms for better sensors will be the theme
We have a 2D bar code reader in OpenCV findDataMatrix() and drawDataMatrixCodes()
- similar to libdmtx
Release target: Feb.
- Bug clean up
- Need to do a documentation push
  - Expand user guide
  - Need index of functionality to functions. For example Camera pose => what functions will do that for you
Website
Vadim’s visit
- edge detection/alighnment
- look at mvtech functioality
- Book version 2 scope

NO MEETING NEXT WEEK

Vadim and Victor are in flight to Khronos
I’ll be at Xilinx meeting

Vadim

. (API cleanup)
- A few obsolete algorithms from features2d and objdetect moved to the legacy module. Moving most of the C interface from core, imgproc, video and other modules to legacy is in progress. In goal is to have purely C++ interface in all the modules except for the legacy.

2. (API design for the CV standard)
- Developed proposal on the API for the standard. Sent by a separate e-mail. The API is inspired by the OpenCV C API, but has a few important differences, such as an abstract high-level algorithm interface.

3. (New Tests)
2 new tests have been developed - for cv::countNonZero() and cv::boundingRect() functions [Alexander Reshetnikov]

4. (OpenCV repository)
code.opencv.org is now basically ready-to-use. We now have (empty) bug tracker, regular backups, e-mail notifications and all other important features working well. For code repository we chose SVN for now; will switch to mercurial (or may be git) after the release. Next week we will start using the new hosting [Alexander Shishkov, thanks to Kirill Kornyakov for help with the setup]

Anatoly (GPU)

Accomplishments & statuses:

1) Kinect Fusion (not included in last week report)

· Implemented free camera mode in Kinfu (issue #474, the patch attached was outdated). Added scene generation from arbitrary generate scene view from arbitrary camera pose.

2) Kinect Fusion

· Finished matching cubes debugging. Now it’s possible to save 3D mesh.

· Investigated issue with keyboard call back in PCLVIsualizer and added workaround (the callback is not called for ‘keyDown’ event for “1,2,3”.. keys for unknown reason)

· Implemented two Kurt’s requests.

o Depth generation from arbitrary camera pose.

o Tsdf Integration pausing if camera movement doesn’t exceed a given value. Such feature allows to put Kinect onto table/robot eliminating TSDF data spoiling due to Kinect noise.

· As we experience a big flow of new feature requests for KinFu, main class is becoming very large. In order to simply further development a refactoring was done. The monolithic class was split into several. Now there are ‘Marching Cubes’, ‘Raycaster’, ‘TsdfVolume’, ‘ColorVolume’, ‘KinFuTracker‘ classes. Further interface refactoring is possible because many people (pcl-users, opencv team) want to combine different camera tracking algorithms. Final interface is to be discussed, I am not ready to propose it now.

3) OpenCV GPU

· Minor bug fixes (OpenGL render functions, gpu::transformSmart with mask, and canny)

· Added support of BORDER_REFLECT extrapolation mode to gpu::cornerHarris and gpu::cornerMinEigenVal.

· Implemented GoodFeaturesToTrack keypoints detector (without minDistance threshold). It is used by optical flow.

4) Stitching module

· Created the initial version of the stitching module documentation.

5) Autocalibration

· Fixed failures occurred when there were no points enough to estimate homography between point clouds.

· Did experiments with weighting the minimized functional terms with match confidences – it didn’t improved results significantly on small datasets.

· Updated the non-linear refinement step in the stereo autocalibration: now it’ll run one more time with the slightly different initial guess if the previous result abs(skew)>20.

· Created a new dataset using the LG camera (26 stereo pairs).

o Re-evaluated the autocalibration method on that datasets (relative errors are in -30%..30% range). After removing poor runs using the ‘abs(skew)>20’ criteria (we may assume that skew equals to zero) we had got the following results:

tx
ty
tz
rx
ry
rz
fx
fy
ppx
ppy
skew
avg
1
-0,0155284
-0,1327194
0,001100184
0,002393738
-0,0033920
2663,2816
2662,6424
956,92464
717,30248
-0,4743531
stddev
0
0,00185383
0,001822756
1.986E-005
0,000236871
1.887E-005
27,99120006
29,22979905
18,95194518
28,36769967
1,77984025
.
o Performed leave-one-out experiments with the OpenCV parrern-based calibration (26 images dataset) and the stereo autocalibration (26 stereo pairs dataset), results are below:

OpenCV pattern-based
tx
ty
tz
rx
ry
rz
fx
fy
ppx
ppy
skew
avg
1
-0,03685741
-0,11728993
0,000770294
0,042409292
0,000811499
2592,357308
2592,357308
1071,669231
742,5432308
0
stddev
0
0,010705909
0,027289007
0,000183807
0,000549123
0,000247277
8,212480043
8,212480043
1,453255444
0,980664114
0
Autocalibration:
tx
ty
tz
rx
ry
rz
fx
fy
ppx
ppy
skew
avg
1
-0,0239647
-0,11295489
0,001120478
0,001595205
-0,00325213
2673,465385
2665,769615
958,7160385
691,1352962
-0,7841123
stddev
0
0,008663641
0,050577501
0,00014148
0,002707494
0,000213615
105,2758247
102,406806
74,72880433
152,8306583
11,9984605
Autocalibration after removing poor runs
tx
ty
tz
rx
ry
rz
fx
fy
ppx
ppy
skew
avg
1
-0,02187779
-0,1103365
0,001109818
0,001577398
-0,00322225
2666,06125
2666,528333
943,7357083
731,2466667
-2,0498633
stddev
0
0,004703124
0,05176818
0,000140794
0,002821836
0,000192586
60,84442258
62,12071192
48,88150676
45,30437112
5,55359341

Plans:
· Implement min distance threshold for GFFT detector. Start Lucas and Kanade on GPU.

· Write autocalibration summary

· Consider implementing other 3D features or another PCL task (registration?)

Action Items

Gary

Start functionality list

Radu

Vincent

Alexander

Vadim

figure out 2D Barcode functions

Victor

Kurt

Ethan

From Last Time

Gary

Get code.ros.org for a dump of the bug tracker database.

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-01-31

Agenda

Everyone at Khronos meeting
Might be a change in operations since Gary has cofounded: Industrial Perception
Release at endof Feb.
- Concentration on bug fix and docs

Minutes

Vadim

1. (Bug fixing)
- 16 tickets in our bug tracker have been closed: ##1345, 1415, 1422, 1442, 1486, 1512, 1514, 1518, 1519, 1521, 1531
+2 more patches from users have been applied.

2. (API design for the CV standard)
- Memory Management section has been added to the proposal document. A few minor changes made the in headers.

Victor

RGB-D visual odometry [Maria]
Improved the convergence of the initial algorithm implementation:
- added occlusions processing
- fixed the gradient computation on the border of the valid area of the warped image.
Optimized the initial algorithm implementation (cpu, speed-up is 3.5x). It takes about 3.5 sec now (for 4 levels of the image pyramid, 30 iterations per level).
Made experiments to evaluate visually the quality of the algorithm. Merged 25 sequential frames of our room by running the algorithm on adjacent frames (see the first and the last frame of the sequence in attachments). It produced the good merged cloud of all 25 frames (see the attached video merged_cloud.mp4).
Started to study the PCL implementation of KinectFusion. It’s not clear yet how we should merge ICP-based and RGB-D visual odometry.

Transparent objects [Ilya]
Finished the paper about our current algorithm and submitted it to a conference.
Devised a way how can we estimate poses in transparent clutter. A test silhouette contains boundaries of several transparent objects in this case. So we should align parts of a train silhouette with parts of a test silhouette to find potential matches and get initial poses. This problem is called part-to-part curve matching that is find the longest common subcurve under similarity transformations. Studied papers on this topic. The following are the most relevant and useful for us:
H.J. Wolfson, On curve matching, PAMI, 1990.
M. Cui, J. Femiani, J. and Hu, P. Wonka, A. Razdan, Curve matching for open 2D curves, Pattern Recognition Letters, 2009.
However, Geometric Hashing is more attractive for this problem because it has lower complexity at the test stage and it is more well-known and reliable technique.
Implemented Geometric Hashing for curve matching and initial pose estimation in transparent clutter. See an example of recognition and pose estimation in attachments.

OpenCV bug fixing [Alexander Reshetnikov]
Changed the code structure of last tests.
Fixed a bug in the function cv::JacobiImpl_(), which caused incorrect work of the function cv::eigen() in the case of 1*1-matrices. Now all tests with such matrices are finished successfully (time of execution is 20 ms or less). Changed the procedure of maximum element search in the cv::JacobiImpl_() to use row and column indices of corresponding elements (thanks to Vadim).
Prepared test videos and started to write tests related to positioning in videos (tickets #203, #1335, #1419, #1497). Fixed the ticket #1335.

OpenCV infrastructure [Alexander Shishkov]
Developers site (code.opencv.org)
Added incremental backup of redmine and SVN. Full backup occurs every Sunday, and changes are backed up every two hours.
Fixed problems with IE for some themes
Tried plugins for Buildbot and repository search
Finished scripts for copying our current SVN to code.opencv.org

Buildbot
Added the Report page with full information about the current state of Buildbot

OSQA
Done the initial settings for the web-server
Started to work on the visual part of this site

Other
Gave a talk with Kirill about working with Redmine and Buildbot for OpenCV developers
Updated the document about OpenCV web-sites

Anatoly

Accomplishments & statuses:

1) Autocalibration

· Created the autocalibration summary report (attached).

2) OpenCV

· Updated the stitching module docs.

· Added function ‘printCudaDeviceInfo’ and calls of it to all samples.

· Added a regression test for the multi-band blending method.

· Fixed compilation with CUDA 4.1 ( so for now we support both 4.0 and 4.1). But have not scanned NPP docs for new functions to use in OpenCV or to create wrappers for them in OpenCV API style.

· Fixed OpenCV GPU tickets:

o #1561(compilation)

o #1567 (gpu::device::transform for non-aligned sources)

o #1569 (inconsistent orientation for SURF_GPU)

3) Optical flow on GPU

· Implemented minDistance threshold for GoodFeaturesToTrackGPU point detector using CPU (as it doesn’t take much time even with data transfer).

· Started learning Farneback dense optical flow method. Going to port it on GPU together with pyrLK optical flow.

4) PCL

· Added ‘gpu::copyFields’ functions (experimental), so that user can easy implement different device point cloud conversion. For example, can copy normal_[x|y|z] fields from ‘DeviceArray’ to ‘DeviceArray’.

5) PCL Registration. Decided to learn it as this cloud help with future work and with KinFu registration.

· Familiarizing with ‘pcl::registration’ and ‘pcl::sample_consensus’ in order to decide what can be ported to GPU.

· Trying to port ‘pcl::icp’ to GPU:

a. Implemented pcl::gpu::octree::knnSearch for k == 1. That is because ICP requires only the closest point. Speed-up vs. pcl::octree is 3.5×. Profiling it.

It seems that only thing that can be parallelized here is RANSAC iterations loop. In this case I need to implement SVD on GPU within ‘thread block’ of another ‘group of threads’, so that many SVDs cloud be performed in parallel. But don’t know if it can be done efficiently or not.

Plans:
· Continue with pyramidal Lucas and Kanade and Farneback optical flow

· Continue with PCL registration

autocalib_itseez_report.pdf

Action Items

Gary

Start functionality list

Radu

Vincent

Alexander

Vadim

figure out 2D Barcode functions

Victor

Kurt

Ethan

From Last Time

Gary

(./) Get code.ros.org for a dump of the bug tracker database.

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-02-14

Agenda

NEW WEBSITE!
Vadim is on site
Victor is on vacation
Release
- New functionality
Khronos
OpenCV.org
GSoC going to start again

Minutes

New OpenCV developer website … for developers obviously.
- New user site is coming in March.
Talking about adding possible FPGA functionality … subject to resources, not clear yet
Release beta probably put off into early March :-( because … we delayed Vadim with new stuff
- Vadim Color Canny
- Gary/Vadim Gabor filters
- Gary/Vadim New python example for mouse interaction mouse_match.py
- Vadim Revamped features2D to make it a consitent C++ interface
- Alexander R. fixing and testing the ffmpeg wrappers (improving positioning, # of frames and other banes of software peace)
  - Added unit tests for other functions last week, this week for video
- Some advances in docs and bug fixes
Khronos API advancing … oi vey ist mir … this is a 1 to 2 year story
opencv.org … might happen. Mission might be a mixture of practical (for usefulness_) and grandios (_for motivation):

To provide a free, common and efficient infrastructure for machine perception
and to advance machine perception beyond human abilities

What an org might allow us to do:
- Ongoing funding for a core development team.
- OpenCV would offer paid official phone and email support
- It could sponsor an ongoing intern program where corps can fund a student to work on something such as done for Google Summer of Code
- Offer a paper or thesis award:
  - When a paper or thesis comes out that we like, we would be able to award the student money to turn it into an opencv submission complete with documentation, sample and test code.
- Course ware: paid online courses in various topics in which students can be certified as OpenCV developers, perhaps in different areas: Robotics, mobile, security
  - Paid summer school?
  - Paid corporate training?
- Books — Amazon.com ebooks in topic areas.
- Vision App store (?)
  - For example maybe a toolkit for mobile: AR, calibration, stitching, computational photography
- Donations, sponsorships, challenges, contest sponsorships, bounties on code or docs people want done
- Swag — T shirts, Vadim and Victor action figures, stickers, etc.

Google Summer of Code starting once again.

Anatoly (GPU)

Accomplishments & statuses:

1) OpenCV GPU Farneback optical flow

· Experimented with different approaches of porting the Farneback’s optical flow algorithm on GPU. Now it works about 8-15x faster than the CPU version on 640×480 images depending on parameters. Refactored the GPU optical flow code a lot.

· Investigated difference between GPU and CPU optical flow results: it seems the reason is in difference of resize() function. Implemented purely texture based GPU analog with the bilinear interpolation (built-in one) – it didn’t help.

Issue:

· With Gaussian kernel error difference between CPU and GPU is always higher then with Box filter kernel.

2) OpenCV GPU pyramidal Lucas–Kanade optical flow

· Finished with GoodFeaturesToTrack detector and Pyramidal Lucas Kanade (Sparse + Dense) and integrated it to OpenCV GPU module.

· The speed-ups below are archived:

GoodFeaturesToTrackDetector: 6-8x faster than OpenCV CPU

PyrLKOpticalFlow Sparse: 1.5-2.1x faster than OpenCV CPU (TBB + SSE).

PyrLKOpticalFlow Dense: 2-2.5x faster than BroxOpticalFlowGPU contributed by NVidia.

Test system was Tesla C2050, Core i5-760 2.8Ghz (quad core). OpenCV CPU version is optimized extremely well. For one pyramid level, our implementation is 0-20% faster than CUVILib’s not pyramidal LK implementation.

3) PCL. No significant updates.

· Experimented with RGBD benchmark. Adjusting volume pose relative to scene, increasing ICP iterations numbers and tweaking other parameters fixes misregistration on the dataset. Committed minor fix for normals computation from tsdf (made them smoother), it seems it improves tracking quality according to the tests.

Plans:

· Investigate why the Gaussian-based version of Farneback differs so much from the CPU. Make the final refactoring and commit the code. Document for all optical flow algorithms. Investigate resize implementation (it differs too much from CPU).

· Probably start adding NPP 4.1 functions to OpenCV or working on background substraction.

· Continue with ACM Queue article.

Action Items

Gary

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

From Last Time

Gary

Start functionality list

Radu

Vincent

Alexander

Vadim

figure out 2D Barcode functions

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-02-21

Agenda

Release
CVPR Workshop
GSOC’12
Khronos
Tracking with known camera movement
FPGA discussion

Minutes

Transparent objects [Ilya]

Finished the implementation of the paper "A Geodesic Active Contour Framework for Finding Glass" by K. McHenry and J. Ponce, CVPR 2006.
Added the texture distortion feature by using textons from the paper ”Contour and Texture Analysis for Image Segmentation” by J. Malik, S. Belongie, T. Leung, J. Shi, ICCV 2001.
Improved SVM accuracy by following A Practical Guide to Support Vector Classification from the LIBSVM.
Post-processed results of the Felzenszwalb's segmentation algorithm to remove thin regions on boundaries of objects.
The attached animated GIF shows a Geodesic Active Contour evolution on a training image computed with the implemented algorithm.

RGB-D visual odometry [Maria]

Made a lot of experiments with ICP, RGBD and different versions of merged ICP+RGBD odometry on freiburg1_desk http://cvpr.in.tum.de/data/datasets/rgbd-dataset/download#freiburg1_desk mainly. KinFu ICP is more stable and doesn't break trajectory critically on the data. RGBD is sensitive to the following problems with rgb and depth data:
- Due to insufficient synchronization between rgb and depth data, motion blur, ... 3d points don't have their real colors.
- Matched frames can be far from each other.
In such cases RGBD gives a pose far from ground truth. So RGBD breaks the trajectory several times, but it has less than ICP median drifts of translation and rotation per frame for well synchronized data (see previous report). Tried several approaches to merge ICP and RGBD. Found that more stable version of merging is the following. RGBD pose is used only if both points error and photo consistency error are less than the same ones for ICP pose. But sometimes this merging strategy gives a trajectory that has larger absolute error than ICP.

OpenCV tests and bug fixing [Alexander Reshetnikov]

Applied some solutions for fixing the bugs associated with frame count computing and video clip positioning (feature #850 and report from the ticket #120).

Results of frame count and positioning tests (default testing infrastructure is Ubuntu x64, actual frame count is 125):

Closed tickets # 850, 1154.

OpenCV infrastructure [Alexander Shishkov]

- conversation with hetzner.de support about upload speed on our server.
- modified Apache settings for OpenCV SVN server; now full checkout of OpenCV trunk (about 1GB) from office takes about 10 minutes
- modified UUID on local SVN mirror, so it is possible to do relocation from local mirror to the public repository, migrated local mirror to Apache, forbade commits to the local mirror
- updated wiki on Buildbot project (#1073), updated Android NDK on Ubuntu slave (#1245), added link to the trunk documentation to the Buildbot main page(#1239), added highgui tests tot the Android builder(#1244)

Other:

- fixed ## 1599, 1602, 1604, 1363, 1492, 1339, 1614
- started the work on FFMPEG problems ## 1615, 924

Anatoly

Accomplishments & statuses:

1) Optical Flow

· Revealed and fixed the bug in the high level Farnebeck GPU function which led to difference between CPU and GPU results. Found out that cv::GuassianBlur() on CPU uses some hard-coded sigma values in case of small kernel sizes. That also caused difference between the CPU and GPU. Ensured the GPU version behaves the same way.

· Integrated Farneback GPU to OpenCV. It works 13..22x faster than the CPU version (Intel Core i3 vs. NVIDIA GTX 470) on 640×480 size images.

· Miscellaneous:

o Added accuracy tests, performance tests, docs.

o Created sample, updated performance sample (samples are convinced as API example for developers)

o Updated demo pack with the optical flows. (Demo pack was convinced as use-case examples for users).

2) OpenCV GPU

· Revealed incorrect behavior of linear filters for kernel sizes >=11×11. Fixed it so that it supports kernel’s size up to 32, also new implementation is a bit faster.

· Added support of different border extrapolation modes into my implementation of Gaussian blur — to be similar to the CPU version.

· Created wrappers for new NPP arithmetic only functions.

3) Other

· Prepared ACM article draft version. Guess after adding Khronos process it will reach recommended size.

· Hiring activities. I never put this in reports, but I spend time for participating interview for Tegra and so on.

· Started learning video stabilization theory.

4) PCL

· Fixed the BUG in marching cubes in Kinect Fusion

· Minor compilation fixes and tests for different platforms (like C++0x flag in CMake breaks NVCC compilation and so on)

· Started learning code and papers for future people detection GTC demo.

Plans:
· Adding NPP 4.1 functions and GPU video decoding.

· Continue learning video stabilization theory.

· ACM Queue article

· Continue with pcl_people on GPU.

Action Items

Gary

Radu

Vincent

Alexander

Vadim

Victor

Kurt

Ethan

From Last Time

Gary

Start functionality list

Radu

Vincent

Alexander

Vadim

figure out 2D Barcode functions

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-02-28

Agenda

“OpenCV submission to Google Summer of Code”: http://code.opencv.org/projects/gsoc2012/wiki/Gsoc2012

GSOC org application has started (Gary)
Khronos
CVPR Workshop prep (Victor)
Web page for OpenCV.org (Vadim)
Release
Nonprofit
Transparent
Bar codes (James is limited 2 string length 3?)
Xilix/Zync

Minutes

Wiki is on code.opencv.org
GSOC Ideas page is set
- OpenCL Optmization:
  - We have contributors already, get one or more of them to be GSoC students
- Mobile vision app development:
  - Vision apps on phone and tablet. We want a very clear “hello world” type of example to make it easier for the rest to start developing mobile vision applications. People can apply for:
    - iOS (iPhone) examples
    - Android example
- 1D and 2D Barcode readers:
  - We have a basic datamatrix reader. Expand to other kinds of barcodes (1D, other 2D).
- Natural barcodes:
  - OpenCV can read calibration patterns. Calibrate a camera, rectify it’s output. Put a picture next to a calibration pattern in a scene. Find 2D interest points with features2D. Measure them in the image using the calibration pattern. Use the pattern of interest points and their measures to turn the picture into a calibration pattern/barcode so that the computer recognizes the picture and knows where it is in relation to the iamge.
    - Stretch goal, embeded and retrieve watermarks in the image.
- Python example:
  - We’re finding the python interface extremely useful and productive. It needs a lot more examples
- Tutorials:
  - Continue with OpenCV Tutorials
- Inverted index:
  - Help create an inverted index of functionality to functions. That is, say, edge detection would list Sobel, Scharr, Canny etc.
- Train Classifiers:
  - OpenCV has many classifiers. Train them carefully on useful objects (faces, people, cars, bikes, plates, cups, bowls etc) and we’ll allow others to use those trained categories.
- Structure from Motion (SfM):
  - Help adapt libmv for OpenCV.
- Course-ware:
  - We want to start offering vision courses using OpenCV.
    - If you are expert in a topic (say it’s your area of graduate research, or you’ve just finished a project or course in a particular topic and can explain it well)
    - Write up the topic, the math and then develop a step by step example of it using OpenCV.
Khronos meetings
- Victor and Vadim attend phone meetings. Gary may start.
- Mostly it’s covering SOW philosophy
Vincent has ideas for opencv.org (Alexander)
- More tutorials etc
- Want to know what’s happening.
  - Clear roadmap
  - Change logs even in trunk
- Clearer description of modules, what it can do
- Small jobs that others can do
- Brainstorm: Website where people can share ideas of what people want/votes
- Show that project is active. Show for example a page of sponsors
- Forum for developers only (make a tab on the website: contacts, developers/users more prominent)
- Improve docs and tutorials obviously
- Official contact when there is a non-profit
- Want levels 1 is core (image processing) 2 tracking … 3 applications
- Inverted index
Release: Beta eo March, final release eo April
- ROS freeze, March 15
lvm (lib virtual machine)
- C++ running natively in web browser
- people use for OpenCL
Transparent
- Working on segmentation
  - “Perceiving light versus material” by F.A.A. Kingdom (Vision Research, 2008) and “Low-level image cues in the perception of translucent materials” by R.W. Fleming and H.H. Bulthoff (ACM Transactions on Applied Perception, 2005)
  - Contrast, texture distortion, color distortion
We have datamatrix barcodes from James based on ideas from libdmtx
Barcodes
- We have datamatrix due to James Bowman
- It would be nice to have
- circular tag modules
- Put these in opencv apps
Think about opencv apps site
OpenCV with OpenNI driver. Using Asus Pro driver
Bug: Shape context. Vincent file bug about it.
Maria on visual optical flow with rgbd
- Near term goal is to integrate code into OpenCV
  - Even more robust motion estimation with pure translation assumption
  - Robust detection of visual tracking failure
    *

Anatoly (GPU)

Accomplishments & statuses:

1) OpenCV GPU

· Finished VideoCapture_GPU implementation under Linux ( but have not integrated in OpenCV)

· Improved arithm functions dispatching between NPP and our code (~2x faster now).

· Created GPU path for gpu::PyrLKOpticalFlow (cpu version has strange behavior in error calculation, created ticket #1651).

· Revealed some inconsistencies in gpu::convolve() for small kernel sizes specializations (ticket #1639). Fixed error in gpu::convolve() docs.

2) Face detection on GPU. As soon as Anton Obukhov doesn’t longer support FD code, we are going to fix its all crashes for CUDA4.1 by our own.

· Fixed HaarCascadeLoader test failure.

Active issues:

· Visualization test fails.

· Haar classify kernel crash with CUDA 4.1

3) Video Stabilization.

· Implemented trimming which ensures that there are no black border artifacts due to stabilization in output video by trying to relax estimated stabilization transformations. It works better than naive ad-hoc which doesn’t stabilize at all when there are artifacts.

· Implemented Fast Marching Method which is required for in-painting steps. Wrote test comparing it with distanceTransform() (as distance map can be computed via FMM naturally).

· Implemented color in-painting via averaging of neighbor pixels colors. Implemented motion in-painting which propagates local motion into areas where optical flow isn’t known. Experimented with different optical flow algorithms. As it was mentioned in the original paper motion in-painting isn’t perfect technique and there are some artifacts.

4) PCL / Human Tracking

· Euclidian point cloud clusterization. Learned the algorithm and its code, understood bottlenecks. Read several papers about connected components on GPU. No idea how to accelerate the cauterization with GPU. It’s universal and slow (800ms). And even taking into consideration that input cloud is organized doesn’t reduce amount of work to be done. Instead, two lightweight algorithms were implemented on CPU which work 116ms and 14ms (floodfill-like). They produce some approximation of output from original. Experiments showed that 14-ms version is enough and doesn’t break human tracking pipeline. Accelerating this 14ms-version with GPU is also not easy and not reasonable as currently it is not a bottleneck.

· Seeded hue segmentation. The same complexities as with previous (work time 200-300ms, not idea for GPU). The approximate method used for clusterization doesn’t work here (it cuts persons hands). Accelerated CPU code by refactoring and sampling points a points queue inside (some another kind of approximation), so that it works 70-90ms for middle distance objects. Further optimization is possible with OpenMP, so expect work time to be ~30-40ms. Guess, better results are achievable by developing another smarter (than floodwill) 2D segmentation with depth constraints.

· smoothLabelImage. It’s a kind of filter. Created GPU implementation for it, so that run time reduced from 83ms to 1ms.

Overall frame time for human tracking was reduced from 2.5sec to 400ms.

Plans:

· Integrate VideoCapture_GPU to OpenCV. Fix CascadeClassifier kernel crash. Fix gpu::convolve implementation (ticket #1639).

· Video stabilization: Work on exceptional cases handling like poor in the sense of number of features datasets and datasets with large moving objects.

· Implement volume shifting for Kinfu (Kurt’s request) or something other in PCL.

· ACM Queue article

Ilya

Transparent objects [Ilya]

Studied psychological papers on perception of transparent objects. They are useful for us because they describe heuristics used by people to perceive transparent objects and we can derive features from them for glass segmentation. The most useful papers are: “Perceiving light versus material” by F.A.A. Kingdom (Vision Research, 2008) and “Low-level image cues in the perception of translucent materials” by R.W. Fleming and H.H. Bulthoff (ACM Transactions on Applied Perception, 2005). Added 3 features to our algorithm based on read papers:
1. Added Michelson contrast.
2. Added root mean square contrast.

These two features are discriminative because contrast plays the important role in perceiving glass. Look at two examples (episcotister.png and contrast.png):

(images from the paper “Toward a perceptual theory of transparency” by M. Singh and B.L. Anderson (Psychological Review, 2002))
Regions p and q on the first image look as transparent and they have lower contrast together than a and b. However, if they have bigger contrast (see the second image) then this perception is gone. In fact, the surrounding regions look as a transparent layer with the hole in the middle because they have lower contrast now.

3. Added distance between median colors in regions as a feature because a chromatic change is useful for perception too. Our regions are rather uniform after over-segmentation with Turbopixels so we can use just a median color, not full histograms.

Adapted glass boundaries computation for Kinect data. McHenry et al. classify only adjacent regions as glass-background. But our regions are much smaller so we can expect glass-background pairs to be separated by other regions. Added classification of region pairs in a neighborhood and aggregated results with a voting procedure.

See example of our algorithm working on a test Kinect image (the animated GIF):



(Note the algorithm uses RGB data only that is a depth image is not used for segmentation).
RGB-D odometry [Maria]
Refactored the RGB-D odometry implementation to put it in the OpenCV contrib module.

Found the bug in estimation of time of our RGB-D odometry working. The time was normalized on iteration number after some experiment, but the time from the original paper on RGB-D odometry is for all iterations. So optimized the implementation in more than 10 times. Main optimizations:

1. Switched to the fast Sobel version (16S).

2. Optimized image warping.

3. Chose a subset of textured pixels to estimate a transformation.

Now the implementation works the same time as in the paper (~80ms).

Started experiments on the Kurt’s dataset.
OpenCV infrastructure and issues [Alexander Shishkov]
OpenCV infrastructure:

Investigated the new Buildbot API in 0.8.5.

Began to work on FFMPEG builders for the OpenCV Buildbot.

Updated settings and the theme of the test OSQA installation.

Investigated AskBot, began to install it on our hosting (we plan to compare it with OSQA internally).

Done default settings for the GSOC project on code.opencv.org.

Fixed some problems with the code.opencv.org theme.

Communicated with the designer of the opencv.org website.
OpenCV issues:

Done some infrastructure and documentation issues (##1646, 1645, 1644, 1637, 1618, 1546, 1507,  1477, 1466, 1456, 1405, 1398, 1289, 1260, 521).

Updated png and tiff libraries used for OpenCV compilation on Windows (#1609).
OpenCV tests and bug fixing [Alexander Reshetnikov]
Configured ffmpeg 0.10 on the local machine to write and test the new OpenCV wrapper. It is based on the ffmpeg class written by Andrey Morozov.

Checked the work of the Andrey’s ffmpeg class on frame count and positioning highgui tests:

1. The frame count test.

Results (actual frame count is 125):



Results strongly depend on the file format because for different containers we can find the frame count property in different ffmpeg structures (now there is no universal rule to get it).

2. The positioning test.

The class gives fully correct results on positioning tests. However, the class has some problems with reading some formats (also it depends on a local ffmpeg configuration).

Began to write the new ffmpeg wrapper. Achieved compatibility of this wrapper with the old implementation of the ffmpeg wrapper.

Vadim

Action Items

Gary

(./) Finish GSoC application http://code.opencv.org/projects/gsoc2012/wiki/Gsoc2012
Submit ECCV workshop
Contact James about limitations in 2D barcode
(./) Ask Kurt about visual odometry 2D+3D

Vincent

ROS integration with latest possible OpenCV trunk
Describe OpenNI for ASUS prime sense

Alexander

Vadim

Talk to Anatoly about stand along kinfu

Victor

Kurt

Talk to Maria on visual optical flow with rgbd
- Near term goal is to integrate code into OpenCV
  - Even more robust motion estimation with pure translation assumption
  - Robust detection of visual tracking failure

Ethan

Get OpenCV for Asus prime sense

From Last Time

Gary

Work on non-profit
(./) GSoC application

Vincent

Assert ownership of direction of Itseez team
Bug report on samples

Alexander

Vadim

(./) Get Alexander to set up wiki for OpenCV.org

Victor

Get on the Mobile workshop prep.

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-03-13

Agenda

Python — have a volunteer to work on demos etc
Vincent ROS integration with latest OpenCV trunk?
OpenNI for OpenCV trunk
New dense optical flow looks really good. Congrats to Maria!
Kinfu integration. IPI has it stand alone now.
- New optical flow technique means we might not even need kinfu, might have something better?
Ethan’s contributions
- Patches for ASUS, really fast normal estimation (~instant even on CPU)
GSOC — our application is in, we find out what the decision is on Friday at noon, PDT.
- See our ideas page http://code.opencv.org/projects/gsoc2012/wiki/Gsoc2012
Khronos updates
Non-profit progress
User web page status
Asus support. Ethan’s contributions (super fast normals)
Eigen
Rotations convenience functions
- Quaternians, Rodriges, R,t and just some common functions we face (segmentations, histograms, morphology)
Connected components
Bump hunting (Gary writing) Finding density peaks in data
ECCV application
CVPR. Gary’s paper is in. Workshop dev => course ware.
Transparent

Minutes

GSoC … find out on this Friday
Python Abid will start working on some examples. He’s not going in GSoC, we’ll find another for that
Latest OpenCV in ROS, Vincent to call Ethan on an issue
Kinfu in OpenCV in GPU version.
New dense optical flow from Maria is working well, document, samples, great job!
ASUS divers submitted as patches to code.opencv.org
- Fast normal finder
Release April 2
Khronos … lots of nitty gritty detials. This is going to take awhile …
Non-profit
- This is on, many efforts and considerations
- We will make this happen within the month.
Next week Victor in the US
New user page for OpenCV … should be up within a few weeks.
- User site will be: OpenCV.org vs
- developer code.opencv.org
- Look into moin moin for the wiki
3D needs. Filters, normals, voxel filters
Eigen
- Good community. Large library
- 2 branches for optical flow with and without
- OpenCV has small matrices 4×4, 3×3 MatEx structure
  - Add quaternions to MatEx
- Nice geometry features in Eigen
- Yet another dependency
  - It had issues with misaligned data, goes to just C code
    - Might be handled in new version
- Need better support for simple translation
- Add a few routines to MatEx, Gary will put up a page on code.opencv.org to collect what is needed
Need general Levenberg-Marquard
- Simple optimization is needed, some of this is embedded inside machine learning
- Expose, normalize and justify the RANSAC that is there
- lmdir lmdiff
- sparse bundle adjustment
- libmv will give a bunch of these functionalities
Connected components — findContours repeated calls of floodfills
- Kurt has CC for speckle components
VO, Maria is on this.
- Not getting quite the same accuracy as the paper yet (missing details in paper)
- Efficiency looks good, 30-50ms/frame on CPU … could be GPU’d later
Transparent objects
- Ilya has done good work. Gary will collect data from new sets and perhaps from new contacts from Proctor & Gamble
- We will combine several techniques to increase reliability

Ilya

RGB-D odometry [Maria]

Added the RGB-D implementation and the sample to the OpenCV contrib module.
Implemented pure rotation and pure translation cases. Added ability to initialize transformation between frames.
Tested the implementation on the Kurt’s dataset. There is only camera translation along a textured plane there. We don’t have ground truth, but visually the method works well on the data. To compare RGB-D odometry results with given kinfu results we plot trajectories (see trajectories.gif). It seems RGB-D odometry is more accurate on the data, especially if pure translation is estimated. RGB-D odometry for rigid body motion accumulates a bigger error during ~350 frames than for pure translation, but it looks better than kinfu icp results.

Transparent objects [Ilya]

Wrapped SimpleBlobDetector for Python and added a full Python wrapper of findCirclesGrid.
Adapted our old dataset of 5 transparent objects for evaluation of glass segmentation. Dataset images are segmented automatically into glass, background as behind the glass, and other background by using fiducial markers. So ground truth mask is computed. The segmentation is computed using the OpenCV Python interface.
Implemented infrastructure (in Python) to evaluate glass segmentation. It runs the algorithm on all images and computes precision and recall of the glass segmentation. Experiments are on the way.

OpenCV infrastructure and issues [Alexander Shishkov]

OpenCV infrastructure:
- fixed a problem with concurrent work of Android builders on Buildbot (#1274)
- investigated a problem with MinGWLogObserver, tried to fix it, but I still don’t understand the reason of such behaviour (#1303)
- installed newest MinGW/MSYS tools to the new Buildbot server, implemented alpha version of the builder for the FFMPEG wrapper (#1266)
- conversation with the Askbot main developer and different customers of OSQA and Askbot, investigation of Askbot administration. I tend to choose Askbot as framework for OpenCV Q&A forum now.

OpenCV issues:
- added support of XtionPRO/XtionPRO Live, updated a sample and documentation (#1657)
- replaces all references to code.ros.org, opencv.willowgarage.com/documentation, opencv.jp to the actual resources (code.opencv.org, opencv.itseez.com, etc..); fixed some bugs in documentation; updated cheatsheet.pdf (#1672)
- fixed some compilation and documentation errors (#1554)

OpenCV tests and bug fixing [Alexander Reshetnikov]

Added the first version of the new ffmpeg wrapper into OpenCV (for versions from 0.7 to 0.10). Now the used wrapper is chosen depending on ffmpeg version.
Current implementation doesn’t work correctly with C interface (highgui tests with video i/o terminated unsuccessfully with hardware/os exception). So now the main activity is fixing the bugs in C interface of the new wrapper and checking it on existing highgui tests.

Also decided that the highgui module needs an another positioning test where we create specific video (every frame of this is binary image that has byte code of frame number).
This test will be independent of compression algorithms used in different containers and codecs.

Other

March 8 was a national holiday in Russia.

Vadim

Anatoly

Action Items

Gary

Submit ECCV workshop
Put up a wiki page on developer site for all the rotation MatEx operators that we need
- Eigen support
Put in ticket for connected components generalized from Kurt’s speckle filter

Vincent

Call Ethan to get latest opencv in ROS
Help with moving libmv along … use GSoC resources to help??

Alexander

Vadim

Look into moin moin for wiki
Look at pulling connected components from speckle filters (Gary will ticket)

Victor

Kurt

Ethan

From Last Time

Gary

(./) Finish GSoC application http://code.opencv.org/projects/gsoc2012/wiki/Gsoc2012
Submit ECCV workshop
(./) Contact James about limitations in 2D barcode
- Can be lifted, probably not before next release
(./) Ask Kurt about visual odometry 2D+3D

Vincent

ROS integration with latest possible OpenCV trunk
Describe OpenNI for ASUS prime sense

Alexander

Vadim

Talk to Anatoly about stand along kinfu

Victor

Kurt

(./)Talk to Maria on visual optical flow with rgbd
- Great progress, dense optical flow is fast and seems to work quite well
Near term goal is to integrate code into OpenCV
- In contrib now
Even more robust motion estimation with pure translation assumption
Robust detection of visual tracking failure

Ethan

Get OpenCV for Asus prime sense

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-03-19

Agenda

Google Summer of Code. Priorities, see http://code.opencv.org/projects/gsoc2012/wiki/Gsoc2012
- Select projects, spread the word, talk to students
Ethan stuff,
- fast structured normals
- OpenCV for Asus prime sense
Eigen
Rotation convenience
Khronos updates
OpenCV.org progress
Website … how to put up new initiatives such as seemless rotation and translation forms in code.opencv.org
- Rotations convenience functions
  - Quaternians, Rodriges, R,t and just some common functions we face (segmentations, histograms, morphology)
libmv

Minutes

GSoC Mentors start talking with students
RGBD modules dense and another module sparse
- Convenience function for rotating colored pt cloud and then dealing with it as an image (getting a “birds eye view”)
- Modified Kinfu should come from IPI
- Sparse points such as SFM
Maria Kurt VO
With ASUS Action Pro (“kinect camera”) — should work fine now.
- Now you can open and run both
Eigen
- Conversion functions back and forth
- Need sample, tutorial, doc
Khronos
- Nice to get a standard, might take 2 years … OpenCV will comply with it when the day comes
OpenCV.org
- Local guy Ramji who is helping Gary with filing the paperwork
- Making a case to a potential big sponsor
libmv … Will contact back in April … so, not this release.
On line classes for vision vision.class.org by Malik (March 2012)
Website graphics guy and website maintainer
- Use common framework, maybe drupal
Code freeze next week, Beta release of 2.4 next week
Latest OpenCV is now in ROS, no problems integrating … thanks!
Gary is traveling on IPI business all next week, so no meeting next week, see you in 2.

Ilya Lysenkov

Transparent objects [Ilya]

Added computation of full recall-precision curves to evaluate glass segmentation algorithms.

Experimented with the pure voting algorithm and Geodesic Active Contours. Curves showing advantage of Geodesic Active Contours are attached.
Investigated and fixed two bugs in computation of invalid regions and aggregation of SVM predictions.
Removed hard-coded parameters, used the Python tempfile module to generate temporary files, removed debug info writing.
Combining of the RGB-based and depth-based glass segmentation algorithms is in progress.

OpenCV development and tests [Alexander Reshetnikov]

New wrapper supporting ffmpeg versions from 0.7.x was refactored (changed implementation for opening, closing, setProperty and getProperty methods, grabFrame and retrieveFrame functions).
Also it was checked on frame count test for versions 0.7.11, 0.8.10, 0.9.1 and 0.10. It has given following results (actual frame count in every video is 125):

Started the work with new positioning test (see checking new wrapper on positioning and writing tests.
Compilation errors under Ubuntu with ffmpeg 0.7.x were occurred with new wrapper. Some problems have been already fixed. Open issues with it will be closed in the nearest time.

OpenCV infrastructure [Alexander Shishkov]

Buildbot:
- updated one of the test servers to the Ubuntu 11.10 (#1068)
- added time-stamp to the Report page (#1329)
- fixed problems with highgui tests on GUI (#1300)
- added new builder for updating opencv_ffmpeg(32/64) dlls after changes in FFMPEG wrapper (#1266)
- added new builder for test highgui module with different versions of FFMPEG library 0.10, 0.5.1, 0.7.3, 0.7.8 (#1272)

Other:
- preparation to the OpenCV hackaton (instructions, issues selection, discussions)

KinFu and RGB-D odometry [Maria]

Implemented initial version of approximate synchronization of Kinect image and depth. Synchronized RGBD data is necessary e.g. for RGBD odometry. The synchronization is supported in both single threaded and multi-threaded versions. Single-threaded version may be used when there isn’t TBB and frames are grabbed from video (i.e. frames aren’t skipped). But reading/writing .oni video files aren’t supported now. Multi-threaded version based on TBB can be used to synchronize data grabbed from the camera online.
Experiments with the initial implementation is in progress. We are trying to minimize the number of skipped online-synchronized “image-depth” pairs by balancing work between threads and playing with synchronizer parameters.

Vadim

1. A lot of changes made locally on my machine during winter have been finally committed to SVN repository. It includes quite substantial refactoring of features2d: introduction of Feature2D class that combines feature detector and descriptor extractor, using the super-abstract class Algorithm for Feature2D (Algorithm will also be used as a base for StereoMatcher, BackgroundSubtractor, OpticalFlow etc.), and many other simplifcations.
1.1. SIFT & SURF are moved to a separate nonfree module, so that it’s possible to ship OpenCV binaries free of any patented algorithms (i.e. in future it’s planned to move more stuff to nonfree).
1.2. Brute-force matcher is now threaded using TBB.
1.3. Some obsolete stuff, like Calonder, Ferns etc., as well as C interface of the module, has been moved from feature2d to legacy module.
2. The new module “photo” has been added. Right now it just contains inpainting algorithm; it’s planned to add more computational photography stuff there in future (maybe even before 2.4 release).
3. Enhanced variant LogPolar transform (using so-called Blind Spot Model), contributed by Fabio Solari and Manuela Chessa, has been put to contrib module.

Anatoly

Action Items

Gary

Talk to Kurt et al for what we want pragmatically for rgbd

Vincent

Send emails to find mentors

Alexander

Ilya

Vadim

Create Eigen tutorial/doc, sample for using Eigen

Victor

Kurt

Get back to Maria on VO

Ethan

From Last Time

Gary

Submit ECCV workshop
Put up a wiki page on developer site for all the rotation MatEx operators that we need
- Eigen support
(./) Put in ticket for connected components generalized from Kurt’s speckle filter

Vincent

Call Ethan to get latest opencv in ROS
Help with moving libmv along … use GSoC resources to help??

Alexander

Ilya

Vadim

Look into moin moin for wiki
Look at pulling connected components from speckle filters (Gary will ticket)

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-04-10

Agenda

GSOC
Transparency
Release

Minutes

Demo when Vincent gets back: Maria and Ilya working on.
OpenCV resources
OpenCV.org coming soon hopefully.
- Alexander on another project. Eugene replacing.
- Funding OpenCV
  - Sponsor areas
    - Computational photography, SFM,
    - Architectures (ARM … Xilinx)
  - Summer schools
  - Travel support
  - Workshops
  - Let’s say, by the end of May it’s up and running.
GSoC … mentors need to sign up now for OpenCV
Release
- Beta packages out this week!
- Final release by end of April
- ~300 bugs fixed
- Stabilization was a big issue on this release. Thanks to NVidia Tegra team especially!
  - Tegra team is going to go to quarterly releases, then perhaps monthly
Workshop
- Set up website
- End in hackathon
- Develop OpenCV Schwag
  **

Vadim

Summary
2.4 beta has been delayed by 1 more week to make our buildbot “green”, i.e. eliminate the remaining outstanding problems. The goal has been mostly achieved (>15 failures have been fixed); now the packages are being prepared and will be uploaded to SF this week.
~10 more tickets have been fixed last week.
Refactored API of the recently rewritten Expectation Maximization algorithm; fixed several bugs and provided Python interface for it.
Multiple bug fixes in ffmpeg wrapper (also very problematic part of OpenCV); now some of the related tests pass, but some still fail; we will look at it after 2.4 release.
Documentation hackathon has been just started; we fix documentation-related issues in the tracker and add the missing documentation.
Details:

ffmpeg stabilization [Alexander Reshetnikov; Vadim]

Accomplishments:
1) fixed some problems with VideoCapture in new ffmpeg wrapper: (crashes on avi files, incorrect frame reading);
2) closed 4 tickets (# # 1708, 1755, 1743, 1287), # # 1112, 1736 are in progress;
3) minor changes in highgui structure
4) fixed crashes in video writer in the case of odd-size videos; fixed crashes in VideoCapture in non-indexed containers, e.g. .mpg files; fixed finalization of video streams in video writer; fixed some of ffmpeg tests to avoid obviously invalid cases; disabled too verbose logging of ffmpeg codecs;

Plans:
1) further bug fixing in highgui;
2) complete performance tests for video i/o;
3) participate in documentation Hackathon on this week

Intrastructure, eliminating test failures [Alexander Shishkov]
- fixed 9 OpenCV tests
- installed plugin for hiding emails by default on OpenCV Redmine, changed this setting for current users
- added automatic Redmine reboot with insufficient memory (memory leak in Redmine)
- fixed environment for test_ffmpeg and coverage builders on Buildbot
- modified WinSuperPack builder (fixed some bugs, added additional files to pack)
- modified online documentation theme (removed tool-tip below QuickSearch)
- fixed ##227,1752

Other:
- bought BNC-F connectors
- delivered certificate about “Argus Procam” software registration
- agile and OpenCV+Tegra meeting

Expectation Maximization improvements [Maria; Vadim]

- Found and fixed a few bugs in kmeans.
- Refactored interface of cv::EM. It has a python wrapper now.
- Started to solve the tickets on EM (#1407, #1581).

Plans
More tests on EM, doc on EM.

Misc

Ilya was on vacation last week.

Action Items

Gary

Vincent

Alexander

Ilya

Vadim

Victor

Kurt

Ethan

From Last Time

Gary

Talk to Kurt et al for what we want pragmatically for rgbd

Vincent

Send emails to find mentors

Alexander

Ilya

Vadim

Create Eigen tutorial/doc, sample for using Eigen

Victor

Kurt

Get back to Maria on VO

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-04-17

Agenda

Mentors, choose students here prior to April 20th!.
- We need a repository where we can see this code
Linemod
- Go over details
Release
New website
OpenCV.org, OpenCV non-profit foundation
- All documents are filed, waiting for Employer Identification Number and ficticious name certificate to be returned.
Workshop status?

Minutes

Choose your students.
We have a separate project on code.opencv.org for GSoC projects. We can move things in from there.
- Python samples should go right in
- Docs and tutorials go right in
Linemod
- Post issue in bug tracker
- Better model capture maybe using Maria’s
  - Knowing if the camera is super useful in any case (in contrib)
    - 2 images; 2 depth maps/ 2 masks (regions to be mapped) odometry
Code.opencv.org was attacked
- Got access to server
- This attack was from China. Stop it.
- Alexander cleaned it up
Beta release is out. Final release by end of month
- Change log https://github.com/opencv/opencv/wiki/ChangeLog
- On ROS side, Fuerty. Some build problems with the farm,
  - SURF and SIFT was moved to “non-free” and this messed up the ROS build. But OK
- but Vincent will sync final release to this
New website for users
- Design is ready
- Web programmer was hired away, hiring another web company. Probably 2 weeks.
- Contests etc … need guy on this. Probably case by case
- Itseez hiring web programmer, can use this guy part time initially
- Web framework will be a well known engines (maybe Drupal)
- Redmine for code.opencv.org
Transparent objects
- Experiments show we can recognize in clutter
  - Robot can grasp one by one (but robot has to be aware of where other objects are)
  - Does this make sense for June demo at Willow? Most of the ROS stuff is done for this
    - Probably go for it
- Vincent coming end of May
Last week was documentation hackathon
OpenCV docs have fewer broken links etc

Vadim

OpenCV 2.4 beta is out [OpenCV team]
OpenCV 2.4 beta is finally out. Here is the announce:
http://tech.groups.yahoo.com/group/OpenCV/message/87261 and
here is the changelog: https://github.com/opencv/opencv/wiki/ChangeLog

OpenCV Documentation hackathon is over [Ilya and others]
Last week we had documentation hackathon. Since beta 2.4 has been delayed by one week, and so we had some overlap in schedule, it was not as effective as the bug fixing hackaton. Still, we closed 24 documentation-related tickets in our tracker, 19 of which was closed by Ilya! The documentation will further be improved until and after the final 2.4.

EM improvements [Maria, Ilya, Vadim]
Maria, with help from from Ilya and Vadim, further improved EM implementation. The new implementation has automatically generated Python interface and much better interface. Several EM-related tickets have been closed as a result of this refactoring: #964, #1340, #1351, #1361 (according to our bug tracker, EM was quite commonly used and yet quite problematic part of OpenCV; hopefully, most issues are now resolved).

Improving ffmpeg wrapper [Alexander Reshetnikov]
Implemented 2 new highgui performance tests for video i/o.
Closed the tickets: #1771, #1790, #203, #1736.

Other bug fixes, improvements [Maria, Vadim]
Other mll and features2d bugs/improvements:
Improved MLData to read files with different space count between columns.
Fixed the ERTrees name in saving/loading;
Fixed #1798, #1795, #1805.
Fixed crash in linemod detector, reported by S. Hinterstoisser.
Extended morphological operations to handle more than 4-channel images, 16S and 64F formats, added test that checks supported formats in filtering operations.

Infrastructure [Alexander Shishkov]
Buildbot:
Updated Windows SuperPack builder (added the skipped include files, fixed incorrect include files for the stitching,…).
Transferred Android builders to MacOS.
Added the OpenCV testing with FFMPEG 0.10.2.
Fixed the problem with svnsync on Buildbot.

OpenCV 2.4 beta packages for Windows/Unix are prepared and uploaded to the SourceForge.

Eliminated the consequences of the attack on the code.opencv.org site (checked for rootkits, removed enemy processes, added the using SSH keys, added IP list for SSH).
looks like we are getting popular :)

Added DNS docs.opencv.org (link to opencv.itseez.com).

Reviewing GSoC proposals [Vadim, Anatoly]
Finished reviewing all the submitted proposals. This year we’ve got a lot of proposals, so while the majority of proposals can be easily rejected, there are enough good ones to choose from.

Action Items

Gary

(./) Start a opencv.org “wish list”: What we can do with what funding
(./) Get P&G objects for transparent test

Vincent

See about converting Maria’s tracking to a generic model capture of camera pose. Talk to here
Sync final release with ROS

Alexander

Ilya

Vadim

Write a hello world application for GSoC 2012

Victor

Talk to NVidia about workshop

Kurt

Ethan

From Last Time

Gary

Vincent

Alexander

Ilya

Vadim

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-04-24

Agenda

Name reserved for OpenCV.org
GSoC … Mentors discuss with plans interns
- Focus for object rec planning
- Distribution tracker
Release
New website
Workshop
- ECCV
Post 2.4 plans
- Transparent objects

Minutes

New paper by OpenCV dudes to be presented at RSS:

“Recognition and Pose Estimation of Rigid Transparent Objects with a Kinect Sensor” by Ilya Lysenkov, Victor Eruhimov and _Gary Bradski, RSS 2012.

OpenCV.org is “taken” probably by us. Open Source Computer Vision Library is reserved. Will be filing this week.
GSoC
- Training classifiers
  - Silverware
    - Tracking from last year.
    - There are classifiers there: Multiple instance learning: mil-boost
    - RSS workshop rgbd sensors, plane finding
    - Shape context into the cascade classifier …
    - LARK?
Release. Beta out now
- 2.4 final will be out this coming weekend

New website for users (existing new website for developers is /|here)



Update on the opencv.org site. We have started working with the vendor last week, we expect the first version of the site by the end of the next week and the final version one week later. Given potential delays and time for content creation, it is reasonable to expect site rollout by the end of May.

Hot or not in vision http://www.cvpr2011.org/statistics
- Blob segmentation
Or, create way for users to drive what they want
- http://brainstorm.ubuntu.com/

Instead of yahoo groups, switch to askbot so that user questions are answered quickly
- Then mailing list would be more about announcements: “Hey, I released this”.
Tutorial at CVPR …
- Need to submit also to ECCV
- $2K award for best open source code from CVPR submitted to OpenCV.
- RSS July 9th … need travel funds
Vincent will be local May 21st, Ilya will probably come too
Post 2.4 plans
- RGBD module
  - Linemod
  - structured arrays
- Split OpenCV into many smaller modules by topic
  - Output can be single large library, or small custom fit libraries
    - Keep dependencies to a minimum to allow the user to minimize space if they want
    - Naive users or those who don’t care can produce single dynamic or static library
- Computational photography will be a big topic
  - Samples should be mobile demos in iOS and Android
- Video stabilization, tracking, effects, color or white balance, augmentation
- Mobile application: Food detection … this is hard
- Hot or not in vision http://www.cvpr2011.org/statistics
  - Blob segmentation
- Or, create way for users to drive what they want
  - http://brainstorm.ubuntu.com/

Anatoly

Accomplishments & statuses:



1)      Video Stabilization.

·         Fixed trim ratio estimation bug for the case of homographies motion model.

·         Ported some parts to GPU (overall speed-up ~3x)

o   Added GPU version of wobble suppressor (14x faster than CPU version). Replaced DECOMP_SVD in global motion estimation methods with DECOMP_NORMAL|DECOMP_LU (it works about 2x faster).

o   Added GPU version PyrLK based global motion estimator (overall speedup of global motion estimation is ~8x, that's about 22 seconds for 500 frames).



•     Started working on local outlier rejection method described in the YouTube stabilization method paper (going to check whether it can improve motion estimation quality).


3)      OpenCV GPU.

·         Integrated VideoWriter_GPU and VideoReader_GPU to OpenCV trunk, results see below. Added tests/docs for GPU decoding/encoding.

·         Tested compilation with CUDA 4.2 under Windows and Linux.

o   Issue: One BruteForceMatcher_GPU test fails for 4.2 (match with collection with mask with L1 distance). Need to investigate.



·         Tested performance. The numbers below include NVidia Encoder/Decoder library performance together with our code.


Encoding:
Windows only:            1920x1080                          4096x4096
CPU [XVID]      21.3 ms/frame [FPS :  46.9]      175.4 ms/frame [FPS :   5.7]
GPU [H264]      1.8  ms/frame [FPS : 546.3]        7.4 ms/frame [FPS : 135.5]
For 1920x1080 default parameters was used, for 4096x4096 bitrate was increased.

Decoding:
Windows:       1920x1080 [MPEG4]                 1920x1088 [H264]
CPU       4.6 ms/frame [FPS : 218.4]      2.8 ms/frame [FPS : 359.9]
GPU       2.7 ms/frame [FPS : 350.6]      4.5 ms/frame [FPS : 224.4]

Linux:         1920x1080 [MPEG4]                 1920x1088 [H264]
CPU      10.9 ms/frame [FPS :   91.8]            <doesn't work>
GPU       1.0 ms/frame [FPS : 1002.0]        7 ms/frame [FPS : 142]

VideoReader_GPU can't open 4096x4096 video file. (it seems NVidia doesn’t support).




Plans:

·         PCL: Continue Skeleton Detection optimization with/without GPU.

·         VS: Continue working on local outlier rejection in the global motion estimation step. Try using L1 minimization (using LP simplex solver) instead of L2 when estimating global motion (as described in the youtube stabilization method paper).

·         OpenCV: Fix BruteForceMatcher_GPU. Investigate issues #1831,  #1823.

Ilya

Transparent objects [Ilya]

1. Tested our pose estimation algorithm in transparent clutter to get a proof of concept whether a robot can clean out a table with transparent objects.
So several objects were placed on a table, our algorithm returned pose of the recognized object with the best confidence, this object was removed from the table and this procedure was repeated. Results are shown in the animated GIF:

2. Optimized the pose estimation algorithm. Now silhouettes of a model are computed using dot products with normals instead of morphology. Re-scanned objects to get models with more accurate normals (thanks to Maria for granting access to her computer with kinfu). Experiments show that the optimized algorithm is almost 2x faster than the old algorithm but accuracy is worse. It means we can combine two algorithms together by finding the single pose with the optimized algorithm and then refinining it with the old algorithm to get fast pose estimation with good accuracy. Results are shown in the table:

-----------------------------------------------------------------------------------------------------
|        |  Mean translation error (cm)  |   Mean pose estimation time (s)    |         Speed-up    |
| Object |-------------------------------|------------------------------------|---------------------|
|        |  old   | optimized | combined |   old   |  optimized  |  combined  |  optimized |combined|
|---------------------------------------------------------------------------------------------------|
|bank    |  0.41  |   0.68    |   0.40   |   31.1  |    20.4     |    20.4    |     1.5    |    1.5 |
|bottle  |  1.41  |   1.33    |   1.37   |   22.6  |    10.7     |    12.1    |     2.1    |    1.9 |
|bucket  |  0.34  |   0.71    |   0.45   |   55.7  |    26.4     |    26.3    |     2.1    |    2.1 |
|glass   |  0.48  |   0.57    |   0.50   |   19.7  |    10.4     |    10.7    |     1.9    |    1.8 |
|---------------------------------------------------------------------------------------------------|
|mean    |  0.66  |   0.82    |   0.68   |   32.3  |    17.0     |    17.4    |     1.9    |    1.8 |
-----------------------------------------------------------------------------------------------------

So the final algorithm has the same accuracy as the old one but it is 1.8x faster.

Other:
Our paper "Recognition and Pose Estimation of Rigid Transparent Objects with a Kinect Sensor" by Ilya Lysenkov, Victor Eruhimov and Gary Bradski has been accepted at Robotics: Science and Systems Conference (RSS) 2012.


RGBD Odometry [Maria]:

Started to work on the model capture demo. Before integrating the RGBD Odometry I was going to run the existing capture pipeline on a dot pattern or an ORB template. It is in progress now because there are problems with setting up the working environment and import/compilation/build of the pipeline.

Vadim

OpenCV stabilization & bug fixing [Alexander Reshetnikov, Vadim]

Closed 15 tickets, including 5 related to video capturing.

Further EM improvements [Maria Dimashova]

- added doc on cv::EM, moved doc on CvEM to legacy/doc (#8096);
- resolved failure of EM accuracy test on ubuntu x86 (32-bit);
- resolved failure of EM save-load test.

Background segmentation improvements [Vadim]

Cleaned up MOG2 code; Threaded it using TBB.

OpenCV infrastructure [Alexander Shishkov]

OpenCV Wiki from opencv.willowgarage.com has been converted to the redmine format (temporarily available here: http://redmine.alekcac.webfactional.com/projects/wiki/wiki/index)
Extended buildbot with new tests (nonfree, stitching & photo modules), all performance tests now run in “sanity check” mode to do some more regression testing of the library.
Moved windows superpack builder to a dedicated workstation and brought the build time from 23 hours down to 3 hours! (big thanks to Andrey Kamaev from Tegra team)

Proposal for Vision standard API [Victor, Vadim]

updated our proposal on vision API standard (aka OpenVL) for Khronos F2F in Dublin. The proposal is based on the collaborative work with experts from ARM and Samsung. Victor has presented this proposal on Tuesday.

Action Items

Gary

Talk to Cat about workshop support for OpenCV tutorial
Apply for ECCV tutorial
Kick people to talk to GSoC interns now. Computational photography is high priority
(./) Make sure Alexander Shishkov takes into account something like http://brainstorm.ubuntu.com/ in opencv site

Vincent

Alexander

Ilya

Vadim

When 2.4 final is out, tell Vincent

Victor

Get plan for CVPR tutorial in place

Kurt

Ethan

From Last Time

Gary

(./) Start a opencv.org “wish list”: What we can do with what funding
(./) Get P&G objects for transparent test

Vincent

See about converting Maria’s tracking to a generic model capture of camera pose. Talk to here
Sync final release with ROS

Alexander

Ilya

Vadim

Write a hello world application for GSoC 2012

Victor

Talk to NVidia about workshop

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-05-01

Agenda

Release
New website
Workshop
- ECCV
Post 2.4 plans
- Transparent objects

Minutes

Release 2.4 is out!!! 2.4 Packages log]], [[http://sourceforge.net/projects/opencvlibrary/files
- On the ROS side it is released as well (ROS Fuerte)!
New website
- Designer is on vacation, mock site is up
CVPR workshop (June . Now that 2.4 is out, we can start preparing for this
- iOS is biggest risk because we don’t have someone on this. Vadim is on this.
- ECCV workshop proposal is in.
Post 2.4
- Tegra team has grown: Want to release each month
  - They can help from infrastructure point of view until (hopefully if) OpenCV foundation is formed.
- Short term: Mobile focus to CVPR
- Computational photography
  - Collage, video summary
- Many more modules
  - Library and application side
  - OpenCV apps site. Itseez seeds
    - Might turn into an apps store for OpenCV.org
    - Standard directory structure with email
    - Image stitching started as an app then turned into a module
  - Make a test case. Computational photography is one.
GSoC
- SfM … libmv 3rd guy willing to work for fun from Hartley’s group

Vadim

OpenCV 2.4.0 is out [All the team + GPU + Tegra guys]

Finally, we have shipped it! Thanks to everybody who participated!
The changelog is here:
https://github.com/opencv/opencv/wiki/ChangeLog
The packages are here:
http://sourceforge.net/projects/opencvlibrary/files

The announcement was sent to the mailing lists.

Bug Fixing [Alexander Shishkov, Alexander Reshetnikov, Vadim, GPU&Tegra teams]

~20 more tickets have been closed during the last week. A few more serious bugs in ffmpeg wrappers have been fixed and now it [finally?] works quite well.

Misc.
Holidays: as Kurt once said – lot’s of national holidays is good, it helps to unite the nation. These days we have a bunch of such opportunities. This Monday and Tuesday, as well as the next Monday and Tuesday are holidays, while the last and this Saturday are working days.

Anatoly

Accomplishments & statuses:

1) Video Stabilization.

· Implemented local outlier rejection. Can’t say it increased motion estimation stability a lot.

· Implemented linear programming based motion estimation (it minimizes L1 norm, which is less sensitive to outliers). It works more stable (significantly) than RANSAC version, but slower.

· Replaced CV_RANSAC with CV_LMEDS in findHomography. Visually, it increased stability a bit on some videos.

Issues:

· OpenCV trunk can’t be compiled using gcc 4.6.3 because of internal error in gcc (bug #1860). The bug was revealed during compiling videostab module.

3) OpenCV GPU.

· Refactored gpu::BruteForceMather (made it similar to new cv::BFMatcher).

· Fixed NVidia/NCV.Visualization test (drawRects in CascadeClassifier_GPU).

· Fixes bugs #1831, #1823, #1759. Investigated and fixed bug in Face Detection that was committed by Tegra team when they were fixing warnings everywhere.

4) OpenCV Demo Pack preparation for GTC

· Fixed several bugs (like ‘window size can be set to zero at real time’)

· Updated optical flow sample, split it to 2 samples – dense and sparse, added frame interpolation demonstration to dense sample.

· Stereo demo was split into two:

o BM match only (CPU vs. GPU), which runs on high resolutions.

o BM vs. BP vs. CSBP which runs on lower resolutions due to memory limitations.

Plans:

· Prepare datasets and stitching demo. Possibly rework feature detection demo.

Plans:

· OpenCV: Finish with GTC demo pack (update feature detection/matching samples ).

Other:

Alexey will be on vacations 2-31 of May. After he will joint Tegra team. Instead, Marina will come to GPU project from Tegra team.

Action Items

Gary

Mail ideas to Anatoly Baksheev
Mail Ilya pictures and interact. Thursday.
Contact Carol about GSoC who may be trying to be part time.

Vincent

Alexander

Ilya

Vadim

Find out status of new user website

Victor

Kurt

Ethan

From Last Time

Gary

~~Talk to Cat about workshop support for OpenCV tutorial~~ Waiting on EIN
(./) Apply for ECCV tutorial
(./) Kick people to talk to GSoC interns now. Computational photography is high priority
(./) Make sure Alexander Shishkov takes into account something like http://brainstorm.ubuntu.com/ in opencv site

Vincent

Alexander

Ilya

Vadim

When 2.4 final is out, tell Vincent

Victor

Get plan for CVPR tutorial in place

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-05-08

Agenda

CVPR tutorial (May 15th Early registration deadline
Travel
Foundation: EIN number received
Website
- Contribution
GSoC: May 21 official start, can start earlier.
- One student on agglomerative mean shift segmentation
New stuff
- Inverted index
  - Need a template for this

Minutes

Use fieldhouse.com for schwag (used by ROS)
- CVPR committee about shipping stuff there
- May 15th early registration deadline
OpenCV.org, now have EIN number
- Click for donations needs to be set up. Use Paypal probably upgraded to business account
- Use fieldhouse for schwag
- Need to set up security for new site
Khronos
- Ponderous discussions of API …
CVPR Tutorial. June 17th. May 15 Early registration deadline
- We want to offer lunch etc, but have to wait until OpenCV has an account and gets funded.
- Put up registration and try to get it going. Deadline for prior registration needed.
- How much time will fieldhouse take to make schwag.
  - Probably a week or 2
- Deadline for our workshop registration will be June 3rd
Travels
- 2 people traveling to CVPR. Victor and Vadim for the tutorial.
- Ilya and Maria over
- Paper accepted for RSS (Ilya, Victor, Gary). Ilya going to RSS to present. Early in July 9-13
GSoC Start your engines interns May 21
- SfM One from Hartly group.
  - b2solver released by Google
- Python bindings (has 2nd internship)
  - If there are samples people want, name them
- Super resolution for email exchange
  - state of the art dense optical flow
  - super resolution for computational photography module
Website
Computer vision mailing list. We should post there.
- imageworld, http://www.vislist.com/

Ilya

Transparent objects [Ilya]
Solved the problem of false positive detections with a very small scale. Tried to make scale normalization of the cost function theoretically grounded by using a Bayesian formulation but it didn't work well in practice. Ended up with simple thresholding on the minimum scale.
Created a new dataset with transparent clutter and ground truth poses to evaluate our algorithm in clutter. The previous optimization with normals doesn't work well on this dataset so the algorithm was optimized by other means (for example, suppression of poses after alignment to a table plane, removing iterative refinement of the closed-form solution, sampling a test silhouette). Results on this dataset (the success rate is the proportion of cases when the translation error is less than 2cm):

-------------------------------------------------------------------------
Object Success rate Mean translation Mean pose estimation
error (cm) time (s)
-------------------------------------------------------------------------
bank 0.86 0.4 17.5
bottle 0.98 1.1 13.1
bucket 1.00 0.2 18.0
glass 0.88 0.6 8.7
sour_cream 0.95 0.4 13.6
-------------------------------------------------------------------------
mean 0.93 0.5 14.2
-------------------------------------------------------------------------

So results are slightly worse than pose estimation in clutter-free environment but still they are good enough for grasping. The optimized algorithm works the same time as on the dataset without clutter.

Also accuracy of initial poses computed by geometric hashing was evaluated alone. The accuracy is measured as the minimum Chamfer Distance between projected silhouettes and the ground truth silhouette. Results (the success rate is the proportion of the cases when this Chamfer Distance is less than 5 pixels).

------------------------------------------------------------------
Object Success rate Mean Chamfer Mean number of
Distance (px) initial poses
------------------------------------------------------------------
bank 1.00 0.7 27.0
bottle 0.98 1.1 41.7
bucket 1.00 1.8 33.8
glass 0.66 1.5 20.2
sour_cream 1.00 0.8 32.0
------------------------------------------------------------------
mean 0.93 1.2 30.9
------------------------------------------------------------------

So geometric hashing produces the accurate initial pose for our algorithm in the most of cases.

Other:
I got the Australian visa to attend RSS 2012.

RGBD odometry [Maria]
Built the ecto-based capture pipeline. It required some effort: to install ubuntu 10.04 (my installed 10.10 isn't supported by ros fuerte), to install ros fuerte, to fix several build problems of the pipeline - thanks a lot to Vincent for this.
Played with the pipeline scripts which are used for model capture based on orb template.
Experimented a little with the pipleline sample of rgbd odometry. The algorithm diverges in several cases:
1) there are moving objects in the scene or changing light (the algorithm supposes a statical scene)
2) there is a motion blur
3) rgb and depth are not synchronized (the ecto openni wrapper doesn't support such synchronization, approximated by time stamps).
Worked on improving the rgbd odometry accuracy:
1) Median filtering of rgb and depth increases slightly the accuracy on the TUM dataset.
2) Implemented a cpu version of icp from kinfu (only camera pose estimation, without a scene volume). Initial implementation uses a random subset of close points and works ~0.15s on one core of core i5 (I think it can be optimezed up to rgbd odometry time ~0.08s). The implemented icp gave the relative median pose error on the TUM dataset comparable with the rgbd method, but its trajectory was further from ground truth than rgbd's and kinfu's ones. Tried to combine rgbd and icp (to minimize not only photo consistency error but spatial error too) but again the way that improves the merged approach quality on the TUM dataset wasn't found. Will try the icp for model capturing (maybe it will be useful).
3) Started to study BA from Vincent.

Other:
We have 2 national holidays these weeks.

Vadim

Closing tickets [Vadim Pisarevsky]
6 tickets have been closed, mostly orphan ones.

opencv.org [Alexander Shishkov; our (contracted) web masters]
The first draft of opencv.org is ready (the link was sent by a separate e-mail; not to be disclosed yet).
Thanks everybody who provided feedback on the design and content.

OpenCV roadmap meeting [OpenCV, GPU & Tegra teams]
There was the meeting at Itseez regarding the possible and feasible release schedule for this year. I can not find any summary sent, but here is a short summary:

OpenCV 2.5 is expected to be out around Nov 1st.
Before that there will be at least two 2.4.x releases, may be more.
Also, we plan to make 2 hackathons – in the end of June and in September.
After some friendly conversation we seem to agreed to migrate to git in order to simplify the simultaneous work of 3+ teams working on OpenCV. One of the primary reasons to choose git vs mercurial – github and its “pull request” feature, using which we can react on most feature requests and bug reports with known solutions in a timely and friendly way – “please, do it yourself and submit a pull request”. It’s definitely better than leaving such tickets without attentions for years.

Action Items

Gary

Call Silicon Valley bank
Send Ilya transparent items photos

Vincent

Contact Bianca about fieldhouse and producing runs of schwag
Push for travel access

Alexander

Ilya

Vadim

Victor

Get Vincent’s comments in to website designer

Kurt

Ethan

From Last Time

Gary

(?) Mail ideas to Anatoly Baksheev
Mail Ilya pictures and interact. Thursday.
(./) Contact Carol about GSoC who may be trying to be part time.

Vincent

Alexander

Ilya

Vadim

Find out status of new user website

Victor

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-05-15

Agenda

CVPR tutorial (May 15th Early registration deadline
Travel
Website
- Contribution
GSoC: May 21 official start, can start earlier.
New stuff
- Inverted index
  - Need a template for this
OpenCV world
Box2D demo with OpenCV?
Transparency
- RGB and depth images of bottles
Vincent is at ICRA, Victor out with stomach bug. Last week had 2 days of hollidays … so, minimal week

Minutes

CVPR Tutorial
- Android part is being helped by NVidia team
- Vadim looking at iOS — porting Android demos to iOS.
  - iOS install instructions are here
New proposed website site: http://opencv.top7y.ru/
- Provide feedback on it
- Want something like opencvmeetingnotes
- Make youtube videos of functionality in new releases. For example, the creation of 2.4 is here
  - Expand opencv projects channel
Need to fill a GSoC vacancy
Travel
- Ilya had Visa. Might come for June
- Maria will get a visa in July
OpenCV world
- There are already 20 modules an growing
- Nice modularlity but can make linking a pain
- World is a module that includes all others like Frodo’s one ring Makes linking easier
  - Can be customized to contain only the modules that user wants to make it smaller
  - Will split some modules into even smaller modules to make them even smaller
    - Needs to be documented
    - Installation guide. Important especially for Windows OS
Transparancey
- Optimizing … sped up 3x so ~5s/frame
- Works in clutter
- RSS presentation (Ilya is going but paying for 50%)
Intel likely to start getting involved
- Application engineers from NN center

Vadim

Only 3 working days (Thursday, Friday and Saturday) this week due to Russian hollidays.

Still, here is the progress:

Bug fixing [Vadim et al]
2 bug in OpenCV 2.4 have been fixed: #1918 and #1931.
Other flavors of cv::split and cv::merge have been added (to use InputArray & OutputArray) to support vector<Mat >, as requested by Vincent.

Also 5+ bugs have been fixed by other teams at Itseez that work on OpenCV.

opencv-world [Vadim and Andrey]
The new super-module modestly called opencv_world has been introduced. It combines all the other modules selected by user and thus it’s much more convenient for redistribution, comparing to ~20 shared libraries libopencv_core, libopencv_imgproc etc. The module takes ~10-15Mb and it is linked from the same object files (their union) as the other modules, so the overall OpenCV compile time increased by just a few seconds. Big thanks to Andrey Kamaev from tegra team who came with the implementation idea and then significantly improved and fixed the initial version of the cmake script.

iOS port [Vadim]
I started playing with iOS port of OpenCV; With very minor modifications OpenCV 2.4.0 was built for iOS using the instructions from
http://docs.opencv.org/doc/tutorials/introduction/ios_install/ios_install.html
Next steps would be to port some of Android samples to iOS.

OpenCV site [Alexander + our contracted web masters]
The final draft of opencv.org has been created: http://opencv.top7y.ru/
You are welcome to check it out and provide feedback.

Welcome Evgeny!
Welcome the new member of our team, Evgeny Talanin, who joined OpenCV team last week. He will replace Alexander Shishkov. However, there is a good chance that Alexander will stay with us till the end of the month.
Meanwhile, Alexander Reshetnikov is on vacation till the beginning of June.

Anatoly

Accomplishments & statuses:

1) OpenCV GPU

· Fixed build under MacOS (disabled video decoding, it’s not supported). Closed tickets #1754, #1846.

· Prepared draft of work plan for next 2 months.

Issues:
· GPU team still doesn’t have any Kepler to test it with GPU module.

· Some compilation errors under Mac (for static libs), to fix this need to wait until Mac Pro from our build farm will be free.

· We only check compilation for MacOS, but never executed tests there, since we don’t have Mac with CUDA capable GPU. Regarding PCL, we even never compiled it on Mac.

2) Background substation

· Investigated current OpenCV CPU implementation, learned paper. (We going to port both current functionality and new VIBE).

Plans:
· Continue work on background substation, implement first GPU version. Test if merging (using Python script) all cu-files to one reduces GPU module binary size.

· Migration to Git. Prepare mini-training for Itseez teams.

Ilya



Transparent objects [Ilya]:

Optimized the pose estimation algorithm. A lot of optimizations were implemented e.g. caching results of computations and sharing them in pose estimations for different initial poses, computing a jacobian for inliers only, scaling an image to get an approximate final pose and then refining it on the original image, using downsampled edgels in geometric hashing.

Results (the success rate is the proportion of cases when the translation error is less than 2cm):

		Mean	Mean pose estimation time (s)
Object	Success	translation	-——————————————	Speed-up
	rate	error (cm)	Old	New

------------------------------------------------------------------------------------ | bank | 0.88 | 0.4 | 17.5 | 6.8 | 2.6 | | bottle | 0.91 | 1.2 | 13.1 | 4.3 | 3.0 | | bucket | 0.98 | 0.2 | 18.0 | 6.5 | 2.8 | | glass | 0.92 | 0.7 | 8.7 | 2.9 | 3.0 | | sour_cream | 0.97 | 0.4 | 13.6 | 4.7 | 2.9 | ------------------------------------------------------------------------------------ | mean | 0.93 | 0.6 | 14.2 | 5.0 | 2.8 | ------------------------------------------------------------------------------------

So the optimized algorithm works with approximately the same quality but it is 2.8x faster.

RGBD odometry [Maria]:
Experimented with mesh_object. Thanks to Vincent for resolving the problem with it.
Refactored the merged method rgbd + icp. Evaluated it on the TUM dataset again (see the graphics of translation errors in the attachment).
Results (the description of the errors can be found here: http://cvpr.in.tum.de/data/datasets/rgbd-dataset/online_evaluation):

RGBD ICP RGBD + ICP
Absolute trajectory error (median), m 0.387107 0.443915 0.066765
Relative transnational error (median), m 0.012085 0.013843 0.015936

RGBD has the smallest relative median error (frame-to-frame). But it also has one bad pose breaking the camera trajectory, so it has the high absolute trajectory error. RGBD + ICP has slightly worse relative median error, but its absolute trajectory error is much less, it does not break the trajectory. So on the TUM dataset merged approach is more stable than pure RGBD. But the dataset has motion blur and it’s the most likely reason of the one bad pose. ICP has several poses breaking the camera trajectory.

Other:
Maria has applied for the USA visa. The interview will be on July 3 (it is the nearest free time).

Action Items

Gary

Ask Adrian ab Eduard or KuangQi
P&G transparent items to Ilya

Vincent

Alexander

Ilya

Vadim

Mail Eduard if he can do a project
Ask if Mooly Eden is involved in NN contacts

Victor

Kurt

Ethan

From Last Time

Gary

(./) Call Silicon Valley bank
Send Ilya transparent items photos

Vincent

Contact Bianca about fieldhouse and producing runs of schwag
Push for travel access

Alexander

Ilya

Vadim

Victor

Get Vincent’s comments in to website designer

Kurt

Ethan

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-05-22

Agenda

Opencv.org
Workshop
GSoC
Future plans

Minutes

GSoC
- CODING BEGINS!
- Eduard
- Repo for code .. Alexander says we’ll make one in a couple of days using Git
Tutorial
- https://sites.google.com/a/itseez.com/mobile-vision-with-opencv/
- Get this linked to CVPR site
- Vadim has Apple developer account, working on iOS
  - Port samples to iOS
OpenCV.org
- Prototype website:
  - http://opencv.alekcac.webfactional.com/
  - Needs answers and wiki
- donations etc for the site
- Will use Citi
- Get money in for email address
- Documentation
- Google is $50/year per account
  - Itseez expense for now?
  - Go with Google or create our own server. Google easier with groups
    - Transition to paid account later
  - Notifications can be done with google groups
  - need account to log in
RGB+D odometry
- Maria Kramer’s method for dense images.
- Frame to frame odometry that is in Kinfu.
- Playing with how to weight these
- libmv? How much overlap. They are sparse. GSoC students on.
- Go with Maria’s stuff and when it breaks go to sparse
- SFM sparse package
- Extend the stuff to more views, more camera types. No noise and noise. Missing entries.

Vadim

Bug fixing [Vadim, Alexander Mordvintsev]
closed 6 tickets. made FaceRecognizer classes derivatives of Algorithm, so that different properties can be set using the unified interface.

GSoC
The students have just started the work; there is no meaningful reports yet; the work on iOS samples started; Alexander Mordvintsev ran existing Python samples and fixed a few problems in the samples, in the wrappers and in OpenCV itself to make sure the samples run.

OpenCV site [Alexander Shishkov + our contracted web masters]
Another final draft :) of http://opencv.alekcac.webfactional.com/. It addressed some of the concerns and requests, but not all. Still, the internal representation is pretty simple (wordpress-based) and can be easily edited by us (which needs to be done anyway). Today we had meeting at Itseez regarding the further steps.

Other than that, wiki has been partially converted to redmine format: meeting notes (https://github.com/opencv/opencv/wiki/Meeting_notes), Changelog (https://github.com/opencv/opencv/wiki/ChangeLogHistory)

Anatoly

Accomplishments & statuses:

1) OpenCV GPU

· Tested OpenCV GPU 2.4 compilation for MacOS manually (since buildbot works with trunk), fixed probable compilation errors (which one user reported, but we can’t reproduce), closed ticket #1901.

· Performance tests framework.

o Yes, we can use them to measure CPU-GPU speed-ups. To do this, we’ve implemented 90% of GPU test cases using CPU functions (in order to compare them using the python scripts). Added tests cases for 24 uncovered functions.

o Going to finish remaining cases, prepare complete log and send to NVIDIA tomorrow. There are some surprising results for us.

3) Background substation

· Implemented first GPU version of FGD Foreground detection (~ 5-6x speedup).That is old OpenCV algorithm, not VIBE.

4) Other

· Hiring. Interviewed one candidate in ‘PCL for Tegra’ project. He is good, probably will hire him.

· Spend some time for learning GTC videos, posters, etc.

· Prepared materials for Git mini-training, plan to conduct it when Git hosting is ready.

Plans:
· OpenCV performance testing, investigation, optimization.

· Continue with FGD Foreground detection on GPU (test implementation on different cards and different OS, try to optimize implementation).

· Recording BodyBarts video for NVIDIA (after fixing acquisition problems). Continue work on Pseudo Convex Hull.

Ilya

Transparent objects [Ilya]:
Finished the first draft of the camera ready paper for RSS 2012. Created a private git repository for it so we can easily collaborate on the paper.
Searched and studied new papers related to transparent objects. There are 2 interesting papers:
1. Zhong Lei, K. Ohno, M. Tsubota, E. Takeuchi, and S. Tadokoro. Transparent object detection using color image and laser reflectance image for mobile manipulator. IEEE International Conference on Robotics and Biomimetics (ROBIO), December 2011. Light detection and ranging (LIDAR) data are used to segment unknown transparent objects: highlight spots from a RGB image are used to find candidate areas of transparent objects and then the GrabCut segmentation algorithm is applied to a depth image and a laser reflectance intensity image to compute the final segmentation. The experimental setup consist of a 2D LIDAR device, a registered RGB camera and a pan-tilt unit so it is more complex than Kinect which we use.
2. Meriaudeau, R. Rantoson, D. Fofi, and C. Stolz. Review and comparison of non-conventional imaging systems for three-dimensional digitization of transparent objects. Journal of Electronic Imaging, May 2012. This paper covers different approaches to scanning transparent objects which were published after the excellent previous review on this topic: I. Ihrke, K.N. Kutulakos, H.P.A. Lensch, M. Magnor, and W. Heidrich. State of the art in transparent and specular object reconstruction. In EUROGRAPHICS 2008. Results are still the same: there is no robust and reliable technique to scan transparent objects in general case.

Other:
Bought flight tickets to/from the USA. I still have my passport in the UK visa center but they officially guarantee that I have at least 90% chance to get it the next week and I have 98% by statistics in March.

RGBD odometry [Maria]:
Installed more dependencies of the ecto capture pipeline. So now all the capture scripts work for me. Captured a couple of objects using an orb template to practice with the pipeline, got their meshes in the browser.
Replaced the existing pose estimation in the pipeline by rgbd odometry. Saw the first results and it seems I applied transformations in the wrong order somewhere, investigating this now. A warped image was stable in that experiment and it means rgbd odometry gave accurate poses.

Other:
The interview for the visa has been rescheduled for June 22.

Action Items

Gary

Check with Carol if Eduard is in
Go to guy who manages tutorials and get the link on the CVPR page above
Try registration on our site

Vincent

Send email summarizing libmv work

Ilya

.

Vadim

Look into email accounts: Google or some other? Commercial or some other?

Victor

Make sure answers and wiki are connected to the new opencv website

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-05-29

Agenda

Workshop
New website
RGBD demo

Minutes

CVPR Vision on mobile devices workshop
- Note the prerequisite section
Alexander will come to CVPR
Intro: how to build ,step by step introduction. Android is there, making iOS tutorials too (inpainting, face detection)
What should the mobile app be?
- Face detection
  - might not be real time on mobile. Instead 2 threads: Detect and track
  - For iOS, we can use standard face detection code because it supports pThreads
  - Either that or do ORB + some homography
We’ll order schwag site
Schwag store=
OpenCV website, launch is ~ Early June
Model capture
- Getting to be high priority
- Get good odometry

Vadim

[Bug fixing; OpenCV team et al]
18 tickets have been closed; 7 of them – by OpenCV team.
Most of them are related to documentation and various build/install issues.

[New functionality: Vadim]
Now all the OpenCV data structures can be stored to text strings and restored back


//==== storing data ====
string buf;
{
FileStorage fs(".xml", FileStorage::WRITE + FileStorage::MEMORY);
fs << "date" << date_string << "mymatrix" << mymatrix;
buf = fs.release();
}
//== reading it back ==

{

FileStorage fs(buf, FileStorage::READ + FileStorage::MEMORY);

fs[“date”] >> date_string;

fs[“mymatrix”] >> mymatrix;

}

Such functionality can be useful for mobile platforms and in large-scale systems that use databases to store data instead of regular files.

[Build infrastructure: Evgeny, Alexander]
Updated our build environment to OSX 10.7, new VMWare, new Linux distros (including Ubuntu 12.04LTS), added parallel MinGW+TBB build, moved some builds to different machine, so now it’s much, much faster. The ultimate goal is to have “build-test before commit” feature to make the trunk version of OpenCV always more or less stable.

[opencv.org, code.opencv.org: Alexander]
a few tweaks to opencv.org have been made to opencv.org: http://opencv.alekcac.webfactional.com/
created SVN repository for GSoC 2012 projects at code.opencv.org. Unfortunately, git still requires more time to configure it properly.

[iOS port: Eduard, Vadim]
Eduard Feicho now officially started working on the project (thanks to Gary!), he setup all the environment and built OpenCV for iOS using the already supplied with OpenCV scripts.
I created a simple app that reads video from camera and processes it using OpenCV function (laplace); will now proceed with something more complex, e.g. face detection.

Anatoly

Accomplishments & statuses:

1) OpenCV GPU

· Reimplemented all GPU performance test cases with CPU functions in order to compare CPU vs. GPU. Complete log was sent earlier.

o Minor improvements in OpenCV performance testing framework for functions than run fast.

o Spend some time for investigation the results, can present outcome in separate email or during Skype meeting.

o Marked items in the log where NPP is called.

· Tested GPU module with CUDA 5.0, fixed some problems (LUT), closed issue #1982 (CUDA5.0 compilation errors). We are ready to release 2.4.1, waiting when Tegra team will be ready.

2) Background substation

· Fixed build problems for old architectures.

Plans:
· Continue with FGD Foreground detection on GPU (optimization).

· Recording BodyBarts video for NVIDIA.

· Continue work on Pseudo Convex Hull.

Ilya



RGBD odometry [Maria]

Found the bug mentioned at the last time in my initial integration of the RGBD to the ecto

capture pipeline. Made experiments to scan some models using the RGBD, see an example in the

attachment. Tried two ways of scanning: 1) static scene + moving Kinect and 2) rotated table with

an object + static Kinect. In reality it is hard to provide an ideal static scene for the whole frame

(e.g. due to other people in the room). So the RGBD was not stable in both cases and its other

results were not so good as the attached example. To satisfy this condition of the RGBD we may

run it on pixels of a table with an object only and use the second way of scanning.

Tried to get a mask of a table with an object (to pass it to the RGBD) using PlaneFitter for each

frame. But PlaneFitter returns unstable axes (rotation) if there is even a relatively small object on

the table (it is unexpected behavior of the fitter). Moreover, it slightly slows down the capturing.

So to get the stable mask the ORB-based poser is run first. The RGBD processes pixels of this

mask only which provides a stable part of a scene.

Experimented with the RGBD working on the stable part of a scene. It still does not give accurate

models because it accumulates frame-to-frame errors of a camera pose. Investigating the reasons

of the errors now.
Other:

Found and fixed a bug in ecto_opencv. This bug produced the incorrect ORB-based poser after

updating the ecto_opencv.
Transparent objects [Ilya]

Revamped the draft of the camera-ready paper for RSS.

Prepared a poster about our work with transparent objects.

Adapted a paper about our previous approach and submitted it to a conference.

Started to refactor the transparent_objects module.
Other:

Ilya will go to the USA this weekend. He will leave the office for ~1.5 month:

June 02 – June 15   Preparing the demo at Willow Garage, Menlo Park, USA

June 16 – June 21   Attending CVPR as a visitor, Providence, USA

June 25 – June 29   Participating in the BMVA/EPSRC Computer Vision Summer School (site), Manchester, UK

July 2 – July 6      Participating in the Microsoft Research PhD Summer School (site), Cambridge, UK

July 9 – July 13    Presenting our paper at 2012 Robotics: Science and Systems Conference (site), Sydney, Australia

July 14 – July 17    Vacation, Sydney, Australia

Action Items

Gary

Call Secretary of state in California to see if we can expedite a previously filed corporate filing.
Call Cat to give here a “heads up”
Schwag sites

Vincent

Check out purchase of usb keys with data
Try to get cash from Willow

Ilya

.

Vadim

.

Victor

Check out purchase of usb keys with data
Find if groups are possible with a free account
- else have to pay for a couple of accounts.

Ethan

.

From Last Time

Gary

~~Check with Carol if Eduard is in~~
~~Go to guy who manages tutorials and get the link on the CVPR page above~~
~~Try registration on our site~~

Vincent

Send email summarizing libmv work

Ilya

.

Vadim

Look into email accounts: Google or some other? Commercial or some other?

Victor

Make sure answers and wiki are connected to the new opencv website

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-06-05

Agenda

Minor release
Website
Workshop
Udacity
RGBD
Transparent progress

Minutes

Minor release started. 2.4.1 is out, 2.4.2 ~July, 2.4.3 in August, 2.4.4 ~Nov
- Pretty much automated procedure, tegra team helps so we can get out more continuous releases
  - packages for linux, windows, mac and android
Website
- Should be out this weekend in Beta
- Fixed some security permissions
- Try to get non-profit corporate filings done [Gary]
Workshop
- There will be foils for Android part
- Vadim on foils for iOS part
- Preparing T-Shirt and USB key. Probably 100 copies
- Victor is out. Vadim June 14. 15th on site. Vincent and Ilya Sat. Morn.
Started to talk with Udacity for possible OpenCV courseware.
- UCSD reading group.
- Vision-class.org
  - Also Malik’s https://www.coursera.org/course/vision
OpenCV.org Foundation could sponsor improvement of vision pages on wikipedia
- See the ideas google docs for more
RGBD module [Vincent and Ilya]
- fast normals
- fast plane finding
- ICP
- Code to clean up NANDS “patch nands”
Object capture is a priority going forward
Transparency
- Reproduced results from the Summer onto the robot
  - About 5 secs/frame
GSoC Reports
- 2 students on iOS
  - 4 projects using camera etc Want to improve before CVPR
- Alexander working on Python samples
  - Documentation: Every example should have a help that tells what it does and how to call it
  - Deva Ramanan: Parts based code Hilton port to objdetect module

Vadim

[OpenCV 2.4.1: Alexander Shishkov and Andrey Kamaev, OpenCV team]
The first stabilization release in the 2.4.x series is out. Here is the complete ChangeLog:
The GPU module now supports CUDA 4.1 and CUDA 4.2 and can be compiled with CUDA 5.0 preview.
Added API for storing OpenCV data structures to text string and reading them back:
1 //== storing data ==
2 FileStorage fs(“.xml”, FileStorage::WRITE + FileStorage::MEMORY);
3 fs << “date” << date_string << “mymatrix” << mymatrix;
4 string buf = fs.releaseAndGetString();
5
6 //== reading it back ==
7 FileStorage fs(buf, FileStorage::READ + FileStorage::MEMORY);
8 fs[“date”] >> date_string;
9 fs[“mymatrix”] >> mymatrix;
cv::calcOpticalFlowPyrLK now supports precomputed pyramids as input.
Function signatures in documentation are made consistent with source code.
Restored python wrappers for SURF and MSER.
45 more bugs in our bug tracker have been fixed

[Buildbot: Evgeny Talanin]

created pre-commit script that runs three selected builds and only then allows to check the changes into the OpenCV repository.

[OpenCV sites: Alexander Shiskov]

1. some minor fixes to the prototype of opencv.org were made. The site will likely be revealed on this weekend.

2. installed and tested the new version of Askbot. It seems that all the known problems with it have been solved. It means that answers.opencv.org should be ready at the same time as opencv.org.

[GSoC 2012]
1. Eduard Feicho created 3 sample projects for iOS that will further be turned into templates for OpenCV-based applications for Apple mobile devices:
https://github.com/Duffycola/gsoc2012

- IntroCamera that shows how to use the camera/photo library (UIIImagePicker) and the video camera with autofocus (using the AVFoundation framework).
- IntroOpenGL (and LaserTowers), which should be an example OpenGL ES project.
- IntroOpenCV, which was supposed to be a hello world example for opencv+iPhone project.

the projects are now in very early alpha stage and are to be improved and cleaned up.

2. Alexander Mordvintsev created digits.py sample that demonstrates the use of OpenCV neural nets for hand-written digits recognition.

Anatoly

Accomplishments & statuses:

1) OpenCV GPU

· Fixed some bugs in GPU module under Windows (convertTo alignment error, compare with scalars error).

· Fixed issue #1991 (GPU interpolation in remap, resize, warp…).

3) Background substation

· Tried implementation with fixed point arithmetic’s, it’s a bit faster, requires less memory, but produce less accurate results.

Plans:
· Continue with FGD Foreground detection on GPU (optimization).

· Recording BodyBarts video for NVIDIA.

· Continue work on Pseudo Convex Hull.

Other:
· We have Russian holidays next Monday and Tuesday, and working day at Saturday instead.

· Marina has joined us. Alexey moved to Tegra team.

Ilya

RGBD odometry [Maria]
Investigated the accuracy of the RGBD again because:
1) As it was mentioned in the last report I did not get sufficiently accurate models using the initial RGBD integration to the ecto capture pipeline;
2) The comparison with the TUM paper in February showed that our RGBD implementation gave the same errors as in the TUM paper only if I synchronized rgb and depth with a smaller time interval (0.01s) than the interval used in the TUM benchmark by default (0.02s).
So it seemed that there are unimplemented ways to improve our RGBD quality. But there was a bug in my conversion of computed camera pose to the format used in the TUM benchmark, i.e. the problem was out of the our RGBD. Recomputed the results of my previos experiments:

paper’s results our results

RGBD GICP RGBD KinFu ICP (without volume, c++ version) RGBD + ICP
freiburg1/desk (573 frames)
translational_error.median (m) 0.0053 0.0103 0.0058 0.0066 0.0054
rotational_error.median (deg) 0.0065 0.0154 0.0059 0.0111 0.0055
absolute_translational_error.median (m) - - 0.1371 0.4439 0.0661
freiburg2/desk (2893 frames)
translational_error.median (m) 0.0015 0.0062 0.0022 0.0077 0.0021
rotational_error.median (deg) 0.0027 0.006 0.0033 0.0058 0.0032
absolute_translational_error.median (m) - - 0.2432 2.3209 0.2176

Сonclusions from the updated results:
1) our RGBD has accuracy close to the paper’s accuracy even we use the default parameters of both the benchmark and the RGBD (from contrib module);
2) RGBD + ICP has the smaller errors (frame-to-frame and absolute) than the our pure RGBD and pure ICP.

With more confidence in our RGBD quality I am choosing the method for the global refining the camera poses and captured model, reading the papers about this now.

Other:
Maria is on vacation for a week from June 4.

Transparent objects [Ilya]
Prepared the final draft of the paper for RSS.
Improved the poster by taking into account comments from colleagues (thanks a lot to all who provided their comments).
Continued to refactor transparent_objects module.
Helped Vincent with his implementation of the fast computation of normals.
Set up the working environment on my Willow machine and reproduced pose estimation of transparent objects on a robot with the old algorithm (without grasping yet). Thanks to Vincent for help.

Action Items

Gary

Try to get corporate filing done

Vincent

Get intern to add “patch nands”

Ilya

.

Vadim

.

Victor

.

Ethan

.

From Last Time

Gary

~~Call Secretary of state in California to see if we can expedite a previously filed corporate filing.~~
~~Call Cat to give here a “heads up”~~
~~Schwag sites~~

Vincent

~~Check out purchase of usb keys with data~~
~~Try to get cash from Willow~~

Ilya

.

Vadim

.

Victor

Check out purchase of usb keys with data
Find if groups are possible with a free account
- else have to pay for a couple of accounts.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-06-12

Agenda

Minutes

Vincent: schwag sent yesterday
Ilya: integrated algorithm for transparent clutter. Works on the robot
website: out this week. Everything seems ready.
Vadim: get packages for Android and iOS. We’ll put them on the keys. 0 install: just copy. Put links to develop for both platforms and how to install device
GSOC: iOS and Python: good progress, bugs fixed in Python wrapper. Non-local means implementation
logging in OpenCV: nothing, look into glog.

Vadim

[iOS port; tutorial preparations: Vadim, Eduard and Charu]

last week most of the time was spent in improving OpenCV support on iOS and preparing some demos.
First of all, a script that creates opencv2.framework, an iOS framework, has been created.

It turned out that iOS simulator does not simulate ARM instructions, it’s i386 “virtual machine” with special versions
of Cocoa libraries. In other words, when you build an app for simulator, it’s i386 CPU (aka 32-bit Intel arch); when
you build the same app for real device, it’s ARM. So, we needed 2 different sets of OpenCV libraries to make an
OpenCV-based app to compile and run on both simulator and device. If we take into account that there are
already ~20 modules in OpenCV, setting all the include and library paths in an application may be really boring
and not forward-compatible thing. But now using the script we can create a “universal framework” called
opencv2.framework. It combines both versions of the library, together with the headers. It’s super-easy to add
this framework to application, just like one adds system frameworks. Xcode automatically links application with the
correct variant of the library, depending on the target.

Meanwhile, GSoC students, Eduard and Charu, did excellent job on creating new iOS samples. Right now we have
more or less good working face detection demos. One works with still images from the photo library, another one
works with live video from camera. The screenshot is attached.

[GSoC Python interface improvements; Alexander]

Alexander Mordvintsev continues work on python wrappers and samples; he added description to digits.py,
added new sample on laplacian pyramid; fixed 2 bugs in the wrappers and other part of OpenCV causing
bugs in python samples.

Anatoly

Accomplishments & statuses:

1) OpenCV GPU
Found reasons and fixed thousands of warnings about “undefined anonymous” in device code under Linux.
(need to implement copy-constructor for device functors themselves).
Resize with INTERP_ARA flag (usually used for big scale coeff, where bilinear or others behave very bad.
Algorithm example: SURF)
Added simple implementation of cv::gpu::resize with INTERP_AREA interpolation type. Average speed-up
25-65x for scale coeff = 3..5. (GTX470 vs Core i5).
Fixed bugs in CPU version of resize function related area interpolation path. Changed behavior of CPU
resize implementation according right formula for tile with not full area.
For large scale coeffs (>10x) speed-up is very little. Working on smarter version for this case.
Spend time for dense optical flow (PyrLK) optimization ( we focues on sparse before, dense method
was a bonus)
Using shared memory for caching first image – it gives maximum 1.2x speedup in some cases.
Using texture memory for second image – it doesn’t gave any speedup.
Using float source + texture built-in linear interpolation – it gives ~2x speedup.
Video stitching application. Updated to support interface changes in opencv_stitching module. Tested
the app and sent to NVIDIA.

2) Background substation
Integrated FGD foreground/background subtraction to OpenCV (total 6x speedup).
Plans to work on VIBE

Plans:
Test new dense optical flow implementation, try some other ideas. Think how to make sparse optical
flow faster.
Implement faster resize with INTEP_AREA flag for large scale coeffs.

Ilya

Transparent objects [Ilya]:
Merged a branch with the algorithm of pose estimation in transparent clutter into the master branch.
Experimented with it on the robot and made sure it works correctly.
Documented public API of classes for recognition and pose estimation. Implemented a simple sample
how to use our algorithm and described it in the Ecto Recognition Kitchen documentation. Cleaned the
code of our algorithm, submitted the camera-ready paper to RSS 2012.
Investigated and solved the issue with too large memory requirements of our algorithm. The reason
was STL doesn’t release memory of deleted unordered_multimaps but holds it for future objects. We had
several temporary but huge hash tables in our algorithm and so memory was not released after their deletion.
Switched to using cv::Ptr to create hash tables only once and it solved the problem.
Vincent fixed integration of Ecto with the grasping pipeline after changes in Fuerte and the robot was
able to grasp a transparent object after these fixes.

Other:
Maria is on vacation.

Action Items

Gary

Try to get corporate filing done
Work on CVPR foils

Vincent

Get intern to add “patch nands”
Help with cvpr workshops

Ilya

.

Vadim

Work on CVPR workshop foils

Victor

Work on CVPR workshop foils

Ethan

.

From Last Time

Gary

~~Call Secretary of state in California to see if we can expedite a previously filed corporate filing.~~
~~Call Cat to give here a “heads up”~~
~~Schwag sites~~

Vincent

~~Check out purchase of usb keys with data~~
~~Try to get cash from Willow~~

Ilya

.

Vadim

.

Victor

Check out purchase of usb keys with data
Find if groups are possible with a free account
- else have to pay for a couple of accounts.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-06-19

General Meeting

Modular development

Minutes

Should modules be more distributed?
- Like linux packages
- Keep main part small and simple as possible.
- Download opencv core, download package. Click on cmake box and try to build. We wouldn’t stand behind ex-modules
- Want OpenCV “brand” to mean, the code has been curated, it’s something that works. Reduce the size.
- If OpenCV means an entity/community. If code is written for a paper, we can host if you use the library
  - Default feature detectors
More acknowledgement for contributions.
- Badges for contributors.
  - On discussions forums
- Ack in doc? Like cite for paper for contributors too.
- Physically thank top contributors
Needs for foundation:
- Web programmer. Web crashes, ask bot
- Someone who takes care of community. Such as thanking contributors. Encouraging bug fixes WITH badges
Points from
Money
- Rotating bill weighted by %
- Bug fix priorities
  - levels of support
    - Phone support(?)
  - board position
  - developer control
Infrastructure
- Hire another engineer to fix bugs
- Buildbot is taking full time right now.
- 3 software engineers
Priorities
- SfM, Bundle Adj., RGBD, VO, Computational photography, Segmentation
- So far, OpenCV includes classic algorithms. Policy on state of the art?

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-06-26

Agenda

GSoC Status
Website
Foundation discussion and mission
CVPR harvest
FREAK
Khoronos — Victor is now Chair! of the new Vision API group.

Minutes

Getting OpenCV.org filed:
- There are several services listed:
  - www.Service@GetItFiled.com ,
  - www.sierraexpress.net/sacramento-sos-expedited-filings.aspx , as well as
  - www.sundocumentfilings.com.
- the SOS offers expedited service on their website as well but I could not follow how one was to use it.
- The SOS website is: www.sos.ca.gov
Mobile workshop at CVPR was standing room only. Here’s the iOS slides
FREAK is in. Descriptor extractor but using SURF for detector
AGAST is at Code
GSoC
- iOS port Eduard had an exam, but did a lot before CVPR which helped a lot for the workshop
- Camera is working for iOS, needs to go into HighGUI
- Simpleflow coming along
  - Single scale example with algorithmic tests is done
  - Going to multiple scale
  - Might also get a side benefit of a faster bilateral filter
Website is good to go. Deploy any time.
- Do a once over
Khronos committee is now fully formed and working on a vision standard. Victor is the chair.
Will work with Stefan Hinterstoisser for final version of LINEMOD by September
Priorities ’12
- iOS
- libmv
- More calibration improvements 2D, 3D, mix
- Model capture
- VO
- optimization —
  - multi-threading on Windows and Mac using their native tools
  - Allow swapping out TBB for OpenMP for those that don’t have
  - OpenCL use wherever possible
  - Many more SSE intrinsics
  - In general, we’re moving this into single optimization class so that all code becomes the same but can specialize to architecture
    - We will then immediately comply with Khronos when that standard is finished.
- Mobile
  - iOS (coming along nicely)
  - Computational photography

Vadim

[Tutorial about iOS+OpenCV programming: Vadim, Eduard and Charu]

Tutorial on using OpenCV on iOS has been prepared and given at CVPR’2012 in
Providence, RI. These are the slides:
http://code.opencv.org/svn/gsoc2012/ios/trunk/doc/CVPR2012_OpenCV4IOS_Tutorial.pdf.

The tutorial went well (though, because no attendees had any experience with iOS programming
and, therefore, no iOS developer account, they only experiment with iOS simulator, so not all the
samples could be run). Before ECCV’12 we should do something about it.

Eduard and Charu continue to refine iOS port of OpenCV, support for color input from camera
has been added and few bugs have been fixed.

[FREAK feature detector+descriptor]
The new feature detector, contributed by EPFL, has been integrated into OpenCV trunk, features2d module.

[Buildbot: Evgeny]
the new builders have been added: with Qt support, OpenGL, Windows with GPU;
email notifications problem resolved, now receiving information about every build
finished pre-commit script that should test working copy before committing it to trunk.
Unfortunately, since we now on SVN, the script is not very convenient to use – it’s not
quite transparent for developer. Hopefully, with transition to git it will be solved.
Ilya is traveling.
Here is update from Maria:

Cleaned up an odometry evaluation application:
Removed median filtering that was applied to image and depth. For previous RGBD
implementations this filtering improved results on TUM benchmark. But current implementation
is slightly accurate without this filtering (I suppose due to M-estimator added last week).
Found experimental uncommented code in the application. Without it RGBD has less errors on
TUM benchmark.
There are no updated RGBD results, since Maria is on sick leave since June 21th.

Also, Maria came to Moscow for interview and could not get visa :(

Marina

Accomplishments:

OpenCV, Vlad.
ViBe Background Subtraction. Fixed ViBe implementation for large resolutions.
MOG. Implemented two CUDA variants of algorithm.
Integrated all algorithms into the gpu module. Added accuracy and performance tests, documentation,
samples. The performance results are below.

OpenCV, Marina.
Read Anton’s gem about GPU Haar cascade, investigated possibilities of LBP cascade
implementation on GPU.
Added ability to read new xml format of OpenCV classifiers to GPU classifier.
Implemented draft version of LBP classifier on CUDA.
Integrated scan in device layer of gpu module.
Issues: None

Plans:
Add new demos for background subtraction to GPU demo pack.
Test and optimize LBP cascade classifier implementation.
Other:

Vlad will be on vacation starting from July 2
Background subtraction performance results for video with resolution 768×576

VIBE CUDA MOG CPU MOG CUDA MOG2 CPU MOG2 CUDA
Avg. FPS 2140 50 1200 135 1500

Ilya



Traveling

Action Items

Gary

Check-in where we are globally in GSoC
Get corporate non-profit filing done

Vincent

.

Ilya

.

Vadim

Mail Kirell Benzi about AGAST code
See about getting AGAST in http://sourceforge.net/projects/agastpp/

Victor

.

Ethan

.

From Last Time

Gary

Try to get corporate filing done
~~Work on CVPR foils~~

Vincent

Get intern to add “patch nands”
~~Help with cvpr workshops~~

Ilya

.

Vadim

~~Work on CVPR workshop foils~~

Victor

~~Work on CVPR workshop foils~~

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-07-03

Agenda

GSoC, Miderm evals in July 9 – 13
OpenCV Foundation now established, search for “OPENCV” https://businessfilings.sos.ca.gov/
- Issues
Primesense
Khronos
AGAST, FREAK
Model capture priority
Workshops ECCV, ACCV?
Cloud connections
Python PIL connections
Get Malik’s Grab Cut

Minutes

GSoC Get mid term evals in 9th-13th
- libmv moving along. Triangulation with multiple images, reconstruction
- Python … amazon cloud example through python
  - Maybe connect him with the cascade project to train in cloud
- Simple flow, single scale. Victor will sync this week.
Foundation is established
- Interface to http://opencv.org/
  - Alexander Shishcov
- Paypal
- Schwag
- Get starter funds
Primesense for OpenNI 2.0
- How is OpenNI supported. We just need to link to the correct one.
Khronos
- Victor is chair, survived back wounds. Next meeting August face to face
AGAST — feature part of BRISK
- Best of ORB and BRISK paper Intern on “yet another binary discriptor” but on better distance function
  - OpenCV foundation could host a mini-conference/hackathon and come out with a “best of breed”
Model capture
- Ilya’s kin

Vadim

[OpenCV 2.4.2: Tegra and OpenCV teams]

The next OpenCV update is almost released. The code has been frozen and the packages are being created at the moment.
Here is the redmine page for the release:
http://code.opencv.org/projects/opencv/versions/6

in short, the new release will get about 45 bug fixes, OpenCV service for Android, OpenCV framework for iOS, comprehensive documentation and samples for the FaceRecognizer, some more GPU optimization etc.

Big thanks to Tegra team who actually did most of the work on preparing the release.

[OpenCV site: Alexander Shishkov]
Finally, http://opencv.org and http://answers.opencv.org are now running!

[Contributions]
pretty interesting background subtraction algorithm GMG:
“Visual Tracking of Human Visitors under Variable-Lighting Conditions for a Responsive Audio Art Installation”, A. Godbehere, A. Matsukawa, K. Goldberg
has been contributed by the algorithm authors, along with the sample and tests. The screenshot is attached.

[GSoC 2012]
Charu and Eduard continue to work on iOS samples.
Charu started working on camera calibration + augmented reality demo for iOS. Camera calibration part has been written and is being debugged now.
Eduard is working on improving and accelerating VideoCameraController class that captures video from camera on iOS.

Alexander Mordvintsev continues to improve digits recognition Python sample – with better feature selection and tricks from CVPR’2012 paper:
R. Arandjelovic, A. Zisserman
“Three things everyone should know to improve object retrieval”
the performance is now much better and SVM trains much faster.
Descriptions to 2 other Python samples have been added.

[Buildbot: Evgeny]
binary compatibility between 2.4.1 version and 2.4 branch is now regularly tested. Overall verdict is ‘compatible’ but still no public webpage exist
added iOS framework builder, it works and seems to be good for using in XCode projects;
configured Android builds to be run simultaneously, and Android tests to be run in parallel with any non-Android builds. Total duration of build queue on Bigmac reduced by ~1.5 hours;
modified Ubuntu x64 QT builder: installed number of video I/O libraries, upgraded Qt to 4.8.2;
minor activities: run tests via run.py, correct removing old MegaPack builds result etc.

Anatoly

Accomplishments & statuses:

1) LBP Face detection

· Implemented simple GPU pipeline for LBP classifier. Going to measure performance and analyze bottlenecks.

· Fixed a few bugs in cascade loading function.

2) OpenCV GPU

· Improved CPU/ GPU similarity of results for nearest neighbor interpolation (resize, remap, warpAffine, etc.). Merged to 2.4 branch.

3) Background subtraction

· MOG implementation was reintegrated in 2.4 branch (previous merge was with mistakes)

Plans:
· Complete GPU implementation of connected component function for classifier hypothesis filtration. Measure performance and bottlenecks. Think how to make it faster.

· Possibly try another idea for cv::gpu::resize with INTERP_AREA flag.

· Roman is going to learn Android/Tegra 3 this week and PCL bodyparts sources.

Other:

· Vlad is on vacations during whole July. Anatoly came back from vacations and have not answered all emails.

· We have meeting this Wed, don’t we?

Ilya

RGBD Odometry [Maria]

Closed TODOs with my code in ecto_opencv/rgbd:
- Refactored and converted odometries to cv::Algorithm. Now there are 3 classes for RGBD, ICP and merged approach.
- Added doxygen doc.
- Added tests for 3 odometries.
- Added sample for evaluation on TUM benchmark.

OpenCV fixing for incoming 2.4.2

Fixed #1764, #1834, #1879, #2043, #2082.
Added test for opponent brief.

Other:
Ilya attended BMVA Computer vision (site) in Manchester (UK). He moved to Cambridge (UK) to attend Microsoft Research PhD Summer School 2012.

Action Items

Gary

Send out reminder on midterm evals for GSoC people
Contact Thrun about .org

Vincent

Zisserman Contacts
Ping primesense back about using OpenNI 2.0

Ilya

.

Vadim

Get example of how to use cloud for training cascade for my student

Victor

*

Ethan

.

From Last Time

Gary

Check-in where we are globally in GSoC
~~Get corporate non-profit filing done~~

Vincent

.

Ilya

.

Vadim

Mail Kirell Benzi about AGAST code
See about getting AGAST in http://sourceforge.net/projects/agastpp/

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-07-10

Agenda

AVG
Foundation progress
- New website publications
Priorities
GSoC
- Student evaluations need to be turned in
MOSSE

Minutes

Install AVG — no.
OpenCV 2.4.2 has been released on July 4th:
- Change log Release 2.4.2
GSoC
- “Nice tracking video (Youtube)”: http://www.youtube.com/watch?v=FirtmYcC0Vc
- Student evals in
- Vadim today or tomorrow
- Python going pretty good
- iOS will have an app showing many demos
Foundation
- Still waiting on banking keys, codes for full access. As soon as we have this, make more noise
Priorities
- MOSSE
- Eulerian Mag
- Mobile acceleration. TBB
Taking a theoretical look at descriptors …
Tracking
- Look at tracking code from last year’s GSoC, some of it was good, get it in
- Contrib or Video
- Meanshift is in contrib
- Meanshift alg is in video
- VO 35% faster
Segmentation engine
Matlab
- Mexopencv
- Headers only to make it easy, Matlab to Mat
Release
- 2.4.2 Release is out now
- 2.4.3 End of Summer
- 2.5 end of year
Hackathon developer guide.
- Sept
Khronos F2F in July in Texas

Vadim

[OpenCV 2.4.2: Tegra & OpenCV teams]
OpenCV 2.4.2 has been released on July 4th:
https://github.com/opencv/opencv/wiki/ChangeLog

[GSoC2012, iOS: Eduard & Charu]
Eduard and Charu continue work on the OpenCV for iOS samples.
We decided to prepare free apps by the end of the project: “OpenCV Demo” and
may be an “OpenCV Image Effects” and put them to AppStore, in order to
popularize OpenCV.

Here is screenshot of the current version of “image effects” demo.

The idea is to make “image effects” runnable on simulator, so that people can play
with iOS and OpenCV without developer account or real ios device. “OpenCV Demo”
however, will be video based demo, demonstrating face detection, elements of AR, video
effects etc.

[GSoC 2012, Python samples: Alexander]
created very nice sample for planar object tracking using ORB features:
http://www.youtube.com/watch?v=FirtmYcC0Vc
made the how-to on using PiCloud
started experiments with Kinect (using OpenNI-MS bridge, but this could possibly
be replaced with highgui at some point)
[Python bindings: Eric Christiansen and Vadim]
A part of contrib module is now wrapped in Python too; it means that chamfer matching,
variational stereo correspondence, tone mapping and, most importantly, face recognizer
are now available for Python users.

[Buildbot: Evgeny]
done first tests of the pre-commit script and received feedback from the Tegra team.
wrote proof-on-concept hook for git to run pre-commit (actually, pre-push) testing when
we move to git.
started installation of a new, much more powerful redmine server at Hetzner.de, where
we create opencv git repository. Optimistic estimate of the transition time – 1-2 weeks.

Anatoly

Accomplishments & statuses:

1) LBP Face detection

· Improved GPU version for LBP classifier cascade (Improved register usage in
LBP classifier, fixed compatibility with early CUDA architectures).

· Implemented GPU version of hypothesizes grouping (Union-find connected components)
to eliminate remaining GPU/CPU data movement in cascade pipeline.

· Current version is up to 2x faster than TBB optimized CPU version
(GTX470 vs. Intel i5-2500k 3.3Ghz quad core)

Plans:
· Continue work on LBP optimization, write custom kernels to support for deep trees.

2) OpenCV GPU

· Fixed #2130, #2147, fixed codecs initialization bug.

· Added destination matrix size checking in integral and resize functions to avoid extra
GPU memory reallocation.

4) Background subtraction

· No update

Other:

· Learned DTAM paper as candidate for implementation
(http://www.youtube.com/watch?v=Df9WhgibCQA)

Plans:
· Continue LBP cascade classifier optimization.

· PCL4Tegra: Write a naive version of the algorithm and integrate it into the Android demo app.

· Try another optimization idea for ‘cv::gpu::resize’ with INTERP_AREA flag for large
scales (after LBP work has been finished)

Ilya

RGBD odometry [Maria]
Made some optimizations of odometries code (Mat elements access, precomputing,
conditions order, etc). That speeded up the algorithms 25-35%.

Chose the paper
Highly Accurate 3D Surface Models by Sparse Surface Adjustment by
M. Ruhnke, R. Kümmerle, G. Grisetti and W. Burgard, ICRA 2012)
for the global refinement of camera poses and point clouds for a model capture task.
Experimented with the paper’s model of Kinect depth errors due to quantization in our odometries.
There are not results yet.

OpenCV [Maria]
Simple fix #2118.

Other:
Ilya attended Microsoft Research PhD Summer School 2012. The school was starring
Antonio Criminisi, Jamie Shotton, Tony Hoare and many others.
Ilya moved to Sydney (Australia) to participate in the RSS 2012 conference and
present our work on transparent objects.

Action Items

Gary

Set up first board meeting
Ping Achin
Car detector A.T.

Vincent

Look into forming an API for a tracking module
See what tracking code should go in

Ilya

Present our work on transparent objects at RSS.

Vadim

*

Victor

.

Ethan

.

From Last Time

Gary

~~Send out reminder on midterm evals for GSoC people~~
~~Contact Thrun about .org~~

Vincent

~~Zisserman Contacts~~
~~Ping primesense back about using OpenNI 2.0~~

Ilya

.

Vadim

~~Get example of how to use cloud for training cascade for my student~~

Victor

*

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-07-15

Agenda

Adrian sez:



StereoBM and StereoSGBM classes not being derived from a common base class.

I realize that both of these objects take slightly different initialization variables, but there are many

better ways to handle this.  In fact, most of the rest of the library uses some kind of initialization

structure which is itself derived from some common base class, such that each derived stereo

algorithm can be initialized by its own custom initiailizer.

Update on foundation.
- Paypal up
- Board
Credit cars or just Paypal? … $300/year cost
Watch lists on websites?

ASIFT
*

Minutes

Stereo: refactor to inherit from base class
- Can’t do it right now due to backward compatibility for 2.4*
- Will do in 2.5
ASIFT
- Can apply to other detectors: Orb, FREAK
- Implemented in python right now, worth moving into OpenCV
Next 2.4 release, 2.4.3 will be in Sept.
- 2.4 focus is
  - increasing optimization,
  - better camera support,
  - GSoC for new stuff.
  - Python much improved, especially samples (good GSoC work this year!)
  - SfM
  - Mobile is much better as well
  - Folding in last year’s GSoC tracking in. Vincent cleaning it up.
  - Photo module for computational photography
  - Better modular infrastructure
- 2.5 might start Nov. Adrian rename conventions
Need youtube, facebook, twitter
- Need a person for this. Board meeting issue, leverage community
- Adrianna has left Citi. Christine as new person.
Itseez team alternates “bug” duty to close out old bugs and deal with new to keep this much more up to date.
Look into FAST code 5-9 as well as what we have 9-15

Vadim

[GSoC2012, iOS: Eduard & Charu]
Both students have successfully passed mid-term evaluation.
They wrote the reports for the passed part of the projects (http://code.opencv.org/svn/gsoc2012/ios/trunk/doc/).

In particular, Eduard created a draft version of iOS OpenCV demo app that
demonstrates various video effects:

[GSoC2012, Python: Alexander Mordvintsev]
Alexander has also passed the mid-term evaluation.
These is update from him:
spotted the problem with SURF features
modified the find_obj sample:
support SIFT, SURF and ORB features
BF and FLANN matching
interactively explore the matching points
added ASIFTdemo (which in fact is also "AORB" and "ASURF")
similar to find_obj demo, but with affine sampling
works well in rather difficult scenarios

[OpenCV, features2d: Maria, Alexander Mordvintsev]

Maria and Alexander (who works on GSoC 2012 OpenCV Python samples) have found some
severe problems in the latest version of SURF, SIFT and ORB. It appears that the coordinate
system (left or right) used for KeyPoint::angle estimation is not specified, and different
detectors use different ones. The guys mostly fixed the problem, added new meaningful
(in addition to simple regression tests) tests that cover the problem.

These are details from Maria:
===
1) Specified the KeyPoint::angle (in the rst doc and in the code comments).
2) Implemented new unit tests of 5 types:
- a simple test of output keypoints from FeatureDetector (validity of pt, angle and size),
- tests on rotation invariance for FeatureDetector and DescriptorExtractor,
- [additionally] tests on scale invariance for FeatureDetector and DescriptorExtractor.
3) Synchronized Feature Detectors and Descriptor Extractors according to given
KeyPoint::angle specification. Thanks to Alexander Mordvintsev for the fix of SURF descriptor.
4) The tests found some possible problems: ORB shows low scale invariance (especially descriptor),
FREAK shows low rotation and scale invariance. Opened the issue #2176 to find the reasons of this
(a problem in the tests or in the algorithms).
===
There are still some problems with scale and rotation invariance that we will look into.

[OpenCV + OpenCL: user contribution]
Finally, ocl module that lived in a separate branch for a while, has been merged into trunk.
The module contains some simple and fairly complex algorithms (like stereo correspondence,
face detection) accelerated using OpenCL. Architecture of the module is similar to CUDA-based gnu module.
Currently, the code can be classified as "good" in Linus Torvalds terminology; in other words, it compiles :)
Unfortunately, all the tests fail on my mac, since it uses CPU as the default OpenCL device for some reason,
while the module has only been tested on a dedicated graphic cards. The problem will be investigated.

[Migration to GIT: Evgeny, Alexander Shishkov]

We successfully installed Chiliproject on our new server at hetzner.de, migrated our bug tracker, wiki there
and installed a test git repository. We plan to finish migration to the new server and git this week.

Anatoly

Accomplishments & statuses:

1) LBP Face detection

· We continue investigation of performance bottlenecks and brainstorming optimization ideas.

· Implemented simultaneous processing of different scales. Current version is up to 2.8x faster (
GTX470 vs. Core i5-2500k, TBB)

· Tried the idea with jobs list which is compacted after each stage. Speed-up is 2.4×. Main reason
is low GPU occupancy at small pyramid levels.

· After all investigations, developed (in-mind) the method with constants grid size (its description
I sent yesterday) and working on its implementation.

Plans:

· Try to offload the algorithm to the GPU using OpenGL ES shaders.

3) OpenCV GPU

· Learned ‘Non local means denoising’ paper.

· Merged documentation and android service from trunk to 2.4.

· Background subtraction: No update

4) Other

· Final preparation for OpenCV migration to Git which will be done in 1-2 days.

· Hiring activities, most likely we have a candidate to fill the second PCL 4 Tegra position.

Plans:
· Continue LBP cascade classifier optimization.

· PCL4Tegra: optimization ‘bodyparts’ with GLSL.

· Previous work review for speed limit sign detection algorithms. Implementing non-local means
denoising on GPU.

· Try another optimization idea for ‘cv::gpu::resize’ with INTERP_AREA flag for large scales (after LBP
work has been finished)

Ilya

Transparent objects [Ilya]
Participated in the RSS 2012 conference and presented the paper “Recognition and Pose
Estimation of Rigid Transparent Objects with a Kinect Sensor” by Ilya Lysenkov, Victor Eruhimov
and Gary Bradski (5 min spotlight talk + 1.5 hour e-poster + 5 min talk at NICTA headquarters).

Other:
Our paper on pose refinement of transparent objects was accepted to the largest Russian computer
vision conference GraphiCon (http://en.wikipedia.org/wiki/GraphiCon). This paper covers the algorithm
that we developed ~1 year ago before switching to Kinect.

OpenCV [Maria]
All last week I worked on OpenCV because of detected problem with the KeyPoint::angle (#2043).
There was not a strict specification of the KeyPoint::angle before (units, a range of values and coordinate
system). Feature Detectors and Descriptor Extractors have used different own specifications of an angle
and it was a potential bug in a feature detection pipeline.
1) Specified the KeyPoint::angle (in the rst doc and the code comment).
2) Implemented new unit tests of 5 types:
- a simple test of output keypoints from FeatureDetector (validity of pt, angle and size),
- tests on rotation invariance for FeatureDetector and DescriptorExtractor,
- [additionally] tests on scale invariance for FeatureDetector and DescriptorExtractor.
3) Synchronized Feature Detectors and Descriptor Extractors according to given KeyPoint::angle specification.
Thanks to Alexander Mordvintsev for the fix of SURF descriptor.
4) The tests found some possible problems: ORB shows low scale invariance (especially descriptor), FREAK
shows low rotation and scale invariance. Opened the issue #2176 to find the reasons of this (a problem in the
tests or in the algorithms).

Last week was also my turn to be on duty of OpenCV track:
done #2165, #2152, #2155, #2156, #2159, #2160, #2167, #2043,
cancelled #2151, #2107, #2174, #2172.

Plans:
Back to model capture task.

Action Items

Gary

Contact the book guy to deposit directly to our account
Get wire information to Vincent
Set up first board meeting
Get Adrian to do a google doc on naming changes
Confirm paypal link to bank

Vincent

Send email to Roberta about wire transfer
Confirm paypal link to bank

Ilya

.

Vadim

Mail to Eugene not to lose wiki notes
Ask them to look into moin-moin for wikis
See about getting ASIFT warping method into Features2D

Victor

.

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-07-24

Agenda

website outage code.opencv.org
GSoC update
Getting Google connected to opencv.org for payments
Foundation core board meeting this Friday
- Statement of information
- Needs and funding
- Undergraduates
- Full time Web/PR guy
- Fund raising
- Other legal requirements
- Goals
- Other corporate support
- Android
- PR
New features
Freak sample, use
ECCV mobile tutorial
ACCV?
Subpixel
Vacations
- Gary to 13th
- Victor next week
- Week after next Vadim
  - August may be subject to meeting interruptions

Minutes

http://code.opencv.org is back up!
- We have git, mirror on git hub etc
  - Need backup to happen properly
  - Git doesn’t need this of course — we have git hub and a mirror
We need to make a guide for developers
- Vincent will start a documentation wiki for this, see below
Code to opencv
- New versions of FAST
- BRISK port
- Super fast normal finding and plane finding (total for both, 5ms)
GSoC
- iOS going: Video samples and photo examples. Planar object tracking demo. Alexander’s example from python
- Alexander implemented MOSSE tracker. Needs to be ported to C++
- Simple flow (dense optical flow) See Middlebury benchmarks.
  - Single and multi-scale. VGA resolution. But it’s slow right now. He’s optimizing now
- New clustering method being developed … research but maybe it pays off
- Have trained cascade for forks
  - Get him to collect a database of profile and semi-profile faces
    - Performance numbers. Need script to train, test and report ROC, positive, false positives etc
    - Zisserman: Good frontal face detector, need profile
- SSE optimization Core I-7
OpenCV needs
- Skeleton (guarantee that the library does not decay)
  - Vadim
  - FT for tech support of opencv resources, tools. Mid-level (SW engineer)
  - Accounting
- Advance
  - FT web/PR/contest run/outreach person
  - 2 FT would make noticeable difference (SW engineer)
  - Limited time
    - Technical write
    - Travel support
    - Marketing support
    - Contests
    - Awards
    - Education, workshop
What are the “gets”
- Name/PR/Good will
- Optimization
- Support (at some level)
  - Sell support at lower levels
  - Architectural consideration (demo applications for new platforms)
- Influence on direction
- XYZ summer of code
Features
- Resampling
- Multi-baseline stereo

Vadim

[GIT & New Server: Alexander, Evgeny]

We finally moved to the new server (http://code.opencv.org now points to it).
The migration took a bit longer than we expected, because of the differences between Chiliproject and
Redmine and other subtle nuances.
Also, because Redmine themes are not fully compatible with Chiliproject’s, we did some rough customization
of the interface.
We already tested various features, including the bug tracker, wiki, git and everything’s working well.

[GSoC Python: Alexander Mordvintsev]

Alexander continues work on tracking samples in Python. Here are the details from him:
MOSSE tracker http://www.cs.colostate.edu/~bolme/publications/Bolme2010Tracking.pdf
I successfully managed to implement MOSSE correlation based tracker; it was compared with some videos
http://www.youtube.com/user/bolme2008
from the paper’s author and I’ve got very similar results. It is quite robust and fast. The code is still need to be polished. S
ee issues section also.
notes on ASIFT
Not actually a progress, just a case of unexpected performance of ASIFT sample. Once I took a few images from a
small aircraft. I tried to run asift.py on downscaled versions of these (aero1.jpg and aero3.jpg). The result is match.png.
Commsnd line was
“asift.py —feature sift-flann aero1.jpg aero3.jpg”.
I didn’t expect it to find anything in the case when I myself hardly spot matches on the grayscale versions of images.
But it did found 42 homography inliers (out of 134 matches).

Issues
I found a couple of issues with DFT implementation in OpenCV.
In order to divide and take conjugates of spectrums I used DFT_COMPLEX_OUTPUT flag for cv2.dft. It seems to leave
half of a resulting array uninitialized, when the source is real:

a = np.zeros((8, 8))
A = np.zeros((8, 8, 2))
A[:] = 123
cv2.dft(a, A, flags=cv2.DFT_COMPLEX_OUTPUT)
print np.sum(A == 123) # expect everything to be zero, but some values are uninitialized

This is not a problem, when transforming back with DFT_REAL_OUTPUT flag, but gives me lots of warnings when I
manipulate spectums with numpy (overflows and invalid values). I’ll add this bug to a tracker when it becomes available.

Also I found, that in theory I could use CCS-packed spectrum for my operations, but OpenCV misses the capability of taking
conjugates and division of spectrums, represented in this way (while it still can multiply them, even with conjugates with
cv2.mulSpectrums). Should we add this?

[GSoC iOS: Eduard and Charu]

Eduard is working on the planar object detection demo for iPhone. The main problem now is very low performance.
We will have to profile the code and see if it can be brought to realtime somehow.
Charu added several new effects to the “image effects” demo. See the attachment.

[Bug fixing: Maria]

Fixed bugs:
- the bilateral filter for uniform image of CV_32F type,
- Lsh index constructor in the Flann.

Supported users in the answers.opencv.org. It results to several improvements

(automatically keeping image types the same),
(allowing the norm to be passed in),
(better tests for FLANN)
(better doc of cascade detection)
and a bug #2183 (Add a norm type support to the FlannBasedMatche)

[User contributions]

Grigory Frolov, intern from Intel Nizhniy Novgorod, contributed a few SSE2/SSE4.2-optimized functions for OpenCV.
It has been integrated to OpenCV trunk, sorry, I meant, to the master.
Wongun Choi, PhD student working with Caroline, contributed the patch for calculating confidence measures from HOG
people detector (the patch is for GPU-optimized version). The patch is now analyzed by the GPU team.

[Papers from CVPR 2012: Vadim]
Studied a few interesting papers from CVPR’2012 that, we believe, may be worth implementing, since they all introduce
some techniques that can be useful beyonds the algorithms, where the techniques are used. The slides that I presented
last Friday have been shared separately. For the reference, the papers discussed are:

FREAK: Fast Retina Keypoint (already in OpenCV)
Real-time scene text localization and recognition.
A non-local cost aggregation method for stereo matching
Three things everyone should know to improve object retrieval

Anatoly

Accomplishments & statuses:

1) LBP Face detection

· Implemented the idea with fixed grid size – speed-up is the same(~3x). it seems that we are compute
bound (IPC is near 2.0, and control flow divergence ~33%).

· Integrated the functionality in OpenCV CascadeClassifier_GPU class. Added support of async. invocations
for LBP only (It’s possible because LBP is implemented fully on GPU).

· Investigated approach in the paper which Shalini send us. For now we don’t see how we can apply
feature-centric approach to optimize our code.

3) OpenCV GPU

· Background subtraction: no update

· Non-local means denoising: no update.

Plans:

· Try another optimization idea for ‘cv::gpu::resize’ with INTERP_AREA flag for large scales (after LBP work
has been finished). (Marina)

· Implementing non-local means denoising on GPU (Anatoly)

· Start working on Connected Components labeling on GPU (Marina)

Ilya

RGBD Odometry [Maria]
Studied the chosen paper (Highly Accurate 3D Surface Models by Sparse Surface Adjustment by
M. Ruhnke, R. Kümmerle, G. Grisetti and W. Burgard) for global pose and point clouds refinement,
ready to start implementation.

Played with the OpenCV bilateral filter on a depth image. It improved results of ICP, but the quality
of RGBD and RGBD+ICP remained the same. Moreover ICP gives the smallest translation error on TUM datasets
(they contain rich 3D structures), but it still has the largest rotation error. So I tried a new scheme of
RGBD and ICP merging: we run ICP at first and then RGBD but with the pure rotation model.
This merging gives the smallest translation error from ICP and small rotation error after refinement by RGBD.

The filter works ~0.28s on a VGA frame (with similar to KinFu parameters). It’s ~4-4.5 times slower than
our RGBD or ICP odometries. We can accelerate it technically by using TBB, SSE or algorithmically by
trying the quasi bilateral filter from the CVPR’12 paper that works O(1) per pixel (thanks to Vadim for
the talk about it on the Itseez seminar).

OpenCV [Maria]
Fixed bugs:
- the bilateral filter for a uniform image of CV_32F type,
- Lsh index constructor in the Flann.

Supported users in the answers.opencv.org. This resulted to several improvements
#2181, #2182, #2184, #2185 and a bug #2183.

Transparent objects [Ilya]
I’ve finally returned to Russia to my beloved transparent objects.
Prepared and submitted a camera-ready paper “Pose Refinement of Transparent Rigid Objects With a Stereo Camera”
by Ilya Lysenkov and Victor Eruhimov which was accepted to the main conference program at GraphiCon 2012.
We got good reviews with 3 overall ratings: Strong Accept, Accept and Accept.
Re-created infrastructure used at Willow to work with transparent_objects:
updated to fuerte and the latest version of ecto.
Started to experiment with Line2D on transparent objects.
Sometimes it works quite well (see attached success.png) but
sometimes it is distracted by other gradients (failure.png). Quantitative evaluation is on the way.

Action Items

Gary

Contact the book guy to deposit directly to our account
Check the bank account
Contact Google about vendor
Get a list from J-Y of what he wants

Vincent

Ask Roberta about transfer of money if it isn’t there (Gary to check the account)
Start a new wiki page: core developer guide

Ilya

.

Vadim

See about getting ASIFT warping method into Features2D

Victor

Send to Eugene whether they want to do a GPU version of Simple Flow

Ethan

.

From Last Time

Gary

Contact the book guy to deposit directly to our account
~~Get wire information to Vincent~~
~~Set up first board meeting~~
~~Get Adrian to do a google doc on naming changes~~
~~Confirm paypal link to bank~~

Vincent

~~Send email to Roberta about wire transfer~~
~~Confirm paypal link to bank~~

Ilya

.

Vadim

~~Mail to Eugene not to lose wiki notes~~
~~Ask them to look into moin-moin for wikis~~
See about getting ASIFT warping method into Features2D

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-07-31

Agenda

board meeting
mess in samples folder
status bilateralFilter
GSOC
BRISK

Minutes

board meeting: went well
samples: images should not be stored in the root but in a data folder
several pull requests
examples of previous contributions by corporations: summer (Intel: patches using SSE, NVidia: GPU optimized face detector, Face Space in contrib, Google, OpenCL Chinese Academy of Science with AMD)
now that the different FAST are in there, we can put BRISK
hooks for different bundle adjuster is possible
GSOC Vadim: add functionality and camera interface/image reading/writing in highgui. Perf test on iOS
parallelization: support for TBB, OpenMP through a generic interface
bilateralFilter: replace NaN with anything (max ?) and not 0 should work

Vadim

[GIT, new code.opencv.org server: Evgeny]

- set up backup script for code.opencv.org data;
- configured github and local (thanks to Andrey Senin) mirrors;

github mirror appeared to be very useful feature; in a few days with already have at least 6 pull requests! That is, this technical tool really facilitates contribution process!

[GSoC OpenCV + iOS: Charu, Eduard]

Charu added 4 more effects to image effects iOS demo: cartoonish, focus, contrast/brightness adjustment, pixellation.
Eduard integrated video camera module into HighGUI.

Another project, not related to GSoC, but related to iOS: Timnit Gebru from Stanford ported some of OpenCV performance tests to iOS, so now we can profile OpenCV there. The code is at some early stage and has not be contributed yet.

[Parallelization: Ilya L, Vadim]

Non-template implementation of parallel_for primitive function has been implemented by the new intern Ilya Lavrenov. It opens very good opportunities for us – we can have different backends for the parallelization framework (e.g. GDC on Mac, OpenMP on Linux/Windows etc.) and the backends could be “hidden” in opencv_core. To test the new engine, Ilya optimized bilateralFilter (floating-point flavor of it) and got 2.7 speedup on dual-core Core i5.

[More contributions]
Grigory Frolov, summer intern from Intel, contributed some more SSE optimizations for the following OpenCV functions: invert(), boundingRect(), cvHaarDetectObjects() (i.e. the older C implementation of our cascade detector).

Vincent contributed 5/8 and 7/12 variations of FAST. Integrated via github pull request.

Ilya

RGBD odometry [Maria]
Added support of a transformation type to all odometries in ecto_opencv (rigid body motion, pure rotation, pure translation). It was in RGBDOdometry from OpenCV contrib before, now ecto_opencv also supports this.

Implemented the building of a depth pyramid with the maxDepthDifference threshold (similar to KinFu) and more appropriate processing the invalid depth values. But it gives almost the same accuracy (improvement is less 0.2% in my experiments) and it’s slower than used before cv::buildPyramid in ~10 times (because my version has additional checks and does not use SSE). So cv::buildPyramid was remained in the odometries.

Tried to compare our odometries with KinFu ICP on the TUM dataset. It would allow us to estimate what refinement of previous frame by the volume TSDF gives for ICP. Compiled PCL but KinFu falls with a segmentation fault in the same environment where it worked in the beginning of this year. Shared my computer with Anatoly to fix the problem.

Other:
Maria will be on vacation from July 31 for 1-2 weeks.

Transparent objects [Ilya]
Accomplished several experiments to evaluate accuracy of pose estimation with Line2D when working with transparent objects (thanks to Maria Dimashova and Stefano Fabri for useful information about Line2D).

1. Simulated train data vs. real train data
Generated artificial templates for Line2D and compared results when real train data are used. The same train poses for templates are used in both cases. Results (success rate is ratio of cases when the estimated pose is closer than 3cm to the ground truth pose):
-——————————————-
| Train data | Success rate |
|———————-|———————|
| Real | 64.0% |
| Simulated | 64.9% |
-——————————————-
Success rate is similar but simulated train data provide much more freedom in generating templates for different scales and rotations so simulated train data were used in subsequent experiments.

2. Accuracy of pose estimation with Line2D
Evaluated accuracy of pose estimation with Line2D on a dataset of 5 transparent objects. Line2D is able to locate an object in an image correctly but it is not able to distinguish between close templates robustly so accuracy of pose estimation is low. The following experiment demonstrates this issue:
42000 simulated train templates were generated for each object. Naive sampling is not uniform in SO group so the algorithm from the following paper was used: Generating Uniform Incremental Grids on SO Using the Hopf Fibration by A. Yershova, S. Jain, S. M. LaValle, and J. C. Mitchell, International Journal of Robotics Research, 2009. These generated templates were sorted according to their proximity to poses in the test dataset and they were used for Line2D training in this order. Accuracy of pose estimation with regard to number of used templates is shown in the attached line2d.png (success rate is ratio of cases when the estimated pose is closer than 3cm to the ground truth pose). One can see that if templates corresponding to test poses are used only then accuracy is high but if we start to add other templates then accuracy degrades significantly. Test poses are not known beforehand so accuracy of pose estimation with Line2D will be low in practice. For comparison our algorithm has success rate 0.98 on this dataset.
These results show that Line2D cannot be used for final pose estimation but perhaps it can be used to generate initial hypotheses of poses so additional experiments will be accomplished to check this.

Other:
Prepared and gave a talk about our work on transparent objects to Intel Summer School students.

Action Items

Vincent

share document about the board
file bug report
add .rst for FAST
bug report for latest OpenNI (cap_openni.cpp)

Vadim

how to check inputArray is outputArray

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-08-07

Agenda

tracking library: we have MIL, online Boosting how about feature based ? (KLT and so on from libmv)
Eric wants to know more about the wrapper scripts (so that he can help with Java and maybe get the JavaCV guys on board)
stereo code Vadim and I saw at CVPR

Minutes

adding feature tracking from libmv
wishlist of things we would like to get in OpenCV to attract new developers: simple bugs, simple features
Evgeny: bugs
Ilya: does get emails on the build farm
BA for Maria and Vincent: OpenCV needs interactive calibration. We have great automatic tests for calibration. We can use the Matlab toolbox for that too. Sample for calibration with a function that spits back the current calibration and the error.
Khronos: design of pipelines
Eric and Andrey will talk

Vadim

[GIT repository: Evgeny]
- added experimental pre-push hook for OpenCV master branch. It runs several light builders with buildbot that build OpenCV and run tests on Mac, Linux and Windows. After some testing we decided to change the strategy: run the tests after pushes and merge changes from master to the to-be-created branch “stable” nightly if all the 40+ buildbot builders succeeded.
- added iOS and binary-compliance builders.

[GSoC OpenCV + iOS: Charu, Eduard]
Charu missed this week because of rellocation.
Eduard is still working on merging highgui OSX camera capture with the newly added iOS camera capture code.

[GSoC OpenCV + Python: Alexander]
Alexander created augmented reality demos; see the video: http://www.youtube.com/watch?v=pzVbhxx6aog

[Parallelization: Ilya L]
Ilya started the work on threading geometrical image transformation functions: remap, warpAffine, warpPerspective, resize, undistort. First version gives ~2.7-3.3 speedup on Core i5 machine on FullHD images.

[More contributions]
Grigory Frolov repaired SSE code in haar face detector haar.cpp and added AVX branch. The code has not been to OpenCV yet, but it gives ~30% performance boost on some configurations. Vincent contributed documentation for 5/8 and 7/12 variations of FAST.

Ilya

Transparent objects [Ilya]

Compared Line2D with our current approach to generation of initial pose hypotheses in transparent clutter. Results (success rate is ratio of cases when the estimated pose is closer than 6cm to the ground truth pose. Note that errors are big because it is just initial poses, they will be refined by subsequent steps of the algorithm):

Line2D	Our current algorithm
-———————————————————————-	-———————————————————————-

Object | | Mean error | | Mean error | | Success rate |-----------------------------------| Success rate |-----------------------------------| | | translation (cm) | rotation (deg) | | translation (cm) | rotation (deg) | ----------------------------------------------------------------------------------------------------------------| bank | 1.00 | 1.5 | 35.4 | 0.97 | 1.7 | 30.7 | bottle | 0.32 | 4.8 | 63.1 | 0.62 | 3.6 | 47.4 | bucket | 0.96 | 3.1 | 96.6 | 1.00 | 2.2 | 17.8 | glass | 0.22 | 2.8 | 71.3 | 0.60 | 2.4 | 53.6 | sourCream | 1.00 | 2.8 | 63.8 | 0.95 | 2.4 | 16.3 | wineglass | 0.24 | 3.9 | 67.7 | 0.98 | 2.7 | 38.0 | ----------------------------------------------------------------------------------------------------------------| mean | 0.62 | 3.9 | 66.3 | 0.85 | 2.5 | 34.0 | ----------------------------------------------------------------------------------------------------------------- These results show that our current algorithm outperforms Line2D for the task of generation of initial pose hypotheses.

Adapted Line2D for pose estimation of transparent objects by using parts of our algorithm (glass segmentation and non-maximum suppression of 3D poses).
Collected new datasets with transparent clutter.
Used Vincent’s RgbdPlane from ecto_opencv to find a support plane. Now transparent_objects supports 3 methods to find a table: RgbdPlane, plane segmentation from PCL and Fiducials.
Checked Vincent’s data of transparent objects from a PR2 robot for applicability of our algorithm.
Prepared and submitted the eventual RSS camera-ready paper for publication in the RSS proceedings by MIT Press.

Plans:
Evaluate accuracy of pose estimation with regard to percentage of occlusion.

Other:
Maria is on vacation.

Action Items

Vincent: finish BRISK

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-08-14

Agenda

Just back from vacations, no discussions, just reports

Minutes

.

Vadim

[GIT repository: Evgeny]
- created “stable” branch of OpenCV repository. The branch “master”, to which we all commit, is automatically getting
merged into “stable” once buildbot successfully built OpenCV in 40+ different configurations and the correctness tests
have passed successfully.
- changed the automated testing of user commits. Now the code is put to master anyway, and after that OpenCV is run
through express testing (3 builders vs 40+ in the full testing mode). In the case any of those builders fail,
user gets e-mail notification.

[GSoC OpenCV + iOS: Charu, Eduard]
- Charu added inpainting into the “OpenCV image effects” demo for iOS. From the screenshot below we can see that
without texture generation, which more advanced inpainting algorithms include, the results look a bit unnatural,
but still the demo could be fun to use.

- Eduard finished integration of video capturing code into highgui. The API is still being refined.

[GSoC OpenCV + Python: Alexander]
Alexander created Wiener deconvolution sample.

Besides, he, our contributor Philipp and Vadim fixed several bugs in Python wrappers: ##2100, 2242, 2257.

[Parallelization: Ilya Lavrenov]
Ilya added multi-threaded processing to cv::resize; on 2-core Core i5 with hyperthreading we get consistent
speedup over 2x.

[More contributions]
Grigory Frolov, intern from Intel, contributed SSE-optimized bilateral filter. The code has not been integrated yet,
so there no performance numbers yet.

[Vacations]
Evgeny will be on vacation from tomorrow till 6th of September. Maria got back from vacation.

Anatoly

WR

Ilya

Transparent objects [Ilya]
Implemented a script to collect ground-truth poses of occluding transparent objects in semi-automatic way. Labeled
our datasets using this script.
Implemented computation of occlusion percentage that is proportion of a test object silhouette which is occluded
by other transparent objects in a test image. Implemented evaluation of pose estimation accuracy with regard to this
occlusion percentage. See evaluation.png for results (success rate is ratio of cases when the estimated pose is closer
than 2cm to the ground truth pose). These results show that pose estimation is robust to occlusion < 40% but success
rate drops to zero if occlusion > 60%. We need to use orientation of edges to increase robustness of the algorithm.
Started to integrate Fast Directional Chamfer Matching into the Levenberg-Marquardt ICP framework.
Found a big area of papers on edge-based 3D tracking that are directly relevant to our algorithm (we call this
step “pose refinement” but it is essentially the same problem as in tracking). Studied classic and state-of-the-art
works. There are real-time approaches that works with surface edges (Real-time visual tracking of complex structures
by T. Drummond and R. Cipolla, PAMI 2002) and also extensions to silhouette edges (Exploiting Occluding Contours
for Real-Time 3D Tracking: A Unified Approach by Gang Li, Yanghai Tsin, and Yakup Genc, ICCV 2007). It is reasonable
to implement these papers at some point.

Plans:
Finish integration of Fast Directional Chamfer Matching into the Levenberg-Marquardt ICP framework.
Improve results of evaluation with regard to occlusion percentage.
Collect more data.

Other:
Maria was on vacation.

Action Items

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-08-21

Agenda

GSoC
Foundation
ECCV
Roadmap

Minutes

Have student’s write summary
- Need write access to opencv.org
- Alexander Shishkov
- We need to put up a blog about the GSoC accomplishments
ECCV
- Same foilware
- Rome U for a workshop?
- Need travel funds in the foundation ….
- Need to talk to Cat at Google
Will look at trained cascade classifiers
Roadmap
- 2.4.3 posponed to first of October (one month since nothing much happens in August)
- 2.4.3 will include most of GSoC results (binary compatible with 2.4)
- 2.5 scheduled next spring
  - Some compatibility may be broken, such as micro-modules
  - Kinect fusion being ported. Since this is modular, might make 2.4.3. Will be a whole new and much faster version, that will work as a 2.5 module
Samples
- Make an python script to self document what functions are there
- Hackathon for putting documentation headers for script to read
Transparent objects
- ICRA deadline Sept 7th. Pose estimation in transparent clutter
  - Rotation of edges for better — fast directional chamfer matching … similar to MERL paper

Vadim

[Optimization: Ilya Lavrenov, Vadim, Grigoriy Frolov]
- Ilya has finished optimization of warpAffine (2x), warpPerspective (2x), remap (3.2x) and resize (up to 12.77x for 2×2
bilinear resize) functions. In parentheses these are speedup factors for fullhd images on Core i5 machine. For 2x
decimation (i.e. the output image area is 25% of the input image area) we now have a special code (C version for now) that
does decimation. With SSE2 intrinsics it should be possible to get another substantial improvement.

- Now OpenCV makes use of native parallel for implementation on Windows (via Concurrency library, a part of MS runtime)
and MacOSX/iOS (via Grand Dispatch Central) when TBB is not available. So far, the new parallel for is used only in a few
functions (image warping and bilateral filter), but we plan to convert other TBB-based parallel for’s to the new primitive. In
order to do that, we also added cross-platform synchronization primitives (cv::Mutex class that is actually a spinlock on most
platforms).

- The patch from Grigoriy Frolov, intern from Intel, for SSE3 optimized bilateral filter has been integrated to OpenCV. The
speedup is not huge, around 20%. The code can be further improved.

Meanwhile, Maria has made a small yet important fix in bilateralFilter to make it handle NaN’s values more gracefully (which is
important for RGBD odometry)

- We fixed CMake scripts for iOS to tell the compiler to build optimized code. Previously, we apparently passed “-O0” flag to
GCC, resulting in quite slow execution. Now it’s “-O3 -ffast-math”. Together with GDC-powered parallel_for, we should get
much better performance of our samples.

[GSoC: Charu, Eduard, Alexander, others]

iOS:
- We finally integrated Eduard’s classes for video capturing into highgui.
- Image filters and video filters samples are basically ready. The final week is devoted to reports.

Python:
Alexander put all the remaining code in work to our repository. Overall, it was excellent project with about a dozen of new
exciting samples.

Non-local means:
Victor Passichenko, GSoC student supervised by Victor and Leonid Beynenson from NVidia team, has submitted implementation
of non-local means image denoising algorithm. It has been already integrated to opencv.

“Simple flow” dense optical flow algorithm:
Yuri Zemlyanskiy submitted implementation of the state-of-art dense optical flow algorithm. The GSoC project was lead by
Victor.

iOS Port coming to an OpenCV near you soon.

Anatoly

WR

Ilya

Transparent objects [Ilya]:
Integrated Fast Directional Chamfer Matching (FDCM) into the Levenberg-Marquardt Iterative Closest Point framework (LM-ICP)
by computing jacobians using the 3D distance transform (both for silhouette and surface edges). FDCM uses orientations of edges
so it allows to get better results in clutter.

Improved filtering of outliers by considering points with high weight only.

Used estimation of edges orientations from the paper “Estimation of Curvature and Tangent Direction by Median Filtered
Differencing” by J. Matas, Z. Shao, and J. Kittler. It work with similar quality as the corresponding part of the FDCM approach
but it is faster in 3.2x times.
Collected a new dataset with 4-5 transparent objects on each image and labeled it.

See the attached plot for results of the improved algorithm (success rate is ratio of cases when the estimated pose is
closer than 2cm to the ground truth pose).

Model capture [Maria]:
Studied the g2o framework (read the paper and doc, studied its examples).
Read the paper about RGB-D SLAM that uses g2o (“An Evaluation of the RGB-D SLAM System” from ICRA 2012). Their odometry
doesn’t have anything in common with RGBD odometry which we implemented. Studied their code (available as open-source)
as a more practical and complicated example of g2o usage than most of the demos from the g2o framework.

Started experiments with the graph of poses from our odometries. To get experience with g2o I built a simple graph with
edges connecting each pose with several previous poses. But this doesn’t decrease the absolute translational error on the TUM
benchmark and makes it even larger. Studying the reason of this and g2o more.

OpenCV [Maria]:
Replaced the value that was set instead of NaNs in bilateralFilter_32f by some more appropriate value to get correct results
for depth maps (thanks to Vadim for the hint). This constant will be made the optional parameter of cv::bilateralFilter when we can
break OpenCV binary compatibility in 2.5 version (reported in the issue #2269). Switched to use original cv::bilateralFilter in
ecto_opencv (it’s possible with some preprocessing).

Other:
Maria returned from vacation on August 14th.

Pose recognition in clutter of transparent objects.

Action Items

Gary

Talk to Cat at Google
ticket 2.4.3 for auto doc of samples

Vincent

.

Ilya

.

Vadim

Look at trained cascade classifiers to decide on what should be included

Victor

.

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-08-28

Agenda

GSoC travel
OpenCV on mac
ECCV
Still have to put up Blog
Foundation

Minutes

GSoC travel: wait for Gary
OpenCV on Mac: wait for Gary. Gary should write a tutorial about it as it is pretty easy. Vadim could help as he works primarily on Mac.
ECCV: if we cannot send people, the team needs to teach people how to present. Tutorial is on September 7th.
GSOC: everything ended well. Very complete GSOC. What did we learn ? Next year : make sure students can work full time on it (filter out students that do not have much time). We need to keep track of what to integrate.
Still have to put up Blog: wait for Gary
Foundation: wait for Gary

Vadim

[Optimization: Grigoriy Frolov]
- Grigoriy, intern from Intel, has finished the summer practice at Intel. He contributed 2 more patches, for the box filter and Haar face detector with SSE & AVX optimization. The patches will be integrated into OpenCV.

[GSoC: Charu, Eduard, Alexander, others]

iOS:
The final report has been sent. Charu and Eduard wrote tutorials about using OpenCV in iOS apps. Tutorials will be included into offical OpenCV build tree and will be uploaded to docs.opencv.org.

Python:
Alexander wrote the final report/blog entry.

Object Detection Cascades:
Cascades for detecting profile faces and silverware, trained by Attila Novak, who was supervised by Gary, have been integrated into the library. The detectors work quite well (although, they are trained for specific orientation of an object, i.e. profile faces should be directed to the right, silverware items should be oriented vertically).

Simple Flow:
Yuri Zemlyanskiy – GSoC student, supervised by Victor, has submitted simple flow dense optical flow algorithm, which is not a realtime, but reportedly quite accurate algorithm. First review round has been passed and the updated, more efficient implementation, has been submitted. It will be integrated to OpenCV, video module.

[Contributions]
7 patches from users have been integrated.

Anatoly

WR

Ilya

Transparent objects [Ilya]:
Used normals to compute silhouettes instead of morphology. It is faster (overall speed-up of the algorithm is 1.4x) and can be analyzed analytically.
Computed jacobian when normals are used to compute silhouettes. Checked validity of results by comparing with a numerical jacobian. See the attached plot for results of the improved algorithm.
Started to prepare a presentation for the Itseez seminar about our latest algorithm.

Model capture [Maria]:
Continued to study g2o and experiment with it on poses from our odometry implementations. Found and eliminated the reason why my simple graph of 3D poses (with edges connecting each pose with several previous poses) didn’t give improvement last week. Some results of the camera poses refinement on TUM benchmark:

frames count translational_error.max
(m, frame-to-frame) absolute_translational_error.rmse
(m, the whole trajectory)

before g2o after g2o

freiburg1_desk 573 0.032586 0.10342 0.052493

freiburg2_desk 2893 0.025508 0.29847 0.107332

freiburg1_360 755 0.057243 0.280819 0.306784

This simple graph can improve accuracy of poses for the video sequences with relatively accurate motion estimations. But it’s powerless if there are large errors in odometry. Improvement of the graph structure is in progress, e.g. a loop closure by applying Dijkstra algorithm as it’s done in implementation of RGBD SLAM [http://vision.in.tum.de/_media/spezial/bib/endres12icra.pdf]. I’m going to reproduce their implementation graph construction but adapt it for our odometries (their code is available as a package for old ROS electric). This can not solve all problems due to inaccuracy of odometry but it will increase robustness of our algorithm.

Action Items

Gary

Talk to Cat at Google
write a Mac tutorial to set things up (maybe with Vadim)

Vincent

ask for a travel request for ECCV for Vincent and one guy from Itseez

Ilya

.

Vadim

.

Victor

.

Ethan

.

From Last Time

Gary

Talk to Cat at Google
~~ticket 2.4.3 for auto doc of samples~~

Vincent

.

Ilya

.

Vadim

~~Look at trained cascade classifiers to decide on what should be included~~

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-09-04

Agenda

website, wiki — so that users can add information. Right now, install to Mac is pretty bad. How to install with QT? What version with what? etc
ECCV
New code … docs?
Transparent
Model capture
FABMAP
Mac
GSoC visits
Vendor datatus

Minutes

Wiki for users? Installation on Mac is missing. Installation is getting pretty standard.
- We need a user wiki again. Evgeny do do?
- Duplicated obsolete information
- Developer
ECCV
- OpenCV soon to be a vendor — was a mail slip up.
- I will be there
- Alexander Shishkov, Kirill from NVidia about Android(?) Might not come.
- Need to cost this out and see where we stand
New code Make sure there is documentation and examples
- FABMAP — documentation
- Simple flow. Short description and sample code
Model capture
- Loop closure — extend
Transparent
- Take advantage of specularities
- Use the model fitting that exists
- RGBD + pre pose needed.

Vadim

[GSoC: Yury Zemlyanskiy]
Finally, we’ve integrated simpleflow dense optical flow algorithm into OpenCV. It needed couple of
review rounds; and the latest version looks great (if only quite a bit slower (more than order of
magnitude?), comparing to the near-realtime Farnerback’s algorithm). Still, it would be interesting
project to implement dense optical flow benchmark in opencv test system, similar to our features2d
nd stereo correspondence test frameworks.

[Code Contributions]

OpenFABMAP (http://code.google.com/p/openfabmap/), implementation of FAB-MAP algorithm, has been
integrated into OpenCV. The code has been submitted by the author who agreed to change the license
to BSD. The code includes documentation and sample code.
a small yet useful extension to PCA algorithm that computes the number of components to retain from
the specified standard deviation has been integrated via github pull request.
another useful contribution came from NVidia tegra team; OpenEXR source code is put into OpenCV build
tree, wrapped into our CMake build engine, so that highgui is compiled with OpenEXR support by default
on several platforms (Windows, MacOSX, Android).
parallel version of the retinafilter, a part of OpenCV contrib module that was contributed and is being
maintained by Alexandre Benoit, has been created by Alexandre.

[Testing: Ilya Lavrenov]
More strict tests of image warping functions have been created by Ilya Lavrenov. The tests for resize,
warp affine and warp perspective have been
stabilized, remap test is in progress.

Marina Kolpakova



Accomplishments & statuses:
2) Image Denoising (Anatoly)

Fixed CPU/GPU results differences in bilateralFilter. Speedup is
20-35x (C2080, i5-750, TBB, SSE optimized).

3) Hough Transform (Vlad)

Implemented a GPU version of the generalized Hough Transform for
the translation, translation + scale and translation + rotation
cases (6-10x speedup).

4) Pedestrian Detection (Marina)

Prepared soft cascade (same as cascade in the original paper).
Implemented reading, writing, and XML representation for soft
cascade.
Implemented a detector that uses this type of cascade.

Plans:

Investigate the OpenNI stability problem. (Roman)
Implement a GPU version of the generalized Hough Transform for
translation + scale + rotation case (fast version). (Vlad)
Test detection rate; integrate ICF (Integral HOG-like channel
features in the training application) features. (Marina)
Get acquainted with Android architecture and the Kiwi library.
(Andrey)

Other:

Anatoly is on vacation until 2012-09-17.

Ilya

Transparent objects [Ilya]
Finished the presentation about our algorithm and gave a talk about it at the Itseez seminar.
Thanks to all for activity, questions and feedback.
It helped to understand how to present the algorithm and write a paper about it.
Started to write a paper about our latest results.

Model capture [Maria]
Added to the graph construction several steps/features from RGBD SLAM
[http://vision.in.tum.de/_media/spezial/bib/endres12icra.pdf]. Now the graph structure is the following:
- each node is a frame pose,
- 3 types of edges for each node:
1) edges connecting the pose with several previous poses,
2) edges to several keyframes (if there are close such ones)
3) edges to nodes that are found by Dijkstra algorithm (if there are close such ones; excluding edges from 1) )
But results of such more complicated graph are similar to the results of last week. The possible problem of this is
that our odometries are less stable to a large transformation between frames than the method from the paper (I’m not
sure that we have a chance for a loop closure with such sensitivity to the difference between frames). I’m investigating
this and the ways to improve the graph results (e.g. initialization of our odometries by a pose from features2d + 3d-3d).

Other:
Maria took two days off for personal reasons (Aug 30, 31).

Action Items

Gary

Send note to Vadim about creating a user wiki (with log in and password)
See about forwarding old site to new site
Cost out ECCV
Send Victor an invite letter for GSoC Mentor summit. Date + week past

Vincent

.

Ilya

.

Vadim

Update tutorials
- Make sure we have installation on mac

Victor

Talk to NVidia team about giving tutorial
Cost out ECCV

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-09-11

Agenda

Vincent: sparse matrices. Hilton told me that sparse matrices are barely supported (you cannot add/multiply them). Is there a plan to support them, maybe using sparsuite?
Should the website still point to Yahoo groups?
Google analytics on our sites
- Google ads?
OpenCV retrospective and going forward
- Want to document what’s changed in 6 months in terms of
  - infrastructure (build system, modules, bug fixes, new websites, downloads, git …)
  - speedups
  - functionality, including new GSoC things, transparency
- Going forward: new areas:
  - SfM, VO, VSLAM
  - Make a case to M and S
  - V needs.
  - Robot specific OR
ECCV plans, expenses
- Rome visit
Adrian on devzone
Forum … answers opencv.org
- Forum for answers … new algorithm
- Open a forum for undergrads
Ilya paper for ICRA

Minutes

Retrospective. Want to prove utility. Plane detector
- ROS stuff, output
- OpenCV past 6 month. Prepare talk
  - 2 researchers, Ilya and Maria.
  - Website, git, bugs, optimization, new functionality, GPU, Android, iOS
  - GSoC, Maria odometry being used
- Good for general robot support in new application areas: research of course, health, retail, lodging
  - Slam, object recognition, arm calibration given cad model. Prepare a generic talk for this. 160K downloads/month
    - operating doc summary
    - MC&S. Project, tasks: health:
      - Suggestions grasp objects better, interact with people
    - L
    - S Navigation, obj rec. Need lighter weight
Google analytics. Docs 1000 new users/day
ECCV
- Gary stay with Vincent
- T shirts, stickers
- Nadri also at Rome
- Sefano Fabri
Ticket is opened about sparse matrices

Vadim

[Bug Fixing]
12 tickets in OpenCV bug tracker have been closed. It includes crashes in the core and imgproc modules, in Python bindings, errors
in documentation and build scripts etc.

[Tests]
Ilya Lavrenov has nearly finished work on the new tests for image warping functions: remap, warpAffine, warpPerspective, resize.
The test for resize with area interpolation is still in progress.
Recently, Ilya has optimized those functions, and we plan some further optimization, since they are critical. And for that we need
more strict tests, which Ilya did.

*Roman Donchenko (GPU) *



Accomplishments & statuses:
2) Hough Transform (Vlad)

Finished with the Generalized Hough Transform and integrated the
code into OpenCV. There are two implementations:

o Ballard for the translation, translation + scale and translation +
rotation cases.

o Guil for translation + scale + rotation case.

The GPU implementation is 6-15x faster than CPU with TBB (Tesla
C2050 vs Core i5-760).

3) Pedestrian Detection (Marina)

Added support for shrinkage to OpenCV soft cascade.

Investigated Benenson’s demo.

Implemented ICF (Integral channel feature) in the OpenCV traincascade app.

Collected some pedestrians datasets for training.

Issue: datasets needed for training cyclist and motorcyclist cascades.

Plans:

Add depth visualization and recording to the demo app; ask libusb
maintainers for advice on the data loss. (Roman)

Continue learning the Java Native Interface and get acquainted with
the Kiwi library. (Andrey)

Try to improve performance of several primitive functions (ex.
pyrDown, pyrUp). Investigate Super Resolution algorithms. (Vlad)

Continue with training. (Marina)

Other:

Anatoly is still on vacation until 2012-09-19.

Ilya

Model capture [Maria]
Read several papers about robust back-end for pose graph SLAM because I did not get improvements from graph
optimization for some TUM datasets last week ago (I suppose the reason is outlier and close to outlier edges). The
interesting paper was “Towards a Robust Back-End for Pose Graph SLAM”, ICRA 2012. Conclusion: outlier edges is
very hard problem having a particular importance for a long-time complex trajectory, so we don’t plan to solve it
for our task now. At first we will suppose that the odometry does not break a trajectory (uncritical number of outliers)
and we have special kind of trajectory (one turn of/around the table).

Implemented a masker of the train images to find the pixels of the table with the object. The reason is that our
odometries suppose stable scene, so we run them only on masked area. The masker is used Vincent’s RgbdPlane
(that copes very well with its task) and pcl convex hull and prism.

Experimented with the last weeks pose graph on some old TOD-like base (turning table with fiducials). Tried all
our odometries on it. The ICP and RgbdICP was not stable for such kind of scene (large plane of table) whereas Rgbd
works well even if there is relatively poor texture on the table (circles grid). The simplest structure of the graph
showed the more accurate results on the base (one edge to the previous frame and loop closure with the first frame).
See example of results in attachment: before and after graph optimization for the cup; only after graph results for
other objects. Notes: 1) circles grids existing in the images are not used, they are only texture; 2) strange pink points
around the table are my hands.

Transparent objects [Ilya]
Finished the first draft of our paper and sent it to Victor for a review.
Cleaned the code used in the paper experiments.

Action Items

Gary

Make some foils and share with Victor
Write Stefano to set dates for Rome

Vincent

Figure out a way of getting the number of clones from github talk also to Vadim
Set up Victor and Ilya talks
Schwag store (password with capital words and numbers).

Ilya

.

Vadim

Figure out infrastructure, optimization, bug advances, build, modules, GSoC stuff over last 6-9 months
Figure out a way of getting the number of clones from github talk also to Vincent
Adrian on devzone

Victor

Get Google Analytics on the site
Prepare OpenCV talk

Ethan

.

From Last Time

Gary

~~Send note to Vadim about creating a user wiki (with log in and password)~~
See about forwarding old site to new site
~~Cost out ECCV~~
~~Send Victor an invite letter for GSoC Mentor summit. Date + week past~~

Vincent

.

Ilya

.

Vadim

Update tutorials
- Make sure we have installation on mac

Victor

~~Talk to NVidia team about giving tutorial~~
~~Cost out ECCV~~

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-09-18

Agenda

schwag
Hilton’s code
2.4.3
ECCV
speed priority
OpenCV on Amazon
repo for common stuff (logos)

Minutes

Shishkov will update the Mac tutorial
NVidia will fund own person for ECCV.
talk at the university of ROME: scheduled
tutorials ready for ECCV. Get USB keys done the day before
schwag: contacted the online store, waiting for bug fix
ECCV schwag: T-shirts a bit cheaper, stickers design ready
Hilton’s code for training: ready to go, we need training.
2.4.3 delayed: tests are failing, bugs need to fix, hackathon at the beginning of October. Release November 1st.
OpenCV.org: add logos/stickers in the downloads section
Eric: Java. Beta module: deactivated for now. Needs more testing,really cool
Maria and Ilya: actually joint work for training

Vadim

WR

Anatoly

WR

Ilya

WR

Action Items

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

From Last Time

Gary

.

Vincent

send Vadim Hilton’s repo
put logos on website

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-09-25

Agenda

ECCV tutorial
FABMAP … other useful things from Willow such as features …
Foundation
- Intelligence Co
- Audience
- GSoC
Website improvements
- cheatsheet?
Hilton’s code
Transparency
VO
- Model capture
Basic line fitting stuff for industrial. Use nlopt? Or duplicate that functionality
- look up what is done.

Minutes

ECCV tutorial
- Tegra team are preparing for Android
- Everyone meet the night before to burn USB keys etc
FABMAP engine BoW image retrieval already in GitHub
- in contrib.
- 3rd party things will move to external modules
- Extra is place holder for test data etc. Contains training sets — cascades for latent SVM
  - New cascades are in data/objcascades: Profile faces and silverware. Pretty good results for profile. Silverware works on only vertical, so need to rotate images
Foundation
- Tune OpenCV main algorithms for MIC, contests
- Marketing budget … motivating use of algorithms/contests, tutorials
- grants
GSoC
- Ilya is ready, Victor still needs visa …
Website
- Cheatsheet is in code.opencv.org resource section. Needs to be off of docs and opencv.org
Hilton’s code, multi-part detector Deva Ramanan’s work
- Detector in C++, trainer is in Matlab/Octave
  - need to move this along in separate module & Hilton improve along with thesis. Source and binary compatibility
Transparent
- Accurate models ~1mm. But need to improve segmentation. Not independent segmentation, but info from several images
- Doesn’t depend on symmetry
- We’ll discuss transparent when Ilya is here
Model capture
- RGBD odemetry for poses
- Think about ways mesh and texture map
line matching
- Chamfer map — opencv sample/cpp/chamfer finds Apple logo
- distance transform
Andrea going to ECCV for tutorial. Need to meet the evening before

Vadim

[Bug Fixing]
over 15 tickets in OpenCV bug tracker have been closed.

[Tests]
Numerous OpenCV test failures (~10) on Windows, Mac, fresh Ubuntu versions have been fixed. They were
mostly caused by the incompatible libjpeg library. The plan is to move to .png files in OpenCV tests to avoid
such problems in the future.

[Optimization]
Evgeny added cross-platform threading to Cascade Detector (which is used for face detection, for example)
and HOG object detector.

Anatoly

WR

Ilya

Model capture of transparent objects [Ilya]
Improved the model capture of unpainted transparent objects. Developed a more robust algorithm
to remove artifacts on top of a model and implemented a 2-step coarse-to-fine approach to produce
models.

Collected a new dataset with 10 transparent objects to experiment with model capture.
Evaluated the algorithm using an object which has a shape of a truncated cone (see attachments
for an example of a train image and the computed model). This shape allows to produce the ground
truth model analytically and compare it with the created model. Measure of quality is partial directed
Hausdorff distance that is a percentile of distances between a point of the first model and the nearest
point of the second model. Results:

Partial directed Hausdorff distance (cm)

Percentile |----------------------------------------------------| | model --> ground truth | ground truth --> model | -----------------------------------------------------------------| 1.0 | 0.276 | 0.300 | 0.8 | 0.101 | 0.102 | 0.6 | 0.068 | 0.075 | 0.4 | 0.041 | 0.055 | 0.2 | 0.017 | 0.038 | ------------------------------------------------------------------ These results show that accuracy of the created model of this object is 3mm in the worst case and 1mm for 80% of points. However, the algorithm is not robust with other objects yet and can produce s ignificant artifacts if segmentation of an object is not stable.

Two assumptions are made when reconstructing objects:
1. Object stands on a support plane.
2. Object is hollow and has open top (the fundamental limitation of the approach is inability to r
econstruct concavities).

Other:
Arranged travels to the GraphiCon conference to present our work on transparent objects and to the
USA to attend the GSoC summit and make a new demo on the PR2 robot at Willow Garage.

Model capture of opaque objects [Maria]
Complemented the graph structure to refine locations of object points too. We model each object point using
its 3D coordinates and minimize its distances to corresponding points from several frames. The current approach
gives slightly more accurate models than the models from the graph for the camera poses only. It seems the results
can be better if we consider the local structure of each model point too (e.g. as it’s done by Ruhnke).
See examples of reconstructed models in the attachments (Note: circles grids in the training frames are not used
for the odometry).

Action Items

Gary

Contact Audience
Alexander from Itseez, Andrea for Tegra coordinate with Vincent for loading USB keys
Other: grants

Vincent

What is the status of the new work on feature detectors descriptors.
Ask Hilton to prepare multi-part detector as a module (docs, tests …)
- Maybe he can have ongoing part in this as his thesis advances, his stuff is already out there

Ilya

.

Vadim

Send email CC’ing me, Vincent, Andrea and Alexander to coordinate meeting night before
Mail to Alexander to be able to put a cheatsheet up and get yourself permissions for changing the site

Victor

Case for foundation. Help with financials. Late Nov.

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-10-23

Agenda

Getting off lots of travel
Other methods of funding
Willow check
Code status

Minutes

OpenCV linux boxes … with service contracts
Service contracts
- Build bot
  - 1. Make sure build bot functions for a particular build /once a year (we just run build bot and builds that specific build. fails are your problem)
  - 2. Want to keep build to be consistent for each release of OpenCV. Build pass all tests. /year (builds are our problem with releases)
  - Call center
  - Movies and tutorials. Setting up environments
Fitzg at MS
- MS Accepted license, can use internally
At GSoC collaborate across graphics, image types
Willow Octo map grids, calibration — arm alignment, arm planning. Grasping
- parts based detector
Working on 2.4.3
- End of this week probably
- Make all the tests pass
- Main highlights
  - GSoC python samples, denoising, new optical flow, new cascades for profile faces and silverware
  - lots of bug fixes.
  - Mobile support Android 4.1, iOS 6 (should get Windows mobile in)
  - parallelization, desktop Java, OpenCL (Chinese Academy of Science … but really plan this for 2.5, make transparent, if libraries are available, it will be used automagically).

Vadim

[OpenCV 2.4.3rc preparations]
12 more tickets have been closed; over 2 dozens of pull requests have been integrated
(most of the integration work done by Tegra team).

There are just a few things to do before 2.4.3rc is ready:

1. make our buildbot “green”. This is what we are currently working on. Most of the
problems are actually “false alarms”, but investigation of fixing some other problems
really unveiled problems in OpenCV OR the environment. For example, we found that
GCC 4.6 in Ubuntu 12.04 x86 is pretty buggy; when OpenCV is built with “-O3” option,
many tests fail on Ubuntu 12.04 x86 (and pass successfully on any other configuration).

2. make the actual packages for Unix/Mac, Windows, Android, iOS; test them; upload them to SF.

3. write the release notes (that’s my TODO); post the announcement.

[Automatic testing of pull requests: Evgeny]

Evgeny wrote a script (it’s beta for now and available only within Itseez) that allows
github.com/opencv/opencv admins to test each pull request using buildbot. In other words,
the patch from the specified pull request is applied to the specified branch, and this code is
compiled and tested in 3 configurations (Ubuntu, Windows, Mac). The next step is to make
buildbot to test each new pull request automatically and post specially formatted comment
to github. This way the process of pull request integration should be simplified a lot.

Anatoly

WR

Ilya

Model capture demo [Ilya]
Finished preparations to the Willow visit (got instructions from Maria how to use her model
capture code, collected with Maria a new dataset with a moving Kinect on a box to simulate PR2).
Attended the Google Summer of Code Mentor Summit.
Installed Maria’s model capture on a Willow machine. Collected a dataset for model capture
from a PR2 robot. However, the robot was controlled manually using RViz and it resulted in
too jagged movement. Plan to use the joystick for robot control and re-collect the data.

Model capture [Maria]
Captured with Ilya a dataset of a new type: rotate a Kinect around the table with an object
(our previous datasets had a fixed Kinect and the rotated table). Current model capture code
coped with such case of a moving camera too (odometry gave good poses and this is the base
step to get accurate models). So we hope that this’ll work on the robot. See the result in the
attachment (note: circle grids are not used).
Another iteration of the code refactoring was done.
Faced with the following problem on the new dataset: on stage of simultaneous refinement
of camera poses and model points we got several NaNs in Hessian of camera poses during second
iteration. I checked that we don’t pass NaN values to the graph and other hypotheses about the
reasons of NaNs, but without success yet. Unlike all our previous datasets (~25) where we didn’t
have this problem, the new one has bigger object, as result the graph has ~5 times more vertices
(~0.5M). Added the check of iterations with bad results to apply only success refinement.
Continue to investigate the reason of NaNs.

OpenCV [Maria]
Added 3 new tests on core module: cv::merge, cv::split, cv::phase.

Action Items

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-10-30

Agenda

Release 2.4.3
New MSER
DAFT functionality
Atom
Surface/Win phone
Transparency
Model capture

Minutes

Release candidate 2.4.3 is out https://github.com/opencv/opencv/wiki/ChangeLog#243-rc
Chinese Academy of Science working on OpenCL with …
Mobile I-chips
- Port, tune flags
- Work interoperability: new sensing, perception data formats
- Work on tutorial style examples
- Work on real demo (image retrieval, summarization?)
- Optimize based on above bottlenecks plus stitching, calibration, Features2D, Calib type stuff
New MSER Matas. Evginy working on ER retrieval. Letters or structural shapes don’t correspond to MSER but ER.
- Image has few millions of ER. Extract and analyze each region in constant time. Prune the uninteresting. Get CC of things that are similar to shapes that you are looking for
RGBD module in OpenCV for simple convenience functionality. Alexander look at it.
- Features 3D to encapsulate this data.
- DAFT in WG. More general version in Ecto repository.
- Need feedback on what should be there
Synthetic view of an object. Visualizer for 3D data
SFM module too, probably should be separate from RGBD
Transparency
- Collected data of bottles.
- Opaque object model capture
Model capture
- Model capture going. Using ROS, publishing odometry online model learning
  - Robot captures model, spits out point clould
  - Vincent on auto training for linemod
  - Review on Thurs. 5 points. Same people … maybe one more.
    *

Vadim

[OpenCV 2.4.3rc]
The release candidate has been released. The changelog is here: https://github.com/opencv/opencv/wiki/ChangeLog#243-rc
The final 2.4.3 is expected on Nov 2nd. Big thanks to Tegra team who helped with the release (as they do regularly during this year)!

[Development process]
Evgeny continues work on the technical aspects of the new workflow. It will be a page (it’s now in alpha version) that will allow
developers to see the status of their patches. Perhaps, commits from regular experienced contributors will not require the use o
f too heavy pull request mechanism.

Anatoly

WR

Ilya

Model capture demo [Ilya]
Wrote a demo application for a robot which grabs data from a RGBD sensor and constructs a model using Maria’s algorithm.
Visualized odometry poses in rviz. A robot path is tracked with Maria’s odometry and it is visualized in rviz simultaneously (see a screenshot).
Experimented with model capture on different robots and objects. Results are good and quite robust (attached several examples).
Thanks to Maria for help with the demo.

Other:
Collected several datasets of transparent objects at IPI.

Model capture [Maria]
Experimented on the dataset from a robot to understand the differences with our datasets that can be considered in the algorithms.
Implemented online frame-to-frame processing.
Implemented refinement of no-keyframes poses within segments of already globally refined trajectory of keyframes.
The goal is to fill missing and invisible information in keyframes set and to get more accurate filter of outliers in the model
point cloud. First result of no-keyframe poses refinement is unsatisfactory. Investigating the reason of this now.

Action Items

Gary

.

Vincent

Set up RGBD module feedback on list
Set up SFM module feedback on list.

Ilya

Make transparent bottle data available to me

Vadim

.

Victor

.

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-11-06

Agenda

Fun with funding
2.5
Background subtraction

Minutes

3 people allocated to new port
- Many of the optimizations should be applicable
- Need to set up SOWs and normalize the levels
2.4.3 is out
- This is pushed out to ROS build farm, no problems found!
2.5
- SOW this week
  - Transition to micro modules
  - Other people can more easily add modules and share
    - Makes the whole thing a distributed library like Boost
  - Better support for higher level languages (Matlab, Octave)
- Long list. Share by End of Week
- Separate C API
- Future proof new interface
- Khronos compliant
Transparent data, how to prepare. Discuss with Ilya later
New background subtraction code. Looks good
Victor is scheduling his visit week of 12. Khronos visit.!! Will coordinate time and G meeting.

Vadim

[OpenCV 2.4.3]
The final OpenCV 2.4.3 is out. The announcement is put to http://opencv.org. After the release candidate we have
fixed in total ~10 bugs together with the Tegra team (mostly various compile problems) and made our buildbot
(build.opencv.org) green. Well, sometimes it gets orange or red because of some internal instabilities, but in general
we fixed most of the test failures by 2.4.3. Currently, we are planning 2.5; the draft version of the proposal should be
ready this week.

Anatoly

WR

Ilya

Demo of model capture
Collected a new dataset from a PR2 robot for subsequent experiments with model capture at Itseez. The dataset contains
33 bag files; the robot makes one loop around a table in each bag file.
Conducted more experiments with the model capture. Attached several examples of new results (the bear was scanned
while lying on its back). However, these models don’t have points for regions which a robot didn’t see. For example Mini-mouse’s
chin is missed.
Vincent and me finished the preparation of the demo and showed it at Willow Garage. RGBD odometry, model capture and
LINE-MOD were demonstrated as Vincent already wrote. The demo went well in overall, however, tracking was not robust and
loop wasn’t closed because depth was corrupted by skylight as we checked after the demo. So we had to use a precomputed
model which we created in completely the same setup as the demo but at the night before. Still we showed several good
models produced with a robot so people could see and appreciate results of the algorithm. Also other parts of the demo
went without such problems.
Thanks a lot to Vincent and Maria for crucial help with the demo.

Model capture of opaque objects
Found the reason why my last week refinement of no-keyframe poses gave bad results. Checked several approaches to
do this refinement. The final version is rebasing no-keyframes within globally refined keyframes and optimization of camera
poses graph for each keyframes segment of the trajectory. This approach is very fast and gives accurate results (see the attachment:
green point clouds are all globally refined keyframes, other color point clouds are all no-keyframes). The other checked versions
were more complicated (with ICP and RGBD edges), but did not show improvement and they worked unacceptably long time
(even by segments).
Implementation of view angle histograms of model points (for more stable outliers filtering; using all frames) is in progress.

Other:
Monday was a national holiday in Russia.
An extended abstract about our work on transparent objects was written for a Summer School on “Robot Vision and Applications”.

Action Items

Gary

.

Vincent

Funding

Ilya

Scripts or how to prepare the transparent bottle data.

Vadim

Send out 2.5 specs

Victor

.

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-11-13

Agenda

New funding
I funding
2.5
CLM

Minutes

Android port works perfect for Vincent on first try. Kudos to the Android team!
OpenCV Manager will select what’s needed, pull in the correct version na
The new descriptor is performing extremely well, beating all prior
- logpolar trick for working with scales
Maybe we need to get 2.4.3.1 fix for 3×3 inversion for signal precision breaks down with illcondition. This was turned on by SSE
optimization but has to be taken out.

Vadim

[post OpenCV 2.4.3: Bug fixes]
Several bugs, found in 2.4.3 have been fixed: bad accuracy in 3×3 matrix inversion, broken update method in
LBPH face detector, data races in parallel video reader/writer, broken build on Windows with OpenGL enabled …

[Infrastructure]
Evgeny continues to improve the automated OpenCV build and testing infrastructure:
- integrated developer branches testing functionality into web GUI;
- improved the distributed testing of the submitted pull requests.
- fixed several try jobs merging into a single build;
- modfied other build definitions: e.g. no “success” if builds with warnings;
- “merge button” proof of concept prepared. The idea is that the pull requests that passed all the tests and got approved could be
automatically merged into the repository.

[OpenCV 2.5: the Proposal Draft is ready and has been discussed]
Here is the document that I prepared: https://docs.google.com/a/itseez.com/document/d/1L4pCxMXLNyjFvZAbH-xDZghQCKDhbckjBAYFssPHWD8/edit
It has been briefly discussed with several developers from various teams. OpenCL team is also in the process of reviewing it.

[The proposed changes in OpenCV roadmap: 2.4.4 and 2.5.0]
Since there is a lot planned for OpenCV 2.5, it would be very challenging to prepare some alpha release by February. So instead the
suggestion is to:
release 2.4.4 in February
release 2.5.0 in May

Anatoly

WR

Ilya

Ilya is on vacation from Nov 7 till Nov 16.
Maria is on vacation from Nov 7 till Nov 20.

Other:
Ilya won a $850 travel grant to attend “Robot Vision and Applications” Summer School. Thanks to Vincent for writing a recommendation.

Action Items

Gary

Set up funding allocation meeting with Victor/Vincent
Ping I on getting new project going
Get creative going under ubuntu
Read & comment on OpenCV 2.5 plan

Vincent

Read & comment on OpenCV 2.5 plan
Surface noise level of create

Ilya

Read & comment on OpenCV 2.5 plan

Vadim

Read & comment on OpenCV 2.5 plan

Victor

Read & comment on OpenCV 2.5 plan

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-11-20

Agenda

C++ exceptions
blog items
wikipedia
2.5
- Matlab/Octave interface

Minutes

Evgeny did cool work so that we can see, via the build bot whether pull requests work before we pull them
Google hates exceptions. Apparently there is no gcc pluggin that handles exceptions that is not GPL (although, note that Linux kernel is GPL).
- But GPL free exception support in C++ is coming. Maybe leave for now
Send blog items to Alexander Shishkov
Updated links around the web, need to get someone on this full time
Matlab/Octave interface — perfect for next Google summer of code
Put comments into 2.5 document prior to publicizing it

Vadim

[Infrastructure]
Some more results from Evgeny who works on OpenCV infrastructure:
- finally set tdr parameters to tegra-win32 buildslave eliminating timeout-related test failures;
- successfully carried out more tests of “merge button”;
- refetched commit history at code.opencv.org to avoid any whitspaces in ‘Repository’ section;

The latest version our the integration page, where the status for each pull request is attached.

[Edge-aware demosaicing]
Ilya Lavrenov has implemented edge-aware demosaicing algorithm. It has been threaded with nearly
linear speedup on dual-core CPUs. The regression tests are there too.

New system to see if pull requests work via the build bot before pulling

Anatoly

WR

Ilya

Maria will return from vacation on Nov 21.
Ilya returned from vacation on Nov 20. Started to write an extended version of our GraphiCon’2012

paper for post-proceedings which will be published in Springer Lecture Notes in Computer Science.

The new paper will contain more details about our previous work on transparent objects using a stereo

camera.

Action Items

Gary

.

Vincent

.

Ilya

.

Vadim

Ask Alexander Shishkov about putting in blog items
Ask Alexander to investigate whether Google ads would be worth it. Buy servers, infrastructure

Victor

.

Ethan

.

From Last Time

Gary

~~Set up funding allocation meeting with Victor/Vincent~~
~~Ping I on getting new project going~~
~~Get creative going under ubuntu~~
~~Read & comment on OpenCV 2.5 plan~~

Vincent

~~Read & comment on OpenCV 2.5 plan~~
~~Surface noise level of create~~

Ilya

~~Read & comment on OpenCV 2.5 plan~~

Vadim

~~Read & comment on OpenCV 2.5 plan~~

Victor

~~Read & comment on OpenCV 2.5 plan~~

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-11-27

Agenda

Ads?
New organization
Update on new client
Content plans

Minutes

Sorry. Meeting pre-empted by OpenNI discussion and by VIP visit at Industrial Perception.
2.5 Module division proposal below

Vadim

[Infrastructure: Evgeny]
Evgeny has practically finished the work on the pull request testing page:
- “merge button” finally can be used, several pull requests were intgerated;
- fixed merge button behavior in case of any git command failure;
- published pull requests testing page at http://pullrequest.opencv.org, so it is now visible for everyone
(thanks to Alexander Tsyplenkov and Alexander Shishkov)

[Demosaicing: Ilya Lavrenov]
Ilya improved implementation of edge-aware demosaicing algorithm, now the performance should be much better.
bayer2RGBA has been parallelized. The parallel version works fine on Mac/Win and now is being debugged on Linux.

[Bug fixing, Misc]
several pull requests from users are at the negotiation stage (improved capturing from Prosillica cameras, all-in-one
connected component extraction algorithm, …); documentation build problems have been fixed; 3×3 matrix inversion has been fixed

[OpenCV 2.5 discussion]
The rough new subdivision of OpenCV has been proposed:
https://docs.google.com/a/itseez.com/spreadsheet/ccc?key=0AoYFDB08hRs6dHFzMzRtVFppdXFtMzQzQ1JxRmZXd1E#gid=0

it will likely be refined.

[Infrastructure: Build server]
it appears that we may need the build server, but not as urgently as we initially thought, we will try to optimize the amount of
work that the current buildbot does. Terribly sorry for alarming you. We will get back to this question next year.

Anatoly

WR

Ilya

Transparent objects [Ilya]
Finished the paper on transparent objects for the GraphiCon’2012 post-proceedings. Wrote about
our algorithm of edge model creation inspired by bio-informatics and added an example of working
with textureless objects.
Prepared a poster about transparent objects for the IEEE RAS Summer School on
“Robot Vision and Applications”.
Sifted through ECCV 2012 papers. Found one paper on the same problem that I was struggling
for a month in model capture of transparent objects: “N-tuple Color Segmentation for Multi-View
Silhouette Extraction” by Abdelaziz Djelouah et al. Unfortunately, It cannot be applied directly
because they wrote “Failure cases have been observed with configurations where the background and
foreground color distributions are similar”. Going to study it and linked papers more thoroughly
to check for ideas that can be useful for transparent objects too.

Other:
Gave a talk on Projective Geometry in Computer Vision at a student school.

Model capture [Maria]
Returned from vacation on Nov 21. Refactoring the code and reading more papers about model
capture (stable camera tracking, in-had scanning, etc). Both are not finished yet.

Action Items

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

~~Ask Alexander Shishkov about putting in blog items~~
Ask Alexander to investigate whether Google ads would be worth it. Buy servers, infrastructure

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-12-04

Agenda

OpenCV hiring needs
2.5
Donors

Minutes

People asking how donation is used
- We have to make this clear. Soon moving to larger support by Willow, Intel and Google, but other’s asking. So far we have
- http://opencv.org/contribute.html
Noise model for depth for rgbd
- sensor module declare sensor type with noise model.
2.5 have early prototype. But 2.5 is intended to be stable for several years
- By next week get shell to compile
- Pospone book for this. API agreed by API
  - Set of modules have been changed
  - Some functionality moved to different modules
  - C API moved to (new) compatibility module
    - Legacy module for older stuff
  - Style of using class instances, you use smart pointer
    - Like NVidia capture method
  - Look at code samples and start changing

Could use a maintainer



The maintainer’s OpenCV time is spent on three activities.

- Communication (60%)

Mailing list discussions on general design, fielding user
questions, diagnosing bug reports; reviewing, commenting on,
suggesting alternatives to, and rejecting patches.

- Integration (30%)

Applying new patches from the contributors while spotting and
correcting minor mistakes, shuffling the integration and
testing branches, pushing the results out, cutting the
releases, and making announcements.

- Own development (10%)

Scratching my own itch and sending proposed patch series out.

Vadim

[OpenCV 2.4.3.1]
Very minor update to 2.4.3 has been prepared (by Tegra team) and is available from
https://github.com/opencv/opencv/tags. It’s so minor that we will not make any public announce.
Most of the fixes relate to Android.

[Infrastructure: Evgeny]
Some further improvements have been done to the pull request testing page (the “external”
variant of the page is available at http://pullrequest.opencv.org):
- “merge” button gets different colors, depending on the comments and build status
(e.g. when the last comment is not :+1: or :shipit:, the button gets orange rather than green)
- the last comment is printed at the status page
- the behavior is now more robust: we set time limit for each step of precommit build,
thus invalid code does not break the buildbot.

[Demosaicing: Ilya Lavrenov]
Ilya fixed the remaining bugs in the parallel edge-aware demosaicing and added parallel
version of bayer2Gray.

[OpenCV 2.5 concept]
I have started preparing the concept version of OpenCV 2.5, which is now at
https://github.com/vpisarev/opencv/tree/2.5_concept (completely broken for now).
The compile problems will likely be fixed by the end of the week; the tests will take
some more and then we will continue to refine it together.

Anatoly

WR

Ilya

WR

Action Items

Gary

Get accounting info to Vincent
Contact Adrian about book pow-wow

Vincent

Writing down donation pages

Ilya

.

Vadim

Send schedule so we can sync book with 2.5
Send plan also for ML module

Victor

.

Ethan

.

From Last Time

Gary

.

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-12-11

Agenda

2.5
- Book info on 2.5
- ML info
Donation page
Website
Intel
transparency

Minutes

Release 2.5
- Studying partitions and refactoring. Illiminate lessor used stuff. Slim everything down
- Big blocks such as ML, how to handle? Other things are much easier
- Plan for 2.5 to compile in Jan
- How to avoid cycles in dependencies for things that can be members of several categories?
- Module size … most quite small, some
- Compatibillity module to help with transition
  - Beta end of April.
  - Release is planned for end of June
Sync 2.5 with the book
- Telcon Vadim with Adrian and Gary
- Look at github https://github.com/vpisarev/opencv/tree/2.5_concept
Donation page
- Vincent
Website, need instructions on how to modify for all admins
- News
- Blog needs to be set up. Separate site and reference it
- Donation give/get needs to be set up
Evangalizing position will come up in 2013.
- Need to find someone who can step in full time and
  - drive code bounties,
  - website,
  - help with fund raising
Vincent talk at UCSD
- Vincent to France Wed.
- Mid Feb. Set up talks (Pr Nk)
2.5 content
- AR big (duh)
- Model capture
- precise tracking
- SLAM
- Movie. Aggarwal Panorama
- Motion capture
Transparent object
- New (secret) approach is working well and fast
- Combine with existing techniques, use above to initialize
Model capture
- Work with better trajectories.
  - Now we circle objects and reconstruct with path completion
  - Random trajectory next
    - We need ease of use
  - Colored points
    - linemod to work in training with this
    - Meshlab to mesh
Vadim on vacation next week to end of December.
Register OpenCV coverity http://scan.coverity.com/start/

Vadim

[Infrastructure: Evgeny]
- We significantly reduced the load of our buildbot, in order to make it more responsive and free some resources for more
extensive testing. So far, the number of builders have been reduced from 50+ to ~30.
- All the invocations of the buildbot builders are now recorded in a DB, so we have more robust and much more
responsive http://pullrequest.opencv.org.
- A separate project, called meta-opencv, is now created at code.opencv.org. We will use it as a place to work on
OpenCV infrastructure. In particular, we put there our buildbot scripts.

[Demosaicing: Ilya Lavrenov]
Ilya started studying SSE2 intrinsics and as a first assignment he optimized the edge-aware demosaicing algorithm.
The SSE2-optimized version is ~4x faster than C version. Now he is working on optimizing some cases of the resize function.

[OpenCV 2.5 concept]
Vadim continues to work on the OpenCV 2.5 concept at https://github.com/vpisarev/opencv/tree/2.5_concept. So far the
modules hal (it’s almost empty for now), core (except for XML&YAML I/O), imgproc (except for contours and few more functions),
bgfg (100%), optflow (50%) are ready. It can take some more weeks to make it completely ready. Given the coming holidatys,
I expect the prototype to be ready and pass some tests to be ready by the early January.

[Plans before NY]
Evgeny plans to switch to non-intensive maintainance mode of the OpenCV infrastructure and spend more time on the actual
code work, e.g. Matas approach for fast connected component analysis.
Vadim plans to take vacation from the second half of December.

Anatoly

WR

Ilya

Model capture of opaque objects [Maria]
Finished next step of the code refactoring.
Implemented missing functionality: getting a model, reading from / writing to a ply file.
Prepared and gave a talk about model capture at Itseez.
Implemented refinement of an object mask after table segmentation. Sometimes outliers in this mask were
a reason of dangling points in the final model.
Implemented the old idea to compute histograms of view angles for each point using all frames (including
refined no-keyframes) and filter the points basing on such histograms. But it appeared that this filter is not
more accurate on our data than current simple filter by correspondences count.
Made simple experiment to improve a model texture: implemented texturing version that set for 3d point a
color that is weighted sum of its projections color on several frames (the weight depends on point normal).
Again this does not give better texture visually.

Plans:
Read more SLAM papers to go to scanning with arbitrary camera trajectory.

Model capture of transparent objects [Ilya]
Studied papers “N-tuple Color Segmentation for Multi-View Silhouette Extraction” and “Automatic 3D object
segmentation in multiple views using volumetric graph-cuts” in more details. The last paper is very similar to
our current approach. However, both papers don’t solve our critical issues and shouldn’t work well in our case
when background and foreground are extremely similar.
Revised our problem of 3D reconstruction and formulated it more rigorously. It is energy minimization on a
factor graph. The issue is the graph has loops and factors have order of 10-200 terms (it depends on the
algorithm parameters). I don’t know how to find the global minimum for such a graph. Going to try to devise
an adhoc global optimization scheme by exploiting the specific structure of the graph, otherwise use
approximate inference or re-formulate the problem.

Other:
Ilya was on vacation and he attended IEEE RAS Summer School on “Robot Vision and Applications” in
Santiago, Chile (December 3-7).

Action Items

Gary

Send Vincent balance pdf
Send Vadim Intel SOW

Vincent

Donation page
Set up talks for when you come back in Feb.
Check out http://scan.coverity.com/start/
Ping Celia, make sure Willow has OpenCV wire info

Ilya

See about resources for doing vision work for use & publication
- Transparent stuff
- Linemod features that train well from model and work well with real camera test

Vadim

Write Adrian and Gary
Decide on book

Victor

.

Ethan

.

From Last Time

Gary

Get accounting info to Vincent
~~-Contact Adrian about book pow-wow~~-

Vincent

Writing down donation pages

Ilya

.

Vadim

Send schedule so we can sync book with 2.5
Send plan also for ML module

Victor

.

Ethan

.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012-12-18

Agenda

2.5
Text localization
Industrial
Barcode
Connected components general framework? (Jason Newton code)
Book considerations
- ML
  - good examples
- Code examples
Transparent
Model capture
Vacations: This is the last phone meeting of the year. We will convene again Jan 8th

Minutes

2.5
- Some work. Vadim dealt with pull requests mostly
Text localization
- But good technique for connected components/extremal regions where they can be split
  - Different channels, H, S, V. Different thresholds on each channel. Raising threshold, watch the extremal region emerge. Use classifier to reject non-letter components
  - Constant time algorithm … so optimized this is fast constant time
- Would like to make general — the training step. For now, just get it working, then optimized
We have a new fast connected component contributed by Jason Newton. Cool!
- Each algorithm, even when it is “finished,” takes a month to get ready for real inclusion.
  - This is necessary, we no longer want to bloat up on junk. Everything should be engineered, have a reason to exist, work well in the entire framework.
Industrial
- Should there be a module in this area?
  - Probably yes, but would be kept minimal. Lines, templates, barcode aides (not worth doing too much in barcodes, this has already been solved)
  - Barcode normalization, subpixel lines
- question of priorities
- Probably good for longer term interns. Probably too complex for summer of code kind of effort.
- Crowd source funding?
ML
- Letter recognition
  - 6 different algorithms NN, random trees, boosting, svm, bayesian, knn
  - NN: Boolean vector of 26 elements
  - Boosting: Unroll data into a vector. Add categorical variable as last component of the input vector (binary classifier)
  - Random trees: Categorical classifier that gives the letter index
  - K-NN: Simple to use, several nearest features
- Python for hand written digits. Training and testing (svm, knn)
  - Image with various digits, trained and recognized by sliding windows
Transparent
- transparent silhouette in rotation for symetric objects
- Gary’s technique for recognition and pose. Worth investigating more/paper
Model capture continues to improve. Getting accurate texture is hardest part
- Collecting data, hand held
- Examining SLAM techniques for better tracking/loop closure
- Better sensors … would yield better results obviously. Motion blur a problem, global shutter a solution. Kurt may solve, working on it.

Vacations: Gary gone Saturday back 7th; Vadim back 7th Jan; Ilya back 7th.

Vadim

[New functionality]
- Evgeny is now working on implementing “Real-time scene text localization and
recognition” by L. Neumann and J. Matas. The draft non-optimized implementation
of the first most expensive preprocessing stage is ready; the next steps will be:
extensive optimization of the first phase, adding classification and the second stage.

- Jason Newton contributed fast algorithm for connected component extraction, which is
about to be integrated to the master branch (once we eliminate test failure on MacOSX).

[Optimization]
- Ilya implemented SSE2 version of resize area fast. Here are the execution times
in ms to process FullHD image on Core i5 2.5 GHz:

Resize Area Fast CV_8UC1 CV_8UC3 CV_16UC1 CV_16UC4
C version 502 1292 380 1982
SSE2 version 112 510 189 1218
Speed up 4.5 2.5 2 1.6

Now Ilya is studying HOG algorithm and will accelerate it further.

[OpenCV 2.5 concept]
Vadim did some more work on OpenCV 2.5 concept. Some more functions in imgproc have
been rewritten in C++, optflow module now compiles; the work on geometry module has been started.

Anatoly

WR

Vincent



on my TODO:

- not done yet with the foundation/donation page

- talk with YYYY in progress. XXXXX too

- we are now part of coverity ! My intentions were twofold:

- find new bugs in OpenCV: coverity is a static code analysis (the best out there) and they provide free analysis for Open Source software:

http://scan.coverity.com/all-projects.html

(we should be on there soon)

We did find some bugs (e.g. some memory leaks) but only under certain conditions or in C code/rare API.

Still, they should be fixed at some point. I sent a list to Vadim about the leaks.

- being on coverity is also a way to show that we care about industrial quality. Linux, Python, OpenOffice, PHP … are on coverity.

1100 industrial companies pay for it : Adobe, ARM, EA games, EADS, LG, Lenovo ….

I still need to setup automatic uploads of a build once a week. Should be trivial, I just need internet ….

- Switchover on funding for the new year is on track

Ilya

Transparent objects [Ilya]
Discussed with Victor our problem of energy minimization on a factor graph
in reconstruction of transparent objects. Decided to not use the graphic models
framework because we have high-order factors in our graph and it is a too difficult
problem for standard algorithms. Started to think about an alternative approach.
Supported Willow Garage in usage of our transparent objects pipeline. Experimented
with training and detection on their data. Training on unpainted transparent objects
didn’t work because it is still in development so a simple semi-automatic training was
implemented as a workaround. Detection worked ok, video with results was prepared and sent out.

Plans:
Develop reconstruction of unpainted transparent objects which will be less generic but
more robust (it will use the rotational symmetry assumption).

Model capture [Maria]
Read a lot of papers about SLAM to replace current one loop closure by an arbitrary graph structure
from a handheld sensor. Still there is no final decision which algorithm we should implement but we want
to have more robust odometry, false poses filter and a loop detector.
Switched to use Xtion Pro Live instead of Kinect because for the current odometry we need high FPS of
synchronized rgb and depth from a handheld sensor. Implemented a sample to capture the data from Xtion in
the base format we use for the model capture now.
Captured several datasets from the handheld device. The capture conditions of the first object were close
to ones we satisfied before (on a robot): one loop, relatively slow camera movement. But this time there was
camera jiggle from me and movements of my hands. The object was reconstructed successfully (see the attachment).
The capture conditions of other datasets were more extreme (fast movement, etc.). In these cases sometimes
trajectory was broken. Analyzing the problems of the odometry and how we can increase its robustness in more
extreme capturing.

Current capture setup and results

Action Items

Gary

Transparent write up to Ilya

Vincent

.

Ilya

.

Vadim

.

Victor

.

Ethan

.

From Last Time

Gary

~~Send Vincent balance pdf~~
~~Send Vadim Intel SOW~~

Vincent

Donation page
Set up talks for when you come back in Feb.
Check out http://scan.coverity.com/start/
Ping Celia, make sure Willow has OpenCV wire info

Ilya

See about resources for doing vision work for use & publication
- Transparent stuff
- Linemod features that train well from model and work well with real camera test

Vadim

Write Adrian and Gary
Decide on book

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2012

OpenCV Meeting notes for 2012 year

2012-01-10

2012-01-17

2012-01-24

2012-01-31

2012-02-14

2012-02-21

2012-02-28

2012-03-13

2012-03-19

2012-04-10

2012-04-17

2012-04-24

2012-05-01

2012-05-08

2012-05-15

2012-05-22

2012-05-29

2012-06-05

2012-06-12

2012-06-19

2012-06-26

2012-07-03

2012-07-10

2012-07-15

2012-07-24

2012-07-31

2012-08-07

2012-08-14

2012-08-21

2012-08-28

2012-09-04

2012-09-11

2012-09-18

2012-09-25

2012-10-23

2012-10-30

2012-11-06

2012-11-13

2012-11-20

2012-11-27

2012-12-04

2012-12-11

2012-12-18

Clone this wiki locally