Build a visual 3D recognition/search engine - QC-137

Preferred Disciplines: Computer engineering, mathematics, computer vision, Master PhD, Post-Doc
Company: Applications mobiles overview inc. (VyoO)
Project Length:  flexible, up to 3 years (40-50 Accelerate units)
Desired start date: June 2018
Location: Montreal (primary location), Quebec City
No. of Positions: Flexible (10-20)
Preferences: Language: bilingual

About Company:

Mobile Applications Overview Inc. is a company founded in France in 2015. It relocated its main R&D activities to Montreal, Canada. French offices are now a subsidiary of the company. Its founding executives have more than 20 years of experience in the field of business creation, development and management, real-time 3D communication and reconstruction, and development in the field of new technologies. The main commercial activity of the company revolves around the development of technological solutions to recognize objects through AI filtering, by processing of video media that allow the generation of 3D data (3D model reconstruction). The applications developed by the company can be used in several fields of industry including the analysis of road anomalies, industrial manufacturing (automotive, aerospace, industrial machinery, railway industry), watchmaking and jewellery, luxury accessories, ... Moreover, the technology developed could also be used in the field of learning and knowledge sharing (visual 3D search engine).

Since its creation in 2015 the founders have injected nearly CAD 535 K in their project and have just finalized a round of funding of CAD 650 K which values ​​the company to CAD 6.4 M. A new round of financing is planned for September 2018 to continue sustaining the development of the project and its deployment on the market.

Project Description:

Searching, identifying, connecting an object to the network is one of the major issues of years to come. As we can see the World Wide Web becoming more and more mobile, this recognition processes and techniques have to be adapted to mobile users and thus mobile devices. Several techniques have yet been developed to do so, such as 2D picture analysis, optical character recognition (O.C.R.), QR-Codes or Bar-Codes, geolocation, color recognition. They prove very useful and efficient in particular cases, like O.C.R. for books, geolocation for monuments or QR-codes when present, but lack of efficiency in most cases. Indeed, objects in today’s life are mainly defined in 3D and 3D parameters have to be taken in consideration to recognize them. Those parameters include peaks, tops, edges, shapes, reliefs.

An object of the invention is to propose a method for recognition of an object using 3D parameters, yet without scanning the object in 3D. With other words, the present invention’s purpose is not to develop a 3D scanner but to use three dimensional parameters to recognize objects.

Background and required skills

Research Objectives/Sub-Objectives:

  1. Geometrical 3D matching (No ML) (8 months, immediate start)
  • Sparse reconstruction in cloud points or meshes. Basis techniques
    • SFM (structure from motion) Algorithms > Points Cloud
    • SLAM (simultaneous localization and mapping) approach > Meshes
  • Pre-processing of the reconstruction. Basic approaches
    • Segmentation
    • Automatic Cleaning / Denoising
  • Automatic matching with database. Basis approaches
    • 3D Database Setup
    • 3D reconstructed models auto formatting (points clouds up-sampling and ply (example) generation, mesh triangulation and stl (example) generation)
    • Points Clouds or meshes automatic matching. Based on PCLs, Cloud Compare…
  • Integration of developed computer vision modules into unified framework and testing of usability
  1. From 3D Geometry to 3D feature vectors for Machine Learning (12 months, immediate start)
  • Local pre-processing of points clouds / neighborhood construction and analysis
  • Identification of shape descriptors (radius of curvature, local geometry, structure information)
  • Delivery of meshes as vector fields for analysis and processing
  1. Machine Learning on 3D Geometrical Classes (12 months, starts 6 months after start of project)
  • Using the point cloud directly as a feature vector and an input to the Neural Network.
  • Using the parameters of the point cloud, which is a preferred option since it is more efficient. To do so further studies shall be done to investigate the best invariant parameters to be used; for instance.
    • Geometric structure parameters
    • Environment parameters
    • Curvature information (it is considered to be the best invariant parameter for detection part of objects)
  • Scale-invariant feature transform (SIFT) for 3D. It can be matched to other SIFT point cloud, in addition to providing a confidence measure for that match
  • In addition, the option of combining both the 2D and 3D features into a single feature vector will be considered
  1. Dense reconstruction from 2D pictures, standalone application/B2C deployment (24 months, 12 months after start of project)
  • Scalability of the integrated solution
    • Big data
    • Server-side computing
  • Dense reconstruction from 2D pictures
  • Deployment of B2C application
  • Head development: Post-doc identified (currently finishing PhD at ETS)
  1. Market analysis, strategy setup and strategic deployment (18 months, 8 months after start of project)
  • Industrial manufacturing (aerospace, automotive, rail etc.)
  • Retail manufacturing (furniture, fashion garment & accessories)
  • Public works/parcs (e.g. Ville de Québec, voieries)

Additional Notes

Estimate regarding researcher needs.

Objectives 1-2:

  • 2 x Master
  • 2-3 x PhD
  • 1-2 PDF (or PhD, to be discussed)

Objectives 2-3:

  • 2-3 X Master
  • 2-3 x PhD
  • 1-2 PDF (or PhD, to be discussed)

Objectives 5:

  • 1-2 X Master

Methodology:

  • Comprehensive supervision/management of researchers from start to finish
  • Usage of existing libraries
  • Access to industrial 3D databases sourced from clients/partners in manufacturing (aerospace, automotive, rail etc.) and retail (furniture, fashion garment & accessories) industries
  • Separation of code:
    • Device side (light) computing
    • Server side (heavy) computing
  • During a first phase, basic module will be developed in parallel towards standalone functionality
  • In a second phase, functionalities will be further developed towards a common framework (local and cloud)

Expertise and Skills Needed:

  • Knowledge of Python, Java, C/C++, C-sharp
  • Machine learning, with focus on image analysis recognition
  • Knowledge/Previous Studies in cloud computing will be a plus
  • Development and testing under the following environments:
    •  Windows
    • Linux
    • Mac OS
    • Android (Android studio)
    • iOS (X code)
  • Source Control: Git (Bitbucket)
  • Computer vision
  • Usage of existing libraries:
    • Open CV
    • Visual SFM
    • Point cloud library (PCL)
    • Geometry factory
    • Cloud compare
    • Unity
    • Etc.
  • Knowledge of both French and English will be a plus
  • Knowledge of 3D modeling techniques and formats (PLY, STL, OBJ, FBX)
  • Knowledge of AR libraries will be a plus (ARkit, ARcore, Vuforia)

 

For more info or to apply to this applied research position, please

  1. Check your eligibility and find more information about open projects.
  2. Interested students need to get approval from their supervisor and send their CV along with a link to their supervisor’s university webpage directly to Gabriel Garcia Curiel  at ggarciacuriel(a)mitacs.ca.
Program: