Object recognition and activity learning using kinect

Object recognition based on shape and function

2011 -

This project explores a new approach to computational object recognition by borrowing an idea from child language acquisition studies in developmental psychology.  Whereas previous image recognition research used shape to recognize and label a target object, the model proposed in this thesis also uses the function of the object resulting in a more accurate recognition.  This thesis makes use of new gaming technology, Microsoft’s Kinect, in implementing the proposed new object recognition model.  A demonstration of the model developed in this project properly infers different names for similarly shaped objects and the same name for differently shaped objects.


The objective of this research is to combine the Kinect sensor with machine learning techniques to implement an object recognition model that uses both shape bias and function bias to learn the names of objects in a manner similar to how human children acquire names of objects.


How human learn name of objects

  • Human children use mainly two biases when learning name of objects:
    • Shape bias
      • Generalize name of the object if the shape is similar
    • Function bias
      • Generalize name of the object if the function of the use seem to be the same

Simulation of biases (Grabner, Gall, & Gool, 2011)

  • Shape bias:
    • Grab 3D models that are labeled with a same name (e.g., chair) and run a machine learning algorithm to train the classifier.
  • Function bias:
    • First define the use of the object; .e.g., chair is a object to sit on.
    • Then, let the program learn the .posture of sitting.
    • If the object is sittable, the object.is more likely to be a chair.


Shape bias with Kinect

  • Take the depth info. of the environment.
  • Use RANSAC algorithm to remove the ground the object is placed.
  • Name the objects and run machine learning to train the classifiers.
  • Google 3D warehouse should enhance the learning (Lai and Fox (2010))

Function Bias with Kinect

  • Kinect has a feature to track 20 different joints of human body.
  • Easily be able to draw skeletal body image
  • So, program can learn the use of object through the movement of the body.

activity learning using kinect skeletal data

Object recognition using both shape and function bias

object recognition with shape and function bias


Akihiro Nakashima,
Sep 10, 2012, 6:14 AM