Indian Sign Language Recognition

Sign Languages are a set of languages that use predefined actions and movements to convey a message. These languages are primarily developed to aid deaf and other verbally challenged people. They use a simultaneous and precise combination of movement of hands, orientation of hands, hand shapes etc. Different regions have different sign languages like American Sign Language, Indian Sign Language etc. We focus on Indian Sign language in this project.

Indian Sign Language (ISL) is a sign language that is predominantly used in South Asian countries. It is sometimes referred to as Indo-Pakistani Sign Language (IPSL). There are many special features present in ISL that distinguish it from other Sign Languages. Features like Number Signs, Family Relationship, use of space etc. are crucial features of ISL. Also, ISL does not have any temporal inflection.

In this project, we aim towards analyzing and recognizing various alphabets from a database of sign images. Database consists of various images with each image clicked in different light condition with different hand orientation. With such a divergent data set, we are able to train our system to good levels and thus obtain good results.

We investigate different machine learning techniques like:

• K-Nearest-Neighbours
• Support Vector Machines
• Convolutional Neural Network

Getting Started

Prerequisites

Before running this project, make sure you have following dependencies -

• Dataset (Download the images from this link)
• Python 3.6
• pip
• OpenCV

Now, using pip install command, include following dependencies

• Numpy
• Pandas
• Sklearn
• Scipy
• Opencv
• Tensorflow

Running

To run the project, perform following steps -

1. Put all the training and testing images in a directory and update their paths in the config file common/config.py.
2. Generate image-vs-label mapping for all the training images - generate_images_labels.py train.
3. Apply the image-transformation algorithms to the training images - transform_images.py.
4. Train the model(KNN & SVM) - train_model.py <model-name>. Note that the repo already includes pre-trained models for some algorithms serialized at data/generated/output/<model-name>/model-serialized-<model-name>.pkl.
5. Generate image-vs-label mapping for all the test images - generate_images_labels.py test.
6. Test the model - predict_from_file.py <model-name>.
7. To obtain Better Results, train the model using Convolutional Neural Network which can be done by running the cnn.py file after activating Tesorflow.

Accuracy with CNN

The best accuracy was achieved by CNN using Momentum Optimizer and a learning rate of 0.3.The Plot for the same is depicted bleow.

To-Do:

• Improve the accuracy if possible by collecting more data and applying various CNN architectures like VGG16,Le-Net5 etc.