Progress with my Handwriting Recognition Project

I am back with more updates on my handwriting recognition project!

In my last post, I talked about developing Scribble-pad for accepting user input that we can process for handwriting recognition. Another way of capturing input can be by a phone camera, so the user can just click a picture of the text. I’ll post about the camera input in a separate post, but today I would like to show you how I did some image processing on the image we saved from the Scribble-pad.

Going for the complete handwritten word seemed too ambitious to me, so I’m going to start by recognizing a letter first!

Logic

First we allow the user to write a letter and save it to a file using our Scribble-pad.

A possible way to recognize the letter could be to compare the saved image with all the other english alphabets and numerals. If it matches to a threshold percentage, it’s more likely to actually be that character. There can be some errors because the style and size of the handwriting of every user would differ. We can do some amount of image processing to detect if the user’s handwriting is too small to compare and enlarge it to a working size, however nothing can be done if the user has bad handwriting. (So, my engine wouldn’t work if you have a really bad handwriting)

Implementation

So following the logic I outlined above, we would need source images of all the characters of english letters and numerals. Where do I get these from? The answer is to generate these images from Qt.

Here’s how I generated the images for upper case characters A-Z. These images were generated as follows,

ascii_counter = 65;
QFont font("Mangal",12, QFont::Bold, false);

for(int i = ascii_counter; i<91; i++)
{
QPixmap pix(20,20);
pix.fill(Qt::white);
QPainter p(&pix);
p.setPen(QColor(0,0,0));
p.setFont(font);
p.drawText(QPoint(3,18), QString(QChar(i)));
pix.toImage().convertToFormat(QImage::Format_Mono);
pix.save(FileRoot+ "img_"+ QString::number(i) +".bmp");
}

Similarly, I generated images for lower case characters a-z and numerals 0-9. I used the image size as 20x20px, just because I feel they were sufficiently good for comparison. Besides, the bigger the resolution, the bigger the size of the images!
Note, I also experimented with the font that is used to generate these images. I am using Mangal, size 12, Bold font here, but we can generate the character set for multiple fonts if we want our engine to be more efficient. For example, Lucida Console handwriting font would be good for running handwriting.

Now, to compare the user scribbled image (let’s load it in desImg) with the images that I generated. Here’s how I’ll do the comparison,

QPixmap recalculateResult()
{
QPainter::CompositionMode mode = QPainter::CompositionMode_Screen;

QPainter painter(&resImg);
painter.setCompositionMode(QPainter::CompositionMode_Source);
painter.fillRect(resImg.rect(), Qt::transparent);
painter.setCompositionMode(QPainter::CompositionMode_SourceOver);
painter.drawImage(0, 0, desImg);
painter.setCompositionMode(mode);
painter.drawImage(0, 0, sourceImg);
painter.setCompositionMode(QPainter::CompositionMode_DestinationOver);
painter.fillRect(resImg.rect(), Qt::white);
painter.end();

return QPixmap::fromImage(resImg);
}

Generated Images


Note that a resultant image (resImg) is generated which is the comparison of the two images. With recalculateResult(), I have drawn both the destination image(desImg) and the source image (sourceImg) over the resultant image (resImg) with the drawImage property of QPainter and have also set the CompositionMode which is used to specify how the pixels in one image, the source, are merged with the pixels in another image, the destination.
By placing them over one another would help in checking how many pixels of the desImg match the sourceImg.
The pixels that are overlapped would be painted black and the once that did not overlap would be plain white.
The function returns a generated pixmap that would store the comparison result of the two images.

Next, I save all the resultant images from the comparison function – recalculateResult()

for(int i = ascii_counter; i<91; i++)
{
sourceImg= QImage(FileRoot + "img_" +QString::number(i)+".bmp");
resImg = QImage(20, 20, QImage::Format_ARGB32_Premultiplied);
genPix = recalculateResult();
genPix.save(FileRoot+"resImg\\"+QString::number(i)+".bmp");
}

Resultant Images

Now, what needs to be done is to check all the saved resultant images and pick out the one with maximum black pixels or the one with minimum white pixels. This is the one which would be the best match with the user’s image.

QImage countImg;
int lowest_white=20*20;
int highest_black=0;
int index;
int index_b;

for(int i = ascii_counter; i<91; i++)
{
countImg = QImage(FileRoot + "resImg\\" +QString::number(i)+".bmp");
int white=0;
int black=0;
for (int j=0; j<countImg.width(); j++)
{
for (int k=0; k<countImg.height(); k++)
{
QRgb getRgb = countImg.pixel(j,k);
if (qRed(getRgb)>=200 &&
qGreen(getRgb)>=200 &&
qBlue(getRgb)>=200)
{
white++;
}

if (qRed(getRgb)<=20 &&
qGreen(getRgb)<=20 &&
qBlue(getRgb)<=20)
{
black++;
}

}
}
if (white<=lowest_white)
{
lowest_white = white;
index = i;
}

if (black>=highest_black)
{
highest_black = black;
index_b = i;
}
}

// Prints the index of the image with lowest white pixels
qDebug()<<"WHITE"<<index<<QString(QChar(index))<<lowest_white;
// Prints the index of the image with highest black pixels
qDebug()<<"BLACK"<<index_b<<QString(QChar(index_b))<<highest_black;

The index and index_b store the letters with minimum white pixels and maximum black pixels respectively.
The variables lowest_white and highest_black store the number of pixels of white and black color respectively.

If the resultant image has its pixels red, green or blue, color count greater than 200 is considered white while less than 20 is considered to be black.

Here’s how a test run was done and its results were obtained:

A’s scribbled image – 

A’s resultant image – 

Result obtained with the run

WHITE 109 “m” 382

BLACK 65 “A” 9

Here, A(65) was found to have maximum number of black pixels matching the desImg and also “m” having the least white pixels when compared to desImg.

So, that’s what I have been working on. Until next time, ciao!

Advertisements
This entry was posted in Code Example, Nokia, Projects, Qt, Qt Quick, Symbian and tagged , , , , , , , . Bookmark the permalink.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s