E-card platform JibJab sought a better way to crop images for a new physical book product. Mission improved the process's accuracy in an effort that won a 2021 Top Projects Award.
A machine learning consulting project helped e-card platform JibJab overcome technical obstacles and launch a product line.
The Los Angeles-based company had traditionally offered customers a simple interface and cropping tool to upload photos, capture head images in an oval shape and place them in personalized digital content. That method worked well enough, but JibJab was launching a new offering -- a physical coffee-table book that called for higher-fidelity images.
Matt Cielecki, vice president of engineering at JibJab, began exploring machine learning (ML) as the way to boost image quality. He tinkered with pre-trained models but ran into limitations. Models built as academic projects couldn't be used commercially, for example.
Matt Cielecki
Serendipitously, a representative from Mission, a managed cloud services provider also in Los Angeles, contacted Cielecki to let him know about its AI services. Mission had helped the e-card company years earlier on an AWS migration and cost optimization project, he noted. That history and the timely outreach paved the way for a new initiative: JibJab hired Mission in 2021 to build a ML algorithm from scratch for image cropping.
Mission built the first version of the algorithm, and the resulting ML model, in about nine weeks. But the algorithm has gone through multiple iterations since then. The cycle of algorithm testing and model refinement continues.
"We are rolling it out to some users to get feedback, and we are starting to see some good results," Cielecki said. "We think we have some room for improvement, but we are definitely offering a much better product."
Training and refinement
Mission's task was to train a ML computer vision algorithm that could detect faces within uploaded photos, accurately crop a person's full face and hair and ignore any background elements. The initial goal was to achieve 85% accuracy with the ML-based image cutting technique.
In the first step toward that objective, Mission used two annotation tools -- LabelMe and Amazon SageMaker Ground Truth -- to label images and create a data set for training the algorithm. Mission used data augmentation techniques, such as adding blur, sharpening and rotation, to expand the labeled data set from 1,000 images to 17,000.
Next, Mission used Facebook AI Research's Detectron2, running in SageMaker, to detect objects within images and perform instance segmentation. The latter went beyond detection, using a more granular approach that helped closely define the shape of a particular object -- in JibJab's case, a person's face and hair.
Ramping up the accuracy of this process involved training and retraining a ML model. Once Mission put a model through its paces on JibJab's training data set, the company ran it on new image data. This process uncovered edge cases -- situations in which the algorithm fell short of fully identifying faces and hair. Mission tweaked the training data set, which is used to revise the model.
This aspect of the initiative drove home the pivotal role of data engineering in artificial intelligence. "It changed my perspective on what an AI project actually is," Cielecki said. "It's less of a coding and more of a data problem."
Ryan Ries
Ryan Ries, who leads Mission's data, analytics and machine learning practice, cited the importance of a diverse training data set. The training process revealed unanticipated issues, such as bright light which washes out part of a person's face or long, flowing hair that defies accurate cropping.
Determining why a model failed on certain data sets was the path to improvement. Ries described the investigative process as learning, "Why is this an edge case and why is the algorithm doing what it is doing and how do I retrain the data set?"
Current use and future plans
JibJab currently uses Mission's image-cutting method in its Starring You Books product line. A customer who wants to create a personalized book uploads an image, which is stored in AWS S3 bucket. AWS' computer vision platform, Rekognition, detects a face in the image and conducts an image-quality check, considering, among other things, whether the facial image is large enough to achieve a good result.
We have additional training and optimizations to implement before the model is ready for everyone on a broader scale.
Matt Cielecki Vice president of engineering, JibJab
If the image passes muster, a Detectron2 model performs instance segmentation to zero in on the face and hair for the final image. The last steps are image post-processing and positioning the image in the personalized book.
Mission's image-cutting algorithm has reached 90% accuracy. With ongoing improvements, Mission and its customer have upped the target to 95%. As the model improves, JibJab will consider how the image cutting method could fit into other product lines.
"Our primary goal is to get the model in a place that can generate print-quality head cuts for all of our users, capturing all of the unique contours and shapes of their faces," Cielecki said. "We have additional training and optimizations to implement before the model is ready for everyone on a broader scale."
Matt Cielecki
Ryan Ries
