​ Using Machine Learning for Identification of Art Paintings
EECS 349: Machine Learning
NORTHWESTERN UNIVERSITY, 2016 SPRING
MOtivation
When it comes to art, most people may have heard of Water Lilies by Claude Monet or The Starry Night by Vincent van Gogh. They are the most iconic work of Impressionism and Post-Impressionism. In fact, there are dozens of genres of art in contemporary and modern times, each with unique painting and expressing style that cannot be replicated.
​
The purpose of our project is to roughly judge which genre a given painting belongs to. Since there are numerous genres of art. In order to make our project practical, we have decided to narrow it down to five most well-known ones: Impressionism, Cubism, Abstractionism, Surrealism and Pop Art. Therefore, given a painting, our project is expected to tell which genre (in the five above) a given painting is most likely belongs to. Our project is of great significance as it is providing people with a handy approach to getting basic knowledge of art. For those who know little about art (like most of us), the only way to do this seems to be turning to their art teacher. Is there a faster and more convenient way? We believe with our project, the only thing you need to do is to take a picture of painting with your phone, and you will hopefully get the answer within seconds.
​
solution
Scale-invariant feature transform (SIFT) features and Histogram of Oriented Gradients (HOG) feature are considered for this classification problem. A linear Support Vector Machine (SVM) was the nused to classify images based on these two features extacted from the images.
​
SIFT descriptor is computed from several interesting locations which could be referred to key points. For each image, we get a matrix (128*k) to represent it (k is the number of key points). But different images might have different k. We apply the Bag-of-words model (BoW model) here to normalize the matrix (1*100) before training with SVM.
​
â–²Fig.1 Bag of words
HOG technique counts occurrences of gradient orientation in localized portions of an image. In specific, we used HOG2*2 for this problem. An image has been decomposed into small squared cells, giving a 31-dimensions descriptor for each cell. Then, 2*2 neighboring HOG descriptors are stacked together, providing more descriptive power. By querying all images into equal size, the inputs of SVM are set to the same dimension.
​
After obtaining the desired feature data, we used them as the inputs to a linear SVM to train the classifier. All the steps were implemented with Python + OpenCV.
​
â–²Fig.2 Impressionism with SIFT and HOG (click on picture to view original)
​
â–²Fig.3 Cubism with SIFT and HOG (click on picture to view original)
​
â–²Fig.4 Abstractionism with SIFT and HOG (click on picture to view original)
​
â–²Fig.5 Surrealism with SIFT and HOG (click on picture to view original)
​
â–²Fig.6 Pop Art with SIFT and HOG (click on picture to view original)
​
​
data training & testing
We collected 500 paintings, 100 paintings for each genre. Then we divided paintings in each genre into training set and validation set with the ratio 4:1. The collection of the training set in all the genres constructs the whole training set. So is the validation set. We applied the built-in function in OpenCV to extract the SIFT feature and HOG feature of all the paintings. As mentioned in the last section, we kept the HOG feature but normalizing SIFT feature using K-means Algorithm before training with SVM. In the testing phase, we evaluate the performance of our classifier on the validation set while restricting the training set to 25%, 50%, 75%, 100% the size of the original training set to get the learning curve shown below. For each amount of training data, we took the average over multiple runs, using the different subset of the training data each time.
​
â–²Fig.7 Identification results
​
key results
â–²Fig.8 Performance of SIFT
​
​
â–²Fig.9 Performance of HOG
​
The overall accuracy for both features are about 80% when using the whole training dataset. As can be seen, both learning curves steadily increase as training data size increases and we didn’t observe overfitting maybe because our total number of paintings is not large enough. In general, the prediction accuracy of the classifier trained with HOG outperforms that trained with SIFT. This is because we reserved all the feature data of HOG descriptor during the training while pre-processing the original data of SIFT feature before importing to the SVM(K-means is applied to maintain consistency of the feature space), from which important information of the paintings could be lost so that the prediction performance of the final classifier may be downgraded. It’s interesting to notice that training with HOG will result in very low prediction accuracy for the Abstractionism (around 5~10%).
This is a website for the course project of EECS 349: Machine Learning at Northwester University.
team members
SHENGJIE ZHU
shengjiezhu2017@u.northwestern.edu
​
​
SHUYU KONG
​
shuyukong2020@u.northwestern.edu
​
​
YEYI PENG
​
yeyipeng2015@u.northwestern.edu
​
​
YUANWEI ZHANG
​