Streamlit

Streamlit is an open-source tool for building data apps in Python. The app is coded in Python and deployed as a web app. Streamlit greatly reduces the work required to prototype and deploy a data visualization or machine learning app. I only recently started working with Streamlit but so far have really enjoyed using it. To further my understanding of Streamlit, I decided to create a machine learning app in my free time that would be of interest to anyone wanting to explore the applications of machine learning without wanting to learn a bunch of coding. The machine learning topic that I thought would be widely popular is neural style transfer. I decided to call my app Pic2Paint since you can use the style transfer to make an input picture look like a painting, but you can use style transfer is any way you want.

Pic2Paint App

One area of machine/deep learning that I really enjoy is the application of neural network models to art and the creative process. A breakthrough in this area came in late 2016 with this paper Image Style Transfer Using Convolutional Neural Networks. This paper presents the idea of using a pre-trained neural network to approximate a measure of an image’s content and style. The content and style measures can then be used in an optimization problem to create a new image that has the content of the content image but the style of the style image. I thought it would be fun to implement this approach in an app so that anyone could try this style transfer with minimal coding or understanding of the underlying algorithm.

Instead of coding the approach from scratch, I figured it would be easier to modify existing implementations. After some searching I found that there are plenty of code examples on the internet that implement the approach but I chose to use the example in the PyTorch forums. I chose this implementation since it seemed pretty well commented and I wanted to ultimately use PyTorch in my implementation as well. I modularized the existing code and made some minor modifications in an attempt to reduce the GPU memory overhead. The existing code was not optimized for memory utilization and trying to perform the style transfer on even a moderately sized image would quickly exceed the 11GB of RAM available to me on my GPU. Once I had reduced the memory overhead, I went about building the app.

One feature I wanted the app to have, was the ability to preserve the color of the content image. This feature is not part of the original style transfer algorithm. I had an idea about how to add this feature, but I figured it would also be worth seeing if someone else had already solved this problem. In my search I found this paper Preserving Color in Neural Artistic Style Transfer. It was exciting but also disappointing. It was exciting that someone had already looked into this problem but also disappointing because my solution to the problem is Approach 1 presented in the paper. It’s always very hard to be the first with an original idea in a popular field. It was pretty straightforward to add this feature to the app.

Coding the app itself was really enjoyable and presented few challenges. Streamlit makes it very easy to implement widgets and display results. One feature that was added recently that I had been waiting for was the ability to upload files. Prior to this feature being added, one would have to somehow upload the data for processing to the host machine running the app. The file upload feature now makes it even easier to abstract the behind the scenes from the end user. The biggest challenge of building the app was deciding the layout for the app. I ultimately went with using the sidebar for control and the main content area for displaying the data and results. I have very little experience with user interface/experience but this layout felt like it was the most intuitive.

I am not currently hosting the app for others to use since that could potentially cost a lot of money if it became popular, or if someone decided to abuse its use for free processing. However, I have put the source code up on my GitHub for others to download and use. If you give the app a try, let me know what you think! Below are some example results of the algorithm.