IMAGE STYLE TRANSFER MODEL WITH CONTENT PRESERVATION

Ruixue Liu, Shengbei Wang, Weitao Yuan

Abstract


The image rendered in another image style, called style transfer, is one of the most interesting applications of deep learning. Style transfer plays an increasingly important role in basic computer vision research and industrial applications. The method of iterative optimization [1] and training feedforward convolutional neural network [2-4] are used to continuously achieve fast and good effects on style transfer. However, most of these methods ignore an essential problem that the content and contour information of the original images could be lost after the transfer process. Throughout the history of style transfer, although the performance is constantly improving, pursuing higher flexibility and speed, an essential problem is ignored, that is, the main content and contour are unavoidably blurred after style transfer. It is found that in many application fields, the content/contour of the original image has great importance and thus should be kept as clear as possible after style transfer. In this paper, a sub-structure named Important Content Contour Extraction (ICCE) is proposed to generate masks and therefore preserve the clear contour and content after style transfer. The Feature Extraction Module (FEM) for extracting advanced features is added in image style transfer network which is trained by the perceptual loss function. We conducted some experiments to verify the effectiveness of the proposed method. Experiments demonstrated that the proposed framework achieves a good compromise in speed, flexibility and quality. The experiment results showed that the proposed model had good ability for style transfer.Meanwhile, it could preserve clear content and contour of the original images, which was better than [1-4].


Keywords


style transfer, feature extraction, content preservation, feedforward convolution network.

Full Text:

PDF

References


L. A. Gatys, A. S. Ecker, and M. Bethge, “A neural algorithm of artistic style,” CoRR abs/1508.06576 (2015).

D. Ulyanov, V. Lebedev, A. Vedaldi, et al., “Texture networks: Feed-forward synthesis of textures and stylized images,” in ICML 2016, New York City, NY, USA, June 19-24, 2016, 1349–1357 (2016).

D. Chen, L. Yuan, J. Liao, et al., “Stylebank: An explicit representation for neural image style transfer,” inCVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 2770–2779 (2017).

X. Huang and S. J. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in ICCV 2017, Venice,Italy, October 22-29, 2017, 1510–1519 (2017).

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States., 1106–1114 (2012).

J. E. Kyprianidis, J. P. Collomosse, T. Wang, et al., “State of the ”art”: A taxonomy of artistic stylization techniques for images and video,” IEEE Trans. Vis. Comput. Graph. 19(5), 866– 885 (2013).

A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” in Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001, Los Angeles, California, USA, August 12-17, 2001, 341–346 (2001).

M. Ashikhmin, “Synthesizing natural textures,” in Proceedings of the 2001 Symposium on Interactive 3D Graphics, SI3D 2001, Chapel Hill, NC, USA, March 26-29, 2001, 217–226 (2001).

L. A. Gatys, A. S. Ecker, and M. Bethge, “Texture synthesis using convolutional neural networks,” in Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, 262–270 (2015).

F. Luan, S. Paris, E. Shechtman, et al., “Deep photo style transfer,” in CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 6997–7005 (2017).

R. Mechrez, E. Shechtman, and L. Zelnik-Manor, “Photorealistic style transfer with screened poisson equation,” CoRR abs/1709.09828 (2017).

J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and superresolution,” in ECCV 2016, Proceedings, Part II, 694–711 (2016).

V. Dumoulin, J. Shlens, and M. Kudlur, “A learned representation for artistic style,” CoRR abs/1610.07629 (2016).

A. Mahendran and A. Vedaldi, “Understanding deep image representations by inverting them,” in CVPR 2015, Boston, MA, USA, June 7-12, 2015, 5188–5196 (2015).

H. Zhang and K. J. Dana, “Multi-style generative network for real-time transfer,” CoRR abs/1703.06953 (2017).

Y. Li, H. Qi, J. Dai, et al., “Fully convolutional instance-aware semantic segmentation,” in CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 4438–4446 (2017).

Y. Li, H. Qi, J. Dai, et al., “Fully convolutional instance-aware semantic segmentation,” in CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 4438–4446 (2017).

S. Liu, J. Jia, S. Fidler, et al., “SGN: sequential grouping networks for instance segmentation,” in ICCV 2017, Venice, Italy, October 22-29, 2017, 3516–3524 (2017).

M. Bai and R. Urtasun, “Deep watershed transform for instance segmentation,” in CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 2858–2866 (2017).

K. He, G. Gkioxari, P. Dollar,´ et al., “Mask R-CNN,” in ICCV 2017, Venice, Italy, October 22-29, 2017, 2980–2988 (2017).

S. Ren, K. He, R. B. Girshick, et al., “Faster R-CNN: towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing Systems 28, December 7-12, 2015, Montreal, Quebec, Canada, 91–99 (2015).

J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in CVPR 2015, Boston, MA, USA, June 7-12, 2015, 3431–3440 (2015).

R. B. Girshick, “Fast R-CNN,” in ICCV 2015, Santiago, Chile, December 7-13, 2015, 1440–1448 (2015).

R. K. Srivastava, K. Greff, and J. Schmidhuber, “Training very deep networks,” in Advances in Neural Information Processing Systems 28, December 7-12, 2015, Montreal, Quebec, Canada, 2377–2385 (2015).

Y. Wang, R. J. Skerry-Ryan, D. Stanton, et al., “Tacotron: Towards end-to-end speech synthesis,” in Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, 4006–4010 (2017).


Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright © 2019 International Educational Applied Scientific Research Journal