浏览全部资源
扫码关注微信
网络出版日期： 2024-08-15 ，
移动端阅览
吴萌,郭歌,孙增国等.自适应卷积约束与全局上下文推理的墓室壁画修复[J].中国图象图形学报,
Wu Meng,Guo Ge,Sun Zengguo,et al.Tomb mural inpainting with adaptive convolutional constraints and global context inference[J].Journal of Image and Graphics,
目的
2
墓室壁画作为地下文物，由于环境湿度、地仗沉降等因素，局部区域出现了脱落、裂缝、霉变等多种病害，导致画面部分缺失。但现有深度学习的修复方法通常在单一维度或固定区域进行信息重建，无法充分捕获稀疏的壁画特征和修复多样化的复杂病害，修复时会出现内容缺失、结构错乱等问题。对此，本文提出了一种自适应卷积约束与全局上下文推理的墓室壁画修复。
方法
2
该方法基于端到端的编码器-解码器架构，首先设计多尺度增强卷积模块从频域和空域联合分析图像特性来充分捕获全局结构和局部纹理；同时在修复路径中加入融合差分卷积的增强激活单元来引入边缘先验信息，提高模型的绘制精度。其次，考虑到纹理和结构在绘制过程中的模式差异，在编码器-解码器间设计基于注意力交互引导的多尺度特征聚合模块，来加强全局稀疏信息的表征能力和相关性，并自适应选择增强有效特征。此外，为了获得真实准确的结果，在特征传递过程中利用自动掩码更新迭代来预测复杂缺失信息，引导解码器精确绘制多样化的损伤区域。
结果
2
本文采用客观评价指标：峰值信噪比（peak signal-to-noise ratio，PSNR）、结构相似性（structural similarity index，SSIM）、学习感知图像块相似度（learned perceptual image patch similarity，LPIPS）在章怀太子墓“马球图”数据集上进行三类模拟病害和真实病害修复实验，并与最新的6种主流方法进行了比较。实验结果表明本文方法修复的壁画图像在主观视觉和客观评价上均有明显的提升。如相较于指标排名第二的模型，对于随机缺失区域的壁画修复，峰值信噪比和结构相似性的均值分别达到31.7602dB和0.9577，各指标的样本均值分别提升了2.3653dB，0.0128，12.75%。
结论
2
本文提出的图像修复模型可以有效修复多种复杂病害，可为手工绘制专家的物理修复提供参考，这进一步表明了该方法在保护和传承数字文化遗产的有效性和适用性。
Objective
2
As an important cultural heritage of ancient civilization， with the changes of the years， due to factors such as environmental humidity and ground settlement， the local areas of the murals in the tomb have fallen off， cracks， mud spots， mildew and other diseases， which seriously affect the sustainable development of the protection of mural cultural relics and the activities of appreciation， cultural creativity and cultural dissemination. Due to the influence of underground environmental factors， in order to protect the precious mural relics， researchers usually adopt the means of block excavation to transfer the mural paintings in the tomb and restore them， and the restoration of murals in the past requires professional restorers to draw manually， which requires a high level of manual operation， and the repair cycle is lengthy and inefficient. Therefore， in the face of the complex semantic environment and the lack of diverse information， the restoration method based on deep learning has been gradually applied to the information reconstruction of murals to provide scientific and technological protection for murals. However， most of the existing methods usually perform restoration in a single dimension or a fixed area， which cannot fully capture the sparse mural features and repair multiple complex diseases at the same time， resulting in semantic inconsistencies or incoherent structural results. In order to solve the above problems， this paper proposes a tomb mural inpainting model with adaptive convolutional constraints and global context inference， which can repair various types of damage and diseases， and produces a rich database of digital cultural heritage.
Methods
2
Based on the end-to-end encoder-decoder architecture， the model firstly designs a multi-scale enhanced convolution （MEConv） module in the encoder path for content constraints， extracts different features of the image from the frequency domain and spatial domain at the same time to complement each other， and the enhanced activation unit fused with differential convolution is added to the repair path to introduce edge prior information， so as to correct the adaptive multi-scale feature mapping， which can more effectively capture the global and local latent feature information. Secondly， considering the differences in the patterns of texture and structure in the mural painting process， a multi-scale feature aggregation （MSFA） module was added between the encoder and the decoder， and through the fusion of multi-scale feature components and the dynamic selection of attention mechanism， the adaptive selection and enhanced effective information of the global context were strengthened， and the representation and generalization ability of the original feature map were strengthened， and the drawing accuracy was improved. In addition， in order to obtain true and accurate results， in the process of feature transfer， the model compensates for the difference between the underlying and high-level features through automatic mask update and jump connection in each layer， and guides the decoder to accurately draw the missing content.
Results
2
In order to verify the effect of network disease repair， this paper used the objective evaluation indexes： peak signal-to-noise ratio （PSNR）， structural similarity index （SSIM）， and learned perceptual image patch similarity （LPIPS） to carry out three types of disease remediation experiments on the constructed polo mural dataset， and compared them with the latest four mainstream methods. The experimental results show that the images restored by this method have obvious improvement in subjective vision and objective evaluation， and the content and style of the generated murals are closer to those of the real murals. For example， compared with the second-ranked algorithm， the mean values of each index were increased by 0.1432dB， 0.0167 and 2.17% respectively for mural restoration in the epitaxial damage area. For the mural repair in the irregular damage area， the mean values of each index increased by 0.4132dB， 0.0304 and 15.11%， respectively. For the mural restoration in the random damage area， the mean values of each index increased by 2.3653dB， 0.0128 and 12.75%， respectively.
Conclusion
2
The image inpainting model proposed in this paper can not only fully capture the latent features of different characteristics of the image， but also capture the long-term contextual semantic features in the deep features. The model can effectively deal with a variety of complex mural diseases， and restore images with consistent semantics， rich details， complete content and natural coherence， providing a feasible solution for the digital restoration and display of murals. In order to restore the real damaged mural images， in the future work， we will collect more high-definition tomb murals， and fully consider the mural painting and historical background of different periods and different years， so as to facilitate the model to learn more accurate and comprehensive mural content and style， and improve the restoration performance of the model.
壁画修复多尺度增强卷积模块多尺度特征聚合模块增强激活单元差分卷积病害修复
mural restorationmulti-scale enhanced convolution modulemulti-scale feature aggregation moduleenhanced activation unitdifferential convolutiondisease restoration
Bertalmio M， Sapiro G， Caselles V and Ballester C. 2000. Image inpainting//Proceedings of the 27th annual conference on Computer graphics and interactive techniques. New York， USA： ACM： 417-424 ［10.1145/344779.344972http://dx.doi.org/10.1145/344779.344972］
Chen S， Amir A A， Shum Hubert P H. 2024. HINT： High-quality INpainting Transformer with Mask-Aware Encoding and Enhanced Attention. IEEE Transactions on Multimedia. 26： 7649-7660 ［DOI： 10.1109/TMM.2024.3369897http://dx.doi.org/10.1109/TMM.2024.3369897］
Criminisi A， Perez P and Toyama K. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing， 13（9）： 1200-1212 ［DOI： 10.1109/TIP.2004.833105http://dx.doi.org/10.1109/TIP.2004.833105］
Gatys L A， Ecker A S and Bethge M. 2016. Image style transfer using convolutional neural networks//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. LasVegas， USA： IEEE： 2414-2423 ［DOI： 10.1109/ CVPR.2016.265http://dx.doi.org/10.1109/CVPR.2016.265］
Guo D S， Gu Z R， Zheng B， Dong J Y and Zheng H Y. 2024. Unified framework with iterative prediction for image inpainting and outpainting. Journal of Image and Graphics， 29（02）： 0491-0505
郭冬升，顾肇瑞，郑冰，董军宇，郑海永.2024.面向图像内补与外推问题的迭代预测统一框架.中国图象图形学报， 29（02）： 0491-0505［DOI： 10.11834/jig.230144http://dx.doi.org/10.11834/jig.230144］
Hu J， Shen L and Sun G. 2018. Squeeze-and-excitation networks//In Proceedings of the IEEE conference on computer vision and pattern recognition （CVPR）. Salt Lake City， USA： IEEE： 7132-7141 ［DOI： 10.1109/CVPR.2018.00745http://dx.doi.org/10.1109/CVPR.2018.00745］
Jiao L J， Wang W J， Li B J and zhao Q S. 2019. Wutai mountain mural inpainting based on improved block matching algorithm. Journal of Computer-Aided Design and Computer Graphics. 31（1）： 118-125 ［DOI： 10.3724/SP.J.1089.2019.17102http://dx.doi.org/10.3724/SP.J.1089.2019.17102］
Johnson J， Alahi A and Li F F. 2016. Perceptual losses for real-time style transfer and super-resolution//Proceedings of the 14th European Conference on Computer Vision. Amsterdam， the Netherlands： Springer： 694-711 ［DOI： 10.1007/978-3-319-46475-6_43http://dx.doi.org/10.1007/978-3-319-46475-6_43］
Lan Z， Yan C P， Li H and Zheng Y D. 2023. HDA-GAN： hybrid dual attention generative adversarial network for image inpainting. Journal of Image and Graphics， 28（11）： 3440-3452
兰治，严彩萍，李红，郑雅丹.2023.混合双注意力机制生成对抗网络的图像修复模型.中国图象图形学报， 28（11）： 3440-3452 ［DOI： 10. 11834/jig. 220919http://dx.doi.org/10.11834/jig.220919］
Liu G， Reda F A， Shih K J， Wang T C， Tao A and Catanzaro B. 2018. Image inpainting for irregular holes using partial convolutions// Proceedings of the European conference on computer vision （ECCV）. Munich， Germany： Springer： 85-100 ［DOI： 10.48550/arXiv.1804.07723http://dx.doi.org/10.48550/arXiv.1804.07723］
Liu H， Jiang B， Song Y， Huang W and Yang C. 2020. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations//In Computer Vision–ECCV 2020： 16th European Conference. Glasgow， UK： Springer International Publishing： 725-741 ［DOI： 10.48550/arXiv.2007.06929http://dx.doi.org/10.48550/arXiv.2007.06929］
Liu T J， Chen B W and Liu K H. 2023. Lightweight Image Inpainting by Stripe Window Transformer with Joint Attention to CNN//In 2023 IEEE 33rd International Workshop on Machine Learning for Signal Processing （MLSP）. Rome， Italy： IEEE： 1-6 ［DOI： 10.1109/MLSP55844.2023.10285983http://dx.doi.org/10.1109/MLSP55844.2023.10285983］
Nikolaos Karianakis and Petros Maragos. 2013. An integrated system for digital restoration of prehistoric theran wall paintings//In 2013 18th International Conference on Digital Signal Processing （DSP）. Fira， Greece： IEEE： 1-6 ［DOI： 10.1109/ICDSP.2013.662 2838http://dx.doi.org/10.1109/ICDSP.2013.6622838］
Pathak D， Krahenbuhl P， Donahue J， Darrell T and Efros A A. 2016. Context encoders： Feature learning by inpainting//Proceedings of the IEEE conference on computer vision and pattern recognition （CVPR）. Las Vegas， USA： IEEE： 2536-2544 ［DOI： 10.1109/CVPR.2016.278http://dx.doi.org/10.1109/CVPR.2016.278］
Peng X L， Zhao H Y， Wang X Y， Zhang Y Q， Li Z， Zhang Q X， Wang J， Peng J Y and Liang H D. 2023. C3N： content-constrained convolutional network for mural image completion. Neural Computing and Applications， 35（2）： 1959-1970 ［DOI： 10.1007/http://dx.doi.org/10.1007/
s00521-022-07806-0］
Rakhimol V and Maheswari P U. 2022. Restoration of ancient temple murals using cGAN and P2022Conv networks. Computers & Graphics， 109： 100-110 ［DOI： 10.1016/j.cag.2022.11.001http://dx.doi.org/10.1016/j.cag.2022.11.001］
Shen J H and Chan T F. 2002. Mathematical models for local nontexture inpaintings. SIAM Journal on Applied Mathematics， 62（3）： 1019-1043 ［DOI：10.1137/S0036139900368844http://dx.doi.org/10.1137/S0036139900368844］
Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition ［EB/OL］. ［2022-09-01］. https: //arxiv.org/pdf/1409.1556.pdfhttps://arxiv.org/pdf/1409.1556.pdf
Singh U， Somsukla M， Anil Saini and Dhiraj. 2023. Ancient indian murals digital restoration through image inpainting//In 2023 10th International Conference on Signal Processing and Integrated Networks （SPIN）. Amity University， Noida： IEEE： 635-640 ［DOI： 10.1109/SPIN57001.2023.10116111http://dx.doi.org/10.1109/SPIN57001.2023.10116111］
Su Z， Liu W Z， Yu Z T， Hu D W， Liao Q， Tian Q， Pietikäinen M and Liu L. 2021. Pixel Difference Networks for Efficient Edge Detection//Proceedings of the IEEE/CVF International Conference on Computer Vision （ICCV）. Montreal， Canada： IEEE： 5117-5127 ［DOI： 10.1109/ICCV48922.2021.00507http://dx.doi.org/10.1109/ICCV48922.2021.00507］
Suvorov R， Logacheva E， Mashikhin A， Remizova A， Ashukha A， Silvestrov A， Kong N， Goka H， Park K and Lempitsky V. 2022. Resolution-robust large mask inpainting with fourier convolutions//In Proceedings of the IEEE/CVF winter conference on applications of computer vision （WACV）. Waikoloa， HI， USA： IEEE： 2149-2159 ［DOI： 10.1109/WACV51http://dx.doi.org/10.1109/WACV51
2022.00323］ Woo S， Park J， Lee J Y and Kweon I S. 2018. Cbam： Convolutional block attention module//In Proceedings of the European conference on computer vision ECCV）. Munich， Germany： Springer： 3-19 ［DOI： 10.1007/978-3-030-01234-2_1］ YuZ T， ZhaoC X， WangZ Z， QinY X， SuZ， LiX B， ZhouF and ZhaoG Y. 2020. Searching Central Difference Convolutional Networks for Face Anti-Spoofing//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition CVPR）. Seattle， USA： IEEE： 5295-5305 ［DOI： 10.1109/CVPR42600.2020.00534］ （ZengY， FuJ， ChaoH and GuoB. 2022. Aggregated contextual transformations for high-resolution image inpainting. IEEE Transactions on Visualization and Computer Graphics， 29（7）： 3266-3280 ［DOI： 10.1109/TVCG.2022.3156949］ （ZhenZ X， HeZ W and LuZ M. 2024. DEA-Net： Single Image Dehazing Based on Detail-Enhanced Convolution and Content-Guided Attention. IEEE Transactions on Image Processing， 33： 1002-1015 ［DOI： 10.1109/TIP.2024.3354108http://dx.doi.org/10.1109/TIP.2024.3354108］
相关文章
相关作者
相关机构
京公网安备11010802024621