基于Paddle2.0实现视频分类模型GSM

本文复现了CVPR 2020的GSM视频分类模型，用Paddle2.0在Diving48数据集实现。该模型通过分解3D卷积核实现时空域视频理解，参数量小。项目处理了数据集，将视频转帧、生成训练测试文件，经训练测试，最佳模型在测试集准确率达0.223。

项目背景

GSM是2020年CVPR上的一篇论文Gate-Shift Networks for Video Action Recognition中提出的视频分类模型。本项目即对其进行复现。

项目简介

本项目使用paddle2.0实现了视频动作分类GSM模型，并在Diving48数据集上进行了训练和验证。

免费影视、动漫、音乐、游戏、小说资源长期稳定更新！ 👉 点此立即查看 👈

Diving48数据集的划分是按数据集游戏的划分方法进行划分的。

模型简介

Gate-Shift Networks for Video Action Recognition (GSM) 的核心是对进行时空域视频理解的3D卷积核进行分解的一种方法。该方法和已有方法的比较如下图所示。从图中可以看出，该模型跟其它网络模块相比具有参数量小的优点。

           

图1 GSM和其它网络结构比较图

模型细节

模型实现细节如下图所示。

           

图2 GSM实现细节图

GSM模块可以加在已有网络的不同部分，论文作者通过实验得出，将GSM模块加在下图所示的网络分支示例中具有较好的效果，即卷积核尺寸较小的分支中。

           

图3 GSM的plug-and-play示意图

论文原文：GSM: Gate-Shift Networks for Video Action Recognition

数据集介绍

本项目使用Diving48数据集进行训练和测试. 数据集游戏为Diving48数据集

Diving48是跳水竞赛不同动作的细粒度视频数据集。该数据集包含48类不同的跳水标准动作，共18k左右个视频片段。由于不同的跳水动作可能在起跳、腾空和入水的三个不同阶段中的任一个阶段发生不同，需要模型学习视频中的长时间段间的特征关系。动作示例如下图所示：

           

图4 Diving48动作示意图

       

这48个类别中的每一个都是由三个不同的动作构成的，分别是起跳动作 (dive groups), 空中动作(somersaults and/or twists), 入水动作 (dive positions)。 下图展示了这48个类别的不同动作组成与互相间的关系：

           

图5 Diving48中不同动作间关系示意图

文件结构

模型实现及训练、验证

解压视频数据集

In [ ]

import osif not os.path.exists('dataset/Diving48'):    os.makedirs('dataset/Diving48')

登录后复制    In [ ]

!tar -xzf data/data56350/Diving48_rgb.tar.gz -C dataset/Diving48

登录后复制    

创建标签文件

仅在预测时使用

In [ ]

import jsonimport numpy as nplabel_dir = {}i = 0with open('data/data56350/Diving48_vocab.json', 'r') as f:    vocab_list = json.load(f)    for vocab in vocab_list:    label_dir['_'.join(vocab)] = i    i += 1np.save('work/label_dir.npy', label_dir)print(label_dir)

登录后复制        

{'Back_15som_05Twis_FREE': 0, 'Back_15som_15Twis_FREE': 1, 'Back_15som_25Twis_FREE': 2, 'Back_15som_NoTwis_PIKE': 3, 'Back_15som_NoTwis_TUCK': 4, 'Back_25som_15Twis_PIKE': 5, 'Back_25som_25Twis_PIKE': 6, 'Back_25som_NoTwis_PIKE': 7, 'Back_25som_NoTwis_TUCK': 8, 'Back_2som_15Twis_FREE': 9, 'Back_2som_25Twis_FREE': 10, 'Back_35som_NoTwis_PIKE': 11, 'Back_35som_NoTwis_TUCK': 12, 'Back_3som_NoTwis_PIKE': 13, 'Back_3som_NoTwis_TUCK': 14, 'Back_Dive_NoTwis_PIKE': 15, 'Back_Dive_NoTwis_TUCK': 16, 'Forward_15som_1Twis_FREE': 17, 'Forward_15som_2Twis_FREE': 18, 'Forward_15som_NoTwis_PIKE': 19, 'Forward_1som_NoTwis_PIKE': 20, 'Forward_25som_1Twis_PIKE': 21, 'Forward_25som_2Twis_PIKE': 22, 'Forward_25som_3Twis_PIKE': 23, 'Forward_25som_NoTwis_PIKE': 24, 'Forward_25som_NoTwis_TUCK': 25, 'Forward_35som_NoTwis_PIKE': 26, 'Forward_35som_NoTwis_TUCK': 27, 'Forward_45som_NoTwis_TUCK': 28, 'Forward_Dive_NoTwis_PIKE': 29, 'Forward_Dive_NoTwis_STR': 30, 'Inward_15som_NoTwis_PIKE': 31, 'Inward_15som_NoTwis_TUCK': 32, 'Inward_25som_NoTwis_PIKE': 33, 'Inward_25som_NoTwis_TUCK': 34, 'Inward_35som_NoTwis_TUCK': 35, 'Inward_Dive_NoTwis_PIKE': 36, 'Reverse_15som_05Twis_FREE': 37, 'Reverse_15som_15Twis_FREE': 38, 'Reverse_15som_25Twis_FREE': 39, 'Reverse_15som_35Twis_FREE': 40, 'Reverse_15som_NoTwis_PIKE': 41, 'Reverse_25som_15Twis_PIKE': 42, 'Reverse_25som_NoTwis_PIKE': 43, 'Reverse_25som_NoTwis_TUCK': 44, 'Reverse_35som_NoTwis_TUCK': 45, 'Reverse_Dive_NoTwis_PIKE': 46, 'Reverse_Dive_NoTwis_TUCK': 47}

登录后复制        

视频转为jpg格式图片

视频转换需要较长时间，不低于1个小时，建议转换到非data文件夹以便后续启动项目后可以直接使用。

In [10]

!python work/extract_frames_diving48.py

登录后复制        

ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv  libavutil      54. 31.100 / 54. 31.100  libavcodec     56. 60.100 / 56. 60.100  libavformat    56. 40.101 / 56. 40.101  libavdevice    56.  4.100 / 56.  4.100  libavfilter     5. 40.101 /  5. 40.101  libavresample   2.  1.  0 /  2.  1.  0  libswscale      3.  1.101 /  3.  1.101  libswresample   1.  2.101 /  1.  2.101  libpostproc    53.  3.100 / 53.  3.100Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/nOlRwoxsDJ0_00555.mp4':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf57.71.100  Duration: 00:00:03.18, start: 0.000000, bitrate: 838 kb/s    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 830 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)    Metadata:      handler_name    : VideoHandler[swscaler @ 0x2204380] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/nOlRwoxsDJ0_00555/frames%05d.webp':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf56.40.101    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)    Metadata:      handler_name    : VideoHandler      encoder         : Lavc56.60.100 mjpegStream mapping:  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))Press [q] to stop, [?] for helpframe=  191 fps=0.0 q=24.8 Lsize=N/A time=00:00:03.18 bitrate=N/A    video:703kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknownffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv  libavutil      54. 31.100 / 54. 31.100  libavcodec     56. 60.100 / 56. 60.100  libavformat    56. 40.101 / 56. 40.101  libavdevice    56.  4.100 / 56.  4.100  libavfilter     5. 40.101 /  5. 40.101  libavresample   2.  1.  0 /  2.  1.  0  libswscale      3.  1.101 /  3.  1.101  libswresample   1.  2.101 /  1.  2.101  libpostproc    53.  3.100 / 53.  3.100Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/OFxuiqI5G44_00406.mp4':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf57.71.100  Duration: 00:00:03.77, start: 0.000000, bitrate: 1028 kb/s    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1021 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)    Metadata:      handler_name    : VideoHandler[swscaler @ 0x18f1400] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/OFxuiqI5G44_00406/frames%05d.webp':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf56.40.101    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)    Metadata:      handler_name    : VideoHandler      encoder         : Lavc56.60.100 mjpegStream mapping:  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))Press [q] to stop, [?] for helpframe=  226 fps=0.0 q=24.8 Lsize=N/A time=00:00:03.76 bitrate=N/A    video:862kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknownffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv  libavutil      54. 31.100 / 54. 31.100  libavcodec     56. 60.100 / 56. 60.100  libavformat    56. 40.101 / 56. 40.101  libavdevice    56.  4.100 / 56.  4.100  libavfilter     5. 40.101 /  5. 40.101  libavresample   2.  1.  0 /  2.  1.  0  libswscale      3.  1.101 /  3.  1.101  libswresample   1.  2.101 /  1.  2.101  libpostproc    53.  3.100 / 53.  3.100Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/LkluZoNfKu8_00121.mp4':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf57.71.100  Duration: 00:00:09.40, start: 0.000000, bitrate: 816 kb/s    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 814 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)    Metadata:      handler_name    : VideoHandler[swscaler @ 0x1b7d3c0] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/LkluZoNfKu8_00121/frames%05d.webp':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf56.40.101    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)    Metadata:      handler_name    : VideoHandler      encoder         : Lavc56.60.100 mjpegStream mapping:  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))Press [q] to stop, [?] for helpframe=  235 fps=0.0 q=24.8 Lsize=N/A time=00:00:09.40 bitrate=N/A    video:922kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknownffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv  libavutil      54. 31.100 / 54. 31.100  libavcodec     56. 60.100 / 56. 60.100  libavformat    56. 40.101 / 56. 40.101  libavdevice    56.  4.100 / 56.  4.100  libavfilter     5. 40.101 /  5. 40.101  libavresample   2.  1.  0 /  2.  1.  0  libswscale      3.  1.101 /  3.  1.101  libswresample   1.  2.101 /  1.  2.101  libpostproc    53.  3.100 / 53.  3.100Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/Q--70-ELqLw_00068.mp4':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf57.71.100  Duration: 00:00:06.56, start: 0.000000, bitrate: 774 kb/s    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 771 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)    Metadata:      handler_name    : VideoHandler[swscaler @ 0x2108060] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/Q--70-ELqLw_00068/frames%05d.webp':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf56.40.101    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)    Metadata:      handler_name    : VideoHandler      encoder         : Lavc56.60.100 mjpegStream mapping:  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))Press [q] to stop, [?] for helpframe=  164 fps=0.0 q=24.8 Lsize=N/A time=00:00:06.56 bitrate=N/A    video:606kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknownffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv  libavutil      54. 31.100 / 54. 31.100  libavcodec     56. 60.100 / 56. 60.100  libavformat    56. 40.101 / 56. 40.101  libavdevice    56.  4.100 / 56.  4.100  libavfilter     5. 40.101 /  5. 40.101  libavresample   2.  1.  0 /  2.  1.  0  libswscale      3.  1.101 /  3.  1.101  libswresample   1.  2.101 /  1.  2.101  libpostproc    53.  3.100 / 53.  3.100Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/_tigfCJFLZg_00205.mp4':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf57.71.100  Duration: 00:00:04.00, start: 0.000000, bitrate: 1545 kb/s    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1541 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)    Metadata:      handler_name    : VideoHandler[swscaler @ 0x1d4e780] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/_tigfCJFLZg_00205/frames%05d.webp':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf56.40.101    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)    Metadata:      handler_name    : VideoHandler      encoder         : Lavc56.60.100 mjpegStream mapping:  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))Press [q] to stop, [?] for helpframe=  100 fps=0.0 q=24.8 Lsize=N/A time=00:00:04.00 bitrate=N/A    video:479kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknownffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv  libavutil      54. 31.100 / 54. 31.100  libavcodec     56. 60.100 / 56. 60.100  libavformat    56. 40.101 / 56. 40.101  libavdevice    56.  4.100 / 56.  4.100  libavfilter     5. 40.101 /  5. 40.101  libavresample   2.  1.  0 /  2.  1.  0  libswscale      3.  1.101 /  3.  1.101  libswresample   1.  2.101 /  1.  2.101  libpostproc    53.  3.100 / 53.  3.100Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/sk8TafuB3lU_00696.mp4':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf57.71.100  Duration: 00:00:01.27, start: 0.000000, bitrate: 1286 kb/s    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1276 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)    Metadata:      handler_name    : VideoHandler[swscaler @ 0x1f20b00] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/sk8TafuB3lU_00696/frames%05d.webp':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf56.40.101    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)    Metadata:      handler_name    : VideoHandler      encoder         : Lavc56.60.100 mjpegStream mapping:  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))Press [q] to stop, [?] for helpframe=   76 fps=0.0 q=24.8 Lsize=N/A time=00:00:01.26 bitrate=N/A    video:199kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknownffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv  libavutil      54. 31.100 / 54. 31.100  libavcodec     56. 60.100 / 56. 60.100  libavformat    56. 40.101 / 56. 40.101  libavdevice    56.  4.100 / 56.  4.100  libavfilter     5. 40.101 /  5. 40.101  libavresample   2.  1.  0 /  2.  1.  0  libswscale      3.  1.101 /  3.  1.101  libswresample   1.  2.101 /  1.  2.101  libpostproc    53.  3.100 / 53.  3.100Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/xbQCwTHcGN8_00158.mp4':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf57.71.100  Duration: 00:00:03.56, start: 0.000000, bitrate: 1580 kb/s    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 1576 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)    Metadata:      handler_name    : VideoHandler[swscaler @ 0x1bbd3c0] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/xbQCwTHcGN8_00158/frames%05d.webp':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf56.40.101    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)    Metadata:      handler_name    : VideoHandler      encoder         : Lavc56.60.100 mjpegStream mapping:  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))Press [q] to stop, [?] for helpframe=   84 fps=0.0 q=24.8 Lsize=N/A time=00:00:03.36 bitrate=N/A    video:406kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknownExiting normally, received signal 2.ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv  libavutil      54. 31.100 / 54. 31.100  libavcodec     56. 60.100 / 56. 60.100  libavformat    56. 40.101 / 56. 40.101  libavdevice    56.  4.100 / 56.  4.100  libavfilter     5. 40.101 /  5. 40.101  libavresample   2.  1.  0 /  2.  1.  0  libswscale      3.  1.101 /  3.  1.101  libswresample   1.  2.101 /  1.  2.101  libpostproc    53.  3.100 / 53.  3.100Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'dataset/Diving48/rgb/l6ZoMjD1UtA_00156.mp4':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf57.71.100  Duration: 00:00:05.60, start: 0.000000, bitrate: 908 kb/s    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480 [SAR 4:3 DAR 16:9], 905 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)    Metadata:      handler_name    : VideoHandler[swscaler @ 0x209c500] deprecated pixel format used, make sure you did set range correctlyOutput #0, image2, to 'dataset/Diving48/frames/l6ZoMjD1UtA_00156/frames%05d.webp':  Metadata:    major_brand     : isom    minor_version   : 512    compatible_brands: isomiso2avc1mp41    encoder         : Lavf56.40.101    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 250x250 [SAR 16:9 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)    Metadata:      handler_name    : VideoHandler      encoder         : Lavc56.60.100 mjpegStream mapping:  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))Press [q] to stop, [?] for help

登录后复制        

生成训练和测试的数据文本文件

In [ ]

!python work/process_dataset_diving.py

登录后复制        

15026/15027

登录后复制        

在之前训练基础上微调

In [ ]

!python work/train.py --epoch 5 --batch_size 32 --seg_num 16 --seg_len 1 --pretrain True --weights 'pretrained/inception_pretrained' \--data_dir 'result/best_model.npz' --best_model_save 'best_model/best_model' --model_arc 'inception'

登录后复制        

You are using Paddle compiled with TensorRT, but TensorRT dynamic library is not found. Ignore this if TensorRT is not needed.W0119 22:07:28.029369 15355 device_context.cc:320] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1W0119 22:07:28.034546 15355 device_context.cc:330] device: 0, cuDNN Version: 7.6.video number:15027video number:1970/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/norm.py:634: UserWarning: When training, we now always track global mean and variance.  "When training, we now always track global mean and variance.")Loss at epoch 0 step 0: [3.657299], acc: [0.25]Loss at epoch 0 step 200: [3.6588974], acc: [0.25]Loss at epoch 0 step 400: [3.6196995], acc: [0.28125]The best result is 0.22030456Loss at epoch 1 step 0: [3.5962734], acc: [0.3125]Loss at epoch 1 step 200: [3.5641773], acc: [0.34375]Loss at epoch 1 step 400: [3.6840277], acc: [0.21875]The best result is 0.22081219Loss at epoch 2 step 0: [3.4678974], acc: [0.4375]Loss at epoch 2 step 200: [3.4990504], acc: [0.40625]Loss at epoch 2 step 400: [3.622469], acc: [0.28125]The best result is 0.22335026Loss at epoch 3 step 0: [3.6552052], acc: [0.25]Loss at epoch 3 step 200: [3.5325003], acc: [0.375]Loss at epoch 3 step 400: [3.6104255], acc: [0.28125]Loss at epoch 4 step 0: [3.6843047], acc: [0.21875]Loss at epoch 4 step 200: [3.6228292], acc: [0.28125]Loss at epoch 4 step 400: [3.7475114], acc: [0.15625]Final loss: [3.6923075]time duration is  5594.123642206192

登录后复制        

图示训练和验证过程

In [ ]

## 画图函数import numpy as npimport matplotlib.pyplot as pltdef draw_train_eval_process(title,step,train_reward, eval_reward, label_train_reward, label_eval_reward, color_train, color_eval, label_y):    plt.title(title, fontsize=24)    plt.xlabel("epochs", fontsize=20)    plt.ylabel(label_y, fontsize=20)    plt.plot(step, train_reward,color=color_train,label=label_train_reward)     plt.plot(step, eval_reward,color=color_eval,label=label_eval_reward)     plt.legend()    plt.grid()    plt.show()

登录后复制    In [ ]

result_gsm = np.load('result/best_model.npz')gsm_train_rewards = result_gsm['all_train_rewards']gsm_eval_rewards = result_gsm['all_test_rewards']steps = [i for i in range(len(gsm_train_rewards))]draw_train_eval_process("GSM training and eval",steps,gsm_train_rewards,gsm_eval_rewards, "GSM trainning accuracy", "GSM eval accuracy", "red","green", "gsm_train_eval_accuracy")

登录后复制        

登录后复制                

用保存的最佳训练模型测试

In [ ]

!python work/test.py --weights 'best_model/best_model'

登录后复制        

You are using Paddle compiled with TensorRT, but TensorRT dynamic library is not found. Ignore this if TensorRT is not needed.[INFO: test.py:  132]: Namespace(batch_size=1, dataset='diving48', infer_topk=1, modality='RGB', model_arc='inception', seg_len=1, seg_num=16, weights='best_model/best_model')W0120 00:08:33.052891 25454 device_context.cc:320] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1W0120 00:08:33.058135 25454 device_context.cc:330] device: 0, cuDNN Version: 7.6.video number:1970测试集准确率为:0.22335025668144226

登录后复制