I have been struggling to resolve this issue with VLC not using my 1030's hardware decoding with VLC using VDPAU. Hope someone here can help me solve this issue. Same problem for H264 and H265. VLC falls back to CPU rendering just fine, but I need to figure out why it isn't able to load VDPAU hardware decoding for my Nvidia GT 1030.
A bit of info to get started...
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic
4.15.0-39-generic
nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.77 Driver Version: 390.77 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GT 1030 Off | 00000000:01:00.0 On | N/A |
| 35% 28C P8 N/A / 30W | 116MiB / 1996MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 943 G /usr/lib/xorg/Xorg 62MiB |
| 0 1090 G /usr/bin/gnome-shell 52MiB |
+-----------------------------------------------------------------------------+
Information string: NVIDIA VDPAU Driver Shared Library 390.77 Tue Jul 10 22:09:49 PDT 2018
MPEG1 0 65536 4096 4096
MPEG2_SIMPLE 3 65536 4096 4096
MPEG2_MAIN 3 65536 4096 4096
MPEG4_PART2_SP 3 8192 2048 2048
MPEG4_PART2_ASP 5 8192 2048 2048
HEVC_MAIN 153 262144 8192 8192
HEVC_MAIN_10 --- not supported ---
HEVC_MAIN_STILL --- not supported ---
HEVC_MAIN_12 --- not supported ---
HEVC_MAIN_444 --- not supported ---
/usr/lib/i386-linux-gnu/vdpau/libvdpau_nvidia.so
/usr/lib/i386-linux-gnu/vdpau/libvdpau_nvidia.so.1
/usr/lib/i386-linux-gnu/vdpau/libvdpau_nvidia.so.390.77
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.1
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.390.77
VLC media player 3.0.4 Vetinari (revision 3.0.4-0-gf615db6)
[000000000210d3b0] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
Qt: Session management error: None of the authentication protocols specified are supported
Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: No such file or directory
Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: No such file or directory
[00007f6c48c06a30] main decoder error: buffer deadlock prevented
Edit: Question was in regard to getting VDPAU hardware decoding working, not VA-API. Either way, even after following the instructions for VA-API, still cannot get hardware decoding to function properly, whether VLC is set to use VA-API or VDPAU. Errors can be found in comments to OP.
Error logs:
VAProfileMPEG2Simple : VAEntrypointVLD VAProfileMPEG2Main : VAEntrypointVLD VAProfileMPEG4Simple : VAEntrypointVLD VAProfileMPEG4AdvancedSimple : VAEntrypointVLD <unknown profile> : VAEntrypointVLD VAProfileH264Main : VAEntrypointVLD VAProfileH264High : VAEntrypointVLD VAProfileVC1Simple : VAEntrypointVLD VAProfileVC1Main : VAEntrypointVLD VAProfileVC1Advanced : VAEntrypointVLD
Qt: Session management error: None of the authentication protocols specified are supported libva info: VA-API version 0.39.0 libva info: va_getDriverName() returns -1 libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)
[00007f55080a34c0] vaapi_drm generic error: vaInitialize: unknown libva error libva info: VA-API version 0.39.0 libva info: va_getDriverName() returns -1 libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)
[00007fae100a37a0] vaapi_drm generic error: vaInitialize: unknown libva error Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: No such file or directory [00007fae1cc0c400] main decoder error: buffer deadlock prevented QObject::~QObject: Timers cannot be stopped from another thread