Connecting NativePlayer rendering with pp::Graphics3D issue

[rockinggit]

Hello,

I am trying to set up NativePlayer Elementary Stream player rendering with openGL to improve its performance.

My basic set up of NativePlayer is working.
I changed line in es_dash_player_controller when creating player to like below.
player_ = make_shared(Samsung::NaClPlayer::MediaPlayer::BindToDisplayMode::DoNotBindPlayerToDisplay);

So that player will not draw itself.

for trial I first tried to do rendering with pp::Graphics2D which works.
but when i tried to connect with pp::Graphics3D it does not display anything.
Can someone tell me if an doing something wrong. I know i can use shaders etc with openGL calls but that is not useful it seems as Nacl::Player does not provide buffer which it is going to draw on display. So i am trying this way and Given NaclPlayer option of DoNotBindPlayerToDisplay , It seems to me it should be able to connect with pp::Graphics3D rendering.

my code is something like below: (for simplicity of reading i have given only changed/new functions. changes are in #ifdef USE_GRAPHICS3D)
/////////////////////////////////////////////////////////
#include "ppapi/cpp/graphics_3d.h"
#include <GLES2/gl2.h>
#include "ppapi/lib/gl/gles2/gl2ext_ppapi.h"

pp::Graphics3D context_; /**< Graphics3D context. */

void NativePlayer::DidChangeView(const pp::View& view) {
LOG_ERROR("DidChangeView");
const pp::Rect pp_r{view.GetRect().size()};
if (rect_ == pp_r) return;
rect_ = pp_r;
int32_t new_width = view.GetRect().width() * device_scale_;
int32_t new_height = view.GetRect().height() * device_scale_;
#ifdef USE_GRAPHICS3D
device_scale_ = view.GetDeviceScale();
pp::Size new_size = pp::Size(view.GetRect().width() * device_scale_,
view.GetRect().height() * device_scale_);
if (context_.is_null()) {
if (!CreateContext(new_size)) {
LOG_ERROR("Couldn't initialize GLES library!");
return;
}
} else {
// Resize the buffers to the new size of the module.
int32_t result = context_.ResizeBuffers(new_width, new_height);
if (result < 0) {
LOG_ERROR("Unable to resize buffers to %d x %d!", new_width,
new_height);
return;
}
}
LOG_ERROR("Initialized module's view with resolution %dx%d",
new_width, new_width);
#endif
pp::VarDictionary message;
message.Set(Communication::kKeyMessageToPlayer,
static_cast(Communication::MessageToPlayer::kChangeViewRect));
message.Set(Communication::kKeyXCoordination, pp_r.x());
message.Set(Communication::kKeyYCoordination, pp_r.y());
message.Set(Communication::kKeyWidth, pp_r.width());
message.Set(Communication::kKeyHeight, pp_r.height());
LOG_DEBUG("View changed to: (x:%d, y: %d), (w:%d, h:%d)", pp_r.x(), pp_r.y(),
pp_r.width(), pp_r.height());
DispatchMessage(message);
#ifdef USE_GRAPHICS3D
MainLoopIteration(0);
///////////////////////////////
#endif
}

#ifdef USE_GRAPHICS3D
void NativePlayer::MainLoopIteration(int32_t) {
if (context_.is_null()) {
LOG_ERROR("Graphics3D context is null");
return;
}
glClearColor(1.0, 0.0, 0.0, 1);
glClearDepthf(1.0f);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glEnable(GL_DEPTH_TEST);
int ret = context_.SwapBuffers(cc_factory_.NewCallback(
&NativePlayer::MainLoopIteration));
if (ret != -1)
LOG_ERROR("SwapBuffers returned %d\n", ret);

}
bool NativePlayer::CreateContext(const pp::Size& new_size) {
if (!glInitializePPAPI(pp::Module::Get()->get_browser_interface())) {
Logger::Error("Unable to initialize GLES PPAPI!");
return false;
}
LOG_ERROR("CreateContext");
const int32_t attrib_list[] = {
PP_GRAPHICS3DATTRIB_ALPHA_SIZE, 8,
PP_GRAPHICS3DATTRIB_DEPTH_SIZE, 24,
PP_GRAPHICS3DATTRIB_WIDTH, new_size.width(),
PP_GRAPHICS3DATTRIB_HEIGHT, new_size.height(),
PP_GRAPHICS3DATTRIB_NONE
};
context_ = pp::Graphics3D(this, attrib_list);
if (!BindGraphics(context_)) {
Logger::Error("Unable to bind 2D context!");
LOG_ERROR("Unable to bind 2D context!");
context_ = pp::Graphics3D();
return false;
}
glSetCurrentContextPPAPI(context_.pp_resource());
LOG_ERROR("CreateContext Success");
return true;
}
#ifdef USE_GRAPHICS3D
}

Any input are highly appreciated.Thanks

[rockinggit]

And i tried with changing
glSetCurrentContextPPAPI(context_.pp_resource());
to glSetCurrentContextPPAPI(view_.pp_resource()); because it is view of pp::Instance even then it does not work.

[GrandTeaBrewer]

Hello,

using NaCl Player alongside OpenGL shouldn't affect Player's performance. Rendering to texture is not supported - you do can use both Player and OpenGL at the same time, but while those two components can coexist, they are separate (for example: Player will play video and OpenGL will render user menu, player controls, etc.).

If you intend to use NaCl Player alongside Graphics 3D, you should use Compositor interface and use background plane layer, which will show video from the player.

[rockinggit]

Hello,

Thanks for your reply. Actually my main aim is to improve performance for live streaming on NaclPlayer. From your reply it seems to me that videorendering using openGL is not supported because texture is not supported.

And If i want to use openGL for controls rendering , i have to use it with pp::Compositor ? Do we have any example for how to use pp::Compositor class ?
Any inputs /suggestions you can give how to reduce latency for live stream on NaclPlayer?

[GrandTeaBrewer]

Actually my main aim is to improve performance for live streaming on NaclPlayer. From your reply it seems to me that videorendering using openGL is not supported because texture is not supported.
...
Any inputs /suggestions you can give how to reduce latency for live stream on NaclPlayer?

Regarding live streaming in NaCl Player, I think low latency mode introduced in pepper_63 (products 2019+) is what you're looking for. Unfortunately previous products doesn't support low latency mode :(. Also, it's not really a matter of performance - multimedia pipeline in "normal latency" NaCl Player is set up so that it buffers some data, assuring playback is as smooth as possible but introducing some latency at the same time (which is VOD playback optimization).

If you use low latency mode, each frame will be presented on the screen ASAP. As a side note, if you use NaCl Player in low latency mode, pipeline clock is controlled by the app pushing frames according to the desired framerate.

pepper_63 isn't available on Samsung Developers yet, but should be any time now... anyway, you can already download it with Tizen Studio Toolchain Download Manager.

And If i want to use openGL for controls rendering , i have to use it with pp::Compositor ? Do we have any example for how to use pp::Compositor class ?

You can check the compositor demo from Pepper SDK: https://chromium.googlesource.com/chromium/src/+/32352ad08ee673a4d43e8593ce988b224f6482d3/ppapi/examples/compositor/compositor.cc
In order to mix pp::Compositor with NaCl Player you'll need to do something like this:

background_plane_layer_ = pp::CompositorLayerSamsung(compositor_.AddLayer());
background_plane_layer_.SetBackgroundPlane(pp::Size(width_, height_));

[rockinggit]

I tried with pepper_63 toolchain. Application builds and runs on 19 and older TV's as well.

However player->IsValid() gives false on older models. on 19 TV player object is valid however video is not playing due to some other errors.

Can you confirm player invalid is due to LOW LATENCY is not supported on older platforms ?

/////////// Documentation says something like this.
/// Player will be invalid if backend fails to initialize. This can happen if
/// unsupported MediaPlayerMode is used when constructing player.
/// If player is invalid all method calls will fail.
bool IsValid() const;
////////////

Please confirm.

[GrandTeaBrewer]

Yes, indeed. Low latency mode is supported since 2019 TVs.