Stream video from Android device over UDP in JAVA application

So, go to the finish line. We have already learned how to stream video from android to a VLC player, now it remains only to integrate the window with the video into the JAVA application and start to steer the robot.







The open source project VLCJ CAPRICA will greatly help us with this .

The vlcj project provides a Java framework to allow an instance of a native VLC media player to be embedded in a Java application.

The idea of ​​the guys is simple, but ingenious (really pepper). Instead of tormenting with FFmpeg libraries and more, you should immediately call a specialist the core of a normal, functional and professional VLC media player. And call it directly from the JAVA application.



Who cares, we ask for a cat.



Since, in this voyage, there are enough pitfalls, we will begin, as usual, with a very simple one and only then move on to the trivial.

Install VLCJ Package

First of all, check your version of the VLC media player installed. We don’t need a fresh version, it cuts what is required for udp stream. This was already mentioned in a previous post . Therefore, we download version 2.2.6 of Umbrella and at the same time carefully check our JAVA package. They must match in bit depth. If the player uses a 64-bit architecture, then the JDK must be the same. And it won’t take off.



After that, you can already download the VLCJ library package itself







Please note that we need the vlcj-3.12.1 distribution (zip) package. It is he who works with players version VLC 2.2.x. You can unzip it anywhere, the main thing is not in the folder of the VLC itself, because two files coincide by name. And if you rewrite them, it will all end in complete failure.



Next, we create a project in the IntelliJ IDEA IDE (if you have a different IDE, I can’t help anything) and write down the necessary dependencies for integrating VLCJ libraries.







We do just that for files:



jna-5.2.0



jna-platform-5.2.0



vlcj-3.12.1



Then we create the only class and write the next tiny program in it.

import java.awt.*; import java.awt.event.WindowAdapter; import java.awt.event.WindowEvent; import javax.swing.JFrame; import javax.swing.JPanel; import javax.swing.SwingUtilities; import uk.co.caprica.vlcj.component.EmbeddedMediaPlayerComponent; import uk.co.caprica.vlcj.discovery.NativeDiscovery; import uk.co.caprica.vlcj.player.MediaPlayerFactory; import uk.co.caprica.vlcj.player.embedded.EmbeddedMediaPlayer; import uk.co.caprica.vlcj.player.embedded.videosurface.CanvasVideoSurface; public class BasicPlayer { public final JFrame frame; public static String mrl; public static MediaPlayerFactory mpf; public static EmbeddedMediaPlayer MediaPlayer; public static CanvasVideoSurface videoSurface; public static Canvas canvas; public static void main(final String[] args) { new NativeDiscovery().discover(); mrl = "D:\\ttt.mp4"; SwingUtilities.invokeLater(new Runnable() { @Override public void run() { BasicPlayer vp = new BasicPlayer(); vp.start(mrl); } }); } public BasicPlayer() { frame = new JFrame("My First Media Player"); frame.setBounds(200, 100, 540, 340); frame.setDefaultCloseOperation(JFrame.DO_NOTHING_ON_CLOSE); frame.addWindowListener(new WindowAdapter() { @Override public void windowClosing(WindowEvent e) { System.out.println(e); MediaPlayer.release(); mpf.release(); System.exit(0); } }); JPanel contentPane = new JPanel(); contentPane.setLayout(new BorderLayout()); canvas = new Canvas(); mpf = new MediaPlayerFactory(); videoSurface = mpf.newVideoSurface(canvas); MediaPlayer = mpf.newEmbeddedMediaPlayer(); MediaPlayer.setVideoSurface(videoSurface); contentPane.add(canvas, BorderLayout.CENTER); //        frame.setContentPane(contentPane); frame.setVisible(true); } public void start(String mrl) { MediaPlayer.playMedia(mrl); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Yes, while we are trying to play just a file (as seen from the code). It is better not to start with udp - it won’t work. And the file is played completely if, of course, you have not forgotten to place it with the appropriate name in advance where you need it. I think that even for the most novice javista it will not be difficult to understand the above code.



All new is:



call for VLCJ

  new NativeDiscovery().discover();
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

and creating the media player instance itself

  mpf = new MediaPlayerFactory(); videoSurface = mpf.newVideoSurface(canvas); MediaPlayer = mpf.newEmbeddedMediaPlayer(); MediaPlayer.setVideoSurface(videoSurface);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

And then we just add it to the desired graphic panel:

 contentPane.add(canvas, BorderLayout.CENTER);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

And everything, the file will be played in this window.



Now try replacing

 mrl = "D:\\ttt.mp4";
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

on

 mrl = "udp://@:40002";
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

as we calmly did in the last post for streaming video via udp connection.

This number will not work here. The window, of course, will open, but it will show a fig, in the sense of a dark screen. Although there will be no error logs. There just won't be anything.

Gotta figure it out

Maybe the H264 codec that we selected in the settings is missing? Stop, but how then did the ttt.mp4 file just play? He can not play with this setting, he is mp4.



Immediately comes the understanding that the VLCJ library runs only the player’s core itself. And what kind of pre-settings were there, she does not know and does not want to know. That is, we need, somehow, when launching the JAVA application, somehow pass the VLC player that we want to explicitly use the H264 codec or, say, we want to rotate the image or something else.



It turns out that this can be done using the MediaPlayerFactory class. Only we launched it without arguments, or even with them. On stackoverflow.com, I immediately found a simple example related to image rotation:

 String[] args = { "--video-filter", "rotate", "rotate-angle", "10" }; mpf = new MediaPlayerFactory(args);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

That is, we transfer something there with a string array to the media factory and it is stored there for the media resource used.



I tried this method to play the file and as usual, nothing worked. It turns out that they forgot to add two dashes and spread it all over the Internet. I had to guess using a similar transform method.



In short, it should be:

 String[] args = { "--video-filter", "rotate", "--rotate-angle", "10" };
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Now our reference file has reverted as it should!







Further it will be quite simple:



To determine the codec, according to the VLC command line, we add the line to the string array:

 "--demux=h264"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Trying the udp channel again





mrl = "udp: // @: 40002";



And this time everything works, indicating the victory of the human mind. Now this video window or several of these windows you can freely port to the graphical interface of your JAVA application.

It would seem a victory?

Not really. A slight bewilderment was caused by temporary delays or, scientifically, lags. At first they are more or less acceptable, but if you have the patience to watch the video to the end, then you will see that the lag by the end of the first minute of the broadcast reaches as much as five seconds. I had enough patience for 10 minutes of shooting, but, strangely enough, the delay did not increase anymore, but remained within the same limits.





Of course, for watching a video from the camera, this will do, but hardly to control a robot. Even the lunar rover reacted twice as fast!



Suspicions immediately fell on the caching processes and they (suspicions) turned out to be true.



The most arrogant was:

  caching for network resources
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

It just eats up almost everything by default, if it is not given to him on time.

Can arrange a lag and:

 caching for cameras and microphones
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Therefore, to avoid multisecond delays, it is recommended to add the following lines to the same string array, separated by commas:

  "--live-caching=100", "--network-caching=500",
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Parameters are set there in milliseconds and therefore anyone can choose them for themselves.



You can also use the key:

 "--clock-jitter=time in milliseconds",
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Then the media player will try to optimize the jitter - twitching the screen. But there, the more time is set, the better it is optimized and this is understandable why. So here it remains only to seek consensus and sometimes see such disgrace in the logs:







He wanted to, you know, fix the jitter, and you set the time interval too small. Now it's my fault.



Now everything seems to be as it should. The delay was reduced to less than one second (although a little less).





As a result, we got a very tiny working code

 import java.awt.*; import java.awt.event.WindowAdapter; import java.awt.event.WindowEvent; import javax.swing.JFrame; import javax.swing.JPanel; import javax.swing.SwingUtilities; import uk.co.caprica.vlcj.discovery.NativeDiscovery; import uk.co.caprica.vlcj.player.MediaPlayerFactory; import uk.co.caprica.vlcj.player.embedded.EmbeddedMediaPlayer; import uk.co.caprica.vlcj.player.embedded.videosurface.CanvasVideoSurface; public class BasicVideoPlayer { public final JFrame frame; public static String mrl; public static MediaPlayerFactory mpf; public static EmbeddedMediaPlayer MediaPlayer; public static CanvasVideoSurface videoSurface; public static Canvas canvas; public static void main(final String[] args) { new NativeDiscovery().discover(); mrl = "udp://@:40002"; SwingUtilities.invokeLater(new Runnable() { @Override public void run() { BasicVideoPlayer vp = new BasicVideoPlayer(); vp.start(mrl); } }); } public BasicVideoPlayer() { frame = new JFrame("My First Media Player"); frame.setBounds(200,100, 540, 340); frame.setDefaultCloseOperation(JFrame.DO_NOTHING_ON_CLOSE); frame.addWindowListener(new WindowAdapter() { @Override public void windowClosing(WindowEvent e) { System.out.println(e); MediaPlayer.release(); mpf.release(); System.exit(0); } }); JPanel contentPane = new JPanel(); contentPane.setLayout(new BorderLayout()); canvas = new Canvas(); String[] args = { "--video-filter", "rotate", "--rotate-angle", "270", "--demux=h264", "--clock-jitter=100", "--live-caching=100", "--network-caching=500", }; mpf = new MediaPlayerFactory(args); videoSurface = mpf.newVideoSurface(canvas); MediaPlayer = mpf.newEmbeddedMediaPlayer(); MediaPlayer.setVideoSurface(videoSurface); contentPane.add(canvas, BorderLayout.CENTER); frame.setContentPane(contentPane); frame.setVisible(true); } public void start(String mrl) { MediaPlayer.playMedia(mrl); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Now you can integrate the video broadcast into my robotic control program, where I used to transmit video in pieces. And I must say the code has been greatly simplified, and the quality has improved by an order of magnitude. In addition to everything to the video stream, we can transmit readings



accelerometers

gyroscopes

lighting level

air pressure

compass readings

temperature

and even humidity



Provided, of course, that all these sensors are available on your smartphone.



And even turn on the headlamp! Automatically! If the lighting level drops.





Hardly anyone is particularly interested, but in case of a link to a github:



for cart

for smartphone