Recording and streaming emu frames

[rockcarry]

What i fool i was.. it was "pixel format", anyway i the emu turns very slow as it encode the video. it's my emu performance thing.

No need to install any other video codec, only using ffencoder.dll is enough.
You can try my ffnes, I upload it. After run a rom in ffnes and close it, test.mp4 will be created in the same directory of *.nes file.

[rockcarry]

Is it worth to try to encode in a different thread?
I made the following: i created a thread like this:

so at the end of the frame i set "video_record" to "1".

The emu run fast, but when i play the video runs too fast, i don't know why.

I suggest you first write some test code for ffencoder, for example, generate random data in audio & video buffer, or some special data such as color bar, sine wave, etc. And then try to use ffencode to encode audio & video into mp4 file. After this, you will exactly know how to use ffencode.

Then you can try to use ffencoder in you emulator, but I suggest you try it in single thread first. In my ffnes, I am using single thread, so I am not sure it can work when using in a different thread. In theory it can work in multi-thread case, because ffmpeg support multi-thread.

And for the process of encoding thread, if you using multi-thread, I sugguest you using event intead of video_record flag and Sleep().

[Anes]

I suggest you first write some test code for ffencoder, for example, generate random data in audio & video buffer, or some special data such as color bar, sine wave, etc. And then try to use ffencode to encode audio & video into mp4 file. After this, you will exactly know how to use ffencode.

Then you can try to use ffencoder in you emulator, but I suggest you try it in single thread first. In my ffnes, I am using single thread, so I am not sure it can work when using in a different thread. In theory it can work in multi-thread case, because ffmpeg support multi-thread.

And for the process of encoding thread, if you using multi-thread, I sugguest you using event intead of video_record flag and Sleep().

Yeah, but c'mon its just a pointer full of pixels.

I tought that "pixel_fmt" in the structure was the INPUT pixel format and now i guess is the OUTPUT.

what pixel format are you using for outputing in your emu??

[rockcarry]

I suggest you first write some test code for ffencoder, for example, generate random data in audio & video buffer, or some special data such as color bar, sine wave, etc. And then try to use ffencode to encode audio & video into mp4 file. After this, you will exactly know how to use ffencode.

Then you can try to use ffencoder in you emulator, but I suggest you try it in single thread first. In my ffnes, I am using single thread, so I am not sure it can work when using in a different thread. In theory it can work in multi-thread case, because ffmpeg support multi-thread.

And for the process of encoding thread, if you using multi-thread, I sugguest you using event intead of video_record flag and Sleep().

Yeah, but c'mon its just a pointer full of pixels.

I tought that "pixel_fmt" in the structure was the INPUT pixel format and now i guess is the OUTPUT.

what pixel format are you using for outputing in your emu??

For pixel_fmt, I am using AV_PIX_FMT_BGRA which is the default value of ffencoder params, which is also the pixel format of win32 32bit bitmap object. It means the pixel format of video data buffer passed to ffencoder_video(encoder, vdata, linesize) is AV_PIX_FMT_BGRA. In bytes order, RGB datas are stored like this:
high low
A R G B

ffencoder will converts input video data to YUV420 format which will be accepted by x264 video encoder. The color converting is done internally by libswscale module of ffmpeg. the output format of color converting is always YUV420, the input format should specified by pixel_fmt. You should specifiy the correct value of pixel_fmt, otherwise the encoding result file will be problem.

[Anes]

Sorry, sorry what a fool i was!!! the "in realtime conversion" was failing cos i was reading from video card memory . That slowed down things.

ffencoder works like a charm!!

[nIghtorius]

Is it worth to try to encode in a different thread?
I made the following: i created a thread like this:

so at the end of the frame i set "video_record" to "1".

The emu run fast, but when i play the video runs too fast, i don't know why.

I suggest you first write some test code for ffencoder, for example, generate random data in audio & video buffer, or some special data such as color bar, sine wave, etc. And then try to use ffencode to encode audio & video into mp4 file. After this, you will exactly know how to use ffencode.

Then you can try to use ffencoder in you emulator, but I suggest you try it in single thread first. In my ffnes, I am using single thread, so I am not sure it can work when using in a different thread. In theory it can work in multi-thread case, because ffmpeg support multi-thread.

And for the process of encoding thread, if you using multi-thread, I sugguest you using event intead of video_record flag and Sleep().

I have a primary emulation thread which isn't connected to the GUI thread at all. All effects/graphics conversion/upscaling is done using a sleeping-dispatch thread. Technically if my emulator finished emulating one frame it dispatches the PPU frame to another thread/threads which takes care of upscaling (2xsai,xbrz, ntsc, 2x, 3x, tv modes, 1:1 -> 8:7 resizing, etc). when it finished the frame it will pause itself and send the data to the GUI thread..

The GUI thread draws the frame in the window and also encodes it to a videofile (using VfW). probably will add your ffencoder.dll to as it also compresses audio!. The reason why I have done this is as follows:

first:
The emulation thread doesn't need to be paused/halted, when the rendering/upscaling/conversion thread is compositing the image. (unless the 2nd frame is ready before the 1st frame has been processed then the emulation thread needs to be paused). After compositing the image the GUI thread takes over and draws it to the screen and if you are recording.. encodes it to a video stream.

And then. while we are encoding to a videostream.. the emulation thread fires another composition request.. meaning we have 3 threads taking care of:

* emulation
* display composition
* and encoding to a video stream.

at any given moment nothing is idle. because when the video encode is complete, the next frame is ready or almost ready from the compositor and the emulation thread is already computing the next frame. Ofcourse using 4xBRZ (1024x672) will bog down the emulator eventually on a old core2quad cpu. At that moment we are just asking too much.

[Anes]

mmm... as it should be now im having problems encoding audio.
The audio is in raw format 16 bit mono pcm data

i have tried filling the structure like this:

Code: Select all

    ZeroMemory(&ffpar, sizeof(ffpar));
    ffpar.filename = szMovieDir;
    // video params
    ffpar.video_bitrate = 512000;
    ffpar.video_width = 256;
    ffpar.video_height = 240;
    ffpar.frame_rate = 60;
    ffpar.pixel_fmt = AV_PIX_FMT_BGR0;
    // audio params
    ffpar.audio_bitrate = 128000;
    ffpar.sample_rate = 44100;
    ffpar.channel_layout = 3;   //this is for stereo

and the encode routine...

Code: Select all

void *data    [8];
int   linesize[8] = {256 * 4};

void AddRecordFrame(unsigned char * pixels, char * audio_stream, int audio_len)
{
    data[0] = pixels;
    ffencoder_video(ffhandle, data, linesize);
    data[0] = audio_stream;
    ffencoder_audio(ffhandle, data, audio_len);
}

It sounds very bad and de-sync.
One thing to mention is that if i use "mono" (0x04) in "ffpar.channel_layout" when i ffencoder_audio() is called it throws me "interger divided by 0" exception.

any idea how to encode the audio?

[nIghtorius]

mmm... as it should be now im having problems encoding audio.
The audio is in raw format 16 bit mono pcm data

i have tried filling the structure like this:

Code: Select all

    ZeroMemory(&ffpar, sizeof(ffpar));
    ffpar.filename = szMovieDir;
    // video params
    ffpar.video_bitrate = 512000;
    ffpar.video_width = 256;
    ffpar.video_height = 240;
    ffpar.frame_rate = 60;
    ffpar.pixel_fmt = AV_PIX_FMT_BGR0;
    // audio params
    ffpar.audio_bitrate = 128000;
    ffpar.sample_rate = 44100;
    ffpar.channel_layout = 3;   //this is for stereo

and the encode routine...

Code: Select all

void *data    [8];
int   linesize[8] = {256 * 4};

void AddRecordFrame(unsigned char * pixels, char * audio_stream, int audio_len)
{
    data[0] = pixels;
    ffencoder_video(ffhandle, data, linesize);
    data[0] = audio_stream;
    ffencoder_audio(ffhandle, data, audio_len);
}

It sounds very bad and de-sync.
One thing to mention is that if i use "mono" (0x04) in "ffpar.channel_layout" when i ffencoder_audio() is called it throws me "interger divided by 0" exception.

any idea how to encode the audio?

Encoding @ 60fps will desync audio eventually as the NES doesn't run perfect 60fps. it is around ~60.098814 frames per second ( source: viewtopic.php?t=492#p3783 )

[Anes]

Encoding @ 60fps will desync audio eventually as the NES doesn't run perfect 60fps. it is around ~60.098814 frames per second ( source: viewtopic.php?t=492#p3783 )

yes, i know that, but there must be some way.

[lidnariq]

Slow down everything by 0.16%. Frame rate would become exactly 60 Hz; CPU now runs at 1.78683MHz; PPU pixel clock becomes 5.36049MHz, &c.

[nIghtorius]

Slow down everything by 0.16%. Frame rate would become exactly 60 Hz; CPU now runs at 1.78683MHz; PPU pixel clock becomes 5.36049MHz, &c.

You could do: 60.098814/60 = 1,0016469

and then do 44100 * 1,0016469 = 44173 for sample rate. (still might have rounding errors)

[tepples]

I seem to remember libavformat (the container layer of FFmpeg) letting the client specify the exact frame rate as the numerator and denominator of a fraction. For example, color NTSC TV runs at 30000/1001 = about 29.97 fps. To calculate the NES's exact frame rate, you need the PPU's dot clock rate and the exact number of dots per frame and the exact dot rate.

• Dot clock rate is 945000000/176 Hz, or about 5.369 MHz. This is exactly 3/2 times the NTSC color burst frequency, which is defined as 315/88 MHz.
• Number of dots is 341*262 in "long" frames and 341*261+340 in "short" frames, or an average of 89341.5 frames
• Dot clock rate divided by number of dots is 945000000/(176*89341.5) = 945000000/15724104. Or reduced to lowest terms, this fraction is 39375000/655171.

[rockcarry]

Don't over-complicate things.

For my ffnes emulator, in roughly video rendering frame rate is about 60Hz, if you are using 48000Hz sample rate for 16bit stereo audio, on each video frame, apu will genrate 48000 * 2 * 2 / 60 = 3200 bytes. It means every video frame, I will draw 1 video frame on screen and render 800 audio samples (3200 bytes audio data) to waveout device. At the same time, if you call ffencoder_audio & ffencoder_video, passing the A/V data buffer and buffer size, the audio and video will be synced very well.

How do you think the recording result of my ffnes, good or bad ?

[nIghtorius]

Don't over-complicate things.

For my ffnes emulator, in roughly video rendering frame rate is about 60Hz, if you are using 48000Hz sample rate for 16bit stereo audio, on each video frame, apu will genrate 48000 * 2 * 2 / 60 = 3200 bytes. It means every video frame, I will draw 1 video frame on screen and render 800 audio samples (3200 bytes audio data) to waveout device. At the same time, if you call ffencoder_audio & ffencoder_video, passing the A/V data buffer and buffer size, the audio and video will be synced very well.

How do you think the recording result of my ffnes, good or bad ?

Audio/Video gets horrible out of sync on your emulator recordings.. As excepted the audio starts to lag. Because your audio runs @ ~60.10Hz and video op 60Hz. This will eventually desync your video.

[Anes]

I finally did it!!

I could mux a mp3 into an avi programatically with vfw and libmp3lame.dll. Audio is not desynced in the final avi. 735 16-bit samples per frame 44100 khz MONO.

Using Windows Video 1 at 50% quality and 128 CBR mp3 i got 5 minutes 315mb of video aprox.
I also testet lagarith but there were not too much difference, indeed Windows Video 1 looks better i think.

Wtf!!!