Here is an odd one - FPS reducing with time


#1

Hi there,

Here is something that I’ve just noticed, but I am not sure how long it’s been doing this.
After a fresh reboot, I looked in NSC (64-bit) at my 3 720p 30fps cameras and all is well. They are all running at or near 30fps.
(There are other cameras on the system too, but all are lesser FPS and res)
After about a day, I check back in on NSC and find that all three cameras have dropped to 15 fps for some reason.
After trying different things (rebooting wifi routers, rebooting cameras, etc.) I discovered that to fix the issue, all I have to do was go into the services panel and do a restart on the Netcam Studio Server (64-bit) service, and everything jumps right back to 30fps.

I can have NSC in the background on my screen, and the services panel in the foreground (while still being able to view the FPS) and if it’s sitting around 15fps, I can click on the service restart, then I can watch the FPS stop for a few seconds, then pick back up at 30fps.

Also, when looking at Netcam Studio Service in Task Manager, while restarting the service, when it comes back up, the CPU and memory usage are essentially the same. I was expecting to see one or the other drop.
The system still has a fair bit of headroom on CPU and memory.

Odd problem but it seems like it’s related to the Netcam Studio services.
There are a couple of things I haven’t looked at yet

  • Does it immediately fall from 30 to 15, or is it gradual over time?
  • How long does it take before it actually hits 15fps?

Any thoughts of anything I can check?

Thanks,

Murray


#2

Hi Murray,
If that was the case this forum would be full of people complaining about this. However, I have a pritty good idea what it is.
What cameras is it?
-Henrik


#3

All three are different

  • Foscam fi9900p
  • Amcrest ProHD 1080P WiFi
  • Generic Chinese 1080p wifi with HI 3516C + Sony IMX222 (Awesome camera BTW)

I was using Software VLC for encoding, but just switched to Hardware_DXVA2.
It’s running stable now, but I will know tomorrow if that was the issue.


#4

I should add that all three are using ONVIF as the video source.


#5

Switch to internal.
Also, update to VLC 3.03 I think it is now. Earlier versions have problems with the latest update of win 10 if you run that.


#6

I am already using VLC 3.03, but it’s funny because I just updated that a few days ago. May be related.
Ok, I’ll leave the Hardware_DXVA2 testing until later and switch to Internal.
I did see a good reduction in CPU though when I switched from VLC to DXVA2.

I’ll get back to you tomorrow with the results of switching to internal.


#7

The reduction of cpu load was interedting since hardware acceleration unfortunately don’t do that much. So lucky you. What processor do you use?


#8

Intel Core i7 2600K o/c’d to ~4Ghz.
I don’t think it was a huge reduction in CPU, but it seemed noticeable anecdotally.
I’ll do some better testing after I confirm the FPS thing is fixed.


#9

Hi Henrik,

I noticed the same issue with my setup, it’s been since the time i started using NCS so approx. February 2018. At first i didn’t really care but now that someone has the same issue I am curious to know if there is a solution.

I am using 7 cheap generic chinese Onvif camera. MISECU brand 720p with 25fps.

Right now running in a VM inside VMWare with a Xeon E3 1275 3.4GHz with only 2 CPU allocated. (Runs pretty smooth except for the FPS drop thing)

I started with a Xeon X3220 that I gave to one of my friend to run his own cameras with NCS too.

After that, I dedicated my actual setup E3-1275 to NCS but I thought that was a waste of CPU power (20% usage top) so decided to virtualize it recently.

Issue was present on all 3 setups I had.

Restarting NCS service seems to resolve the issue for some time then the FPS drop happens again.

Thanks

Mickael


#10

Hi Mickael,
How much is “some time”?
-Henrik


#11

Some time it depends, can be 2 hours or can be a whole day, i can make a test later today if you want as i am going to bed now lol.

Thanks

Mickael


#12

Ya do so. I will setup a couple of 4mp Hikvision and see what happens. In my profuction systems I have never seen this.


#13

I let it run overnight on the internal encoder and all 3 cameras are now sitting at 15 fps again. I haven’t restarted anything in case there was something else you want me to try.
I was able to determine that it took one of my cameras about 6 hours to revert to 15fps. Also, it happens in an instant. I have a 19 second recording that was going along at 30fps, then dropped to 15 at about 4 seconds in, and then every recording after that was 15.
I couldn’t correlate with the other two cameras since they didn’t have a recording at that exact time, except to say that before the “event” they were at 30 and afterwards 15.


#14

I just tried rebooting one of the cameras and it came back up at 30fps. The other two are still sitting at 15.
So we know we can “fix” it be either restarting the Netcam Studio Server (64-bit) service or be rebooting the camera.
2nd Camera:
Disabling and re-enabling the source in the client bumped it back to 30.
3rd Camera:
Editing the video source in the client, then clicking the check mark restores it to 30 as well.

So it looks like anything that breaks and re-establishes the stream will fix things.


#15

Hi,
Thanks for investigation. I suggest that you go into the configuration of the camera and check how they are configured. Quite often in megapixel cameras you can select constant or variable FPS to save data traffic. By default it is usually set to variable. If nothing is going on in the scenario or low light the intelligence of the camera will reduce the FPS. This will also reduce cpu load since the h.264 calculations are rather heavy.
I have several 4mp Hikvision cameras and depending on what they are looking at some are on variable and some on constant FPS. The ones set to constant FPS keeps that. The ones set to variable varies with movement and light.
-Henrik


#16

They were all set to CBR (2048 kbps) and 30fps.


#17

CBR is Constant Bit Rate meaning that it keeps 2048 kbps and other parameter varies depending on movement and exposure etc. I don’t know what camera you have, but 2 Mbps is not much to keep 30 fps with 1080p. So something must change. I am running 1080p at 25 fps with a CBR of 6 Mbps with a medium video quality.
I think you need to do some experimenting to see how it works and what is the important parameter. Check the task Manager for bitrate.
When you restart NCS you reset the the h.264 encoding and it all starts from beginning. Then it’s up to the camera to deliver according to the configuration.
Since you have 3 cameras configure them a little bit different which might speed up results.


#18

Henrik,

Here is some screenshots from my setup

camsettings
CPU
fps25fps17

As you can see, FPS will vary, You were right about the VBR/CBR setting, I was on VBR so that explained the fps dropping to 15 for all cameras.

The screenshot is after setting the camera to CBR. Do I miss something?

In NCS,there is only 1 source enabled during testing.

Thanks


#19

I am only running the 1080p cameras at 720p/30fps, so 2048 kbps should be ok,
All 3 cameras work perfectly at 30fps for several hours with that setting, regardless of motion detection frequency or any other factors.
At some point, all three drop back to 15fps. If I reset the stream in some way, such as restart of the NCS service, or rebooting the camera, then they all jump back to 30fps.
It also does the exact same thing if I have the CBR set to 4096 so I don’t think it’s related.
The other non-rtsp cameras I have running all maintain their initial fps.


#20

EDIT: Something fishy is going on here. I made many tests with my two 4mp Hikvision cameras and in my largest production system and we need to look into this :thinking: Change to software_vlc and see what happens.

Hi guys,
I am running some tests here with two identical cams in parallel, but with some different settings. I got one result after 7h. I am running it again to at least have 2 results hopefully pointing in the same direction. So I will be back tomorrow :slight_smile: