Suddenly high CPU load


#1

I used Netcam Studio with 4 cameras on an old Q6600 running with Server 2008 R2 for quite a while without problems.
Of course CPU load was quite high.
So I changed Netcam Studio to a Xeon E3-1245v5 running with Server 2016 (new installation of Netcamstudio)
The CPU load is quite low (about maybe 15%) when I start Netcamstudio.
But after a while it increases in steps.
Suddenly it’s 25% and then after a while it’s over 95%.
Sometimes this takes days, sometimes just some hours.
Closing and restarting Netcamstudio fixes this again.
The same happens when I use Netcamstudio as service.
I collected some logs of the last run (4 days) and some diagrams.
The pink curve indicates the CPU load which at the end is very high.

The logs and diagrams can be found here Logs


#2

Hi Erik,
Nice server! Right now I have this. In Settings -> Recordings there is a checkbox for recodiing in the background. When that is checked rhe recorded files are first stored in avi and when the processor have some “free time” the ffmpeg codec decodes the avi into mp4 that is the final format. It turns out that this process can use a lot of processor power and send the CPU load very high. So check this first. You can also look in the recording folder where is can be files in both avi and mp4 format.
-Henrik


#3

Hi Henrik,

Background recording is disabled because I read about this here in the forum.
I don’t think it has to do with the recording because the CPU load stays high (>90%) for many hours although there is no recording anymore (you don’t see this in the log files I posted, but it happend before).
The only way to bring it down is to restart Netcamstudio.
The logs show quite a few errors.
I expect that there are circumstances where somethings are not cleaned-up properly.


#4

I am running ncs on 2008R2 for a very long time and it is as you say very stable. Did this start with server 2016 or was it fine for some time? It is of interest since ncs is not tested for 2016 and there can be some new things. Can you also check the version of .NET Framework.
The logs says error in the stream encoder MJPEG. Can you check the Task Manager what process causes this high cpu load. Is it ffmpeg?
Do you get any mp4 files that you can play when this happens?

I m setting up NCS on server 2016 just now and for moment it is fine with one camera. We have had this Exception in TimerMemoryBitmapRefresh in a very early version of NCS caused by scaling of video, but that was fixed.


#5

Hi Henrik,

when there is high CPU load everything still works normal.
I get proper mp4 files.

But it looks like I found the main problem.
I used ProcessExporer and I saw that there are quite a view TCPIP connections active although nobody was connected.
It has to do with the ReverseProxy I started to use recently.
But only in combination with a static authToken.
It works very well but it seems netcamstudio can’t clean up properly when I use a link like this
https://cams.xxxxx.yy/Mjpeg/0?authToken=xxxxxxxxxx and close the browser tab again.
(All traffic from the subdomain “cam” is redirected to the port of netcamstudio).
When I use the normal login via the reverse proxy (in my case something like https://cams.xxxxx.yy/) netcamstudio can do a proper cleanup. So it doesn’t seem to be a general problem with the reverse proxy.
Also when I bypass the reverse proxy and use the authToken directly with the port everything is fine.

When I connect via the reverse proxy and authToken I see the user in Netcamstudio Usermanager
When I disconnect I see that the user is gone in the Netcamstudio Usermanager.
But the ProcessExporer still shows an active connection which is not closed anymore.
So each time I look at a camera a new connection is added and the old ones are not removed anymore.
I disconnected the reverse proxy for a moment so that the connections are forced closed and the CPU load of Netcamstudio dropped immediately.


#6

Hi Erik,
Thanks for the report and excellent that you found the problem with some detective work ;).
-Henrik


#7

Hi Henrik,

I hope there is a way to solve this in Netcamstudio directly because the reverse proxy solution works very well for me.
Please let me know if there is anything else I can do.

Thanks’
Erik


#8

I forward this to the developers.
-Henrik


#9

Finally investigating this one. Really nice finding btw.

I have managed to partially reproduce however as soon as I close the browser the corresponding sessions are all released.

So I could reproduce the problem if I keep the client browser (chrome) running and open since it’s the one that maintains the connection open.