Service stops responding after a while after updating to 1.5.1

Netcam Studio Problems & Bugs
1

Hi,

After upgrading to 1.5.1 I am facing some issues. First of all it’s worth to mention that after upgrading, a big part of my cams were not working anymore. All cams with custom URLs lost their URL. Anyway, that was quickly fixed.

Issue I am now still facing, is that NCS runs for a couple of hours, and then the service goes brain dead.
I have seen the following sypmtoms during the 2 crashen I got today:

  • Web interface stops responding
  • All cams freeze on clients (and client seems to crash as wel)
  • Service keeps running
  • Nothing is written to the logs anymore
  • Mem & cpu usage drops
  • Can’t stop the service via services.msc

These are the logs:

 Error	18:29:39.097	WebServer.RootVideo	Exception in MoonwareServer. Command:http://beomcamsurv01:8100/jpeg/5?authtoken=f2d483a5-ee7a-4723-8009-37e693a75f1e&uniq=0.8648615440813849 from 10.101.50.55
Exception Detail:De objectverwijzing is niet op een exemplaar van een object ingesteld.	1814	1451,23MB
Warning	18:29:39.206	WebServer.ServerRequest	Failed Request for http://beomcamsurv01:8100/Jpeg/5?authToken=f2d483a5-ee7a-4723-8009-37e693a75f1e&uniq=0.8648615440813849 from 10.101.50.55.
 HTTP Error 500 - Internal Server Error	1814	1392,06MB
Warning	18:29:39.206	WebServer.ServerRequest	Failed Request for http://beomcamsurv01:8100/Jpeg/5?resolution=0 from 10.101.50.112.
 HTTP Error 500 - Internal Server Error	1813	1336,51MB
Error	18:29:42.941	NetcamVideoSource(1).WorkerThreadMJPEG	Exception (Web) using http://10.101.4.21:80/axis-cgi/mjpg/video.cgi?resolution=640x480
Exception Detail:Er is een time-out opgetreden voor de bewerking	1819	1409,45MB
Error	18:29:49.128	NetcamVideoSource(1).WorkerThreadMJPEG	Exception (Web) using http://10.101.4.21:80/axis-cgi/mjpg/video.cgi?resolution=640x480
Exception Detail:Er is een time-out opgetreden voor de bewerking	1819	1358,78MB
Information	18:29:49.300	SyncNetwork	Connecting to NAS / Network (\\10.101.2.2\ncs)	1820	1399,18MB
Error	18:29:50.160	LibraryManager	Error in Synchronization Timer
Exception Detail:Toegang tot het pad \\10.101.2.2\ncs\Motion\20170503\18 is geweigerd.	1821	1307,31MB
Error	18:29:52.161	SyncNetwork	Connection to NAS / Network failed (\\10.101.2.2\ncs)!	1821	1316,19MB
Error	18:29:53.332	LibraryManager	Error while trying to clean up remote repository (SyncNetwork):
Toegang tot het pad \\10.101.2.2\ncs is geweigerd.	1820	1364,13MB
Error	18:29:55.160	NetcamVideoSource(1).WorkerThreadMJPEG	Exception (Web) using http://10.101.4.21:80/axis-cgi/mjpg/video.cgi?resolution=640x480
Exception Detail:Er is een time-out opgetreden voor de bewerking	1815	1314,49MB
Information	18:29:55.894	Source(6).StopRecording	File recording completed for Recording_6_20170503_182936_269.mp4 (00:00:19.5309382). Took 250 ms	1813	1392,72MB
Error	18:30:01.363	NetcamVideoSource(1).WorkerThreadMJPEG	Exception (Web) using http://10.101.4.21:80/axis-cgi/mjpg/video.cgi?resolution=640x480
Exception Detail:Er is een time-out opgetreden voor de bewerking	1806	1244,97MB
Information	18:30:04.300	Source(6).StartRecording	Started File Recording [MotionDetector]. Took 15 ms	1808	1238,76MB
Error	18:30:07.415	NetcamVideoSource(1).WorkerThreadMJPEG	Exception (Web) using http://10.101.4.21:80/axis-cgi/mjpg/video.cgi?resolution=640x480
Exception Detail:Er is een time-out opgetreden voor de bewerking	1801	1258,64MB
Error	18:30:13.503	NetcamVideoSource(1).WorkerThreadMJPEG	Exception (Web) using http://10.101.4.21:80/axis-cgi/mjpg/video.cgi?resolution=640x480
Exception Detail:Er is een time-out opgetreden voor de bewerking	1785	1475,56MB
Information	18:30:14.519	Source(6).StopRecording	File recording completed for Recording_6_20170503_183004_284.mp4 (00:00:10.2188584). Took 219 ms	1789	1378,91MB

ErrorInformation	18:37:52.964	Service Host	Initializing Netcam Studio Core	225	98,06MB
Information	18:37:52.964	MoonwareCore.Constructor	Application Folder: C:\Program Files\Netcam Studio - 64-bit	226	98,57MB
Information	18:37:52.964	MoonwareCore.Constructor	UserData Folder: C:\ProgramData\Moonware\Netcam Studio\Server	226	98,57MB
Information	18:37:57.480	BackgroundFFMPEG.Start	Starting Background Encoding Task...	286	109,32MB
Information	18:37:57.683	MoonwareCore.Constructor	1928 Network Camera Templates Loaded...	296	121,73MB
Information	18:37:57.683	MoonwareCore.Constructor	Adding DirectShow Provider...	296	121,73MB
Information	18:37:58.152	MoonwareCore.Constructor	Adding Special Provider...	324	122,06MB
Information	18:37:58.168	MoonwareCore.Constructor	Adding NetCam Provider...	324	122,07MB
Information	18:37:58.183	MoonwareCore.Constructor	Adding Motion Detector...	324	122,09MB
Information	18:37:58.183	MoonwareCore.Constructor	Adding Audio Detector...	324	121,79MB
Information	18:37:58.183	MoonwareCore.Constructor	Adding LPR...	324	122,11MB
Information	18:37:58.199	MoonwareCore.Constructor	Adding MJPEG Stream Encoder...	324	122,11MB
Information	18:37:58.199	MoonwareCore.Constructor	Adding FFMPEG File Encoder...	324	122,12MB
Information	18:37:58.199	MoonwareCore.Constructor	Adding TimeLapse Encoder...	324	122,12MB
Information	18:37:58.214	MoonwareCore.Constructor	Adding Live Stream Encoder...	324	122,13MB
Information	18:37:58.214	MoonwareCore.Constructor	Adding MP3 Audio Stream Encoder...	324	122,25MB
Information	18:37:58.214	MoonwareCore.Constructor	Adding Watermark Filter...	324	122,31MB
Information	18:37:58.214	MoonwareCore.Constructor	15 Source Plugins Loaded in AvailablePlugins [SourcePlugins]...	325	123,66MB
Information	18:37:58.355	MoonwareCore.Constructor	Creating Instance of EmailSender Plugin...	336	122,46MB
Information	18:37:58.558	MoonwareCore.Constructor	Creating Instance of NAS Plugin...	349	122,68MB
Information	18:37:58.558	MoonwareCore.Constructor	Creating Instance of simple FTP uploader...	350	122,72MB
Information	18:37:58.574	MoonwareCore.Constructor	Creating Instance of FTP Plugin...	350	122,74MB
Information	18:37:58.574	MoonwareCore.Constructor	Creating Instance of Dropbox Plugin...	351	122,59MB
Information	18:37:58.589	MoonwareCore.Constructor	Creating Instance of Google Drive Plugin...	352	122,60MB
Information	18:37:59.496	RuleManager	RuleManager Created	368	128,83MB
Information	18:37:59.714	WebServer.Constructor	Initializing Web Server	365	128,68MB
Information	18:38:00.605	MoonwareCore.Constructor	Web Server started on port 8100	404	129,77MB
Information	18:38:00.683	SyncNetwork	Disconnecting from NAS / Network (\\192.168.1.100\Share$)	417	135,16MB
Information	18:38:00.793	UserManager	New User connected: Beomkiosk02 from 10.101.50.112 [1]	446	134,04MB
Information	18:38:00.964	UserManager	New User connected: Beomkiosk03 from 10.101.50.114 [2]	446	133,95MB
Information	18:38:00.980	Source(0).RestartIfNeeded	Re-Started Source(0)	457	133,94MB
Information	18:38:00.996	NetcamVideoSource(0).WorkerThreadAudio	Starting Network Camera Audio Capture Thread	457	138,84MB
Warning	18:38:01.011	WebServer.RootVideo	Unauthorized access to http://beomcamsurv01:8100/jpeg/0?resolution=0 from 10.101.50.112	464	136,43MB
Warning	18:38:01.027	WebServer.ServerRequest	Failed Request for http://beomcamsurv01:8100/Jpeg/0?resolution=0 from 10.101.50.112.
 HTTP Error 401 - Unauthorized	464	136,08MB
Warning	18:38:01.043	WebServer.RootVideo	Unauthorized access to http://beomcamsurv01:8100/jpeg/0?resolution=0 from 10.101.50.114	464	136,03MB
Warning	18:38:01.043	WebServer.ServerRequest	Failed Request for http://beomcamsurv01:8100/Jpeg/0?resolution=0 from 10.101.50.114.
 HTTP Error 401 - Unauthorized	464	136,03MB

Right before the crash I see some NAS connection errors. Al tough I don’t think it’s related, I’ll disable NAS offload for now. Should I enable any extra logging to troubleshoot?

Specs:
i7-2600 quad core 3.4ghz
8GB RAM
Windows 2012 R2
50GB SSD for OS & binaries
850GB HDD for video storage
14 cams (various models & resolutions)

Thanks

Mitch

1 Like
2

logs on non-english os are a bit tricky to read.

please upload on dropbox or somewhere the full today logs.

there are already suspicious entries at the begining of the sample you provided.

3

but I already don’t like this one:
De objectverwijzing is niet op een exemplaar van een object ingesteld

aka object reference not set to an object.

4

Sorry for the non-english :slight_smile:

I’ve sent you a link with the logs.

thanks

5

Well I see really a lot of different errors and problems in there.

First of all the audio source of 1st camera (source 0). Do you use audio on this one ? couple of codec not found errors so probably anyway not working so in this case please disable audio.

The NCS has issues starting the LPR plugin. Are you using the license plate detection ?

As you already noticed the problems to connect to your nas. Wither to be solved or to disable.

Then it was a bit in a strange state and started to report timeouts on most video sources so i would first start with these 3 issues so that at least it can start / initialize properly.

Use Netcam Studio X first, you’ll have better visibility than with Service. Once things work ok, then go back to Service.

6

Hi Steve,

thanks for your input.

On camera with id 1, I had LPR enabled. I disabled it now. That same camera (I guess that is the one you mean with source 0) has an G711_ULAW audio preset enabled. It do has audio (Axis P1344). Not that I hear much, because that camera is in a housing. How can I disable audio in NCS?

I now also removed camera 2, which was broken since a while, and NCS couldn’t connect anyway.

I started as X now. I’ll let it run for a while and send the logs.

Thanks

7

Strange because g711 ulaw is a compatible codec so it shouldn’t return Codec not found when trying to connect audio from this camera.

Please post the camera connection screen here (add source wizzard) and if you can also connect the screen in VLC and a screen shot of the Media Information screen in VLC.

8

Apparently and after checking closer it seems that we do not have anymore the G711 codec in our ffmpeg distribution and this is probably the reason.

Looking forward to fix this asap.

9

I really have the feeling that it’s related to this audio over HTTP (which anyway doesn’t seem to work and I doubt worked before).

So first thing here, please disable the audio by removing the checkbox Audio over HTTP for all the cameras that do have it (apart if some do have audio working) and report back if from this point the software is stable.

I couldn’t replicate the problem here however i’m already one version further in term of ffmpeg so maybe that’s why.

10

Unfortunately my first guess of the missing g711 was wrong so it’s not this. Still have the feeling that it’s related to audio so disabling (anyway not working) audio on sources reporting these errors if the first thing to do until I come with an updated version at least to see if it brings back stability.

11

Hi Steve,

I have done the steps described earlier, and have done nothing with the audio. However it has not crashed since 2 days. I’ll switch back to the service now to see if that gives the same result.

12

So what are the steps you’ve done ?

Anyway if you have non-working audio on some sources better to disable it rather than having ncs trying in loop to reconnect it. It’s not good for the logs and may impact performances.

13

I still didn’t locate the root cause but will have an updated version soon which at least will log more relevant information in the parts i’m suspecting as responsible here.

14

I’ve unchecked “audio over http” on every one of my cameras. It crashed overnight last night, but it’s been up since 9AM (it’s after 4PM) now.

15

Hi,

so what I did:
On camera with id 1, I had LPR enabled. I disabled it…
Removed camera 2, which was broken since a while, and NCS couldn’t connect anyway.
I started X instead of service

So I didn’t touch the audio yet, but it runs smooth (with x).
I just ended x, and tried to start the service, but the service wont start anymore. I reverted to x, and will troubleshoot the service later.

16

Hmm, maybe these audio related errors which I saw occur when a source is broken / unreachable.

I’ll verify this theory. Do you remember what template was used for the broken / unreachable cameras? Was this one also a panasonic ?

17

The crash was after disabling all audio or before ?

Please provide corresponding log files as soon as it crashes (you can upload on wetransfer.com and send them by PM to me and @Henrik)

18

Turned on this morning. It crashed within a few hours. Here are the logs.

https://we.tl/J6IlqNlqHR

Should I go ahead and install 1.5.2 now or wait?

19

Yes please upgrade to 1.5.2 and try to replicate the problem with Netcam Studio X.

There is no reason that it would crash with Service and not with X and at least when it happens with X, it gives you the possibility to send a report in case it’s not handled by the software and would generate a crash.

We don’t have this possibility with the Service so the only way to locate and solve that is by replicating using NCSX. Unfortunately the logs didn’t reveal anything suspicious to me.

Thanks

20

Ok, 1.5.2 installed.