Watchdog for lost camera communication


#1

Today I needed to lookup a recording of an event that happened. Unfortunately, the camera stopped working and was showing a picture of at least 30 hours ago.
I checked the camera itself, and it’s working fine. Restarting NCS solved the issue, and started showing the camera feed again.
I went trough the logs, but can’t find any reference of an issue. I’m monitoring all cameras (ping, http, rtsp), but I didn’t get any alert because there was no issue at the camera side.
Is there any other way I could capture events like this? Is there a way to fire an event if NCS can’t grab images for more then an hour or something?


#2

Hi Mitch,
I am residing in Brasil for the moment where the quality of the electrical system is not that good. So I have experienced a lot. I ended up with a separate electrical system based on UPS for the cameras and switches. This solved mainly all problems. But, it is also a strong correlation between the quality of the camera/price and how it can handle even very short glitches in the energy supply. If the camera is cheap, well they have to cut the cost somewhere and quite often it is in the quality of the electronics. The electrical separation of the analog och digital system are often not so good.

Even if the camera seems fine and is running glitches in the energy supply effects the digital system so that the IP communication with NCS stops. If the camera is running I logged into the camera and rebooted it which made the IP communication with NCS work again. Better than restart NCS for me. With more expensive cameras I never have this problem.

In NCS there is actually a rule for lost communication with a device. Test that one. I am not sure what that is based on. When you have this frozen image do you also get this orange icon for warning in the event list? I get that in my system.

-Henrik


#3

I forgot that there was such rule :no_mouth:. However, when checking, I see that I had configured it before:

But I didn’t get any notification. It looks like NCS didn’t see that the image was not updated.

My server rack and all the cameras are running behind UPS and electricity filtering. All switches, ethernet ports and fiber ports are closely monitored (and where possible redundant), and I also have monitoring on grid voltage, current and phase. So I would know right away if something would happen.
I have a variation of cameras, going from $15 no name Chinese cams to $500 Axis cameras. All of them have from time to time some glitches in NCS. I don’t care if a camera is gone for a couple of frames, but if it stays down for a couple of minutes, I would like to know :slight_smile:


#4

Well, I can´t argue against that :slight_smile:
When we are talking about security this is of cause a very interesting and important part as well and not only hackers and intruders. In principal all necessary information should be in NCS so the question is how to create an intelligent watchdog that discover no activity on a certain camera? Very simple and do not cover everything, but quite often there is clock running in the video. That might be one possibility or … Can the rule be different?
First thing would of cause be to find out why it stops at all. I assume that will need close monitoring of the network traffic. I will do some simple test with my cameras.
-Henrik


#5

When this happens there is normally an event ConnectionLost triggered after 20 seconds which will log that into the event log, trig the rule if required and more importantly try to reconnect the video source.

Now if this time and in this case It wasn’t triggered in your case, I cannot really explain why. This event occurs quite often for most of us since there are many reasons on why netcam studio may loose connection with a camera.

I have never experienced cases with recent (1.5+) versions where NCS didn’t manage to reconnect by itself and it was required to do it manually.

If you have steps to reproduce then we can dig further but like that unfortunately I’m unable to provide you a reason.


#6

I have 2 cams. One is not connected and one have a power failure. They generate a message in the event list:
7th July 2017, 09:52:55 NetcamVideoSource(13).WorkerThreadFFMPEG Connection with rtsp://admin:****@192.168.2.30/ipcam_h264.sdp has been lost. Will try to reconnect in 1000ms… It tries to connect for days.

The Rule manager is set to SourceLost for any source. Send email. No email is sent after 20 sec. It works for MD so it should be correct setup. Is this maybe not a valid source lost?


#7

Actually, I just found out that I have 2 cams not running on NCS in the same way as mitch2k above.
This one on NCS

and if I connect directly to the cam I have this

In the Event list I have this:
7th July 2017, 10:12:26 NetcamVideoSource(13).WorkerThreadFFMPEG Connection with rtsp://admin:****@192.168.2.30/ipcam_h264.sdp has been lost. Will try to reconnect in 1000ms…

This do not generate an email. If I reboot the camera it will come back again.

What info do you need for this.
-Henrik


#8

Well you have the event logs and NCS is trying to reconnect.

Verifying the event is something that I can do. What is more worrying for mitch is that he didn’t have the events and NCS didn’t seem to be trying to reconnect the sources.

So apart the fact that the rule is not triggered (easy to verify for me), NCS seems to behave as it should.


#9

Yes, that´s one way of putting it, but still no video even though the camera is working. When I rebooted the camera then I got an email of lost source. So in some sense it is both correct and not correct. It triggered the rule when reboot. So can it have an “electrical connection” to the camera that do not trigger the rule at the same time as the video part to not work?


#10

Well in the event logs you have this message in loop right ? So NCS is constantly trying to reconnect (but doesn’t succeed until the camera is physically right) ?

In this case it’s really at camera level that something is wrong (and it needed to be rebooted).

What is not totally correct is that after some time we should probably already send the event and notify by email but if it keeps trying to solve the situation by itself, that’s normal…


#11

Well in the event logs you have this message in loop right ? So NCS is constantly trying to reconnect (but doesn’t succeed until the camera is physically right) ?
---- yes, it is in a loop. Yes, it reconnects automatically when cam is rebooted.

In this case it’s really at camera level that something is wrong (and it needed to be rebooted).
– before a reboot I connected with a mjpeg stream and that worked fine. It is the rtsp/h.264 that seems to be the problem. After a reboot that worked also. Is h.264 more sensitive for something than mjpeg? It is more hardware, but …

Yes, the notification after 20 seconds is good so it is possible to solve the problem manually whatever it is.

I missed to restart NCS to see if that solved the problem also. I do that next time. When NCS restarts it sends a command to the camera to connect with the rstp stream. Can that sort of resync or whatever the h.264 compression?

New interesting learning today also!


#12

Restarting Netcam Studio can help but not because it’s sending any special command or anything like that.

IMO it’s more the camera which runs out of sockets or out of streaming channels or something like that and when NCS tries reconnecting every 20 seconds it’s kind of flooding the camera’s I/O.

Letting the camera alone for a few minutes (while rebooting Netcam Studio) may also help it to recover automatically.


No HTTPRequest action on LostConnection event?
#13

The fact that even is not always triggered will be fixed in the next release.


#14

Awesome! Thanks for investigating


#15

Should be fixed here: