I have version 1.3.7.2 syncing perfectly for the most part to both my FTP server and Google Drive. Last night, my FTP server disk became full and the last 4 recordings didn’t get pushed. Once I cleared some space on the FTP server, I expected to see the missing files start showing up, but what happens is only new recordings get synced one at a time. All of the previous files get ignored no matter how many new recordings get synced. Is this the expected behavior?
Not sure if it’s related, but I also noticed that any files that fail to sync to FTP also fail to go to Google Drive. It seems they should be completely separate to me.
Yes, I have been doing some testing, and it looks like there are two distinct issues with FTP. Google Drive has it’s own problems which I haven’t tackled yet.
What I can do is limited since I don’t have physical access to the machine that the webcam is on. About the only control I have over it is forcing an occasional reboot. The netcam server software running as a service (v1.3.7.2) seems to get quirky after a while even though the computer is otherwise very stable. It’s 64-bit Windows 7 Home and it doesn’t have the most reliable network connection (802.11g).
Yesterday, it worked perfectly all day long and then overnight, it failed to upload any of the recordings. They are good and I can play them through the web interface as well as the Windows client with VLC. I rebooted the machine this morning and the current recordings transfer just fine, but all of the failed ones never show up. They all have the synced checkmark set in the interface (FWIW, none showed in in Google Drive, either).
It appears that the behavior with FTP is try once as opposed to an actual sync like I am accustomed to with rsync. I don’t see any way to force it to go back and re-attempt once it puts that check in the box.
The second issue is slightly more understandable and I am not sure what the solution is given the limitations of FTP. Once in a while, the computer will get rebooted during a transfer, which means that when the service restarts, it hasn’t checked the synced bit, but a partially uploaded file exists on the FTP server. In that case, I get an obvious error in the log like the following:
FTPUploader: Upload: Remote file Recording_0_20170105_095702_664.mp4 is already exists
After this, the sync box is checked and there is nothing I can do. Deleting it from the FTP server manually doesn’t force a resync later.
If there were a way to download the files from the server, I could kind of work around these problems, but I cannot figure out how. I can play them in the web client and the Windows client will open up VLC to play them, but actually downloading them seems impossible. Am I missing something?
Now that every overnight recording failed to upload again, I at least have one part of the pattern identified. These overnight files that are failing are all a result of rolling over at the 1 hour limit. The files are not large, all in the 60-120Mb range. Files 10x this size routinely transfer successfully during the day when I have the quality bumped up.
Here are the log snippets from one file that failed to transfer to either FTP or Google Drive:
Recording for more than 01:00:00 [Manual]. Starting a new file
File recording completed for Recording_0_20170106_042150_747.mp4 (01:00:00.1533203). Took 15 ms
Started File Recording [Manual]. Took 0 ms
FTP Uploader >> Upload Thread Starting
FTP Uploader >> Connected
FTP Uploader >> Uploaded 1 Items successfully
FTP Uploader >> Upload Item Exception
Exception Detail:Failed to open file 'C:\ProgramData\Moonware\Netcam Studio\Server\Library\Recordings\20170106\Recording_0_20170106_042150_747.mp4’
Synchronized 1/1 Items using SyncFTP.
FTP Uploader >> Disconnect
Synchronized 1/1 Items using SyncGoogleDrive.
Google Drive Uploader >> Error in UploadItem
Exception Detail:The process cannot access the file ‘C:\ProgramData\Moonware\Netcam Studio\Server\Library\Recordings\20170106\Recording_0_20170106_042150_747.mp4’ because it is being used by another process.
After these failed all night, I had to reboot the computer this morning to get any new files working properly. Finally figured out how to successfully save them with VLC, but it is incredibly labor intensive. Would be nice if there was even a download link in the web client or something.
Hi Scott,
Thanks for more info that makes it more clear what is going on and to figure out a solution.
1.The word synchronization is maybe not correct used. As you say a true sync takes software on both sides to keep track of what is correct transferred and re-transferred if necessary. For example I used a backup system over a VPN to another location. The Internet connection was not stable so the VPN went down and the backup failed. I moved to a true sync system between 2 NAS that works fine since it keeps track of the files and when the link is up or down.
As it says in the log the FTP uploads and when that start the checkbox is marked. However, there is no feedback from the FTP server of success or not. This becomes very clear when the files are to big and takes a long time to transfer relative day time.
2.It seems that your internet connection/LAN is not so stable during night time for some reason. Maybe it is a good idea to monitor that and have a word with your ISP. I use a program called PingPlotter where you get a very clear view of the connection in time. I have used these results several times in discussions with the tech people and quite often the plots help them to solve the problem.
3. Since both FTP and Google Drive have problems during night time something happens in the network. Google Drive is outside your office, but is the FTP on you LAN or outside?
4. An alternative is to avoid the FTP in NCS and use a separate software that sync/copies the files from the Library to the FTP server. There are several of these like File Juggler or some free ones also. You can also set up a separate FTP transfer to see if the problem still exist.
5. “Would be nice if there was even a download link in the web client or something.”. This is also on top of my list. It must be possible to download a file when you have looked at it in the web client. If I understand the process correct the file is actually downloaded to the local computer so saving it feels very close. The thing is that the web client is open source so it can be custom designed so NCS developers are for the moment not so keen on that idea. However, I am planning to look into that myself ;).
To summarize, I would check what is going on with the network connections since it works during the day.
This is what I can think of for the moment. Test and report back and you know where to find me ;).
Good luck,
Henrik
It is definitely not the network connection being unavailable that much. When I say it’s unreliable, I mean it will drop out for a few seconds, maybe a minute here and there when it is under heavy load. If I had my choice, everything would be wired
Anyway, more testing tonight and it most definitely has something to do with running the recordings past their max duration. I was using an hour before, so just now I set it to 15 minutes and hit record until it created 4 recordings in the library. Not one of them showed up on the FTP server. During this time, I was also listening to the audio stream and it didn’t drop out even once. Every single file has the same exception:
FTP Uploader >> Upload Item Exception
Exception Detail:Failed to open file 'C:\ProgramData\Moonware\Netcam Studio\Server\Library\Recordings\20170106\Recording_0_20170106_203258_422.mp4’
FTP Uploader >> Upload Item Exception
Exception Detail:Failed to open file 'C:\ProgramData\Moonware\Netcam Studio\Server\Library\Recordings\20170106\Recording_0_20170106_204758_858.mp4’
FTP Uploader >> Upload Item Exception
Exception Detail:Failed to open file 'C:\ProgramData\Moonware\Netcam Studio\Server\Library\Recordings\20170106\Recording_0_20170106_210259_182.mp4’
FTP Uploader >> Upload Item Exception
Exception Detail:Failed to open file ‘C:\ProgramData\Moonware\Netcam Studio\Server\Library\Recordings\20170106\Recording_0_20170106_211759_488.mp4’
I know these are good files because I can download them manually with VLC through the Windows client.
Interestingly, even the last file that I stopped manually didn’t transfer. But, after stopping that recording, I was able to make 3 more shorter manual start/stop recordings and they all transferred perfectly.
For this test, I had Google Drive disabled and I will test that separately even though it doesn’t do me much good. I was only using it as a backup in case FTP failed, but it appears to fail exactly the same files.
I know this is a longshot, but there is no chance of considering rsync (over SSH) as a sync protocol is there? Dropbox and Google Drive are still an extra step when trying to get the files onto my Linux box.
Hi Scott,
Well, the issue was not wired or not but why it works day time and not night time. Audio is buffered. However, I am setting up the same scenario with FTP sync between my Windows 10Pro and a NAS. I will alter the recording time between minutes and hours. The cam is on wifi. I will disconnect the NAS from the LAN and see what is happening with the FTP sync. I put the rsync on the list for suggestions.
Set up the FTP server on the NCS computer and sync to that and see how that goes.
The time of day was a red herring and I really should not have mentioned it. It’s just the time that I try to set and forget to let the recordings happen. During the day, I can just monitor and hit record whenever I want, so I just wasn’t seeing the behavior then. Now that I know what to look for, I can reliably reproduce with any max recording time setup any time of the day.
Just tested Google Drive with a 5-minute rollover and all the transfers fail, but with a slightly different error:
Recording for more than 00:05:00 [Manual]. Starting a new file
File recording completed for Recording_0_20170107_090907_601.mp4 (00:05:00.1865234). Took 63 ms
Started File Recording [Manual]. Took 0 ms
Google Drive Uploader >> Error in UploadItem
Exception Detail:The process cannot access the file ‘C:\ProgramData\Moonware\Netcam Studio\Server\Library\Recordings\20170107\Recording_0_20170107_090907_601.mp4’ because it is being used by another process.
Synchronized 1/1 Items using SyncGoogleDrive.
Of course, now the sync box is checked and it won’t retry. I would guess either a file lock issue or the record writes to a temp file and hasn’t finished copying when the sync process begins?
Yes, the audio is certainly buffered, but I will get dropouts when the WiFi adapter goes wonky on occasion. I am not concerned about the couple seconds here and there that would go undetected by listening to the audio since they are well within FTP’s tolerance. It’s not a powerful machine, so asking it to record, transfer files and play Call of Duty can bring it to it’s knees sometimes
Hi Scott,
Wel well ;).
I have really been a pain in the as for my computer running NCS and doing everything I can to stress it with several different parameters and recording everything from a minute to an hour, remove LAN from my NAS and wifi for the cam and nothing helps. I cannot reproduce the problems you experience.
The FTP sync checks every five minutes and if the file is still in use by NCS for recording it do not touch that file and it do not produce this “being used by another process”. When recording the file is done and file is closed the FTP sync copies it and the sync box is checked. Just some things:
Have you tested to FTP to the NCS computer?
If you use an external FTP program and try to copy the same files, do that work?
I am sure you have done the classical Windows style and completely remove NCS and install it again.
…
On the positive side I finally found a problem that have been haunting us for some time. So thanks for that ;).
Of the two of us, I am sure I am the bigger pain in the ass on this deal. But, at least I was able to see something interesting through the technological marvel that is Skype That machine is far away and I cannot just reinstall easily.
Anyway, I threw NCS on a laptop here and it behaves like yours. Uploads every recording reliably to both FTP and GD. It’s Windows 10 Pro and I don’t have any other Windows 7 Home machines around here to test with. Watching the library directory in real time, the machine that works records directly to the MP4 file and the broken Win7 machine records to an AVI file and then converts it when the recording completes.
This seems to be the root of the problem to me. That conversion from AVI to MP4 at the end of a recording takes quite a while if the files are large (video quality cranked up and/or the recording is long). When I turn everything down (low quality, 1 minute cycle) the transfers happen very reliably.
This doesn’t exactly explain why manually stopping a recording results in more reliable transfers even when the files are quite large. I noticed that the FTP cycle comes around every 5 minutes normally, but also kicks off shortly after a recording completes. It’s going to take a bit more testing for me to figure out the exact behavior here, but there is a clear difference between stopping a recording and letting it roll past the max duration. Somehow, it doesn’t want to wait for that AVI-MP4 conversion to take place when it rolls over.
Is this AVI temp file something I can change or is it a result of media libraries on the computers?
OK fine, I give up regarding the ass thing
Which version of NCS do you run? In an earlier version of NCS it was possible to do the trans-coding from avi to mp4 in the background. When the computer was busy with a high CPU load the recorded file was first stored in avi format and when the cpu load was not so high the ffmpeg would take the avi to mp4. The idea was good, but in practice it turned out to generate much more problems so this feature (checkbox under Settings -> Recordings) was removed. It was much better to do this in the internal software and save the file in mp4 directly. I don´t remember when that checkbox was removed, but if you have it uncheck that or use the latest version of NCS which is 1.3.7.2 on the download page. If you want you can have version 1.3.8.0 which have been under test for some time and will be released in late January, but it is on your own risk. I am running it in a production machine for some time now and I am happy.
-Henrik
Well…problem solved. That checkbox is definitely available on 1.3.7.2 because it appears that I had selected it at some point. Probably shortly after installation since background encoding seemed like a good thing on on old E8400 At the time, I was just trying to get everything working and never revisited what that meant. Actually even forgetting about it until you mentioned it.
Unchecked it and everything has been working perfectly for two days. I haven’t had a single failure since and some of the test files were upwards of 3Gb.
Here is the crazy thing. If I had actually checked the CPU usage back to back I would have gone back and unchecked it after a single recording. The CPU runs at 5-10% while recording with it unchecked and 50% with it checked. Removing it altogether definitely seems like a sane decision.
I still feel like the failure handling for transfers could use some work, but at least now that there are no failures, it’s not a priority
Thanks for your help and sorry to waste so much time on a simple fix.
for my computer running NCS and doing everything I can to stress it with several different parameters and recording everything from a minute to an hour, remove LAN from my NAS and wifi for the cam and nothing helps. I cannot reproduce the problems you experience.
That machine is far away and I cannot just reinstall easily.
At the time, I was just trying to get everything working and never revisited what that meant. Actually even forgetting about it until you mentioned it.