That´s certainly an interesting a challenging setup that you have and I fully understand why the poor AMD processor is completely choked I hope it survived. The 5MP cameras really use it all.
Normally I would give you some hints to tweak the system to lower the cpu usage, but when I look at the results and your demands I realize that it will most likely not work anyway. However, each setup is unique so have you tested to use software_vlc or hardware_DXVA2 or QuickSync? Any changes using that if the GPU and AMD processor supports that.
So here are some tips from experience.
CPU: for these applications the Intel i-series works best. A high-end i7 takes you a long way, but again from you demands I would go for the i9-series. The more cores/threads the better. I am planning to build my own computer dedicated for many MP cameras and I will use the i9-9900K, but I have not built that computer yet.
Maybe you can start with only one 5MP camera and see what happens and then use 2, 3 … to gain some experience.
Drive: I assume the 1TB is a standard 5400 rpm HDD. When everything is maximized today how busy is the HD? If motion is detected for many cams at the same time and with that resolution it will be a lot of writing to the disk. I have so far not experienced any limitation with HD´s running at 5400 rpm. A 1TB SSD is always fun, but a little bit more expensive.
Graphics card: not that crucial I would say since the mode hardware_dxva2 do not improve the situation that much.
Another way to test this is if you have access to servers with XEON processors. The more cores/threads the better. However, this is unfortunately not fully true since the xeon processor lack the graphics part which seems to be important. I have one system running on an HP server with a xeon 3.2GHz 4 core processor with 20 cams from 640x480 to 720p. That ends up at 95% all the time. For that application it is ok, but the videos are rather low fps and sometimes jumpy. A very important part of using xeon servers is the reliability and stability you get. My system have been running for 5 years with no unplanned downtime. For me that is very important.
A final possibility that might be interesting is to look at a dedicated DVR. My experience is however that these solution also gets rather expensive for high-end systems. Also it is not always possible to connect the cameras that you already have.
That is what I have and there are some other postings about this in the forum, but I think I have covered most of it above.
Good luck and please give us a report how it goes.