You have two files in pem-format: fullchain.pem and privkey.pem
Windows use the pfx format so you must convert pem to pfx.
The pfx format consists of a certificate file xxxx.pfx and a password. In the conversion process from pem to pfx you must manually add a password. Remember that password since it is used in the NCS configuration file.
For the conversion you can either use openssl as ngcard shows above or an online converter website. I don´t know why you get the error with the openssl so I recommend to do this first so you know your pem certificate works. I used this online free service SSL Converter - Convert SSL Certificates to different formats
Add files and password as
In return you get the pfx file and remember the password.
Now follow point 1c and 2 in the post Running Netcam Studio Server on SSL / HTTPS - Windows 10 and 11
The name of the certificate are different in my post.
When configurations are done start NCS X server, blue icon. At the bottom it should now state https://computerIP:8100
When you start the clients make sure to enable https.
When this works all is fine with the certificate. If you want to automate like ngcard go back to the openssl and add a password to -passout pass:xxxxx