After updating the OS on a Mac Mini to 10.12.6 and downloading Server.app 5.3.1 we encountered an error:
It turns out that not all OS/Server.app versions allow direct updates; in some cases you must do a data migration. Apple posts an upgrade matrix here.
The quick fix, if you don't mind configuring the server again, is to simply trash the Server.app from Applications and remove the /Library/Server folder. Once this has been done, you can re-install Server.app and it will launch the configuration manager.
If you get the "unable to update server" error and you need to keep your settings then you are in for a bumpy ride. You have no choice but to do a data migration between two devices. The procedure is explained in the same link as above and entails building a new system, target booting, doing a data migration and then installing Server.app. Honestly, Apple couldn't have made it any harder if they tried.