Files added to a portal as an attachment on an object record as set to "private" by default, so they require a different API call than normal to access
When a file is first added to a portal it is set to "private" by default, so it cannot be accessed by the file manager API endpoints like files in the File Manager. If the file was first added to File Manager then added as an attachment, it will be public and available by these endpoints.
Otherwise you will need to access the file URL using the Files API endpoint "Get URL for Private File" which returns a temporary signed URL where you can access and download the attached file. This URL will last for 1 hour and accessing it after that window will result in a 404. Once the link expires you can call the API endpoint once more to generate a new file.