Having reviewed the documentation, this design satisfies XSEDE and SP security requirements as I understand them. I have just a few comments about the documentation itself:
3. Custom Domains (new in v5.4.FIXME)
This will undoubtedly be addressed, but we will need to have the actual version that contains this fix in this doc. This occurs again later in the document
In 4.1 Endpoint Domain, in Example 5, the example discusses setting an endpoint to appear as data.example.edu, with mapped and guest collctions looking like m-13ea0.data.example.edu and g-8ff7e.data.example.edu, but then goes on to describe the endpoint administrator obtaining a certificate for *.example.edu. I assume that this is a mistake, and that the certificate should be for *.data.example.edu, but if not, it is concerning that one might such a broad wildcard.
In section 4.2 Mapped Collection Domains, in Example 6, it is implied (but not stated) that the certificate for the custom mapped collection domain must be a wildcard certificate (i.e. *.data.project.example.org) when using the --wildcard option so that guest collections have domain names that are subdomains of data.project.example.org. This should be explicit.