This sounds like it’s a problem no matter what method of communication you use, unless you keep no address book and memorize everything.

Phone numbers cost money, which means they’re not easy to create in bulk, and therefore banning or blocking spam numbers is much easier than if it was open sign up.

Signal doesn’t use SMS anymore, and all messages are sent over encrypted Internet protocol. Any servers in between won’t see the phone number, it’s not needed to deliver the message, it’s using an IP address at that point and the entire message metadata is encrypted. Signal is the only one that can see the phone numbers, which they use to identify multiple clients as a single user and route messages accordingly.

If you don’t trust Signal to run an unmodified server without malicious modifications, then why would you trust their build of the APK?

To truly be safe from Signal’s influence you would need to audit the source code and build it yourself.

Personally I have no problem using Signal’s servers