If it helps you this would allow you to use openrouter.
I think the issue is mostly with codec conversion. Twilio handles that which is why they integrated twilio specifically. Avr-agent handles this as well, it took quite a bit of tweaking and eventually he found a module on a github account that handled PCM with different bitrates to/from ulaw.