Please think about not disconnecting the streaming but to implement a Disconnect button in the Cast to Device window.
I'm not sure how others usually do it, but I will look into this. What are the downsides as using the cast button as a toggle?