Not sure I've understood 100% but my understanding of your problem is that in the BADI you've implemented you raise an error which from FB50, FB60 return the user to screen with an error mesage but that the same BADI cause a dump when it raise an error message via MIGO.
One explanation that comes to mind is that a lot of the MIGO processing is done in an update task so that if this BADI is called from within the update task the error message you raise will cause the update task to fail rather than displaying an error message on screen. I don't think this should technically generate a dump but rather that you will see an express document update terminated message and see the update in error in sm13. If this is the problem, the only solution that comes to mind is duplicating the check you are doing related to this error message in MIGO in some enhancement that is called from the local task rather than the update task.