Hi Bo
I met with such behavior and the same error message when I had some rules in SBO_SP_TransactionNotification stored procedure. I'd first check what's inside this procedure.
The reason why you're able to add such document through B1 interface could be, that in UI some FMS queries gets executed and all fields are properly fulfilled.
btw
As for using transaction notification alongside with B1if you can check these explanations by Teun Aben:
Add/update object with B1i with custom businesslogic in Transaction notification
Kind regard,
Radek