Hi Rajesh,
transaction is for DI API only - meaning only transactions made by DI API can be rolled back. You are trying to roll back something you are not doing, but SBO client.
If you want to stop adding of a document in client you could use transaction notification.
regards,
Maik