A transaction is a SIP message exchange between two user-agents that starts with a request and ends with its final response (it can also contain zero or more provisional responses in between). For example, during the termination of a SIP session, one user releases the call by sending a BYE request and the other party replies back with a 200 OK response. This message exchange is called a transaction.
A dialog is a complete exchange of SIP messages between two user-agents. That means that transactions are actually part of a dialog.