I was thinking along the lines of the recipient gaining the sense as the memory is obtained, or the intended person gains the sense first while the memory still plays out like a grainy film.

Since Tanjiro relies more on his sense of smell than other senses, it affects how he perceives his surroundings which including memories. So I was thinking of writing out his memories more abstract and that the scents he picks up can sometimes overpower what he sees or hears. i.e Person A may suddenly smell medical salves and see white sheets before the memory becomes clearer and they see Tanjiro in a hospital bed next to his friends.

Though by the time the memories are swapped back, the recipient will also be giving him back the scents that come from the memory.