Toxic is really too strong a word for it. "Mildly poisonous" is closer to the truth, but it lacks character. It sounds like something that causes a topical rash, not something that's truly detrimental to your life.
Allergen might come the closest. An allergen can be mildly irritating,(more) or potentially deadly. It depends on the circumstances involved; like the amount of exposure or your particular sensitivity.
In this case the allergen isn't something you ate, or something you rubbed up against. It's something your remembering. And specifically, it's something that you can remember not doing. Something that you meant to or wanted to do, but didn't. Maybe it's something that you still mean to do, but never seem to get to. The time is never there, the time is never right, the timing doesn't work out.
Sometimes you can't fight reality. What you want to do just isn't going to happen. Other times, it's you. You didn't make it happen; and now you wish that you had. Every time you do that, you've created a allergen-memory. The more things you fail to do, the more allergen-memories you've created; and the more allergen-memories you create, the more allergen exposure you've brought upon yourself.
At low levels it can be a mild irritant, frustrating or distracting you. However, as the exposure/accumulation grows it can reach toxic levels; crushing you, depressing you, even making it impossible for you to think straight. At high levels it can truly ruin your life.
Then what? What do you do with a mountain of allergen-memories? Which can you resolve, and which can't ever really be exorcised?
As they say, the first step is admitting you have a problem. After that... well, we'll just see.(less)