I've got a couple of dwarf gourami, couple ember tetras, a few rummynoses, a black widow tetra. I also have a young bristlenose and a borneo sucker, but they aren't really affected. They all get along peacefully have been in together for quite some time now.
I get what your saying about associating me with bad stuff, but it really is only when the light is on. When it's off they don't have a problem. I can press my face against their tank and they couldn't care. Unless it has something to do with seeing me different when the lights are on. I'll try the hiding from them and see what happens