Online Photography Rights: Are You Allowed to Link to Me?

This got me to thinking about what my online rights are and I realized, unfortunately, that I have no idea. Are random sites allowed to link to my photos? What if the way that they are using my pics has the potential to be detrimental to the image of the