Call for info: Portland's haunted mansion... or castle?

Hi folks! I have a question that may be too far back for most of you, but just in case someone once upon a time told you a story about it, I figured it was worth asking. Does anyone have any stories/recollections of hearing about a so-called "haunted house" or castle on Munjoy Hill?

It's a crazy Gothic home that was just up the hill from the Observatory.  It was torn down in 1914, but I keep finding vague references to it being haunted (this may purely be because it LOOKED haunted). I know recollections about it were still lurking around as late as the 1940s/50s.

Any thoughts or loose bits of story rattling around would be appreciated, as so far I have very little to go on so far! Thanks guys!!!  ...also of course any other haunted house stories from Portland are always welcome!

Here's what the crazy thing looked like (in other words, no wonder it had a reputation for being haunted):