Fireplace and chimney - value vs function

[oldman]

I'd bet part of that is cold air coming down the chimney. Since cold air is heavier than warm, it'll just sit there until it's warmed. I'd go with an insert. My folks had one and it came in handy more than once when the power went out.