IE 7.0/8.0 have such problems reported in past, try out with IE 8.0 in 7.0 compatibility mode. In IE 9.0 Beta sometimes works like in Firefox, sometimes fails.
EDIT: After deep research regarding the Internet Exploder issues, I found this:
[quote]
Internet Explorer adds an entirely new method to the window object to open a modal window - window.showModalDialog(). If you use that to try to open a modal window then if your visitor is running Internet Explorer then a modal window will be opened. If they are running some other browser then all they will get is a Javascript error.
[/quote]
You should pick FF or IE and recommend it to your users. If the modal box is setup for IE only, then bye, bye to others.
Better put a warning plugin on the main site, that user uses insecure web browser which can cause malfunctioning on they're system and better to change it to Chrome, Opera or Firefox. (works fine with Maltese religious users) Now most of them will never return to IE.
Source:
http://javascript.about.com/library/blmodal.htm