I don't have problem with atheists, it's muslims I am afraid have problem with us.
Being afraid is rarely a wise alternative to being informed. Perhaps you could try finding out whether Muslims, as a class, have a problem with you, before using your fear to back your decision making?
You might be surprised to find that Muslims who have a problem with you are neither more common, nor more fanatical, than Christians who have a problem with you; Jews who have a problem with you; white supremacists who have a problem with you; anti-government loonies who have a problem with you; or a whole bunch of other groups that spawn occasional terrorist nut cases.
The only thing you have to fear is fear itself.