See his short story "The Streets of Ashkelon" in the book Two Tales and Eight Tomorrows published by Sphere Books. It's a story of a man living alone (as the only human) on a planet inhabited by curious, evolving, quite intelligent amphibians. He teaches them science and reason/logic. Then a ship lands and a missionary comes to save the souls of these creatures through Christanity.
In a collection entitled Stainless Steel Visions (1993) containing the above mentioned story, Harrison wrote a preface which includes some very strong statements about his atheism, and the ease (or lack thereof) of getting that particular story published in the anti-atheist environment of America at the time when he wrote it.
Contributor "Wylee" posted some more details on the Message board.
From the "About the Author" section at Harry Harrison's official website there's a "Bluffer's Guide" to the man written by Paul Tomlinson. Here's the section talking about HH's atheism:
Harry Harrison's short story The Streets of Ashkelon has appeared in more than 30 anthologies and in a dozen or so languages: it is probably the author's most widely published story. It is a story which was almost never written, and even after it was written, it seemed it might never be published.
Harrison had had the idea for the story for some time, but never wrote it because he knew there was no market for such a tale. But then he learned that Judith Merrill was putting together an anthology of original stories, all of which would break one of the taboos which had constrained authors writing for the genre magazines of the time (this being the late 1950s, early 60s). The anthology was never published, so Harrison tried to place the story elsewhere, but without success: it remained unpublished for over a year, until Brian Aldiss accepted it for his anthology More Penguin Science Fiction.
What was so terrible that no one wanted to publish the story? The hero was an atheist who tried to protect the inhabitants of an alien world from the influence of a Christian missionary. The story was regarded as being too offensive for a Christian readership.
Harry Harrison is a self-confessed atheist with no sympathy for such attitudes: The Streets of Ashkelon is an angry, and disturbing, story intended to make the reader question assumptions about religious belief.
The official website is located at http://www.harryharrison.com/:
--JS, TAS and "Wylee"