People were frequently surprised to learn that Arthur Clarke and I were good friends. He is considered the doyen of optimistic, technical, Space Age speculative writers, believing our species’ salvation to lie entirely in scientific discovery and engineering invention, his fiction full of detailed explication, sometimes virtually indistinguishable from fact. I am usually portrayed as the iconoclast of the SF “New Wave”, rejecting physics for psychology and favouring social themes over space stories, tending to examine the downside of technology. Yet actually we shared similar ideals. Much of our early work anticipated advances in astrophysics while dealing with the psychic future of mankind.

Many years after our first meeting I gave a party where I introduced Arthur to William Burroughs, the Beat author of Naked Lunch. No one expected them to have a lot in common, but they spent the next few hours together, sipping orange juice, occasionally asking for the music to be turned down because it was spoiling their conversation.

Born two days (and 22 years) apart, we met when I was 15, shortly before he went to live permanently in Sri Lanka. He was humorous, encouraging, egalitarian and generous, as interested in exploring the sea as examining outer space. We would generally meet whenever he was in England, usually at the Globe pub in Hatton Garden, where would-be writers could chat casually with established authors such as John Wyndham, John Christopher and C S Lewis; the SF fraternity had moved to the Globe from the White Horse in Fetter Lane in the mid-1950s. Arthur had already written his light-hearted Tales from the White Hart (1957) in affectionate memory of the Fetter Lane pub. Before the war he and some fellow SF writers had shared a flat in Gray’s Inn Road. His flatmates already called him “Ego” because of his total absorption in the subjects that interested him. He cheerfully accepted the nickname.

Born and raised in Somerset, Arthur came to London in the late 1930s to work as a pensions auditor for the Board of Education, but space travel was already his chief enthusiasm. An active member of the British Interplanetary Society, he grew up reading all the SF he could find, most of it in US pulp magazines, though H G Wells and Olaf Stapledon (the author of the epic Last and First Men) remained his chief influences. He contributed frequently to the pre-war SF fanzines, co-editing Novae Terrae (“new worlds”) in its original form. One flatmate and fellow editor, William F Temple, described him as highly strung and given to “sudden, violent expressions of mirth”.

After working on radar in the RAF during the war, Arthur received a first-class degree in physics and mathematics from King’s College London, and sold a few speculative articles, including one to Wireless World that proposed communications satellites in space. His first sales of professional fiction were to Astounding (later Analog), at that time the most prestigious American SF magazine, specialising in speculation based on hard science, with a strong emphasis on space travel. His later work – including his novella Against the Fall of Night, which became his first novel, The City and the Stars (1956) – appeared in rather more garish pulps such as Startling Stories. His fiction quickly brought him popularity with readers and in less than a decade he became known, with Isaac Asimov and Robert A Heinlein, as one of hard SF’s “Big Three”.

“Hard SF” is distinct from the kind written by Orwell, Dick or Ballard, which specialises in social and psychological speculation. Arthur’s work was distinguished from that of his peers by an almost mystical lyricism and a faith in a future where mankind would rid itself, through science, of its primitive and brutal characteristics. (Unlike Heinlein, with whom he eventually fell out over the American author’s support for Ronald Reagan’s “Star Wars” plans, he had little interest in military space fiction.)

***

At first his factual books, such as The Exploration of Space (1951), were more successful than his fiction. He was soon able to support himself by his writing, becoming a leading expert on rocketry and space travel, ready whenever the media needed a piece about space exploration. He even advised the creators of the running story “Dan Dare, Pilot of the Future”, which appeared in my favourite comic, the Eagle, and whose images prefigured those of 2001: a Space Odyssey.

He developed a keen interest in scuba diving; it was one of his chief reasons for moving to Sri Lanka in 1956 not long after the breakdown of his first and only marriage, which had lasted just a few months. He returned to England often, always staying with his brother Fred, his sister-in-law Babs and his mother, Nora, in suburban London. Occasionally he came with a diving partner, Mike Wilson, and brought film of their expeditions with him. He was extremely proud of his underwater discoveries, which included the lost Koneswaram temple in Trincomalee, an important historical site.

Some time after his arrival in Sri Lanka Arthur developed a profound friendship with the diver Leslie Ekanayake, whose family adopted him. He dedicated his 1979 novel, The Fountains of Paradise, to Leslie, describing him as the “only perfect friend of a lifetime, in whom were uniquely combined Loyalty, Intelligence and Compassion”. In 1977 he suffered a terrible emotional blow when Leslie was killed in a motorbike crash just before his 30th birthday. Arthur continued to live with the Ekanayake family until he died. He was buried next to Leslie. The family and his many friends in Sri Lanka describe Arthur as a gentleman of great generosity and spirituality, even though he was anti-religious and placed mankind’s salvation entirely in its own hands.

There is indeed a quality of spiritual idealism in most of Arthur’s major work, including 2001 as well as much of his non-fiction, an element largely lacking from the writing of his science-fiction peers. In most respects he was perhaps the most complex SF writer of his generation: his scientific training combined with a highly logical mind that was passionately committed to humanity and the natural world. Yet his pride in his achievements was obvious and he continued to earn his nickname.

In the mid-1970s my friend Angus Wilson visited him in Colombo. When he returned home Angus asked me if (like him) Arthur was gay. A keen SF reader, he shared a similar investment in humanity but had been somewhat overwhelmed by Arthur’s “tour” of his house: framed endorsements, pictures taken with presidents and princes, awards on display. Arthur struck him as competitive and “perhaps the most egocentric person I ever met”. Did I think Arthur was afraid that he, Angus, was trying to upstage him in some way? I assured him that Arthur was probably just showing off.

Arthur developed polio in the 1980s, making travel increasingly difficult. Shortly before he was due to be knighted in Colombo by the Prince of Wales in 1998, the Sunday Mirror published disgusting and unfounded gossip about him. I wrote to him to give him my moral support. He thanked me graciously. I should not worry, however, he said. The story was merely an attempt to embarrass his friend Prince Charles. He assured me that another friend, Rupert Murdoch, was looking after the matter. The story was soon retracted with apologies.

***

There are several published accounts of how the 1968 film 2001: a Space ­Odyssey came into being. I understood from Arthur that he was somewhat frustrated by the erratic schedule of its director, Stanley Kubrick. Consequently, the novel, which they were supposed to write before the film appeared, came out after the initial release date. But in the main he seemed happy with the collaboration, even up to the time that rough cuts were being shown. He was, I know, afraid that what with Kubrick’s inability to settle down and collaborate on the novel, with the result that the book was due to come out after the cinematic release, it might look like a novelisation of the film rather than an ­original work.

Based primarily on his short story “The Sentinel”, together with other published fact and fiction, the film was very much a joint effort, although Arthur was overly modest about his contribution. For his part, Kubrick seemed unable to come up with an ending that suited him. When I visited the set, the film was already about two years behind schedule and well over budget. I saw several alternative finale scenes constructed that were later abandoned. In one version, the monolith turned out to be some kind of alien spaceship. I also knew something that I don’t think Arthur ever did: Kubrick was at some point dissatisfied with the collaboration, approaching other writers (including J G Ballard and myself) to work on the film. He knew neither Ballard nor me personally. We refused for several reasons. I felt it would be disloyal to accept.

I guessed the problem was a difference in personality. Arthur was a scientific educator. Explanations were his forte. He was uncomfortable with most forms of ambiguity. Kubrick, on the other hand, was an intuitive director, inclined to leave interpretation to the audience. These differences were barely acknowledged. Neither did Kubrick tell Arthur of his concerns regarding the final version. Where, thanks to Arthur, the film was heavy with voice-over explication and clarifications of scenes, Kubrick wanted the story to be told almost entirely visually.

Without consulting or confronting his co-creator, Kubrick cut a huge amount of Arthur’s voice-over explanation during the final edit. This decision probably contributed significantly to the film’s success but Arthur was unprepared for it. When he addressed MGM executives at a dinner in his honour before the premiere, he spoke warmly of Kubrick, declaring that there had been no serious disagreements between them in all the years they had worked together, but he had yet to see the final cut.

My own guess at the time was that Kubrick wasn’t at ease with any proposed resolution but had nothing better to offer in place of his co-writer’s “Star Child” ending. We know now that the long final sequence, offered without explanation, was probably what helped turn the film into the success it became, but the rather unresponsive expressions on the faces of the MGM executives whom Arthur had addressed in his speech showed that they were by no means convinced they had a winner.

What had impressed me on my visit to the set was the dedicated enthusiasm of the Nasa advisers, who had offices at the studios. You could walk into a room and find a fully equipped spacesuit hanging behind the door. There were star-charts and diagrams on the walls; exploded drawings, models, mock-ups and pictures of spaceships and equipment. I saw Roy Carnon’s paintings of Jupiter and large sketches of scenes that would soon become every filmgoer’s idea of what the future in space would look like. The main set was dominated by a huge, fully working centrifuge, built at vast cost by Vickers-Armstrongs, the British engineering firm. Every technician I met talked about the project with such commitment that I was soon infected by the conviction that we really were preparing an expedition to Jupiter. Computer-generated imagery did not yet exist, and so a great deal had to be built or painted close to full size.

With almost no interest in space exploration, I nonetheless found myself excited by the atmosphere. Yet I did wonder if all the “authenticity” I saw around me might not be overwhelming. Could Kubrick’s singular imagination flourish in this atmosphere? Was that why it was taking so long to complete 2001 and the film was so heavily over budget? I had a slightly uncomfortable feeling that the considerable investment in establishing the reality of interplanetary space travel might produce a film more documentary than fiction.

As it turned out, Arthur did not get to see the completed film until the US private premiere. He was shocked by the transformation. Almost every element of explanation had been removed. Reams of voice-over narration had been cut. Far from being a pseudo-documentary, the film was now elusive, ambiguous and thoroughly unclear.

Close to tears, he left at the intermission, having watched an 11-minute sequence in which an astronaut did nothing but jog around the centrifuge in a scene intended to show the boredom of space travel. This scene was considerably cut in the version put out on general release.

***

If Arthur was disappointed by Kubrick’s decision to cut his dialogue and narrative to the bone, he was eventually reconciled by being able to put everything left out of the film into the novel, meaning that each man was able to produce his own preferred version. The success of the film ensured that the book became a bestseller, as audiences sought answers to questions raised by Kubrick’s version, and Arthur soon got over his disappointment, going on to write three bestselling sequels to his novel, only one of which has been filmed so far.

Inspiring governments to invest in space exploration and schoolboys to become astronauts, 2001 convinced the general public that science fiction could be taken seriously. Until Star Wars sent the genre back to an ­essentially juvenile form, the movie led to a greater understanding of the valuable creative possibilities of all kinds of science fiction. There would not be a more influential film until Ridley Scott’s Blade Runner, with its sober moral resonances. It also proved to Hollywood that good, big-budget SF movies could be money-spinners and garner critical respect at the same time. Without 2001 it is unlikely the genre would have progressed to its current state.

I have one other memory of that visit to the 2001 set. After being given a tour of the studio by the MGM publicist, I was led towards Kubrick’s office just as the director entered the main building. I prepared to meet the man who had contacted me a year or so earlier. I had many questions. Perhaps he would confirm some of my guesses.

Kubrick’s eyes went straight to me and did not leave me as he spoke brusquely to the publicist.

“Get these people off the set,” he said.

We were never face to face again.

“2001: a Space Odyssey” by Arthur C Clarke, introduced by Michael Moorcock, is published by the Folio Society (£29.95)