Hollywood [Sydney Morning Herald]- HISTORY is a continuum but there are various events you can put a pin in that have a huge effect on the future.

The world changed forever for a generation after the assassination of President John F. Kennedy, followed a few years later by the murders of his brother Robert F. Kennedy and Martin Luther King. The strange prologue to these turning points is the death of Marilyn Monroe. Despite thousands of books, films, reports and stories in all media, these tragic events remain a source of unanswered questions for pop culture and for serious scholars.

The best history comes from primary documents from the time that offer a contemporary snapshot, not filtered by passing decades. Last October the FBI released thousands of documents from its 20th century files under the freedom of information act. This has unleashed a torrent of information from a huge investigative body that was run by J. Edgar Hoover, arguably the most powerful man in US history, with his long tenure as the guardian of America’s secrets.

Hoover was obsessed with the private life of celebrities, particularly those with leftist leanings. It appears the FBI was tracking Monroe closely from the Cold War mid-1950s until her death, from the period she met and married the playwright Arthur Miller, who was being watched as a possible communist.

Thousands of documents have been released about Hoover’s boss, the attorney-general Robert Kennedy, but only 97 are about Monroe. Most have deletions and blacked-out passages. It is likely that thousands of other reports about her have not been released, but what we can see now gives an extraordinary window into history. A look at the dark side of the moon.

Most of the Monroe documents are marked “internal security”. They detail three chapters of her life: meeting and marrying Miller; her post-marriage activities including meeting American communists in Mexico; and her high-stakes relationships with the Kennedy brothers up until her death.

Monroe emerges as a highly involved individual politically and, typically of the misogynistic culture of the FBI at that time, the reports repeat that she must have got her leftist leanings and political questions and ideas from Miller. After all, blonde bombshells are not supposed to be asking the president about atomic testing within the US, an current issue at that time. When Miller remarries, a report says Monroe feels like a “negated sex object”.

The voluminous FBI files were declassified over the past 20 years, but were only provided to the individuals who asked for the information under the freedom of information act. All are stamped with various dates, initially marked “secret” and then stamped “unclassified” on the relevant dates. Many are from and to Hoover personally. Subjects range wildly from the spies Kim Philby and Guy Burgess, to Bud Abbott’s collection of 1500 porn films, to Lucille Ball being a member of the Californian Communist Party.

While studying these unclassified files for a film, I came across one explosive document, to my knowledge never previously noted in media accounts, that gives life to decades of conspiracy theory. This extraordinary document is dated as having been received by the FBI on October 19, 1964, is designated “Enclosure 61-9454-28”, and is titled “Robert F. Kennedy”. This is 16 days before the 1964 presidential election and 11 months after the assassination of President Kennedy. It was circulated to five top FBI officers, including Hoover’s right-hand man, Clyde Tolson.

It was forwarded by a “former Special Agent” working in the governor of California’s office.The agent, whose name is deleted, advises that he does not know the source and cannot evaluate the authenticity of the information. The report outlines a conspiracy involving Robert Kennedy, the actor Peter Lawford and others to induce Monroe to “suicide”.

This sounds like the wildest rantings of tabloid journalism except for a few key points: it is an official FBI document received before RFK’s murder and distributed to a small number of high-level FBI officials.

It was declassified only on December 6, 1984, for a person/s unknown and released in October 2006. It is a well written, highly detailed document noting specific phone calls, hotels and names, and reflects the FBI culture of its time, referring, for example, to The Diary of Anne Frank as a “slanted, left-wing picture”.

It announces that RFK has been having a “romance and sex affair” with Monroe over a “period of time” and that Kennedy is “deeply involved emotionally”, “and had reportedly promised to divorce his wife to marry Marilyn”. After Monroe’s contract with 20th Century Fox was cancelled, it says, she called RFK “person-to-person” at the Justice Department and he “told her not to worry about the contract – he would take care of everything”. They later had “unpleasant words” on another (presumably) bugged call and she “was reported to have threatened to make public their affair”.

The report continues: “On the day that Marilyn died Robert Kennedy was in town and checked into the Beverly Hills Hotel. By coincidence, this is across the the street from the house in which a number of years earlier his father, Joseph Kennedy, had lived for a time, common-law, with [the Hollywood star] Gloria Swanson.”

The document says it is “reported” that Lawford made “special arrangements” with Monroe’s psychiatrist, Ralph Greenson, who was treating her for “emotional problems and getting her off the use of barbiturates” and on her last visit to him prescribed 60 tablets of the barbiturate Seconal, “which was unusual in quantity especially since she saw him frequently”.

The story reported becomes ghastly: Monroe’s housekeeper, Eunice Murray, and her press agent, Pat Newcomb, were “co-operating in the plan to induce suicide. Newcomb was rewarded for her co-operation by being put on the federal payroll … of the Motion Pictures Activities Division of the US Information Service.”

Within 48 hours of Monroe’s death Newcomb is reported to have been flown to Hyannisport, Massachusetts, where the Kennedys have a compound, just after Lawford flew there himself. To continue the report: on the day of her death, Kennedy flew from Los Angeles to San Francisco via Western Airlines and checked into the St Charles Hotel. He then made a phone call from the hotel “to Peter Lawford to find out if Marilyn was dead yet”.

The housekeeper called Monroe’s psychiatrist on her behalf, after she had “taken the bottle of pills”, and the psychiatrist “left word for Marilyn to take a drive in the fresh air, but did not come to see her until she was known to be dead. Marilyn received a call from [her stepson, Joe DiMaggio jr] who was in the US Marines stationed at Camp Pendleton … Marilyn told him she was getting very sleepy. The last call she attempted to make was to Peter Lawford to return a call he had made to her. [Her former husband Joe DiMaggio] knows the whole story and is reported to have stated when Robert Kennedy gets out of office, he intends to kill him.”

How did the FBI informant know about these calls?: “[Los Angeles] Chief of Police Parker has the toll call tickets obtained from the telephone company on the calls made from Marilyn’s residence telephone. They are in his safe at LA headquarters.” In addition, eavesdropping or bugging was widely used by the FBI and cleared by Robert Kennedy, so it is reasonable to assume Monroe’s phone was tapped.

The report says the coroner appointing a psychiatric board of inquiry (an “unheard-of procedure”) was instigated “so the findings could be recorded that she was emotionally unbalanced”, to “discredit any statements she may have made before she died”.

In the lurid style of many FBI documents I have read, the report adds: “Marilyn also had an intermittent lesbian affair with … [deleted] … while Robert Kennedy was carrying on his sex affair with Marilyn Monroe … on a few occasions John F. Kennedy came out and had sex parties with … [deleted] … actress.”

The whole eyepopping account ends: “During the period of time that Robert F. Kennedy was having his sex affair with Marilyn Monroe, on one occasion a sex party was conducted at which several other persons were present. Tape recording was secretly made and is in the possession of a Los Angeles private detective agency. The detectives wants $3000 for a certified copy of the recording, in which all the voices are identifiable.”

What does one make of all this? I have no dog in this fight. The FBI document is undoubtedly genuine. Its scandalous, terrible contents are released to the general public with no editorial comment from the FBI. Many of the details have emerged piecemeal over the years in different accounts by different people. Were the media and others just spoon-fed bits of this narrative in an attempt to discredit the Kennedys, or did they come across evidence themselves that lends this account credibility?

One more unclassified piece of this jigsaw puzzle is a 1963 teletype marked “decoded” and “urgent” sent to Hoover soon after the assassination of JFK: “To director: … [deleted] informed that Attorney-General Robert Kennedy made a very secret trip to Los Angeles for a conference with [Chief William H.] Parker during which conference the Attorney-General allegedly told Parker he would replace Director as head of the FBI.” No date for this trip is provided. This is the same Parker the secret memo names as having the Monroe telephone records. Hoover was never removed and Kennedy left the administration of President Lyndon Johnson in 1964.

I started going through all these files working on a film project called Moral Relativity, a thriller about history, and curiosity took me to the files on the Kennedys and then Monroe, never expecting to find anything as startling as the secret account of her death.

I’m familiar with archival materials; in 1972 Lutz Becker and I found Eva Braun’s home movies in Washington. Then in 1978 I assembled a film documentary on the 1960s called The Times They Are A-Changin. The thesis of that film was that the 1960s really started when JFK was killed in 1963. An end of innocence, even if Kennedy was not an innocent.

I obtained permission from Abraham Zapruder’s son to use his father’s famous 8mm footage of the assassination for the first time in a feature; I found footage of an articulate Lee Harvey Oswald interviewed, saying he was a Marxist not a communist; I found film of LBJ rehearsing his resignation speech and home movies of Camelot. The studio, Columbia, took one look at the film and freaked out, closing it down and locking me and the producer, David Puttnam, out of our cutting room and offices. My interest in the 1960s never waned, so going through these files and finding something startlingly new has reinvigorated this interest.

Hoover was in a feud with RFK from the moment they worked together. He chafed at his boss’s youth and later said the only worse attorney-general was Ramsey Clark. Is it possible that the FBI concocted this account or parts of it internally in 1964 to discredit RFK? It’s hardly likely since he was alive when this was circulated within the bureau. Kennedy’s power had been drained with the death of his brother. Hoover never used this document, as far as we know.

Faced with the possibility that RFK might run for president, which he finally did in 1968, Hoover may have been holding this card close to his chest for leverage at a later time.

According to a Kennedy friend and historian, Arthur Schlesinger, Monroe died less than three months after she met Robert. Is all this the elaborate dirty tricks of Kennedy haters from decades ago, or are we getting closer to the historical truth with these new documents?

Right-wing extremists at the time said Monroe’s doctor was a communist so her death was labelled a “red” plot. In 1973 Norman Mailer speculated in Marilyn on FBI complicity in a plot to embarrass the Kennedys. A July 1973 FBI internal memo concludes with, “Action: No action is recommended regarding Mailer’s allegations. Any public statements would merely serve to feed the fires of publicity which Mailer is attempting to stoke.”

The new Monroe files end with a final touch of prurience, with a memo titled “interstate transportation of obscene matter” to Hoover in 1965, which included the following: “[Deleted] … at his office ran a “French-type” movie which depicted Marilyn Monroe, deceased actress, in unnatural acts with an unknown male. [Deleted] informed them he had obtained this film prior to to the time Monroe achieved stardom and that subsequently Joe DiMaggio attempted to purchase this film … for $25,000. This information should not be discussed outside the bureau.”