Those who have been following Libya since the fall of Muammar Gaddafi in October 2011 would not be surprised by what transpired on Tuesday evening. Libya has been a militia mess since the “war for democracy” ended, with almost daily accounts of violence between rogue militias who refuse to disarm for a variety of reasons.

The unstable and precarious state in which Libya finds itself led to one such militia — which US officials suspect was planned — attacking the US consulate in the eastern city of Benghazi, killing the US ambassador along with three embassy staff and 10 Libyan security officers trying in vain to defend them.

While Libyan officials say the US ambassador died from smoke inhalation in the aftermath of a rocket attack, two of the four American staffers were shot dead, suggesting that this was an orchestrated attack set to kill US diplomats.

But the intention of an anti-US militia in Libya should not detract from the more concerning facts for the United States. Despite a concerted effort to appear “on the right side of history” with pro-revolutionary forces in various parts of the Arab world, anti-Americanism still remains at an exceptionally high level across the region.

Dismissing the acts in Libya as that of a minority is difficult to grasp as anti-US protests spread like wildfire across the region courtesy of a short US film deemed offensive of the Prophet Muhammad, and promoted by anti-Muslim US pastor Terry Jones. Starting on Tuesday in Egypt with thousands of people, including non-Islamist elements such as soccer “ultras” fans, the protests have now spread to Tunisia, Sudan, Gaza and US-ally Morocco. Not wanting to miss out on the action, Iran has also called for mass protests, while Egypt’s Muslim Brotherhood has scheduled nationwide demonstrations for Friday. Foes of the Arab world united in condemnation of the US film, from Lebanon’s Shia Hezbollah to Syria’s Sunni Muslim Brotherhood.

Anti-Americanism is a general sentiment in the Arab and Muslim world, a fact wishfully ignored by many Western analysts over the past two years, hopeful that Western stances in Libya, Egypt and Syria had improved their popular position within the Arab street.

The abhorrent attacks on the US consulate in Libya can indeed be attributed to extremist elements in the region. The flurry of condemnation by Libyans and Muslims on Twitter and social networks is testimony to that. But it is only an extremist manipulation of a widely felt sentiment that continues to perceive the US in particular and the West in general as enemies of the Arab and Islamic world.

Despite Western intervention in Libya to oust Gaddafi, the Arab street remains unconvinced that the US is genuinely interested in the democratic, social and economic development of the Arab world. Such misgivings are justified, given nearly half a century of US support for autocratic regimes, continued backing of Gulf monarchies, a hypocritical stance towards the Bahraini uprising, and — the core of all Arab discontent — its unwavering support for Israel.

Anti-Americanism, thus, remains a constant in the Arab world, and easily exploited by extremist groups when necessary, as the latest fiery protests concerning the anti-Muslim US film show. The mood of the Arab street was previously of little concern to the US, which relied on local dictators to suppress all sign of discontent. The Arab Spring has changed those rules, and US policy in the Middle East has failed to grasp the rapid changes taking place in the region. The revolts, lobbed with increasingly assertive positions in the region by global rivals Russia and China, as well as continued Iranian defiance of US interests, show a US Middle East policy sorely lacking cohesion and clarity.

The killing of the US diplomats in Libya is demonstrative of a failing US policy. Washington, with its Western allies, militarily intervened to oust Gaddafi, flooding local militias — some Islamist, some nationalist, and certainly mostly anti-US — with heavy weaponry. What resulted was a breakdown of state authority, with a transitional government friendly to the West simply unable to rein in much more powerful militias that are determined to dictate authority on their own terms.

Extremism is found in every society of the world, even the most democratic, as Norway’s Anders Breivik taught us. But only when there is a breakdown of central authority are extremist elements at their most potent. The United States, with its NATO and Gulf Arab allies, contributed to the breakdown of the Libyan state, and in a repeat of Afghanistan in the 1980s, has armed groups that at the core hold a strong resentment towards the US.

If Washington thought it would receive a stable Libyan ally through the West’s preferred candidate Mahmoud Jibril, the throng of militias that are ready to tear down US diplomatic missions have shown it otherwise.

The Arab street has seen through superficial, opportunistic and selective US support for local revolts that it handpicks based on its own interests. The Libya incident should serve as a wake-up call to US policymakers, Democrat and Republican-alike, that their strategy in the vital Middle East needs a major overhaul.Arab grassroots political culture is emerging, albeit in its infancy and, as it continues to grow, Arab interests will naturally become more independent of US interests in the region. This is a reality the US must acknowledge. US policy in the Middle East is caught between historic approaches of blindly supporting Arab autocrats and Israel, and a new dawn where it is no longer the primary actor.

Regional powers are filling the vacuum left by a decline in US power, and their contest for this space threatens a catastrophic regional war. Iran, Israel, Turkey and the Gulf states led by Saudi Arabia and Qatar are battling in every corner of the region for a greater slice of the pie, driven by logic of mutual fear rather than co-operation.

Amid the geopolitical contest are the rising Arab masses that want a say in shaping the new Middle East. As the region evolves, the US can no longer afford to put itself at the heart of its dilemmas and must reassess its interests in the region.

Uninterrupted energy supplies are at the core of US interests in the region, but so is realigning its position to genuinely engage the Arab street. If the US desires a warm relationship with the Arab world in the long-term, it must abandon its futile support for autocratic regimes, and truly back democratic movements and Arab self-determination across the board.

A readjustment of US Middle East policy also involves Palestine. The US cannot continue to pin itself to Israeli security, for it is a false assumption that Israeli security needs are parallel with American security concerns in the region. The Obama administration’s strong and public reluctance to back an Israeli war on Iran is evident of the need for Washington to distance itself from its close ally. Israel’s deeply felt paranoia over Iran is not shared in Washington,  and the Obama administration — to the frustration of the Israeli government — rightly sees no interest in sending US forces to another devastating Middle East war where the cost for the US far outweighs the benefits.

The Middle East is undergoing a rapid transition that will determine its shape for the next century. The US no longer enjoys primacy in the region, but can still play an active role and meet its interests. While condemnation of the Libya attacks has flowed worldwide, it should not shroud the US from focusing on its main challenge in the region: the Arab street. The Arab masses can no longer be ignored, and autocrats can no longer be relied upon to suppress local discontent. It is rising to become a major stakeholder in regional affairs, and if the US desires a friendly Arab world in the future, it must reshuffle its regional policy and win over the Arab street.

*Antoun Issa is a Beirut-based Australian journalist and the news editor of Lebanon’s Al-Akhbar English