Even though the sword is closest to 安奴綠洲, you cannot enter the valley where the sword is from the side of the Oasis.
To start from Heroes' Audience, follow the purple path on the map. In the sand there will be 沙元素s and 牛頭怪 and in the Oasis there will be 羅沙魯 groups. You will find some "Ancient Remains" (hold [Alt]) and in them there will be the 家族儀式之劍.
An easier way is to go from 占卜之石 (the green path). Northwest from Augury is a teleporter. Go just East of the teleporter and you gain access to the canyon where the remains will be.