For much of its existence, the kingdom of Sicily was under threat. If relations with the papacy were not universally hostile after 1139, there were certainly those at the Curia who had no love for the new kingdom and its ruler, and this tension remained until the treaty of 1156. The Byzantine Empire, which viewed much of southern Italy as rightfully its territory, remained potentially hostile until the late 1150s, and open conflict broke out in 1147–8, and again in 1155–6. The nobles whom Roger and William I had exiled, and their relations, sought to recover their lost patrimonies, a problem that was only solved with the recall of most of those excluded by Queen Margaret between 1166 and 1169. Above all, the German empire remained almost continuously hostile up to the peace conference at Venice in 1177, and the emperors, and especially Frederick Barbarossa, periodically planned the invasion of the kingdom. Furthermore, the hostility of the emperor, who was also king of northern Italy, complicated relations with the north Italian maritime cities, which when freed from imperial interference were important trading partners for the kingdom. While left to themselves, these would almost certainly have preferred peace and the pursuit of profit, Frederick cajoled and bullied them into taking part in his invasion plans. In these circumstances, the military defence of the kingdom was a priority that its rulers dared not ignore.