You are in an ideal place to grow them contrary to most of us:
Climate
Olive trees are a Mediterranean native, and require a fairly balmy climate. In particular, they will not tolerate cold winters; if you experience temperatures below 15° Fahrenheit (-7° Celsius), plan on planting something else!
Olives are shallow-rooted, so windy areas are problematic. You won't notice a problem at first, but when a mature tree, carrying a heavy crop, is exposed to high winds, they're extremely vulnerable. If you're in a windy area, plan on planting something else!
[NOTE: Thomas Jefferson learned the hard way that olives also won't make it in areas that are humid. This rules out most of the US, with the exception of California, Arizona, and Texas. If you're interested in growing olives in Texas, please check with the Texas Olive Oil Council for useful information.]