LOL, a loaded question. :) I started with 200 plants and eventually planted 1,000 - 1,200, I can't remember exactly. That makes me the midwest's largest saffron producer ;)
If you know you like saffron there's never too much. I've used it in breads, desserts and savory dishes. When it's no longer $20 for 0.1g I use it a lot more.
If you don't know if you like it, try some first. You have to be careful to add a little, try it and add more if needed. Too much is awful. You have to get threads, not powdered. The powdered stuff is often less than half saffron. Try to get saffron threads that have a harvest date on them. Otherwise you could easily be getting 6 year old stuff. Saffron will last a long time (years if stored properly) but it does not age like wine :)
Then you have to decide if you want a lot right away or if you want to start slow and gradually get enough over time. Also you have to decide how much space you want to devote and how many 7" deep by 1-2" wide holes you want to make.
I started with 200 and I wish I'd gone for the full 1,000 right away. 200 is one of the shipping sweet spots where you get the most corms per shipping dollar. A lot of people start with 20 but everyone who started with fewer than 50 has told me they were sorry they didn't buy more. So I would say, based on prior years' feedback that 50 is the minimum to avoid regrets. Fewer than 20 and you waste money on shipping but you certainly can do that if you want.
I hope this is helpful. Feel free to ask anything else as I'm sure anything you want to know will help others, too.