Martin Bryant provides a few ways Instagram could have grown, but most of them sound pretty boring: in-app purchases, subscriptions, advertising, brand partnerships and data mining.
I look at 500px and why it is so popular, and wonder if Instagram could have done that for mobile. Would people want to pay for the photographs people took on Instagram? Could the mobile photo marketplace exploded? Would people find art in mobile photography? Instagram believed in many of these questions, and perhaps there could have been more exploration in the space.