Martin Bryant provides a few ways Instagram could have grown, but most of them sound pretty boring: in-app purchases, subscriptions, advertising, brand partnerships and data mining, it could also have gone into administration that happens sometimes to companies, click here to learn more about this.
I look at 500px and why it is so popular, and wonder if Instagram could have done that for mobile. Would people want to pay for the photographs people took on Instagram? Could the mobile photo marketplace exploded? Would people find art in mobile photography? Instagram believed in many of these questions, and perhaps there could have been more exploration in the space.