Data infrastructure for market intelligence