Using Acrobat Pro., the easiest way to determine what is taking up so much space (and possibly change it) is to open the PDF Optimizer under File > Save As Other > Optimized PDF. At the top right of the PDF Optimizer dialog click on the Audit space usage button.
Then, in the Audit Space Usage dialog, you can see what types of elements might be taking up so much space. Like in this example, the images and fonts take up the most space. If necessary, you can use the PDF Optimizer to Downsample the image sizes.