Why is fish oil only harvested from the liver?

My best guess is that is would otherwise be wasted, and fish brains are very small. Hardly worth the effort.