I suspect if you try and claim it back from a travel agent they will argue that it is their service fee for making your application. I would assume they have used the official site to apply rather than a fraudulent one. It may be that if you could prove they didnt tell you that you could apply yourself and save the charge that you would get it back.
At least if you have applied through a travel agent i would doubt any of the worries about fraudsters having passport details would apply.