Looking at the proportions you are using...2 eggs to 1/2 cup whey/1/2 cup almond/1/2 cup flax and 1/2 tsp baking powder

I use...
2 TBS flax/2 TBS almond/2 TBS whey with 1 egg and 1/4 Tsp baking powder

Sounds like not enough egg and not enough baking powder.

I cook mine in the micro for about 2 minutes--comes out very moist and spongey--you may want to try that also instead of the"oven"--the oven works on the principle of dry heat and may be drying the mixture out. Just a thought

Lisa