Position a rack in the center of the oven and preheat to 350ºButter a 9" round cake pan and set aside.
Make the topping:
In a medium skillet over medium heat, melt the butter. Whisk in the brown sugar and salt, cook until bubbling, then slowly add the brandy (it will flame up if you add it too quickly). Cook, whisking, until the caramel is bubbling thickly, 2-3 minutes. Pour the hot caramel into the greased pan. When the caramel has cooled (it will still be fairly liquid), arrange the plum and nectarine wedges in concentric circles, nestling them close together. Set aside while you...
Make the cake batter:
In the bowl of a stand mixer fitted with the paddle attachment, beat the almond paste and butter until combined, scraping down the bowl and beater as needed. Add the sugar, and beat on medium until light and fluffy, 3 minutes. Add the eggs one at a time, beating until combined after each addition.
Sift the flours, baking powder and salt into a medium bowl, and add to the cake batter, beating until combined. (Since there is no gluten, you needn't worry about over-mixing the batter.) Scrape down the bowl and beater, give the batter a final fold with a spatula to make sure it is homogenous, then dollop the batter over the fruit and spread smooth.
Bake the cake:
Bake until a tester inserted into just the cake comes out clean, about 1 hour. Immediately run a thin knife or small, off set spatula around the edge of the cake to loosen it. Place a large plate upside-down on top of the cake pan, put on your oven mitts, grasp both cake pan and plate firmly, and flip the whole thing over so that the cake is right-side-up. Remove the pan. Spackle any fruit or cake that got left behind back onto the cake. Let the cake cool for at least half an hour. Serve warm or at room temperature.
The cake is best within the first 2 days of being baked.
Notes
Adapted from Everyday Greens.If gluten isn't an issue, feel free to use all-purpose flour in place of the rice and oat flours. If gluten is an issue, be sure to use certified gluten-free flours. Sticky rice flour, sometimes called "sweet" or "glutinous," is available at most well-stocked grocers. I usually use Mochiko brand.This cake would be equally good with peaches, apricots, cherries, or even ripe pears. I used yellow nectarines and elephant heart plums here - you want 1 pound total of fruit that is ripe yet firm.Make sure your almond paste is fresh and soft so that it will incorporate into the butter.You can make this in a 10" oven-proof skillet if you like; the sides will be more rounded and the cake a bit more flat (a 9" skillet will likely be too small and result in escaped caramel, as a friend who made this in a 9" skillet discovered).All ounce measurements are by weight.Nutritional values are based on one of eight servings.