Place a foil lined sheet pan in the bottom of the oven and arrange the shelves so you have one shelf in the middle of the oven. The foil will protect your oven from any bubbling over of the butter and brown sugar sauce. Heat oven to 350 degrees.
Melt 8 ounces of butter on low heat in a 12″ cast iron skillet. Add brown sugar, 2 t cinnamon, ground ginger and 2 T dark rum. Cook, stirring until sugar is melted and begins to bubble. Remove from heat and arrange plum halves cut-side-down in the bottom of the skillet filled with the sugar and butter mixture. Set aside while you make the batter.
To make the cake batter, finely grind the almonds in a food processor or blender. Process until almonds looks like fine meal. Mix ground almonds, flour, baking powder and salt in a small bowl and set aside.
In a medium bowl, cream the remaining 8 ounces of butter with an electric mixer until pale, light and fluffy, about 3 minutes. Add granulated sugar and continue to cream for another 3 minutes. Beat in eggs, remaining 3 T rum, vanilla and orange zest until combined. Add 1/3 of the flour and almond mixture, followed by 1/3 of the milk, mixing and adding more flour mixture until all the ingredients are incorporated. Do not over mix. Batter will be thick.
Spoon the batter over the plums and butter-sugar-rum sauce to cover the plums. Bake for 50 minutes until a toothpick inserted into the middle comes out clean. The top of the cake will be golden brown. Allow to cool for 30 minutes before inverting onto a serving plate. Serve warm or at room temperature.