Preheat your oven to 350°F and lightly grease a 7x5-inch baking dish.
In a small bowl, whisk together brown and granulated sugars, cinnamon, nutmeg, and salt. Drizzle in melted butter and mix until well combined. Use a spoon to stir in the flour, a little at a time, until flour is completely incorporated. Place in the refrigerator.
In a small bowl, whisk together flour, baking powder, and salt. Set aside.
In a medium bowl, combine butter and granulated sugar. Cream them together until light and fluffy, about 1 minute. Add egg and vanilla and beat until well-combined. Add sour cream and mix well. Add half of the flour mixture and stir until just mixed. Add 1 tablespoon of milk and stir until just mixed. Repeat with the remaining flour mixture and milk.
Pour batter into your prepared dish and drop crumb mixture over the top in pea-size pieces. It will seem like a lot, but don't worry! It's just the right amount.
Bake for 28 to 34 minutes, until a toothpick inserted into the center of the cake comes out with just a few crumbs. Differences in ovens, baking dishes, and environments may affect baking time, so don't be worried if your cake takes slightly longer to bake. Cool for at least 15 minutes before cutting, and enjoy!