Currently, wildfire spread modeling has drawn a lot of attention from the research community since many countries are suffering from severe socioeconomic impacts of wildfires, every year. Fire spread modeling is a key requirement for effective fire management to deploy fire control equipment and forces at the right time and locations, and plan timely evacuations of residential areas. This paper proposes a new data-driven model for fire expansion which uses reference-based image segmentation for vegetation density estimation and incorporates it into the fire heat conduction modeling. Compared with the conventional parameter collection methods at fire scenes, our method relies on topview images taken by unmanned aerial vehicles, which provides significant advantages of flexibility, safety, low cost, and convenience. Our low-complexity and probabilistic model incorporates the terrain slope, vegetation density, and wind factors with adjustable model parameters which can be easily learned from experiments. The proposed model is flexible and applicable to forests with mixed vegetation and different geographical and climate conditions. We evaluate the fire propagation model by comparing the results with the propagation data available for California Rim fire in 2013.