Property/holiday home is a medium to long term investment. It should be looked at as holiday or a retirement home, if this was ultimately intended.
Speculation, wether property or any other commodity has its down side its just the nature of the beast.
If you do not wish to get caught out. Have plan B & C, and keep your borrowing to a managable level so that you can sleep at night.
(The above does not take into account illegal/criminal activities by all in sundry along the chain )