Unless a player is using catapults on you, then they cannot do any damage. The game rewards the players who are VERY active, and the people who are casual do indeed get left behind. I played many lowbie accounts and brought them up in the ranks from being active and knowing what to do. For the most part, they start you away from any players with catapults, so unless you get super unfortunate and start with a player that has 2-3 more villages than yourself, you should be fine. The game itself is pretty fun, and is very balanced (it is actual THE best balanced game I have ever played), but it does get lopsided with you have a super nerd (ie, me) that plays too much compared to someone who just wants to play an hour or so a day.
However, having said that, I wouldn't recommend this game to anyone due to the amount of time it takes.