How far does that freedom to do what you please extend? Am I not actually free because I can't beat up people I disagree with? Personally I feel the freedom to disagree without the threat of violence is more important.

There is freedom to, and there is freedom from. Don't undervalue freedom from.