You are SO right. Now that I think about it, there are a lot of Pastors that close themselves up, so that they don't have to be accountable to their flock. There are chain of commands where, if a man has a problem, he has to speak to the leader of the men's ministry, who will in turn relay the message to an elder, who will speak to the associate pastor, who will bring it up to the pastor during their monthly meeting. And you pray that nothing was lost in translation. This is one of the reasons why I have problems with megachurches. I'm not saying that God can't use megachurches, but its almost impossible for pastors and leaders to be accountable to their flock when there are so many in attendance.
Another thing I hear from pastors, when you do charge them with something is, "Follow me as I follow Christ". How am I to willingly do that if you're (on a regular basis) teaching something that is unbiblical? Apparently you AREN'T following Christ if you can, with confidence, preach something that isn't true. And I'm not suggesting that every pastor does this on purpose. I believe there are some loving pastors, who truly have a heart for God and His people, but they just aren't whole-heartedly submitting themselves to the Word. They have other things going on, they are intimidated by the congregation, they let tradition stand in the way. Whatever it is, whether its done on purpose or by mistake, these pastors will be held responsible. And not just pastors, but any leader. That's why I NEVER seek position or title. It is a lot of responsibility and unless I am totally convinced and confident that I will be able to lead my group of people (be it Sunday School teacher or a mentor) I will reject the position.
Please continue to pray for the pastors. They really need it.