When my pastor preached that he only hangs around other pastors...is this a biblical thing?...more inside.?

just wondering cuz i thought Jesus sat with sinners and Paul confronted Peter for not eating with the gentiles