Nancy Pelosi says those who do not share the Democrats views will be “collateral damage”