Due the fact that the infallibility of the three is crystal clear...
Due the fact that the infallibility of the three is crystal clear, and Muslims all over the world believe the infallibility of the Holy Prophet and the fact that he is immune from any mistake whatsoever, in form of receiving, protecting it and propagating its message. Therefore, if it is clear that God's book has such pure and firm infallibility, the prophet's household is also immune from any mistake or error.
This is so because in this narration, the prophet's household were mentioned as the those who are accompanying the Holy Quran in guiding and leading the nation, and due to this 'accompanying', both tend to have the same infallibility. In other words, it doesn't make sense to consider fallible individuals as being the ones accompanying the holy Quran that is in the purest form.
The most evident witness to the infallibility of the imams (p) is mentioned by the prophet (p) himself that: "They will not separate in (guidance and leadership) till they join me at the pool (of Kawthar)." If it was so that the prophet's household are not immune from mistakes and in some instances might have been liable to make them, then they have already separated from the Quran which is free from all mistakes, and they have gone astray, whilst the holy prophet himself strongly emphasized to their infallibility, by negating them from any mistake whatsoever.
Of course, what the holy prophet meant by his household were not all his relatives, because all of them were not free from mistakes. It is only a special group of his family (etrat) who possessed this honor, and this position applies to only a few of them. These are the ahl-bait who all along history, were the illuminators of the path of the nation, the protectors of the prophet's tradition and the guardians of the Islamic laws 'Sharia'. Mustadrak Al-Hakim, 3rd volume, p.148.
Al-Sawaeq Al-Muhregha, sect. 11, chapt.1, p. 149. Similar to this context in 'Kanz Al-Amal', 1st volume, section of Al-etesam bil Kitab wal sunnah' p.44 and in Masnad Ahmad 5th volume, p. 182-189 among others. Question 7 : Why do you recite "Ash hadu anna Aliyyan waliyyu Allah" in the call of prayer, thereby witnessing the guardianship of Ali (p)?