How many children did Mary and Joseph raise?

Another question from Face Book

This is one of those points where Protestants and Catholics have different views.

The three oldest Christian churches (Catholic, Eastern Orthodox and Oriental Orthodox) reject the view that Jesus had any siblings. These churches believe that Mary remained a virgin after the birth of Jesus, even though she was married to Joseph. They believe that it would not have been seemly for any human to be born from the same womb as Christ.

The Protestant churches refer to Chapter 13 of Matthew where it says,

54 Jesus went to his hometown and taught the people in the synagogue in a way that amazed them. People were asking, "Where did this man get this wisdom and the power to do these miracles?
55 Isn't this the carpenter's son? Isn't his mother's name Mary? Aren't his brothers' names James, Joseph, Simon, and Judas?
56 And aren't all his sisters here with us? Where, then, did this man get all this?"

Matthew 13:54-56 GWT

The Bible indicates Jesus had 4 brothers: James, Joseph, Simon, and Judas. It also mentions sisters, but we don't have any indication of how many sisters.