I placed by barricade next to the invasion portal (the barricade passed through the portal effects), and for the rest of the game we could not use the portal at all. Not sure if the barricade was the reason for this, but it seems likely.