Yes there will most likely be a problem. You might have got lucky, but I doubt it. These engines are very consistent, so there's a small chance you might be OK...
Now...why? you ask?
When the heads are built, the cam caps are installed on the head (but not the cams). Then, they have a large boring machine that hones all of those journals. This operation makes each and every cam cap slightly different and perfectly matched to the journal they are installed on. When they are finished honing the journals, the caps are marked for their positions and removed so that the cam can be installed.