Castile soap is made from Olive Oil. Though it is great for the babies tender skin, it can coat the cloth fibres and the wipes. This may cause the diaper insides to be sticky and may attract dust. Eventually the wipes will be less absorbent too.

In general a small amount may be fine of course. It could be an issue down the line. But if the amount is minuscule it should be fine.