I haven't used MATCHES before, but you could try this:
If FILE_NAME(Object.UploadedPhoto) CONTAINS ' OR FILE_NAME(Object.UploadedPhoto) CONTAINS ',' THEN
I think your issue might be around this: '[,']'
If you want to include a ' in rules you normally put it in instead of ' '