Which one should be used?
To me, it makes no difference, but I’m not really sure why.
For me, it makes no difference, but I’m not really sure why.
I think the general (most widely-applicable) one would be “to me”. To my knowledge, it can be used in any of these cases where “for me” is used, and in some where “for me” can’t be used.
For me, this is not a difficult problem.
To me, this is not a difficult problem.
Both of the above are fine.
But look at the following pair (the “*” sentence is ungrammatical in this context):
*For me, he is an idiot.
To me, he is an idiot.
The first example doesn’t work as a way to express opinion, but “to me” still works fine.