What’s the difference between ‘In a way of her own’ or ‘in her own way’?
I’m going to make an example for clarity.
Brenda is always very funny, although in a way of her own/in her own
What I want to mean is that Brenda is funny in a peculiar way, so some people might not always find her funny, especially if they don’t have an open mind.
Firstly, the difference between your two suggestions is that one is idiomatic and the other isn’t.
It would be natural to say:
Brenda is funny in her own way.
We just wouldn’t say "in a way of his/her own", as this ngram demonstrates – the phrases just don’t appear in the search.
However, I’m not sure the correct version of this idiom means everything you say you want it to. It essentially means that someone achieves the same end result as others, but their method or ‘way’ of doing it is different or unorthodox.
You said you want it to mean that "some people might not always find her funny". I’d say that was true of almost everyone, unorthodox or not. Not everybody shares the same sense of humour. Also, in your quotation, you say that Brenda is "always very funny". This seems at odds with what you are asking how to say. How can someone always be very funny if some people don’t always find her so? Additionally, saying that someone is "funny in their own way" could also potentially mean that they are eccentric, unusual, and unintentionally funny.
If you are trying to say that Brenda’s humour is unorthodox, and not everybody finds it funny but you do, I would suggest you say something like:
Brenda has an unusual sense of humour, but I always find her very funny.
You might also say her humour is "unusual" or "offbeat".