When specifying possession, my understanding is that one adds an apostrophe if modifying a plural ending with an ‘s’, or adds apostrophe followed by an s if not. How does one specify possession of one of a group?
For example, when discussing a ball belonging to a friend of mine, should I assume ‘friends’ is plural and use:
That’s one of my friends’ ball.
or should I assume ‘one of my friends’ is singular and use:
That’s one of my friends’s ball.
…Obviously, I could just use “That’s my friend’s ball”, but assume in this case that I don’t want to reword as such (maybe I want to point out in the same sentence that I have multiple friends).
English has several ways to indicate possession.
Adding -‘s or -‘ to the end of a word is just one of these ways. In the rest of this answer, I will call this construction the "-‘s genitive". It is not only used to indicate literal possession.
I said in the preceding paragraph that -‘s or -‘ is added to "the end of a word". This accurately describes the way it is pronounced and written, but it doesn’t describe the syntactic rules about how, why, and where it gets added.
The syntax of the -‘s genitive is fairly complicated to analyze. Semantically, the construction applies to a noun phrase. But the -‘s or -‘ isn’t always suffixed to the word that functions as the head of the noun phrase. In noun phrases that don’t end in the head noun, the -‘s genitive attaches either to the last word of the noun phrase, or to the noun phrase as a whole (linguists have debated between these two formulations of the rule).
There are situations where it sounds unnatural to use this construction rather than an alternative way of indicating possession. There are also situations where different speakers have different realizations of this construction.
The multiple forms of the -‘s genitive: a brief summary
You may already know about the variability between the forms -‘s and -‘ (sometimes but not always related to variation in pronunciation). When to use -‘s vs. -‘ after a noun is the subject of the prior question What is the correct possessive for nouns ending in "‑s"? Basically, the form -‘ is always used at the end of a head noun ending in the plural-noun suffix -(e)s. The form ‘s is always used at the end of a head noun, singular or plural, that doesn’t end in a sibilant sound. There is some variability between the forms -‘s and -‘ at the end of certain singular head nouns (or names) ending in the sound /z/ or /s/.
However, your question is not about a noun, but about a noun phrase "one of my friends". As you say, this noun phrase is singular because the head, one, is singular. But it ends in a plural word friends.
There are noun phrases that don’t sound natural with the ‘s genitive
I would say that it is just not usual to use the -‘s genitive with a singular noun phrase ending in a plural noun that is not the head of the noun phrase. And if it is used, native speakers may disagree about how to realize the genitive suffix in this context. I discuss this a bit more in my answer here: What’s wrong with “One of my children’s name IS John”? So I would recommend using an alternative structure, instead of trying to form "[one of my friends] + -‘s-genitive suffix + ball".
Alternative ways to say what you want to say
In writing, you could indicate that you are talking about one of your many friends by using the form "one of my friends’ balls". This is not built on the noun phrase "[one of my friends]": instead, the syntactic structure is "one of [[my friends’] balls]". Here, the -‘s genitive is applied directly after the plural suffix at the end of the head noun friends, so it is clear that the genitive must take the form -‘. This form implies that more than one of your friends has a ball, or at least that your friends collectively have more than one ball.
In speech, "one of my friends’ balls" sounds identical to "one of [[my friend’s] balls]", which is also valid but has a distinct meaning. This form wouldn’t indicate that you have multiple friends, but that you have (at least) one friend with multiple balls.
If you want something that is unambiguous in speech, or if you don’t want to imply that you have multiple friends with balls, you should use a longer rewording like "That ball belongs to one of my friends".