I am only familiar with "faggot" being used as a slur for a gay man or as a bundle of sticks.
Some statements in the preceding paragraphs that provide context:
…A predawn hush had come over the desert basin. Straight overhead, the stars were a sequin shawl flung over blue-black…
…As the Duke watched, the moon dipped beneath the Shield Wall cliffs, frosting them, and in the sudden intensity of darkness…
And the passage itself:
To the east, the night grew a faggot of luminous gray, then seashell opalescensce that dimmed the stars.
The preceding paragraphs suggest to me that "faggot" is describing a color. As an allusion to a burning bundle of sticks I would have expected him to say orange or red. Instead he says "luminous gray".
This book was written 50 years ago so I’m wondering if it had another meaning back then.
I’ve researched this a bit more and I didn’t find any definitive answers.
The most common response people have to this question when asked on other sites is that “faggot” refers to a bundle of burning sticks (e.g. this Quora post).
There is a reference to red farther down the page:
He had never imagined anything here could be as beautiful as that shattered red horizon and the purple and ochre cliffs.
Wikipedia talks about the red/blue characteristics of opalescence:
The material appears yellowish-red in transmitted light and blue in the scattered light perpendicular to the transmitted light.
So I think the passage is talking about a sunrise that starts red (like the yellowish red/orange of opalescence) to the soft orange light of a bundle of burning sticks, and finally to the gray of the Arrakis sky.