Is this sentence correct? It sounds weird:
The accuracy is sub-millimeter.
Is it OK to use “submillimeter” without the “-“?
Yes, it is indeed an adjective.
This is the definition given by Dictionary.com:
less than a millimeter in size
(Spelt “submillimetre” in Australia and the UK)
Judging by a quick Google search, “submillimeter” should not be spelt with a hyphen.