I am writing my PhD thesis, and I want to use the acronym “DOM” referring to “Domain Optimization Method”, as I use a lot that expression throughout the document. At the moment of defining it (i.e. the first time I use it), I write something like: “…solved by using a Domian Optimization Method (DOM)…” My question is if here I should write “Domain”, “Optimization” and “Method” starting with a capital letter.
No, unless all the words making up the initialism are proper nouns. In your case you would have to see if the words in your initialism represent proper nouns. I would advise you to take under advisement the comments under this answer, but for a more definitive answer I suggest looking it up in appropriate literature (i.e. in your university’s library). Wikipedia does have a page on proper nouns, but I don’t think it is very clear.
According to the AMA manual of style:
"Do not capitalize the words from which an acronym or initialism is derived (see, Abbreviations).prostate-specific antigen (PSA) enzyme-linked immunosorbent assay (ELISA) Exception: When the words that form the acronym or initialism are proper names, use capitals as described in […]" 1
By the way, if you are in university, search your library for "style guide", it will probably have one or more books in it on style and this will most certainly be covered, probably under a heading like initialism(s) or (capitalisation).
1 American Medical Association. "Acronyms and Initialisms ." Acronyms and Initialisms – AMA Manual of Style. September 13, 2017. Accessed March 21, 2018. http://www.amamanualofstyle.com/view/10.1093/jama/9780195176339.001.0001/med-9780195176339-div1-113.