Depends.
For an intercom - sure. For critical listening or scientific work, no.
The ultimate goal of audio electronics is to produce sound into the human ear. Faithful reproduction involves both amplitudes and phases, and the biggest phase of all is the direction of the speaker cone. The microphone sees positive (compression) and negative (rarefaction) instantaneous air pressure during each half-cycle of whatever the sound is, and the speaker should mimic this. That is, at a point in the audio when the microphone diaphragm is being pushed inward by positive air pressure, the speaker cone should be moving outward to create positive air pressure. By convention, positive air pressure against the diaphragm is represented by the positive half-cycles of the audio signal, and positive half-cycles push the speaker cone outward. This is why amplifier outputs and speaker terminals are marked with some kind of polarity indication - it matters.
The air-to-microphone-to-signal-to-speaker-to-air relationship can be maintained through the entire signal chain from the microphone in a recording studio to wireless ear buds while jogging by having either zero or an even number of inverting stages. Of course no one can control everything, but each stage along the path can be controlled to be non-inverting between its ins and outs.
So it comes down to quality versus effort (and cost). How much of a "golden-eared wonder" are you (or your customers), what is the application, etc. - ?
ak
BTW, I use the term "golden-eared" only semi-sarcastically. Thanks to a couple of my past lives, I'm pretty good; but I know some people who are outright scary when it comes to audible perception. An easy example is the 1812 Overture ("The Year 1812 Solemn Overture" by Tchaikovsky). There are actual cannon shots in the musical score. The initial wave front from an explosion is compressive, and as such is quite directional. If the signal to the speaker is inverted WRT the microphone, the speaker moves inward rather than outward, creating a "blast" of low pressure - a micro-vacuum. Thing is, evacuation is not nearly as directional as compression. In a controlled listening test, a random dude off the street might not be able to say why "identical" signals A and B sound differently, but over 90% will be able to hear a difference.