If you use a magic cloaking field like for a klingon Bird of Prey, then it doesn't need to have many downsides.
There you only know where
something is as soon as empty space starts firing at you.
Games features
live from limitations, though. I'm not sure how believable the feature in F-177A was because naturally it's top secret but they had an IR "cloaking" system where the engine exhaust was rerouted to heat up internal mass and then dispersed over a wider area to hide from heatseeker missiles. This was limited by the need to eventually cool off said mass.
ECM worked similarly.
An aircraft can't carry the mass to do brute force ECM so they try to alter the
pattern of the RADAR return / emissions to confuse whoever and whatever is looking for the aircraft.
If some seeker figures out the pattern, your ECM just got a lot less effective.
Space ships will always have a heat signature. Space may be "cold" but it sucks at
cooling anything. The only way to get rid of excess heat is through radiation and that only gets effective at high temperatures so the heat sinks on a very active ship would
literally glow and have a monstrous IR signature - or even enter the visible spectrum.
IIRC, Star Citizen or the new ELITE use that bit of hard SF as a feature.
A ship could approach something while only using it's heat sinks on the other side and so be invisible - but that's directional invisibility and useless for disappearing in the middle of some ships.
RADAR absorbing layers aren't armour, either. Getting hit would quickly ruin your stealth paint job. It's also mass that you carry and would come at the cost of something else, like actual armour.
Much like the F-117A, it could be possible to build a stealth fighter. That should seriously degrade it's combat capabilities, though.
Just like stealth aircraft nowadays, it would be limited to
internal missile launchers because external mounts generate way too much RADAR return.
There also is no (passive) GPS in space. If your target emits a strong signature you're in luck but navigating a black asteroid field in black space could be quite nerve wracking.
X3 has that in one or two super-dark sectors where you have to be reeeeeally careful in asteroid fields.
If your sensors "bracket" the rocks that's no problem but that would require to "go active" so no more stealth.
There is no "I" in Tea. That would be gross
.