Overkill usually refers to damaging a unit by far more than the target's actual hit points. It is a waste of a weapon's capabilities, if the weapon could instead attack another target that has much higher hit points. Because damage per second is usually balanced between different factions' equivalent units, the more damaging shots (hence more likely to cause overkill) tend to come from weapons with longer reload times.
For example, one bomb from an Aeon T3 Strategic Bomber does 3450 damage. It is overkill to use this to destroy an enemy Tech 1 mass fabricator, which has only 360 hit points. It is more cost effective to build and use an Aeon T1 Attack Bomber, which does 200 damage and will destroy the target in 2 passes, while using the remaining balances of mass and energy to build units which can attack other targets.
Another type of overkill is to focus too many units on attacking too few targets (or one) . In this scenario, many extra shots may be fired before the killing blow lands, wasting the extra shots (which, as above, could have been fired at units with more health). In this scenario, slow muzzle velocities and long ranges (long flight times) can make overkill more likely, by increasing the chance that units will continue to fire after the killing shot is already en route.
An example of what is not overkill is using a nuke on a large army of ground units. Although a nuke does 70,000 damage while the toughest non-experimental, non-command unit is the Percival with 9,300 hp (at the very least an overkill of 60,700 damage), the nuke's area of effect and inability to be defended against by anything in the field is something that cannot be duplicated by any other attack and the strategic gain is considerable.