Yes. This is an impressive part of the counterintuitive nature of orbital mechanics.

Imagine a highly elliptical orbit (mathematically, this is a conic section), such as a comet. It travels fastest when it is nearest the sun, at perihelion (generic term is “periapsis”), and slowest when it is farthest from the sun, at aphelion (apoapsis). In order to change that orbit from elliptical to parabolic or hyperbolic* a delta v is necessary. If that delta v is applied at aphelion, away from the sun, then the delta v will first have to be enough to create a circular orbit around the sun, and that is more delta v, then create an elliptical orbit in which apohelion becomes perihelion, then create a parabolic or hyperbolic trajectory. The slow speed at apohelion means that the delta v required is not great, but it is still more than needed at perihelion.

However, since the velocity at perihelion is already so great, the orbit is already far beyond being circular, it takes much less delta v to open up the orbit (conic section) to a parabola or hyperbola.

In orbital mechanics, there are two kinds of energy that are tracked most: kinetic energy, the energy of motion, and potential energy, the energy of height above a gravity source. Combining both gives a total energy that remains the same throughout the orbit. Kinetic energy is greatest and potential energy is least at periapsis, and the other way around at apoapsis.

Confusion about the energy requirements may abound, because the propellant (chemical energy) expended by the spacecraft is the same for the same delta v at either apsis. However, the wikipedia article I linked, above, helps to explain this:

At very high speeds the mechanical power imparted to the rocket can exceed the total power liberated in the combustion of the propellant; this may also seem to violate conservation of energy. But the propellants in a fast moving rocket carry energy not only chemically but also in their own kinetic energy, which at speeds above a few km/s exceed the chemical component. When these propellants are burned, some of this kinetic energy is transferred to the rocket along with the chemical energy released by burning.

To help explain the seeming discrepancy of the energy difference, recall that the exhaust velocity of the rocket engine is the same relative to the spacecraft at either apsis. However, the velocity of the exhaust is different relative to the center of attraction, such as the Sun. From the Sun’s point of view (or the universe’s), more energy is added to the spacecraft when its engines fire at perihelion than when they do at aphelion. The energy of the universe is conserved despite the intuited discrepancy.

I hope that helped.

* Both are escape trajectories and will never return to the sun, but parabolic is only just barely enough, theoretically coming to a stop at infinity, while hyperbolic is faster than the minimum needed, never stopping, even at infinity.

Even Aristoteles thought that heavier objects fall faster than light ones. And that stuck for a couple of thousand years. It is funny how counter intuitive it is how things move, while we are intuitive masters at throwing a ball. Even dogs don’t get it but run towards where the ball is at the moment.

“When travelling at a low speed, an increase in velocity adds a small amount of kinetic energy. Yet if you’re travelling at a high speed, the same amount of velocity increase will add many more times the energy.”

Is it really more efficient to push the swing of a child when it is vertical and has the highest speed, than when it stands still while turning around?

That would be wild indeed, since the best time to apply thrusters is at perihelion in order to take advantage of the Oberth effect.
https://en.wikipedia.org/wiki/Oberth_effect