tbench regression on each kernel release from 2.6.22 -> 2.6.28

[Christoph Lameter]

It seems that the network stack becomes slower over time? Here is a list of
tbench results with various kernel versions:

2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03

And linux-next is:

2.6.28(l-next) 2568.74

It shows that there is still have work to be done on linux-next. Too close to
upstream in performance.

Note the KT event between 2.6.24 and 2.6.25. Why is that?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Kok, Auke]

Post by Christoph Lameter
It seems that the network stack becomes slower over time? Here is a list of
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to
upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?

is this with SLAB or with SLUB? SLUB has been known to impact network performance...

Auke


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Christoph Lameter]

Post by Kok, Auke
is this with SLAB or with SLUB? SLUB has been known to impact network performance...

The original testing config was SLAB based so it was used throughout.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[David Miller]

From: Christoph Lameter <***@linux-foundation.org>
Date: Mon, 11 Aug 2008 13:36:38 -0500

Post by Christoph Lameter
It seems that the network stack becomes slower over time? Here is a list of
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to
upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?

Isn't that when some major scheduler changes went in? I'm not blaming
the scheduler, but rather I'm making the point that there are other
subsystems in the kernel that the networking interacts with that
influences performance at such a low level. This includes the memory
allocator :-)
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Christoph Lameter]

Post by David Miller
Isn't that when some major scheduler changes went in? I'm not blaming
the scheduler, but rather I'm making the point that there are other
subsystems in the kernel that the networking interacts with that
influences performance at such a low level. This includes the memory
allocator :-)

Right this covers a significant portion of the kernel. SLAB was used since .22
was pretty early for SLUB. And around 2.6.24 we had the merges of the antifrag
logic.

.25 was the point where HR timers came in. By switching off hrtimers I can get
some (minor) portion of performance back. There must be more things in play
though.

Maybe what we are seeing is general bloat in kernel execution paths due to the
growth in complexity?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[David Miller]

From: Christoph Lameter <***@linux-foundation.org>
Date: Mon, 11 Aug 2008 16:33:43 -0500

Post by Christoph Lameter
Maybe what we are seeing is general bloat in kernel execution paths
due to the growth in complexity?

It could be, and any kind of analysis into this would be great.

I had a change that RCU destroyed sockets and this added a
tiny bit of latency, so I never added it even though it would
have allowed a lot of simplification of socket handling (which
I though would make up for RCU's latency, but it didn't).

[Kok, Auke]

Post by David Miller
Date: Mon, 11 Aug 2008 16:33:43 -0500

Post by Christoph Lameter
Maybe what we are seeing is general bloat in kernel execution paths
due to the growth in complexity?

It could be, and any kind of analysis into this would be great.

perhaps Rick Jones who maintains netperf could enlighten us on some historic
numbers? he usually seems to be happy to prop up new netperf numbers :)


Auke
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Rick Jones]

Post by Kok, Auke

Post by David Miller
Date: Mon, 11 Aug 2008 16:33:43 -0500

Post by Christoph Lameter
Maybe what we are seeing is general bloat in kernel execution paths
due to the growth in complexity?

It could be, and any kind of analysis into this would be great.

perhaps Rick Jones who maintains netperf could enlighten us on some historic
numbers? he usually seems to be happy to prop up new netperf numbers :)

While this is an excellent opening to talk about how netperf
top-of-trunk can now emit keyword=value results easier (ostensibly) to
put into a database then the regular or even CSV output formats, I
cannot fully exploit it by pointing at a database of results :(

rick jones
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Andi Kleen]

Post by Christoph Lameter
Maybe what we are seeing is general bloat in kernel execution paths
due to the growth in complexity?

Wouldn't surprise me. Have you considered doing profiles?

e.g. just oprofiling the benchmark on the different kernels and see
if there's some obvious difference in the CPU consumers?

-Andi
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Christoph Lameter]

Post by Andi Kleen

Post by Christoph Lameter
Maybe what we are seeing is general bloat in kernel execution paths
due to the growth in complexity?

Wouldn't surprise me. Have you considered doing profiles?

If I get the time I will try to do that.

Another way to understand why we are accepting the regressions here may be
that we give more consideration to real time issues and deterministic
performance these days. Hardware speed gains compensate for the additional
bloat? (I ran the old kernels on cutting edge hardware after all).
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Ilpo Järvinen]

Post by David Miller
Date: Mon, 11 Aug 2008 13:36:38 -0500

Post by Christoph Lameter
It seems that the network stack becomes slower over time? Here is a list of
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to
upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?

Isn't that when some major scheduler changes went in? I'm not blaming
the scheduler, but rather I'm making the point that there are other
subsystems in the kernel that the networking interacts with that
influences performance at such a low level.

...IIRC, somebody in the past did even bisect his (probably netperf)
2.6.24-25 regression to some scheduler change (obviously it might or might
not be related to this case of yours)...

[Zhang, Yanmin]

On Tue, 2008-08-12 at 11:13 +0300, Ilpo J=E4rvinen wrote:=20

=20

Post by David Miller
Date: Mon, 11 Aug 2008 13:36:38 -0500
=20

It seems that the network stack becomes slower over time? Here is=

a list of

Post by David Miller

=20
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03
=20
=20
2.6.28(l-next) 2568.74
=20
It shows that there is still have work to be done on linux-next. =

Too close to

Post by David Miller

upstream in performance.
=20
Note the KT event between 2.6.24 and 2.6.25. Why is that?

=20
Isn't that when some major scheduler changes went in? I'm not blam=

ing

Post by David Miller
the scheduler, but rather I'm making the point that there are other
subsystems in the kernel that the networking interacts with that
influences performance at such a low level.

=20
...IIRC, somebody in the past did even bisect his (probably netperf)=20
2.6.24-25 regression to some scheduler change (obviously it might or =

might=20

not be related to this case of yours)...

I did find much regression with netperf TCP-RR-1/UDP-RR-1/UDP-RR-512. I=
start
1 serve and 1 client while binding them to a different logical processo=
r in
different physical cpu.

Comparing with 2.6.22, the regression of TCP-RR-1 on 16-core tigerton i=
s:
2.6.23 6%
2.6.24 6%
2.6.25 9.7%
2.6.26 14.5%
2.6.27-rc1 22%

Other regressions on other machines are similar.

yanmin


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Ilpo Järvinen]

Post by Ilpo Järvinen

Post by David Miller
Date: Mon, 11 Aug 2008 13:36:38 -0500

Post by Christoph Lameter
It seems that the network stack becomes slower over time? Here is a list of
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to
upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?

Isn't that when some major scheduler changes went in? I'm not blaming
the scheduler, but rather I'm making the point that there are other
subsystems in the kernel that the networking interacts with that
influences performance at such a low level.

...IIRC, somebody in the past did even bisect his (probably netperf)
2.6.24-25 regression to some scheduler change (obviously it might or might
not be related to this case of yours)...

I did find much regression with netperf TCP-RR-1/UDP-RR-1/UDP-RR-512. I start
1 serve and 1 client while binding them to a different logical processor in
different physical cpu.
2.6.23 6%
2.6.24 6%
2.6.25 9.7%
2.6.26 14.5%
2.6.27-rc1 22%
Other regressions on other machines are similar.

I btw reorganized tcp_sock for 2.6.26, it shouldn't cause this but it's
not always obvious what even a small change in field ordering does for
performance (it's b79eeeb9e48457579cb742cd02e162fcd673c4a3 in case you
want to check that).

Also, there was this 83f36f3f35f4f83fa346bfff58a5deabc78370e5 fix to
current -rcs but I guess it might not be that significant in your case
(but I don't know well enough :-)).

[Zhang, Yanmin]

=20

Post by Zhang, Yanmin
=20
On Tue, 2008-08-12 at 11:13 +0300, Ilpo J=E4rvinen wrote:=20

=20

Post by David Miller
Date: Mon, 11 Aug 2008 13:36:38 -0500
=20

It seems that the network stack becomes slower over time? Her=

e is a list of

Post by Zhang, Yanmin

Post by David Miller

=20
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03
=20
=20
2.6.28(l-next) 2568.74
=20
It shows that there is still have work to be done on linux-ne=

xt. Too close to

Post by Zhang, Yanmin

Post by David Miller

upstream in performance.
=20
Note the KT event between 2.6.24 and 2.6.25. Why is that?

=20
Isn't that when some major scheduler changes went in? I'm not =

blaming

Post by Zhang, Yanmin

Post by David Miller
the scheduler, but rather I'm making the point that there are o=

ther

Post by Zhang, Yanmin

Post by David Miller
subsystems in the kernel that the networking interacts with tha=

t

Post by Zhang, Yanmin

Post by David Miller
influences performance at such a low level.

=20
...IIRC, somebody in the past did even bisect his (probably netpe=

rf)=20

Post by Zhang, Yanmin

2.6.24-25 regression to some scheduler change (obviously it might=

or might=20

Post by Zhang, Yanmin

not be related to this case of yours)...

I did find much regression with netperf TCP-RR-1/UDP-RR-1/UDP-RR-51=

2. I start

Post by Zhang, Yanmin
1 serve and 1 client while binding them to a different logical proc=

essor in

Post by Zhang, Yanmin
different physical cpu.
=20
Comparing with 2.6.22, the regression of TCP-RR-1 on 16-core tigert=
2.6.23 6%
2.6.24 6%
2.6.25 9.7%
2.6.26 14.5%
2.6.27-rc1 22%
=20
Other regressions on other machines are similar.

=20
I btw reorganized tcp_sock for 2.6.26, it shouldn't cause this but it=

's=20

not always obvious what even a small change in field ordering does fo=

r=20

performance (it's b79eeeb9e48457579cb742cd02e162fcd673c4a3 in case yo=

u=20

want to check that).
=20
Also, there was this 83f36f3f35f4f83fa346bfff58a5deabc78370e5 fix to=20
current -rcs but I guess it might not be that significant in your cas=

e=20

(but I don't know well enough :-)).

I reverted the patch against 2.6.27-rc1 and did a quick testing with ne=
tperf TCP-RR-1
and didn't find improvement. So your patch is good.
Mostly, I suspect process scheduler causes the regression. It seems whe=
n there are=20
only 1 or 2 tasks running on the cpu, the performance isn't good. My ne=
tperf testing
is just one example.


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Christoph Lameter]

Post by Zhang, Yanmin
Other regressions on other machines are similar.

There are AIM7 regressions that are similar to tbench.

2.6.22 28436
2.6.26 23064
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Ray Lee]

On Mon, Aug 18, 2008 at 7:07 AM, Christoph Lameter

Post by Christoph Lameter

Post by Zhang, Yanmin
Other regressions on other machines are similar.

There are AIM7 regressions that are similar to tbench.
2.6.22 28436
2.6.26 23064

Just a shot in the dark -- is this with Group Scheduling on or off?
Off is prefered for benchmarks.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Christoph Lameter]

Post by Ray Lee
On Mon, Aug 18, 2008 at 7:07 AM, Christoph Lameter

Post by Christoph Lameter

Post by Zhang, Yanmin
Other regressions on other machines are similar.

There are AIM7 regressions that are similar to tbench.
2.6.22 28436
2.6.26 23064

Just a shot in the dark -- is this with Group Scheduling on or off?
Off is prefered for benchmarks.

Off
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Zhang, Yanmin]

Post by Christoph Lameter

Post by Zhang, Yanmin
Other regressions on other machines are similar.

There are AIM7 regressions that are similar to tbench.
2.6.22 28436
2.6.26 23064

Mostly, AIM7 has about 4~5% regression on my machines. As AIM7 result is stable,
so 4% is big.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Zhang, Yanmin]

Post by Christoph Lameter
It seems that the network stack becomes slower over time? Here is a list of
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03

What's the hardware configuration? Is it dual-core?

I also track tbench performance with lastest kernels on a couple of quad-core machines,
and didn't find such regression while the results did have fluctuation.

What's the commandline you is using to start tbench? I start tbench with CPU_NUM*2.

BTW, I enabled CONFIG_SLUB since 2.6.22.

Post by Christoph Lameter
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to
upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html