Benchmarking Single Board Computers

We run several SBCs through their paces and compare their performances.

Favorited Favorite 1

Benchmarking is a favorite pastime among many computer enthusiasts. Many tests exist that evaluate various performance aspects of computers, and with the rise in popularity of single board computers (SBCs), we would like to have a method of comparing them to each other.

Raspberry Pi benchmark

We want to test the performance of several SBCs

Testing SBCs is nothing new. However, we wanted to create a set of (somewhat) standardized tests that we can apply to new SparkFun SBCs. We just want to get a comparison of relative performance; our tests do not have to be super accurate, but should show how SBCs perform against each other in the same aspects.

The Phoronix Test Suite provides a useful set of tests, and we chose only a few to construct our array of tests. Additionally, since power is a common concern among SBC users, we measured average current draw for idle and loaded CPU states. If you want to see the full report, complete with test installation instructions and results, click the link:

Results

If you are still here, then we can happily give you a summary of that report. We tested the following single board computers:

Each of the tests are briefly described in their respective sections below.

CPU (Single Core)

Times how long it takes to convert a .WAV audio file to a FLAC file. The raw test results can be found here.

enclode-FLAC results

CPU (Multi-Core)

Computes floating point operations per second (in MFLOPS) by timing how long it takes to solve a pressure Poisson equation. The raw test results can be found here.

Himeno results

RAM

Tests aggregate RAM read and write speed. The raw test results can be found here.

RAMspeed results

Flash Memory

Transfers files to and from flash memory to compute the read and write rates. The raw test results can be found here and here.

Note that the BeagleBone Black Rev C test was slightly different, as we had to use a 512 MB test file (instead of the 2 GB file used for the other tests). There was simply not enough space on the BeagleBone Black's onboard eMMC.

IOzone results

Graphics (2D)

Times how long it takes to draw a set number of filled circles (many thousands) in a window. The raw test results can be found here.

Note that we left out the Edison as it does not have a video out port and therefore no default GUI.

GtkPerf results

Power

Compute average power consumption based on measured current draw (averaged over 10 seconds) and measured DC voltage. We created a current logging device to measure the instantaneous current draw. The raw test results can be found here.

Power results

Conclusion

So, you might ask, which single board computer is the best? Well, that really depends on how you plan to use it. If you want a lightweight server that consumes little power, low power might be your biggest concern (an Edison could be a good choice). Alternatively, if you want a tiny home theater PC, graphics is important to note (in which case, a pcDuino3 or Raspberry Pi 2 would be good options).

Personally, I am still a big fan of the Edison, especially when you consider performance per Watt (in CPU, RAM, and flash memory read/write). If I need some kind of video output, well, that's another story...

We plan to update the SBC Benchmarks tutorial once in a while with a new set of computers (hence the idea of "rounds"). If you are curious about how SBCs perform, definitely keep an eye on that page.

What SBC is your favorite to use right now (if you use them at all)? Are there other tests that you would like to see performed?


Comments 18 comments

  • Something that I feel is being forgotten is that all of these are running different flavors of Linux with different overheads. As such, the amount of time the processor is dedicating to each benchmark is going to be vastly different.

    For example, the Edison is running Yocto Linux which is a very minimalist version of Linux. In fact, it is so minimalist that it is very difficult to install any packages or other softwares on it - they assume you will build a new version of Yocto with any packages you need when you need them. It has very few background processes and as a result has a much greater amount of CPU time it can dedicate to running your benchmark code.

    Conversely, if we look at the Raspi, it is running a comparatively heavy version of Linux (Raspian in this case). Instead of a super bare-bones experience like Yocto, Raspian is designed to be almost kid-friendly. There is A LOT happening in the background to give you endless tools, super easy USB interfaces (keyboards, mice, wifi/bluetooth dongles, ect), a GUI with two video outputs, external RAM, and endless other things. The end result is that the Raspi has a much greater CPU overhead and much less CPU time it can dedicate to a given test.

    When you compare the two without taking this into consideration, it looks like the Edison is the best thing since sliced bread. But, I’d bet you money if you built a version of Yocto for the Raspi (and the other two boards) and ran your tests again, you’d see something completely different. I very highly doubt 400MHz dual core chip can run around 3.5-4 times faster than a 1.5MHz quad core chip (especially in both single and multicore tests). Intel does some great work, don’t get me wrong, but so do the folks that design the ARM architectures.

    You also have to look at the fact that PHP is an interpreted language and not a compiled one. Different instruction sets and chip architectures are going to define just how well a bit of scripted code will run depending on its ability to do things like pipelining, prefetching instructions, and the like. I would personally love to see some benchmarks done in compiled languages like C/C++ and see how they compare to the PHP benchmarks.

    • Yocto indeed enables building minimalistic Linux distros. However Yocto build for Edison has all heavier things that are normally not included in minimalist linux: eglibc (rather than uclibc), real linux utils (rather than busibox), g++/binutils, node.js, systemd, dbus. (Though no heavy GUI). Besides, when running CPU and/or RAM intensive benchmarks, several daemons running in background in Raspbian or Yocto/Edison barely consume CPU cycles.

      You are right that PHP interpreter could be not the best benchmark. It might be better optimized for x86/Atom than for Arm, so running C benchmark could tell us more about bare compute performance difference. So what we got here is h/w + middleware performance, and middleware performance (php interpreter, node.js) differs on different platform and could also be decision point.

      Comparing Mhz for different architectures means nothing. If we assume same s/w, the performance would be proportional to (Mhz * IPC) * (numCores * scalability).

      p.s. I do full time performance measurement and optimization since 2005. p.p.s. disclaimer: I work for Intel.

  • Wow, the Edison really stands out in these tests.

  • Inforce Computing’s SBC should definitely be part of the above Benchmarking report. Their boards have excellent IOs, are rightly priced and come with full free support for Android and Ubuntu Linux based BSPs.

  • Add in the fact that Edison has built in Wifi and bluetooth (which may have been running during the power benchmark?) and its a no brainer for your embedded needs. I am running two of them, both battery powered.

    • Nope! I made sure to turn them off during the power tests. :) I was definitely able to see the current drop by several 10s of mA when I did (at least for WiFi. Bluetooth seemed to be pretty quiet if it wasn't connected to anything).

  • If I am not mistaken, BBB C is the oldest one of the bunch, so it is at disadvantage when compared with newer models. I think older models should also be included in this test, like RPi v1.

    • It is, but it works as a useful comparison to see how the newer boards perform. I would like to do a "Round 0" with things like the RPi v1 B+ and BBB (something like the Rev B if I can get my hands on one).

  • Picking a favorite SBC is pretty difficult because a lot of it boils down to how you are planning on using it. Since they all have little features and quirks that make one just slightly better than another for each individual project.

    Of the three SBCs I have used so far I would have to say my favorite is the Odroid C1. The Banana Pi Pro is a close second and the Raspberry Pi gets an honorable mention for being a solid system to take the plunge into SBCs.

    • I've heard good things about the Banana Pi, but haven't heard of the Odroid before. I just looked it up, and it seems like a solid competitor for the Raspberry Pi.

  • With the 2D graphics test, why did the pcDuino3 outperform the Acadia? The Acadia is just a quad core version of the pcDuino3. It should have at least been the same speed.

    • They're not actually the same. The Acadia is a quad core ARM Cortex-A9, whereas the pcDuino3 is a dual core ARM Cortex-A7, which means a slightly different architecture. However, a lot of that could be software (e.g. Linux or video drivers). I experienced some video problems with the Acadia while running the tests, which makes me believe that it has not been optimized yet.

  • Going into "old geezer" mode: Wow! I remember the [bad old] days when the top "super computers" were rated at ONE MFLOP, and cost millions of dollars, would take up a large room, and need LOTS of A/C. Now we can get 30 or so MFLOPs for about $40, stick it in a pocket, and even run it for an hour or so off a battery. Where will we be in another 35 or 40 years?

    • It's kind of crazy to think about how far we've come. I know that people have told of the "end of Moore's Law," but we keep doing things like adding more cores (which doesn't violate Moore's Law, but gives us "more" computing power). I can't even fathom where we'll be in 40 years!

      • IBM recently reported advances in new chip architecture using Germanium Arsenide. I forget specifics, but they claimed it keeps Moore's law alive. They also said it's still in dev and won't hit the market for a couple years, so do with this info what you will.

        • Perhaps you are thinking of this research? Except it is UC Berkeley and graphene transistors, not IBM and GaAs. But it, too, claims to prolong Moore's Law. I sure hope it's time to get back on the roller coaster and start screaming.

          • I think LC may be referring to this recent announcement from IBM:

            http://www.npr.org/sections/thetwo-way/2015/07/09/421477061/ibm-announces-breakthrough-in-chip-technology

        • Well, I guess I'll be in the market for a new computer in a few years :D That's really cool, though.

Related Posts

Raspberry Pi 3

Recent Posts

Tags


All Tags