Motion Capture in Blender. Toolset selection.

This process is not very complicated but contains many pitfalls. So that you do not step on the same rake as I do, and do not spend too much time experimenting, I will try to describe in as much detailed as possible all the stages and their features. Well, in the first article we will just try to decide on the equipment and software to be used.

However, it is not a fact that the series will continue. If I see that there is no interest in it and it makes no sense to waste my time on continuation, then this article will be the last one. So if you are interested to find out more about different custom development solutions, please leave a message under the article.

Now let’s get closer to the point. Remember those demonstration Hollywood videos in which a person in a special suit runs and jumps, and his movements are transmitted to an animated computer character? This is the very Motion Capture (abbreviated Mocap). That is what we are going to do, but we will only use HTC Vive with a few additional trackers for it to record movements (this is tens/hundreds of times cheaper than specialized equipment).

For the actual capture of movements (Motion Capture) we will use IKinema Orion, for quickly creating a human model - MakeHuman, for transferring recorded movements to the resulting model and subsequent post-processing - Blender is the one. Moreover, some of these stages can be excluded. So, if you do not need specific movements, then it is likely that you can find and use ready-made recordings and transfer them to your model. And if you already have a ready-made model of a person (or a humanoid creature) with a skeleton, you can skip the stage of working with MakeHuman (although I would recommend you to get acquainted with this wonderful program anyway).

The final animation can be used at your discretion - to create cartoons (the full cycle can be done directly in Blender), game development (hello, Unity, Unreal and other game engines) and any other goals.

But first things first. I will try to provide details on each stage. I will give not only ready-made instructions but also the results of my own experiments with other options so that it is clear why exactly the described solutions were chosen. I also do not pretend to be the absolute expert, and if one of the readers has thoughts, how to optimize the process as much as possible - again, you are welcome to leave the comments.

Equipment for motion capture (the very Motion Capture)

Using HTC Vive for Motion Capture

Actually, why the HTC Vive? It is simply because today, among the budget options (which fit in one or two thousand dollars), this is the most adequate and accurate. There have been other attempts to use simple tools for Mocap. One of the simplest options, the ordinary webcams were tried to adapt for these purposes, but the accuracy of the results left much to be desired. From later and more accurate, I tried to use Kinect. But with the advent and relatively wide distribution of virtual reality helmets (and additional accessories for them), it became clear that this solution is much more accurate and convenient. And among the helmets of virtual reality to date, HTC Vive is perhaps the most common and convenient solution (large thanks to the collaboration with Valve and integration with Steam).

So why is VR equipment (in our case, HTC Vive) so well suited for recording Mocaps? Well, because the helmet itself, the controllers that come with it, as well as additional trackers purchased separately, are capable of transmitting their exact position in space, including tilt angles. Though, using a helmet with a wire for mocap is not very convenient. Therefore, we will get rid of it by putting one of the trackers directly on our heads. Also, taking in each hand a controller and hanging additional trackers at several points, we actually get a simple version of the Hollywood costume for motion capture.

Moreover, since we determine not only the location of the equipment in space but also the angles of inclination, many intermediate points (for example, elbows and knees) can be calculated automatically. Thus, we can save on the number of trackers used (each of them costs about $ 150).

Speaking of trackers. Accepted (and in many cases sufficient) minimum of additional trackers is considered to be 4. That is the amount I will use in the examples. We will hang one of them on the head (on the forehead), and fix the remaining three on the feet and belt, and take it into our hands by the controller. That's all, this amount is enough to achieve very decent results. You can read the details about it in the software description.

Motion capture software

So, since we do not claim Hollywood scope and decided so far to abandon the millions of expenses for specialized equipment, we will only consider options for programs that provide motion recording using HTC Vive. Looking ahead, I would warn right away that I actually used the IKinema Orion program to record movements, but before moving on to describing it, I prefer to talk about a few more options.

Vive Setup and Steam VR

Of course, before you start working, you need to configure the equipment. First of all, you need to install software from HTC (Vive Setup) and configure the operation of the bases, helmet and controllers.

Also, it won’t hurt to install Steam VR. This is a software from Valve for working with VR-applications (games and more). It is not related to Mocap, but I would still recommend installing it for configuration and testing of equipment. By the way, on Steam, there are many free VR applications that will allow you to have fun (games) and with benefits (graphics, sculpting). But now is not let’s move forward.

Motion Capture in Blender

Of course, since we are using Blender for work, the ideal option would be to capture the movement directly from it, without using third-party software. But, unfortunately, I and other Software Developers I know have not yet been able to find working solutions. If you succeed, please let me know about it in the comments below. I will be very grateful and supplement the article.

it is clear that technically there is such an opportunity. There is just no craftsman who would like to write an add-on for Blender that was specifically designed to capture movement with the help of Vive-type VR helmets. Add-ons for Blender are written in Python, and if you know this language, you can start writing such an add-on. Lots of people will appreciate that.
In the meantime, I found an add-on for Blender that allows you to work with the viewport by wearing a VR helmet, that is, actually seeing the scene you are working using VR. But this has nothing to do with the Mocap.

Well, for the lack of motion capture tools built into Blender, you have to look for third-party software for these purposes with the subsequent transfer of recorded movements to Blender using the appropriate add-on. Though, we will review that later.

Motion Capture in  Unity 3D

Good news is that add-ons are written to game engines. For example, here is one of these add-ons for Unity (already from the HTC team itself). You can use it for free. I managed to install it on the free version of Unity and test it. At a minimum, it works in real-time. I did not deal in detail with the recording of animations and the possibilities of remapping since simpler option like IKinema suits me more.

Motion Capture in Unreal Engine

Almost the same can be said about the Mocap plugin for Unreal. It is paid, I did not try to install and test it, but I saw a lot of laudatory reviews about it on the network.

Motion Capture in Brekel OpenVR Recorder

Brekel OpenVR Recorder is a program, that, unfortunately, can be used only for Windows, designed specifically for recording the positions of HTC Vive trackers. The program is paid. It costs $ 139, but there is a trial version that allows you to record up to 10 seconds. A rather interesting and noteworthy program, it exports animations in FBX format. The results are opened and work normally in Blender (verified). However, it is not suitable for our purposes for one reason - the program records only the position of the trackers, we do not get any animated skeleton at the output.

Motion Capture in IKinema Orion

And finally, the winner of our hit parade - IKinema Orion. Unfortunately, there is no Linux version either, and the program is quite expensive, costs 400 British pounds (about 530 US dollars). But there is a two-week full-functional trial version. It was more than enough for my purposes.

That's all. Try it and I hope, you will enjoy this experience! Though if you feel like you would trust this task to a dedicated development team, then let's have a discussion!

Best regards,
Dmitry Zhart
Ardas Team