Week 10: Conclusion

This is my last week of SOCIS, and as mentioned in the previous post, this one comes a few days early because I am traveling. This week has been spent fixing bugs and cleaning up code for the most part. I have also started drafting a guide on C++ generation for the GNU Radio Wiki and written some QA code which will be uploaded later this week (forgot to push it before I left).

For me, SOCIS has been a new, and undoubtedly memorable, experience. I’m fairly content with how my project turned out, and it’s been fun to see my changes turn into a useful feature. It’s also been incredibly inspiring to get feedback from the community and see that my work matters.

SOCIS is over, but the next step for me will be to get my code merged into the upstream repository and keep on improving the C++ output functionality. There are still a couple issues remaining that I haven’t been able to fix yet, and there are new features to be added. Also, one of the main objectives of the python3 branch is to switch from Cheetah/XML to Mako/YAML, so I will be helping out with this.

My work can be checked out here, with the “center” of my project in grc/core/generator/cpp_top_block.py and the Mako templates in the directory below. I also have a branch called yaml_blocks_python in which I have ported all the modules’ (including UHD) blocks to YAML except QT and a couple of the blocks in grc/blocks/. 

So, conclusively, I’d like to thank my mentors and the rest of the GNU Radio community for these exciting 2.5 months. It’s been fun! 

Week 9: QT Demo

My coding period for SOCIS is approaching its end, and the pencils down date is less than a week away. I will be traveling next weekend, so my final blog post will come on 28 September. I have spent the past week adding QT support and build options, as well as replacing GRC’s entire block tree with YAML blocks. The latter can be found in my yaml_blocks_python branch.

The addition of QT GUI and build options marks the completion of milestone #2, so that’s an exciting announcement to make. There are still some small QT-related changes to make, some variables need to be renamed, others need to be moved, and the generated files don’t look too elegant yet, but that’s mostly cosmetic. I have written a small demonstration of the project’s current state below.

This demonstration will be very similar to my last demonstration from week 7, so if you want details you’ll have to check that out though I will repeat the commands here (wall of text incoming):

git clone https://github.com/hauk142/gnuradio.git

cd gnuradio

git checkout python3_demo2

git submodule init

git submodule update

mkdir build

cd build

cmake .. -DENABLE_GR_COMEDI=OFF -DENABLE_DOXYGEN=OFF -DENABLE_GR_DTV=OFF -DENABLE_GR_DIGITAL=OFF -DENABLE_GR_FEC=OFF -DENABLE_GR_UHD=OFF -DENABLE_GR_TRELLIS=OFF

make

sudo make install

export PYTHONPATH=/usr/local/lib/python3.6/dist-packages/

export LD_LIBRARY_PATH=/usr/local/lib/

gnuradio-companion

If you made it through this, you can continue by opening demonstration/qtdemo.grc. The flowgraph should look like this:

This is simply a flowgraph with a signal source connected through a throttle block to a QT frequency sink. At the bottom we have two QT entry blocks, giving you an opportunity to change the amplitude and frequency of the signal source. If you double-click the options block you can see the new build options, namely the ability to disable CMakeLists.txt generation, enable static linking and pass options to CMake:

CMake options are passed as shown in the image, with each option separated by a semicolon. This means that

OPT1=VAL1; OPT2=VAL2

in the options block will result in the following line in CMakeLists.txt:

set(OPT1 VAL1)

The options block can be closed, and you can press the execute button to generate, build and execute your flowgraph. GRC will hang for a couple of seconds until you are presented with this window:

That’s it! You’re free to change the amplitude and frequency on-the-fly, just change the value and press enter.

So this leaves me with just the final week of SOCIS remaining. It’s been a fun and challenging project, and I’m excited to get started with the finishing touches. The last week will be spent fixing bugs, writing a couple of Wiki pages, QA and refining the branch.

Week 8: QT GUI

This week has been dedicated to the QT GUI blocks. More specifically, porting them to YAML and handling them in cpp_top_block.py. I have uploaded all the QT blocks in YAML format to my grc_yaml_blocks repository. There are some details missing though, because there is still some Cheetah-style code left to edit.

The QT blocks don’t play very well with the way I’ve designed the YAML files’ cpp_templates, because they have more than just the standard module::block::make(…); line in their make field. This is a bit unfortunate, as it messes with the #include statements as well, since they are generated by looking at the make field. Currently I am trying out moving all the other code into its own field called additional_make. This seems to fit the QT blocks so far, and I might go with this approach if it doesn’t introduce any new problems.

I have also been working on a few bug fixes for the Python generation code. Today’s blog post is of the shorter kind, but the second milestone is in a few days (Thursday), and I’ll write up a lengthier one, hopefully with a demonstration, for that in the weekend.

Week 7: Vacation

I spent the previous week participating in a military exercise, and there’s not much to report because of this. However, I did create a poster of my project for the 2017 GNU Radio Conference:

https://drive.google.com/open?id=0B9ht6cKX6zczUWQxSHhIV3ZBZ28

There are some parts that don’t work quite yet or are bugged, and these are mostly pertaining to parameters, build options and the ‘Execute’ button. Hierarchical blocks don’t work yet either, but this is true for the upstream python3 branch as well.

The next step for me now is working towards the second milestone, which I have, together with my mentors, decided to move from September 17 to September 21. This milestone includes the ability to create QT applications written in C++ from GRC.

Week 6: Milestone #1 Demo

This week’s blog post comes a few days earlier than usual. This is because I have a military exercise starting tomorrow, which means I won’t get any work done until next weekend. However, my SOCIS project has reached its first milestone, and I will dedicate this post to a demonstration of the project’s current state. This demonstration will be a bit more detailed than the one from week 3, so everyone can try out my changes. I will show you a simple flow graph with two signal sources, an add block, a throttle block and a file sink.

First of all, you’ll have to clone my repository with the following commands:

git clone https://github.com/hauk142/gnuradio.git

cd gnuradio

Second, switch to my python3_demo branch and checkout VOLK:

git checkout python3_demo

git submodule init

git submodule update

Next, you are ready to build GNU Radio. For the purpose of this demonstration, the CMake command below disables some components. Feel free to enable them, but it’ll cost you some time when building.

mkdir build

cd build

cmake .. -DENABLE_GR_COMEDI=OFF -DENABLE_DOXYGEN=OFF -DENABLE_GR_DTV=OFF -DENABLE_GR_DIGITAL=OFF -DENABLE_GR_FEC=OFF -DENABLE_GR_UHD=OFF -DENABLE_GR_TRELLIS=OFF

make

After make finishes, you are ready to install and run GRC:

sudo make install

export PYTHONPATH=/usr/local/lib/python3.6/dist-packages/

export LD_LIBRARY_PATH=/usr/local/lib/

gnuradio-companion

The two ‘export’ commands will have to be run every time you run GRC in a new terminal. You can put them in a shell script for easy access, but remember that you need to use source for it to work.

Now GRC should be open and you can continue by opening a file called fsink.grc. You can find it in the demonstration directory from the cloned GitHub repository.

 

Your screen should look something like the image above. This flow graph is almost ready to be executed, all you have to do is double-click the file sink and choose in which file you want your signals to end up. Next, you can press the “play” button at the top, and dismiss the warning. GRC will freeze for a few seconds (will be fixed), and a terminal will appear:

 

Congratulations, you’ve just generated, built and executed your first program with GRC’s brand new C++ output functionality!

It might seem like nothing’s happening, but actually, GNU Radio is filling your selected file with the output from the signal sources. You can press Enter to quit and the file can now be inspected however you’d like. You might notice an error upon pressing Enter. This is an issue that will be fixed once I figure out the cause. Please note that I haven’t added support for GUI flow graphs yet, so you won’t be able to create a flow graph with a file source and a QT plot to view the contents of the file. Also, I have only added a few YAML blocks in this branch, but you can find more in my grc_yaml_blocks repo.

I modified the display_qt example from the gr-qtgui module, opened the file, and got this plot:

 

I have included this program in the plot directory in my python3_demo branch, and you can run it like this:

cd plot

mkdir build

cd build

cmake ..

make

export PYTHONPATH=/usr/local/lib/python3.6/dist-packages/

export LD_LIBRARY_PATH=/usr/local/lib/

./display_qt ‘/home/hkon/signals’

Where signals is the file you generated with GRC.

So that’s it! I hope, if you tried to follow my steps, that you managed to get through the demonstration without running into an error you couldn’t resolve (and if you did, please don’t hesitate to ask) 🙂 The next step for my SOCIS project is adding support for GUI flow graphs.

 

Week 5: Parameters and Variables

This week’s focus has been the transpiling functionality for parameters and variables. This is done using python’s re module and simple regex matching. A more sophisticated approach would be to make use of the ast module, but I have decided not to include this as part of my SOCIS project. The reason for this is that the university semester has begun (the coding period kick-off was shifted due to delays) and I want to keep my work load manageable. I’m keen to keep working on this after SOCIS though!

I have also created a GitHub repository for YAML files for blocks with C++ support. So far, I have uploaded most of the gr-analog blocks, all of the ZeroMQ blocks and a couple of gr-blocks blocks. The addition of C++ support brings some changes to the blocks, which I will show you below:

(larger image here)

The presence of the ‘has_python, has_cpp’ flags enables GRC to check if all the blocks in the flowgraph support the current output language. If not, GRC will refuse to generate code. This is implemented in my python3_dev branch.

Also, the cpp_templates field includes information essential to the generation of C++ code. includes contains the required include statements and callbacks is exactly the same as its Python counterpart. make contains the make statement for the block. You can find it in the top_block’s constructor. The link field simply tells CMakeLists.txt which libraries to link against. Finally, translations fixes the problem of enums or bools in a block’s parameters. Very few blocks use enums, but there are some that use bools, and ‘True’ is written with a lowercase ‘t’ in C++. The parts that need editing are translated when the user generates the C++ code, after the block templates have been rendered.

Week 4: CppTopBlockGenerator

This week’s focus has been the CppTopBlockGenerator, which is a subclass of the standard TopBlockGenerator and can be found in grc/core/generator/cpp_top_block.py. I have also been working on the C++ hier block generation, which still needs some work. The next step is working out a way to take care of parameters and variables. My current plan is that simple expressions will be transpiled, along with some popular objects such as numpy-arrays and similar.

I have also been spending some time trying out different ways to decide which GNU Radio modules to build the C++ files against when generating CMakeLists.txt. This obviously needs some attention since different blocks rely on different modules.

One possibility is to keep a kind of cache in GRC over the installed modules. You can’t create flowgraphs with blocks from modules that you don’t have installed, so this could have been a viable option, but the problem with this approach is that CMakeLists.txt may have to be edited to remove any modules that aren’t required, especially if that CMakeLists.txt is going to be used with a different installation of GNU Radio. 

Another solution would be to “scan” the flowgraph for the modules required to build it. For example, I can parse the #include statements from the blocks’ YAML files to decide which modules are required. This may or may not take some more work, but it generally sounds like a more elegant approach that will give a smoother and more streamlined user approach. 

Week 3: CMakeLists.txt Template and Demo

This week’s focus has been the CMakeLists.txt template. However, the most notable change has been my progress with the CppTopBlockGenerator class. At the moment, GRC is able to produce the planned C++ files, but there are still some details and quirks remaining. I’ll give a small demonstration to show the current state of the project:

As previously mentioned, my work is mainly done on the python3_dev branch, and to compile it and try it out for yourself you will have to apply the small fixes from my Week 1 post. Please note, however, that the fix pertaining to uses_var_id() has already been applied in that branch. When you launch GRC, you will notice a new addition to the options block:

This option governs the output language, and will decide whether your flowgraph produces a .py file or a .cpp file, an .hpp file and CMakeLists.txt. The option is manipulated with a basic drop-down menu:

You can now just click “Generate” as you would normally, and GRC will generate the three aforementioned files. Please remember to set an ID in the options block, otherwise you  will get an error. To demonstrate, I created a flowgraph with a signal source connected through a throttle block to a null sink. GRC yielded the following files (Pastebin link):

top_block.cpp
top_block.hpp
CMakeLists.txt

As you can see, there are still some pieces missing and parts that are Python in there, but it’s something. I’m glad I can finally see some actual results from my work 🙂 The next steps are adding support for hier blocks and further refinement of the CppTopBlockGenerator class, before I take on parameter and variable handling.

Now, if you try to do exactly what I did in this brief walkthrough, you won’t be able to generate the same files. This is because I have added C++ components to the blocks’ YAML files, which aren’t tracked in Git at the moment. The block tree will be converted to YAML eventually, but is still using XML at the moment. You can find the files I have edited here:

analog_sig_source_x.block.yml
blocks_throttle.block.yml
blocks_null_sink.block.yml
options.block.yml
stream.domain.yml

Just replace the autoconverted ones in ~/.cache/grc_gnuradio/ with these, and you should be all set to generate the flowgraph I demonstrated above.

Week 2: Source File Template

This main focus for this week has been the source file template. The templates are coming along pretty well, and most of the work remaining to get C++ output from non-GUI flowgraphs is related to coordinating the Generator classes with the Mako templates. Actually, I did manage to trick GRC to generate a C++ header file earlier today, but it was pretty messy and not quite ready for this blog yet 🙂 Other than that, I have been working on resolving some miscellaneous quirks within GRC.

 

GNU Radio’s master branch currently uses XML for describing the blocks, and Cheetah as the template engine. These are in the process of being phased out however, with Mako replacing Cheetah and YAML replacing the XML files.

This is the YAML file for the Throttle block:

I have omitted the top three lines, describing the id, label and flags. The remainder of the file lists the data required to use the block in your program. The parameters key contains the options for the block. You can find these by double-clicking the block in GRC:

You can find each option in the YAML file, there’s one for each “–“. You can also see the label (the text you see in the GUI), the type, default value and the conditions for hiding when you are viewing your entire flowgraph.

The next two keys are the inputs and outputs, also visible from within GRC:

The third, templates, incorporates the code-related part of the block and does not have anything to do with the GRC GUI. This part defines the code that, together with the other blocks and the Mako templates, make up your generated code. More specifically, the required libraries, the call to the block’s constructor and the callbacks.

Week 1: Header File Template

This week’s theme has been the header file template. The contents of the header file are mainly the #include statements, the top block (or hier block) class declaration, and declarations of other things such as blocks and variables. This does not amount to much code in comparison to the source file, which will include everything else such as constructors, definitions, main() and so on. However, it’s a great starting point. This week’s work hasn’t exactly yielded much of an immediately visible profit, but this is due to the fact that all the blocks still are Python only and the generation code is more or less untouched at this point. The C++ output will come 🙂 There’s also a small practical change, from now on I will be pushing daily commits to a python3_dev branch and keeping the larger changes on the python3 branch.

 

The python3 branch from the main repository does not run without error right now, but I’ve gathered the steps necessary to get it running:

First, you’ll have to edit grc/scripts/gnuradio-companion at line 84:

def check_python_version():
         if sys.verion_info.major > 2:
                 die(RuntimeError(‘No python3 support yet.’), ‘Stay tuned…’)

The program will die here if you’re running it with Python 3, of course, so you’ll want to fix this.

Second, you’ll have to take a look at grc/core/generator/top_block.py at line 206 and 212:

def uses_var_id():
(…)
callbacks[var_id] = [callback for callback in callbacks_all if uses_var_id()]

This creates an error because callback is used within the uses_var_id() method. I suspect this is a Python 3 migration problem. This error will hinder GRC from generating any flow graph files. The fix is easy though:

def uses_var_id(callback):
(…)
callbacks[var_id] = [callback for callback in callbacks_all if uses_var_id(callback)]

So, with those fixes in place, you should be able to run GRC and generate the top block Python file.

EDIT: In fact, there are a couple more steps you’ll need to take to get it running:

To avoid getting the “domain not registered” errors, you’ll have to copy the YAML files stream.domain.yml and message.domain.yml from grc/blocks/ to your block cache. This is probably at ~/.cache/grc_gnuradio/

The next step is adding “generator” at the end of grc/CMakeLists.txt at line 152, like this:

DIRECTORY core gui generator

That’s it, you’re all set!