How to profile Heat using OSprofile

The OpenStack Heat project has been having some scaling issues, and it really helps knowing where your problems are
before trying to solve them. So to help out here are the instructions to get osprofiler (
working with Heat (and the other projects that have support for it).


  • I am assuming you are using devstack
  • Once the inflight patches have landed this will be a lot easier!

Go to the base directory of your projects.

cd cinder
git pull
echo -e "[profiler]\nprofiler_enabled = True\n" >> /etc/cinder/cinder.conf

cd ../python-cinderclient
git review -d 103359

cd ../nova
git review -d
cp etc/nova/api-paste.ini /etc/nova/
echo -e "[profiler]\nprofiler_enabled = True\n" >> /etc/nova/nova.conf

cd ../python-novaclient
git review -d

cd ../heat
git review -d 118115
cp etc/heat/api-paste.ini /etc/heat/
echo -e "[profiler]\nprofiler_enabled = True\ntrace_sqlalchemy = True\n" >> /etc/heat/heat.conf

cd ../python-heatclient
git review -d 118118

cd ../keystone
git review -d 103368
pushd /etc/keystone
echo -e "[profiler]\nprofiler_enabled = True\n" >> /etc/keystone/keystone.conf
mv keystone-paste.ini keystone-paste.ini.orig
wget -O keystone-paste.ini
diff -u keystone-paste.ini.orig keystone-paste.ini

cd ../python-keystoneclient
git review -d 114856

sed -i "s/notification_topics.*/notification_topics = notifications,profiler/" /etc/ceilometer/ceilometer.conf

Note: in the api-paste.ini files above there is a default key “SECRET_KEY” – on anything but a devstack you should quickly change it.
What ever it is, make sure it is consistent and you provide the same thing on the command line (below).

You then need to restart the effected services (apache, heat-*, cinder-*, nova-*, ceilometer-*).

Here is an example run of “heat stack-create” I did:

heat --profile SECRET_KEY stack-create test -f bug1288774/1.yaml 
| id                                   | stack_name | stack_status | creation_time        |
| 1c1a27ac-291e-46ca-bc53-69e026ad9dd1 | test       | _            | 2014-09-04T08:30:24Z |
Trace ID: 105b22f9-f9f0-4526-ac52-79e0dab94c79
To display trace use next command:
osprofiler trace show --html 105b22f9-f9f0-4526-ac52-79e0dab94c79 

And the result?
Have a look here:
I think that’s quite neat!

BTW: the above trace is for a stack with a single server.


How to get Heat to work with a remote Openstack.

Say you have a remote/public install of Openstack and you want to use a local install of Heat
to talk to it. I find this handy when developing, as the remote Openstack can be kept stable
and is not effected by changes made to the development machine.

So lets say you have 2 machines:

  • “rock” ip == (used for base Openstack services)
  • “hack” ip == (used for Heat development)

Install your Openstack as normal on “rock”.

I used devstack to install Heat on “hack”.
My localrc looked like this:



enable_service mysql heat h-api h-api-cfn h-api-cw h-eng

Then run your ./ as normal.

You then need a special environment (not devstack/openrc) to make this work.
go to your “rock” machine and get the tenant_id that you want to work with:

keystone tenant-list
|                id                |        name        | enabled |
| 6943e3ebad0d465387d05d73f8e0b3fc |       admin        |   True  |
| b12482712e354dd3b9f64ce608ba20f3 |      alt_demo      |   True  |
| bf03bf32e3884d489004ac995ff7a61c |        demo        |   True  |
| c23ceb3bf5dd4f9692488855de99137b | invisible_to_admin |   True  |
| c328c1f3b945487d859ed2f53dcf0fe4 |      service       |   True  |

I wanted “demo”.
Now make a file to store your new environment (heat.env).

export HEAT_URL=
export OS_USERNAME=admin
export OS_TENANT_NAME=demo
export OS_PASSWORD=abetterpasswordthanthis
export OS_AUTH_URL=

Now you use this like:

. heat.env
heat list

Note: remember to open up firewall ports on “rock” so that you can access the OpenStack services.

Leave a comment


Get every new post delivered to your Inbox.