How to profile Heat using OSprofile

The OpenStack Heat project has been having some scaling issues, and it really helps knowing where your problems are
before trying to solve them. So to help out here are the instructions to get osprofiler (https://github.com/stackforge/osprofiler/)
working with Heat (and the other projects that have support for it).

Note:

  • I am assuming you are using devstack
  • Once the inflight patches have landed this will be a lot easier!

Go to the base directory of your projects.

cd cinder
git pull
echo -e "[profiler]\nprofiler_enabled = True\n" >> /etc/cinder/cinder.conf

cd ../python-cinderclient
git review -d 103359

cd ../nova
git review -d
cp etc/nova/api-paste.ini /etc/nova/
echo -e "[profiler]\nprofiler_enabled = True\n" >> /etc/nova/nova.conf

cd ../python-novaclient
git review -d

cd ../heat
git review -d 118115
cp etc/heat/api-paste.ini /etc/heat/
echo -e "[profiler]\nprofiler_enabled = True\ntrace_sqlalchemy = True\n" >> /etc/heat/heat.conf

cd ../python-heatclient
git review -d 118118

cd ../keystone
git review -d 103368
pushd /etc/keystone
echo -e "[profiler]\nprofiler_enabled = True\n" >> /etc/keystone/keystone.conf
mv keystone-paste.ini keystone-paste.ini.orig
wget https://dl.dropboxusercontent.com/u/2663410/keystone-paste.ini -O keystone-paste.ini
diff -u keystone-paste.ini.orig keystone-paste.ini
popd

cd ../python-keystoneclient
git review -d 114856

sed -i "s/notification_topics.*/notification_topics = notifications,profiler/" /etc/ceilometer/ceilometer.conf

Note: in the api-paste.ini files above there is a default key “SECRET_KEY” – on anything but a devstack you should quickly change it.
What ever it is, make sure it is consistent and you provide the same thing on the command line (below).

You then need to restart the effected services (apache, heat-*, cinder-*, nova-*, ceilometer-*).

Here is an example run of “heat stack-create” I did:

heat --profile SECRET_KEY stack-create test -f bug1288774/1.yaml 
+--------------------------------------+------------+--------------+----------------------+
| id                                   | stack_name | stack_status | creation_time        |
+--------------------------------------+------------+--------------+----------------------+
| 1c1a27ac-291e-46ca-bc53-69e026ad9dd1 | test       | _            | 2014-09-04T08:30:24Z |
+--------------------------------------+------------+--------------+----------------------+
Trace ID: 105b22f9-f9f0-4526-ac52-79e0dab94c79
To display trace use next command:
osprofiler trace show --html 105b22f9-f9f0-4526-ac52-79e0dab94c79 

And the result?
Have a look here: https://dl.dropboxusercontent.com/u/2663410/stack2.html
I think that’s quite neat!

BTW: the above trace is for a stack with a single server.

Advertisements
  1. #1 by Chintha Govardhan on September 12, 2014 - 1:47 pm

    Hi I am getting the following error with heat

    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging return self._encode(data)
    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging File “/opt/stack/python-neutronclient/neutronclient/openstack/common/jsonutils.py”, line 168, in dumps
    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging return json.dumps(value, default=default, **kwargs)
    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging File “/usr/lib/python2.7/json/__init__.py”, line 250, in dumps
    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging sort_keys=sort_keys, **kw).encode(obj)
    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging File “/usr/lib/python2.7/json/encoder.py”, line 207, in encode
    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging chunks = self.iterencode(o, _one_shot=True)
    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging File “/usr/lib/python2.7/json/encoder.py”, line 270, in iterencode
    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging return _iterencode(o, 0)
    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0x80 in position 0: invalid start byte
    2014-09-12 19:17:30.407 TRACE oslo.messaging.notify._impl_messaging

  2. #2 by Chintha Govardhan on September 15, 2014 - 5:57 am

    The following code in jsonutils.py is solving the issue. I donno whether it is correct solution or not .

    file : neutronclient/openstack/common/jsonutils.py

    def dumps(value, default=to_primitive, **kwargs):

    try:
    json.dumps(value, default=default, **kwargs)
    except Exception as e:
    return json.dumps(value, encoding=’ISO-8859-1′, default=default, **kwargs)
    return json.dumps(value, default=default, **kwargs)

  3. #3 by ahsalkeld on September 14, 2014 - 2:51 am

    Thanks Chintha, I’ll look into it.

  4. #4 by Chintha Govardhan on September 15, 2014 - 9:55 am

    Hi ahsalkeld, I did not pull the changes for ceilometer so why I am getting the above mentioned error.

    I pulled the changes from

    https://review.openstack.org/#/c/100239/

    then, everything is working fine.

    Thank you 🙂

  1. OpenStack Community Weekly Newsletter (Aug 29 – Sep 5) » The OpenStack Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: