Deap dive into using Heat Alarms

So this is just some output from me debugging how Heat’s alarms work with the other parts
of OpenStack (warts-n-all).

First some histroy

Way back, Heat needed alarms to be able to do autoscaling (because it is really useful). The only problem
was we didn’t have Ceilometer back then. So we wrote a minimalist version of Cloud Watch, it was not
suposed to last very long as we fully expected some monitoring-aas to come along. Alas, it has lasted a
lot longer then expected. Also note that we didn’t have Alarms based on notifcations at this point so
all metrics were generated by an agent (cfn-push-stats).

Using Cloud Watch alarms

First off how you can choose the implementation (Ceilometer/Heat).

We use the global environment to configure this. Look in /etc/heat/environment.d/default.yaml
Just uncomment the one you want (We HIGHLY recommend Ceilometer).

resource_registry:
# Choose your implementation of AWS::CloudWatch::Alarm
"AWS::CloudWatch::Alarm": "file:///etc/heat/templates/AWS_CloudWatch_Alarm.yaml"
#"AWS::CloudWatch::Alarm": "OS::Heat::CWLiteAlarm"

Note: the default will change this cycle to use Ceilometer for Cloud Watch Alarms.

There are unfortunately some differences depending on the implementation and whether or not
you are using cfn-push-stats.

A note about cfn-push-stats “–watch”
Basically this was a mistake in implementation as it causes a circular dependancy (this was hidden at
the time of implementation as way back then Ref returned only the resource name, not a resource instance.
(AWS does not do this, they use tags, like we now do)
Basically you should just drop the “–watch” option (it’s just not needed anymore).

Some Cloud Watch examples

CWLiteAlarm with in guest agent

https://github.com/asalkeld/autoscaling-testing/tree/master/AWS_CW_cfn_push_stats_with_watch

https://github.com/asalkeld/autoscaling-testing/tree/master/AWS_CW_cfn_push_stats

AWS_CloudWatch_Alarm.yaml with built in Ceilometer metrics

https://github.com/asalkeld/autoscaling-testing/tree/master/AWS_CW

AWS_CloudWatch_Alarm.yaml with in guest agent

https://github.com/asalkeld/autoscaling-testing/tree/master/AWS_CW_cfn_push_stats

Some Ceilometer Examples

How do these alarms work?


MEMAlarmHigh:
type: OS::Ceilometer::Alarm
properties:
counter_name: cpu_util
statistic: avg
period: '60'
evaluation_periods: '1'
threshold: '50'
alarm_actions:
- {"Fn::GetAtt": [ServerScaleUpPolicy, AlarmUrl]}
matching_metadata: {'metadata.user_metadata.server_group': 'ServerGroup'}
comparison_operator: gt

You will notice the matching_metadata, this is how Ceilometer finds the applicable samples to
calculate the Alarm. When ever we create a server we add metering.InstanceId=x and when ever
we create an autoscaling group we add metering.group=x to the server’s metadata.
Then Ceilometer looks for metadata with the prefix metering. and adds that to the sample metadata.
Have a look here: https://github.com/openstack/ceilometer/blob/master/ceilometer/compute/util.py#L34
Just remember if you are using cfn-push-stats or even ceilmeter client directly this renaming does not happen
and you will need to search for metering. and not user_metadata..

With builtin metrics

https://github.com/asalkeld/autoscaling-testing/tree/master/Ceilometer

With in guest agent

https://github.com/asalkeld/autoscaling-testing/tree/master/Ceilometer_cfn_push_stats

Some potential reasons that your Alarms don’t work:

1) cfn-push-stats is out of date and has a bug – at one point it had a bug that pervented samples from being sent
2) The Ceilometer pipeline interval is too slow (needs to ~60 secs) https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/pipeline.yaml
3) check the matching_metadata to see if it is correct (use ceilometer -d sample-list -m ) check what the metadata actual is.
4) we have a bug in Heat:-O – head over to #heat and ask for help.

Things we need to do

1) make this more consistent
2) add all those combinations to integration tests
3) look at adding Monasca Alarms (if possible)
4) remove the builtin cloud watch implementation (it’s now deprecated)

1 Comment

How to profile Heat using OSprofile

The OpenStack Heat project has been having some scaling issues, and it really helps knowing where your problems are
before trying to solve them. So to help out here are the instructions to get osprofiler (https://github.com/stackforge/osprofiler/)
working with Heat (and the other projects that have support for it).

Note:

  • I am assuming you are using devstack
  • Once the inflight patches have landed this will be a lot easier!

Go to the base directory of your projects.

cd cinder
git pull
echo -e "[profiler]\nprofiler_enabled = True\n" >> /etc/cinder/cinder.conf

cd ../python-cinderclient
git review -d 103359

cd ../nova
git review -d
cp etc/nova/api-paste.ini /etc/nova/
echo -e "[profiler]\nprofiler_enabled = True\n" >> /etc/nova/nova.conf

cd ../python-novaclient
git review -d

cd ../heat
git review -d 118115
cp etc/heat/api-paste.ini /etc/heat/
echo -e "[profiler]\nprofiler_enabled = True\ntrace_sqlalchemy = True\n" >> /etc/heat/heat.conf

cd ../python-heatclient
git review -d 118118

cd ../keystone
git review -d 103368
pushd /etc/keystone
echo -e "[profiler]\nprofiler_enabled = True\n" >> /etc/keystone/keystone.conf
mv keystone-paste.ini keystone-paste.ini.orig
wget https://dl.dropboxusercontent.com/u/2663410/keystone-paste.ini -O keystone-paste.ini
diff -u keystone-paste.ini.orig keystone-paste.ini
popd

cd ../python-keystoneclient
git review -d 114856

sed -i "s/notification_topics.*/notification_topics = notifications,profiler/" /etc/ceilometer/ceilometer.conf

Note: in the api-paste.ini files above there is a default key “SECRET_KEY” – on anything but a devstack you should quickly change it.
What ever it is, make sure it is consistent and you provide the same thing on the command line (below).

You then need to restart the effected services (apache, heat-*, cinder-*, nova-*, ceilometer-*).

Here is an example run of “heat stack-create” I did:

heat --profile SECRET_KEY stack-create test -f bug1288774/1.yaml 
+--------------------------------------+------------+--------------+----------------------+
| id                                   | stack_name | stack_status | creation_time        |
+--------------------------------------+------------+--------------+----------------------+
| 1c1a27ac-291e-46ca-bc53-69e026ad9dd1 | test       | _            | 2014-09-04T08:30:24Z |
+--------------------------------------+------------+--------------+----------------------+
Trace ID: 105b22f9-f9f0-4526-ac52-79e0dab94c79
To display trace use next command:
osprofiler trace show --html 105b22f9-f9f0-4526-ac52-79e0dab94c79 

And the result?
Have a look here: https://dl.dropboxusercontent.com/u/2663410/stack2.html
I think that’s quite neat!

BTW: the above trace is for a stack with a single server.

5 Comments

How to get Heat to work with a remote Openstack.

Say you have a remote/public install of Openstack and you want to use a local install of Heat
to talk to it. I find this handy when developing, as the remote Openstack can be kept stable
and is not effected by changes made to the development machine.

So lets say you have 2 machines:

  • “rock” ip == 192.168.1.88 (used for base Openstack services)
  • “hack” ip == 192.168.1.77 (used for Heat development)

Install your Openstack as normal on “rock”.

I used devstack to install Heat on “hack”.
My localrc looked like this:

HEAT_STANDALONE=True
KEYSTONE_AUTH_HOST=192.168.1.88
KEYSTONE_AUTH_PORT=35357
KEYSTONE_AUTH_PROTOCOL=http
KEYSTONE_SERVICE_HOST=$KEYSTONE_AUTH_HOST
KEYSTONE_SERVICE_PORT=$KEYSTONE_AUTH_PORT
KEYSTONE_SERVICE_PROTOCOL=$KEYSTONE_AUTH_PROTOCOL

MY_PASSWORD=abetterpasswordthanthis
DATABASE_PASSWORD=$MY_PASSWORD
RABBIT_PASSWORD=$MY_PASSWORD

disable_all_services
ENABLED_SERVICES=qpid
enable_service mysql heat h-api h-api-cfn h-api-cw h-eng

Then run your ./stack.sh as normal.

You then need a special environment (not devstack/openrc) to make this work.
go to your “rock” machine and get the tenant_id that you want to work with:

keystone tenant-list
+----------------------------------+--------------------+---------+
|                id                |        name        | enabled |
+----------------------------------+--------------------+---------+
| 6943e3ebad0d465387d05d73f8e0b3fc |       admin        |   True  |
| b12482712e354dd3b9f64ce608ba20f3 |      alt_demo      |   True  |
| bf03bf32e3884d489004ac995ff7a61c |        demo        |   True  |
| c23ceb3bf5dd4f9692488855de99137b | invisible_to_admin |   True  |
| c328c1f3b945487d859ed2f53dcf0fe4 |      service       |   True  |
+----------------------------------+--------------------+---------+

I wanted “demo”.
Now make a file to store your new environment (heat.env).

export HEAT_URL=http://192.168.1.77:8004/v1/bf03bf32e3884d489004ac995ff7a61c
export OS_NO_CLIENT_AUTH=True
export OS_USERNAME=admin
export OS_TENANT_NAME=demo
export OS_PASSWORD=abetterpasswordthanthis
export OS_AUTH_URL=http://192.168.1.88:35357/v2.0/

Now you use this like:

. heat.env
heat list

Note: remember to open up firewall ports on “rock” so that you can access the OpenStack services.

Leave a comment

Follow

Get every new post delivered to your Inbox.