zenoss: creasting zencommand custom plugins

Loads of work for me on Zenoss last few weeks.
FINALLY the class system/zProperties/rrd Templates/etc are making some sense to the inside of my head. We want to monitor buckets of data.


I posted this to here to see what people think:
http://community.zenoss.com/forums/viewtopic.php?p=18670#18670

I've added tons of stuff to Zenoss_configuration page in wiki yesterday, some today.
Up till 3:30am last night - I stopped when windows/mouse died.
I managed to put a zenoss template which contained several graphs and datasources into a ZenPack and then delete it and was not able to load it back. ARGH! So something else to figure out.


I had a couple of awkward problems to solve when getting a zencommand plugin to work so I thought I would share. If my analysis is correct this information might also be useful to others and could possibly be used to enhance the documentation on creating zencommand plugins.

So please take a look and see if this is useful.

J.

== Tip when working with zencommands. ==

If ONE data point in a whole template has an error (can have multiple graphs, data sources and points) then the data does not get stored in rrdtool. Even though testing with "zencommand run" reports a "storing" message. It might be safer to define datasources and graphs in a few seperate templates instead of one template.
In that way an error appearing for one piece of data would not cause all data retrieve to stop.

I'm still finding it a little awkward how best to manage classes and attributes in zenoss.

== Large data values received had a problem being stored if set to AVERAGE, use COUNTER instead ==

Value returned was FramesReceived=74609722 but rrdtool tried to insert 74609722.0.

2008-04-17 12:36:12 ERROR zen.RRDUtil: rrd error not a simple integer: '74609722.0' Devices/frigg.ie.commprove.test/iubTGenStatusTxt_FramesReceived

Fix is easy, use COUNTER instead of AVERAGE for those large data types.

(It could happen that a value monitored for a long time could grow and suddenly cause this kind of problem is a surprising way.)

== zencommand has some problems scaling. ==

... if you quickly hack together a plugin! Smile

"ERROR zen.zencommand: [Errno 24] Too many open files"

I see per-process open file limits of 254 and 258 on solaris 10 boxes.

zencommand fires off multiple processes ( number of servers * data sources ) to retrieve data.

One may edit zencommand.py and reduce MAX_CONNECTIONS
gsed -i s/MAX_CONNECTIONS=256/MAX_CONNECTIONS=16/ $ZENHOME/Products/ZenRRD/zencommand.py

Problems were reduced but not eliminated.
There were 15 servers in list and 3 zencommand datasources 15 * 3 = 45 which is not too many.

If you implement a zencommand plugin that causes zencommand to use up file handles you can run into this issue. I have a generic plugin which is called with different parameters to feed several data sources. The performance template was applied to a list of solaris servers.

The first incarnation of the plugin is a script which calls nagios plugin check_http and also uses a temporary file. I figure plugin is too slow/heavy; bash is spawned and another process to call check_http. Essentially best practice for plugins is to keep them very light and do any messing/work on the server side.

We wish to add more servers and more data sources using plugins like this. So it must scale. There is a limitation on what resources plugins are allowed to use which may not be very obvious.

parental fears and kids independance - public transport

http://www.nysun.com/editorials/why-i-let-my-9-year-old-ride-subway-alone
"Long story short: My son got home, ecstatic with independence."

I came across Mayer Hillman's writings on environment, transport a while ago.
http://www.spokeseastkent.org.uk/mayer.htm
Parental Fears and Loss of Children's independance
http://www.psi.org.uk/people/person.asp?person_id=30
Mayer Hillman

Our minor adventures with the Dublin bus to school can be seen a bit in this blog.
Daire & Kate are now mostly getting the very nice Kelly bus to school.
Occasionally the Dublin bus.
And some other families locally are also getting it so there is mostly a decent bunch of kids travelling the same route and older ones are watching out for the smaller ones.


First Story (Mom lets 9yr old rude NY subway home) Via Bruce Schneier on security
http://www.schneier.com/blog/archives/2008/04/overestimating.html
Overestimating Threats Against Children


And off topic a Myriad is 10000 http://en.wikipedia.org/wiki/Myriad

standards to reduce waste (remote-control, mobile phone charger devices)

Standards in general.

No, not in general. A method of using standards to limit resource use (reduce waste of materials/energy and general). Make the manufacturing/commercial applicance world slightly
more efficient hopefully.

Statement of problem (the easy bit):

Companies have a moral obligation (to the environment and to frustrated users) to make things standard and inter-operable. Particularily in my mind now are:
* infra-red remote controls
* mobile phone chargers and any general chargers/transformers
* mobile-phone and other electronic devices connection cables
* batteries ~maybe~
* light-bulb connections ~maybe~
* in-car devices (radios, ...)

You need a certain amount of flexability to change your interfaces to improve them. e.g. to make things smaller/cheaper/use less power. BUT infra-red remote controls and some accessories for mobile phones in particular are items which are purposefully made incompatible so that the customer is tied to one manufacturer to replace them.

This means lots of accidental revenue for the company for each unique remote control that a customer loses. But it also means there is a waste. Much less of these devices need be made.

Can the benefit to customers and to the environment be evaluated for these cases?
Companies purposefully seek to block competition from an area where they are dominant.
This should be made disadvantageous.

If a standard(S) can be defined for device(D) used to interface to a larger machine or systems(M).
And requirements on standard S can be defined as SR.
If a company has machine or system(M1) with interface(I1) and devices(D1) available implementing I1.
If the requirements of the interface to the system M1 can be defined as a subset of SR but if the interface is not compatible with standard S then a consumer and environment tax should be levied on system M1 especially and also on incompatible devices D1.

In plainer english D might be a remote control, M would be a TV.
D might be a mobile-phone charger, M would be a mobile phone.
M could be a light-bulb and SR a definition of requirements on light output, lifetime and manufacturing cost in resources).

In practice this could be difficult to implement. :) :-P

It can be very hard to get standards accepted.
The engineering and technology industry has to have standards in key areas to facilitate the
advancement of the industry but open standards can reduce the advantage that a closed system and
monopoly gives a company financially. So standards are not developed by the industry in many cases where they should be. Because most of reward of science/technology is through industry we have
an environment where openness is discouraged. This stifles experimentation and general sci/tech/...
advancement.

But, anyway, like, y'know:

1. Specific cases of this method of controlling waste can be legislated for.
e.g. Legislation on the control on sale of light-bulbs which are not(or less!? :))
environmentally friendly is somewhat similar to a control like this.
General legislation might make niche areas more wasteful?
Think about: using cheaper lightbulbs in an environment where bulbs are in more
rugged environment (i.e. our kids bedrooms!) might generate less waste, ...

2. On a larger scale (global or starting as European/American/Russian/Asian/...) standards can
be defined and rules set so that the overhead of setting standards, evaluating benefits of
having standard and decisions on conformance can be made (with less waste! :) )


This is (an attempt at defining) a general method of controlling waste.
Waste which is financially advantageous to entities to promote by accident of having a monopoly over a manufacture of interface-devices.

Brought to you by the letters S, D, R and I, on a Wednesday and probably as a result of consumption of too much caffeine and too much thinking. :-7

etymology of jargon

etymologically interesting

Anything self-referential is amusing (blame/praise starts with GEB - how can one describe systems purely if you have any use/reference of anything in that system in the description of the system - HAHAHAH!) like a joke or pun. Etymology of words about language, jargon: old french jargon garrire bird-chatter gargle. from http://www.etymonline.com/index.php?term=jargon

http://catb.org/jargon/html/introduction.html
http://catb.org/jargon/html/distinctions.html
"communication, and of inclusion, and of exclusion."
And for hacker jargon as a technical language there is a complex level of detail which does seperate it from being just a plainer slang. Not easy to learn/understand unless you learn the whole culture (shared consciousness), descriptive of complex/different ideas/models/thinking... which is very useful and important for facilitation of communication, quite like a real language.

Oh yeah! so elitist :) too.

"we have attempted to ensure that everyone's sacred cows get gored, impartially. Compromise is not particularly a hackish virtue, but the honest presentation of divergent viewpoints is."

This digression started with http://www.faqs.org/docs/jargon/H/hexadecimal.html
"correct term for base 10, for example, is ‘denary’"

   ~"etymologically pure"~   real live language   decimal style words   "too racy and amusing for stuffy IBM", (incorrect and inconsistent anyway! :))
2  binary                    binary
3  ternary or trinary        ternary
8  octonary                  octal                octaval
10 denary                    decimal              decimal
16 sendenary                 hexadecimal          sedecimal             sexadecimal 

Numerical Prefixes: http://phrontistery.info/numbers.html
numerical words in English (possibly common to many languages via maths ?) (mainly from greek and latin )
see Table 1: Latin-Prefixed Numerical Words
see Table 3: Latin Numerical Words: 13 to 1000

Numbers from 1 to 10 in over 5000 languages: http://zompist.com/numbers.htm

Arabic            wāhid  iŧnān ŧalāŧah 'arba`ah xamsah sittah  sab`ah  ŧamāniyyah tis`ah   `ašarah
Cantonese         jat    i     sam     sei      ŋ      lok     tshat   pat        kau      sap
Dutch             een    twee  drie    vier     vijf   zes     zeven   acht       negen    tien
English           one    two   three   four     five   six     seven   eight      nine     ten
French 	          un     deux  trois   quatre   cinq   six     sept    huit       neuf     dix
German            eins   zwei  drei    vier     fünf   sechs   sieben  acht       neun     zehn
Greek             éna    đío   tría    téssera  pénde  éksi    eftá    oxtó       ennéa    đéka
Hindi/Urdu        ek     do    tīn     cār      pā~c   chai    sāt     āth        nau      das
Irish             aon    dó    trí     ceathair cúig   sé      seacht  ocht       naoi     deich
Italian           uno    due   tre     quattro  cinque sei     sette   otto       nove     dieci
Japanese          ichi   ni    san     shi      go     roku    shichi  hachi      ku       juu
Mandarin(Beijing) yi     er    san     si=sz    wu     liu     qi      ba         jiu      shi
Polish            jeden  dwa   trzy    cztery   pięć   sześć   siedem  osiem      dziewięć dziesięć
Runic+            æinn   tvæiR þrīR    fiūriR   fæ:m   sæx     siū     ātta       nīu      tīu
Russian           odín   dva   tri     četýre   pyat'  šest'   sem'    vósem'     dévyat'  désyat'
Spanish           uno    dos   tres    cuatro   cinco  seis    siete   ocho       nueve    diez

Heh heh :) warhammer again in comics:
http://ars.userfriendly.org/cartoons/?id=20080403
http://www.weregeek.com/2008/03/03/

library-check.pl, postfix config to forward/relay mail to smtp server

1. Move across library-check.pl script from suse to ubuntu
2. enable mail from command-line and perl WWW::Mechanize script dependancies
3. solve postfix trouble (i.e. config postfix properly)
4. WAH! 5 + 6 books two accounts 5*23 + 6*29 days overdue. oops.


# Helpful reading:
Relaying Mail to a SMTP Server That Requires Authentication (Ubuntu/Postfix)
http://www.finefrog.com/2008/02/15/relaying-mail-to-a-smtp-server-that-requires-authentication-ubuntupostfix/
http://www.marksanborn.net/linux/send-mail-postfix-through-gmails-smtp-on-a-ubuntu-lts-server/
http://ubuntuforums.org/archive/index.php/t-67397.html
http://www.wormbytes.ca/articles/postfix-with-sympatico-relay
http://www.postfix.org/docs.html


O=/mnt/suseold
export http_proxy=
export https_proxy=
sudo apt-get install mailutils
sudo cpan -i WWW::Mechanize
sudo cp $O/usr/local/bin/library-check.pl /usr/local/bin/
sudo cp $O/etc/cron.daily/run_library-check.sh /etc/cron.daily/
sudo cp $O/etc/cron.daily/run_library-check-fionnuala.sh /etc/cron.daily/


# postfix setup took me a couple of hours - arg!
# simple postfix setup just to forward/relay mail to smtp server (which needs authentication)
# set some vars for use just in commands below
MYSMTPSERVER=www.smtpservername.com
MYACCOUNT=myaccount
PASSWORD=password
MYACCOUNT_SERVER=smtpservername.com
MYEMAIL=${MYACCOUNT}@${MYACCOUNT_SERVER}

# this was the thing I did last which started everything working (and uncomment myorigin = /etc/mailname in /etc/postfix/main.cf):
echo $MYSMTPSERVER > /etc/mailname

sudo vim /etc/postfix/mail.cf

# uncomment this:
myorigin = /etc/mailname

# add relay host with location of passwd info
relayhost = [<MYSMTPSERVER>]

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd
smtp_sasl_security_options = noanonymous

smtp_always_send_ehlo = yes
smtpd_sasl_local_domain =
smtp_sasl_tls_security_options = noanonymous

# comment this line out:
#myhostname = <MYACCOUNT_SERVER>


# store smtp authentication details for postfix use
sudo mkdir /etc/postfix/sasl/
echo "${MYSMTPSERVER} ${MYEMAIL}:${PASSWORD}" > /etc/postfix/sasl/sasl_passwd
postmap /etc/postfix/sasl/sasl_passwd
rm /etc/postfix/sasl/sasl_passwd # I think.
# but ACTUALLY I notice the hash did not seem to work? can still see info unhashed in /etc/postfix/sasl/sasl_passwd.db ?


/etc/init.d/postfix restart


# log of successful send:
tail -1550f /var/log/syslog |grep -v atkbd.c

Apr  4 02:08:29 <MYHOSTNAME> postfix/pickup[12695]: 6320E454092: uid=1000 from=<<MYACCOUNTNAME>>
Apr  4 02:08:29 <MYHOSTNAME> postfix/cleanup[12700]: 6320E454092: message-id=<20080404010829.6320E454092@<MYHOSTNAME>.localdomain>
Apr  4 02:08:29 <MYHOSTNAME> postfix/qmgr[12696]: 6320E454092: from=<<MYACCOUNTNAME>@<MYACCOUNT_SERVER>>, size=384, nrcpt=1 (queue active)
Apr  4 02:08:30 <MYHOSTNAME> postfix/smtp[12702]: 6320E454092: to=<<MYACCOUNTNAME>@<MYACCOUNT_SERVER>>, relay=<MYSMTPSERVER>[<MYSMTPHOSTIP>]:25, delay=1.2, delays=0.03/0.01/1.1/0.06, dsn=2.0.0, status=sent (250 OK id=1JhaPy-00073e-6f)
Apr  4 02:08:30 <MYHOSTNAME> postfix/qmgr[12696]: 6320E454092: removed

smtp auth failed
because smtp user is NOT <MYACCOUNTNAME>@<MYHOSTNAME>.localdomain
"550-Verification failed for <<MYACCOUNTNAME>@<MYHOSTNAME>.localdomain> 550-Unrouteable address"
This is solved by having smtp server name in: /etc/mailname

Apr  4 02:05:35 <MYHOSTNAME> postfix/pickup[12551]: 04E56454092: uid=1000 from=<<MYACCOUNTNAME>>
Apr  4 02:05:35 <MYHOSTNAME> postfix/cleanup[12556]: 04E56454092: message-id=<20080404010535.04E56454092@<MYHOSTNAME>.localdomain>
Apr  4 02:05:35 <MYHOSTNAME> postfix/qmgr[12552]: 04E56454092: from=<<MYACCOUNTNAME>@<MYHOSTNAME>.localdomain>, size=404, nrcpt=1 (queue active)
Apr  4 02:05:36 <MYHOSTNAME> postfix/smtp[12558]: 04E56454092: to=<<MYACCOUNTNAME>@<MYACCOUNT_SERVER>>, relay=<MYSMTPSERVER>[<MYSMTPHOSTIP>]:25, delay=1.2, delays=0.03/0.01/1.1/0.03, dsn=5.0.0, status=bounced (host <MYSMTPSERVER>[<MYSMTPHOSTIP>] said: 550-Verification failed for <<MYACCOUNTNAME>@<MYHOSTNAME>.localdomain> 550-Unrouteable address 550 Sender verify failed (in reply to RCPT TO command))
Apr  4 02:05:36 <MYHOSTNAME> postfix/cleanup[12556]: 2AB34454093: message-id=<20080404010536.2AB34454093@<MYHOSTNAME>.localdomain>
Apr  4 02:05:36 <MYHOSTNAME> postfix/bounce[12559]: 04E56454092: sender non-delivery notification: 2AB34454093
Apr  4 02:05:36 <MYHOSTNAME> postfix/qmgr[12552]: 2AB34454093: from=<>, size=2580, nrcpt=1 (queue active)
Apr  4 02:05:36 <MYHOSTNAME> postfix/qmgr[12552]: 04E56454092: removed
Apr  4 02:05:36 <MYHOSTNAME> postfix/local[12560]: 2AB34454093: to=<<MYACCOUNTNAME>@<MYHOSTNAME>.localdomain>, relay=local, delay=0.02, delays=0.01/0.01/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)
Apr  4 02:05:36 <MYHOSTNAME> postfix/qmgr[12552]: 2AB34454093: removed