Friday, 14 July 2017

Automation with RT CLI

Ticket automation in Best Practical's RT is by far the easiest with the RT CLI.  I shan't re-hash the documentation but will give an example because it wasn't obvious just how easy it is. Like in Scouts, Be Prepared. A bit of prep makes the RT CLI simple to work with.
  1. Setup your .rtrc and .bashrc as a one-off so you can invoke the RT CLI directly
  2. Build a search query in the regular RT Web UI
  3. Automate the function 
Find the "rt" binary:
[support-email@rt ~]$ locate */rt
/opt/rt4/bin/rt
Add it to your PATH in .bashrc:
export PATH=$PATH:/opt/rt4/bin
 Setup your .rtrc file with your credentials rather than giving them on the command line:
[support-email@rt ~]$ cat .rtrc
server http://rt/
user me
passwd xxx
auth rt
Now you can already do some stuff like the examples from the RT CLI page in the wiki:
[support-email@rt ~]$ rt show user/ggee
id: user/832782
Password: ********
Name: GGee
EmailAddress: GGee@example.com
RealName: G Gee
Privileged: 1
Disabled: 0
CF-Employee Department: Applications Software
The last "prep" thing is to create your search criteria. This is far easier in the Web UI like you can build up your Search and then when you click Advanced you can copy that Query text directly and test it out from the CLI:
[support-email@rt ~]$ rt ls -i -q "'Corp Support'" "Status = 'stalled' AND Told < '-1 week'"
ticket/314370
ticket/315571
Now you're ready for some automation.

  • The use of "-i" gives the output in a suitable format for processing
  • The "-q" option specifies a queue and you need to use quotes (') around names with spaces in them, hence on the CLI you get "'Corp Support'"
The above query is Searching for Stalled tickets which haven't been touched (Told) in over a week. We want to change such tickets to Open so that staff pick up these tickets. For this we can setup a job with cron which pipes the tickets found in a search into an rt edit command.
# un-stalls support tickets NOTE: requires valid creds in .rtrc
@daily /opt/rt4/bin/rt ls -i -q "'Corp Support'" "Status = 'stalled' AND Told < '-1 week'"  | rt edit - set status=open
You can automate all kinds of functionality whether routine activities like this example, or to build helper scripts for large operations like to populate some new custom field or otherwise.

Wednesday, 5 July 2017

Data Retention and Percona Archiver

Data retention can be a bit touchy but when the alternative is to let tables grow by GB per week or per day, sometimes you just got to pick an upper limit. In my experience, suggesting something to stakeholders helps to get things rolling.

Magically I've recently "discovered" the Percona Archiver - I've been rolling my own for far too long. This tool is well documented and I shan't repeat the documentation other than to give an example along with some tidbits.


The archiver can move records to a destination table (the --dest option) OR to a file (the --file option). Both are useful and I'll show the file one because that's The Final Solution other than outright launching the nukes with --purge. Give a Select criteria (the --where option) and consider to include table maintenance (--optimize) if you are moving a lot of data.

For clarity: pt-archiver does a DELETE for each record it archives. 


# dump table from N months ago
DELAGE=6
DELYRMO=`date --date "$DELAGE months ago" +%Y%m`
TABLE="calib_aimextractor_log_history_$DELYRMO"
BAKFILE="/mnt/backups/mysqlserver/archive-$DB-$TABLE.txt"

# do not (!) overwrite file with something (-s) in it already
if [ ! -s "$BAKFILE" ] ; then
        pt-archiver --source h=localhost,D=$DB,t=$table --file archive-$table.txt --where "calib_aimextractor_id > 0" --optimize s --statistics
else
        echo "$BAKFILE has something in it, dump has been SKIPPED"
        exit -1
fi
This is a drastically simplified script from what I used to do.

  1. Set the data retention which in this case is 6 months. The "date" command is useful for generating dates or parts thereof like the year, month, day, week whatever you need for both file names and search criteria. 
  2. File target should be some file system location locally or NFS. The file format is suitable for LOAD DATA INFILE
    • Gotcha! Loading data files is a risky thing to do and disabled by default in MySQL. Typically load the data to a non-production server, then manually extract the relevant records and insert them back into prod.
  3. Sanity check you're not stomping a file that's already there. I prefer to be safer than sorrier.
  4. Credentials should be in .my.cnf 
    • Seems obvious when you know to do it, but don't put user creds in scripts, dumbo! I did that too often :(
  5. Gotcha! If using --dest table instead of a file target, specify the host (h) and database (D) because otherwise pt-archiver makes some assumptions which may be very wrong
  6. Optimize your source (s) especially if a large number of rows are being pulled. Consider to also use destination (d) 
There's lots more guidance in the documentation and from other users Online. Some like to process larger numbers of records concurrently like with --limit and --bulk-delete, but the defaults (1 record) have been good to me as this runs relatively fast. Likewise there's options to check your slaves don't get far out of sync which again default behaviour is fast enough for me, but there's lots of powerful options to tune pt-archiver.

Take backups, test, test, test and you shouldn't need Good Luck :)

Saturday, 14 May 2016

Ubuntu automysqlbackup

There is a script called "automysqlbackup" which is a pretty straightforward shell script wrapping up routine MySQL backups. The Ubuntu package is mostly preconfigured so you would not necessarily even have to modify the stock configuration.
  • Gets the maintenance user from "/etc/mysql/debian.cnf" for credentials
  • Dynamically determines what databases are on the system
  • Has a default schedule and backup path (/var/lib/automysqlbackup)
You should consider changing a couple of the defaults found in "/etc/default/automysqlbackup".
  • BACKUPDIR to preferred backup path
  • MAILADDR to an appropriate recipient in case there are errors
It does not remove old backup files so you may want to make a basic script which does remove them. There's a "PREBACKUP" variable so you can hook in such a script. I like this because it runs before your backup so you don't accidentally nuke your fresh backups and keeps things simple:
  • PREBACKUP="find $BACKUPDIR -mtime +90 -delete"
Finally, remember to copy your backups offsite if appropriate. "rsync" to some remote system or otherwise. Use the "POSTBACKUP" script - again a good hook here because it will push out your backups right away after they have been created.

Ubuntu Man page:
http://manpages.ubuntu.com/manpages/wily/man8/automysqlbackup.8.html

Monday, 21 September 2015

CentOS 7

Having started an install for CentOS 7, this is my first time working with the Red Hat Enterprise Linux 7 based system and they've done a few things I'll have to learn.

Goodbye, Sys V init! What an era there's been with init scripts. The newer "systemd" system and service manager replaces the init system along with RedHat's chkconfig and similar tools. The "systemctl" command is kindof similar to "chkconfig", but takes the command name first and the new style service name second:


# systemctl status nfs-server.service

Overview of systemd for RHEL 7
https://access.redhat.com/articles/754933

The other is the new "firewalld" which provides more of set of front-ends to iptables. The command-line tool, firewalld-cmd, can generate the settings changes like to open ports. As the RedHat docs say, this mechanism can load firewall rule changes instead of dumping the whole rule set so you keep open connections and stats.

# firewall-cmd --zone=public --add-port=80/tcp --permanent
# firewall-cmd --reload

Using Firewalls
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/sec-Using_Firewalls.html

I'm sure there's more but mostly cosmetic like how the installer works, which packages are bundled or not (like bind-utils not included in a base install? interesting). Sometimes hard to figure out what to do in a new system when there's big changes that aren't just drop-in replacements for older tools.

Ciao,
Arch

Thursday, 30 July 2015

Update to Password Generator

A couple years ago I created a simple password generator which was just a wrapper for "APG". While it worked fine for me, I did find that I needed passwords which fit a specific pattern so I've added support for the "mode" argument to APG.

https://alia.thenibble.org/passwords/

Voila!

Tuesday, 9 September 2014

Free Signed SSL Cert

I run a few things on an Ubuntu server sitting under my desk at home and have used self-signed certificates usually, but there are free Certificate Authorities including startssl.com which are readily available for personal use like running your ownCloud.

Here's a good write-up on how to do this in Ubuntu:

https://gist.github.com/mgedmin/7124635

And the results look swell:SSL Labs

/etc/apache2/sites-enabled/ssl

NameVirtualHost *:443
<VirtualHost *:443>
        SSLEngine On

        SSLProtocol all -SSLv2
        SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM
        SSLCertificateChainFile /etc/apache2/ssl/sub.class1.server.ca.pem
        SSLCertificateFile /etc/apache2/ssl/alia.thenibble.org.crt
        SSLCertificateKeyFile /etc/apache2/ssl/alia.thenibble.org.pem

        DocumentRoot /var/www/

        ServerAdmin webmaster@thenibble.org

...

Saturday, 12 July 2014

Home network

I'm putting together a diagram of my home network, it may take me a couple passes... Here's a quick drawing made on my tablet.


Popular Posts