Sunday, 20 October 2019

Python Logging

For those of us that start a program with "print" statements to get our bearings, the Python logging library is a drop-in replacement. After creating the logger, you can replace "print" statements with logger.debug('same text as before') or use suitable levels from debug, info, warning, error, and critical. You quickly get logging that can be configured to different levels you can use from debugging during development and clear output for the program running. There's also a built-in handler on logging exceptions to cleanly log exception info during exception handling. The library and cookbook are both clear and illustrative so check these links out:

Python 3 Library - Logging

Python 3 Logging Cookbook

I mentioned shell logging last time here:

I'm a Logger and I'm Okay

The other tool I use all the time in Python is the "assert" call which raises an error with message given. For example to sanity check the resulting size of a blob is matching your file size:

# sanity check the resultant blob is the same size as what's in the database
assert (os.path.getsize(outpath) == len(imgblob)), "File size %d does not match blob size %d for image %d in %s saved to %s" % (os.path.getsize(outpath), len(imgblob), imgid, args.table, args.archivepath)
There's every opportunity to properly handle program output and errors.


Sunday, 3 March 2019

I'm a [logger] and I'm OK!

The "logger" command can send a text string to the syslog. #why-didn't-I-know-that-already

localadmin@mypc:~$ logger Mark
localadmin@mypc:~$ sudo tail /var/log/syslog
Oct 31 15:23:40 mypc systemd[16714]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Oct 31 15:23:40 mypc systemd[16714]: Reached target Timers.
Oct 31 15:23:40 mypc systemd[16714]: Listening on GnuPG network certificate management daemon.
Oct 31 15:23:40 mypc systemd[16714]: Listening on D-Bus User Message Bus Socket.
Oct 31 15:23:40 mypc systemd[16714]: Reached target Sockets.
Oct 31 15:23:40 mypc systemd[16714]: Reached target Basic System.
Oct 31 15:23:40 mypc systemd[16714]: Reached target Default.
Oct 31 15:23:40 mypc systemd[16714]: Startup finished in 44ms.
Oct 31 15:23:40 mypc systemd[1]: Started User Manager for UID 1000.
Oct 31 15:23:46 mypc localadmin: Mark 
In other news I updated my page on useful PC tools


Tuesday, 30 October 2018

Working with DNS settings in systemd-resolved in Ubuntu

In troubleshooting some DNS name resolution issues I started to get more familiar with systemd-resolved in Ubuntu. Specifically if you look at the traditional /etc/resolv.conf file it says something like this:
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
Hah hah! Subtly clever for any old hands in networking because "53" is the standard DNS port so is a little clue to look for something specific.

Go ahead though - run "systemd-resolve --status", it doesn't require root, and it shows you a lot of info (the IPs of name servers have been change to protect the innocent...)
localadmin@ca-yvr-adm2:~$ systemd-resolve --status
         DNS Servers:
          DNS Domain:
          DNSSEC NTA:

Link 1 (eno1)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

The config file is easily found in "/etc/systemd/resolved.conf" and it's quite short and simple and it should look familiar if you have used other systemd configurations before. The configuration out-of-the-box will be blank with all options commented out. I wanted to add additional search domains to the Domains line, in the same space-delimited way you would traditionally do the search line in your resolve.conf.
#FallbackDNS= example.local

Restart the same as you would any other systemd tool, and then re-check your resolved status
sudo systemctl restart systemd-resolved.service 
systemd-resolve --status

OR check your /etc/resolve.conf file because if your only change is to modify the search domains, it also appears there for the resolver to work normally.
         DNS Servers:
          DNS Domain: example.local
          DNSSEC NTA:

Normally you would be done... Here's a couple bonus tricks that may arise.

You can modify DNS settings by interface - furthermore this is handy for testing DNS changes and reverting before making the change in the config file. Check out systemd-resolve --help 

The multicast service may conflict with a .local domain. The symptom I had was that I could resolve a short name like "pc" but could not resolve an FQDN like "pc.example.local". If you are using .local and finding odd DNS resolution results, edit your nsswitch.conf  and move "dns" earlier than the mdns (keep it after "files" though to avoid breaking your hosts file).

I've tried to make this a quick and useful blurb on how to use systemd-resolved and get pointed in the right direction because the documentation available wasn't simple for simple cases - there is certainly a lot of tuning you can do with the resolver tools.


Wednesday, 3 October 2018

Archiving Files (E.g. Deleting Stuff) is My Super Power!

This week one of the things I was working on was archiving a lot of files on a file in order to prune it down ahead of migrating that share to a new location.

In particular this share while only ~100GB has > 3M files on it! Any basic operations like checking folder sizes, applying ACLs, etc are very slow on xM files. Looking at the share with WinDirStat I found that there's a large number of folders each with 20-50K files in each. Furthermore the majority of these files were old (years) and not actively used so we decided to Archive (e.g. Delete) the contents of these folders.

I started by using 7-Zip and there's an option on the Add to Archive screen to "Delete files after compression":

And that's pretty good! ... But not if you've got hundreds of folders to process.

Instead, with a bit of PowerShell, we invoke 7-Zip from a script and use the -sdel switch to remove files after they've been archived. This PS we used to simply to stuff each sub-folder into an archive.

Get-ChildItem . -Directory |  ForEach-Object  {
 $Archive = $_.Name + ".zip"
 $Folder = $_.Name + "\"
 &"C:\Program Files\7-Zip\7z.exe" a $Archive $Folder -sdel

Thursday, 16 August 2018

Let's Encrypt on Blogger

I can't say I fiddle too much with the settings in Blogger, it's kinda "set it and forget it" stuff BUT sometime I think recently the team made it so you can enable SSL certs for custom domains on Blogger and it signs up a cert for you and everything.

In short it's so stupid simple just go do it and do it now

  1. Go to basic settings 
  2. Change HTTPS to Yes
  3. Are we done yet? Why yes, yes we are.

Optionally once the cert has generated (it's not instant) you can also turn on redirect to SSL which again, why not? It's just the next tick box

This is how security should work; it works and its easy. I guess it could be on by default and pushed out but really, just click "security activate!"


Friday, 14 July 2017

Automation with RT CLI

Ticket automation in Best Practical's RT is by far the easiest with the RT CLI.  I shan't re-hash the documentation but will give an example because it wasn't obvious just how easy it is. Like in Scouts, Be Prepared. A bit of prep makes the RT CLI simple to work with.
  1. Setup your .rtrc and .bashrc as a one-off so you can invoke the RT CLI directly
  2. Build a search query in the regular RT Web UI
  3. Automate the function 
Find the "rt" binary:
[support-email@rt ~]$ locate */rt
Add it to your PATH in .bashrc:
export PATH=$PATH:/opt/rt4/bin
 Setup your .rtrc file with your credentials rather than giving them on the command line:
[support-email@rt ~]$ cat .rtrc
server http://rt/
user me
passwd xxx
auth rt
Now you can already do some stuff like the examples from the RT CLI page in the wiki:
[support-email@rt ~]$ rt show user/ggee
id: user/832782
Password: ********
Name: GGee
RealName: G Gee
Privileged: 1
Disabled: 0
CF-Employee Department: Applications Software
The last "prep" thing is to create your search criteria. This is far easier in the Web UI like you can build up your Search and then when you click Advanced you can copy that Query text directly and test it out from the CLI:
[support-email@rt ~]$ rt ls -i -q "'Corp Support'" "Status = 'stalled' AND Told < '-1 week'"
Now you're ready for some automation.

  • The use of "-i" gives the output in a suitable format for processing
  • The "-q" option specifies a queue and you need to use quotes (') around names with spaces in them, hence on the CLI you get "'Corp Support'"
The above query is Searching for Stalled tickets which haven't been touched (Told) in over a week. We want to change such tickets to Open so that staff pick up these tickets. For this we can setup a job with cron which pipes the tickets found in a search into an rt edit command.
# un-stalls support tickets NOTE: requires valid creds in .rtrc
@daily /opt/rt4/bin/rt ls -i -q "'Corp Support'" "Status = 'stalled' AND Told < '-1 week'"  | rt edit - set status=open
You can automate all kinds of functionality whether routine activities like this example, or to build helper scripts for large operations like to populate some new custom field or otherwise.

Wednesday, 5 July 2017

Data Retention and Percona Archiver

Data retention can be a bit touchy but when the alternative is to let tables grow by GB per week or per day, sometimes you just got to pick an upper limit. In my experience, suggesting something to stakeholders helps to get things rolling.

Magically I've recently "discovered" the Percona Archiver - I've been rolling my own for far too long. This tool is well documented and I shan't repeat the documentation other than to give an example along with some tidbits.

The archiver can move records to a destination table (the --dest option) OR to a file (the --file option). Both are useful and I'll show the file one because that's The Final Solution other than outright launching the nukes with --purge. Give a Select criteria (the --where option) and consider to include table maintenance (--optimize) if you are moving a lot of data.

For clarity: pt-archiver does a DELETE for each record it archives. 

# dump table from N months ago
DELYRMO=`date --date "$DELAGE months ago" +%Y%m`

# do not (!) overwrite file with something (-s) in it already
if [ ! -s "$BAKFILE" ] ; then
        pt-archiver --source h=localhost,D=$DB,t=$table --file archive-$table.txt --where "calib_aimextractor_id > 0" --optimize s --statistics
        echo "$BAKFILE has something in it, dump has been SKIPPED"
        exit -1
This is a drastically simplified script from what I used to do.

  1. Set the data retention which in this case is 6 months. The "date" command is useful for generating dates or parts thereof like the year, month, day, week whatever you need for both file names and search criteria. 
  2. File target should be some file system location locally or NFS. The file format is suitable for LOAD DATA INFILE
    • Gotcha! Loading data files is a risky thing to do and disabled by default in MySQL. Typically load the data to a non-production server, then manually extract the relevant records and insert them back into prod.
  3. Sanity check you're not stomping a file that's already there. I prefer to be safer than sorrier.
  4. Credentials should be in .my.cnf 
    • Seems obvious when you know to do it, but don't put user creds in scripts, dumbo! I did that too often :(
  5. Gotcha! If using --dest table instead of a file target, specify the host (h) and database (D) because otherwise pt-archiver makes some assumptions which may be very wrong
  6. Optimize your source (s) especially if a large number of rows are being pulled. Consider to also use destination (d) 
There's lots more guidance in the documentation and from other users Online. Some like to process larger numbers of records concurrently like with --limit and --bulk-delete, but the defaults (1 record) have been good to me as this runs relatively fast. Likewise there's options to check your slaves don't get far out of sync which again default behaviour is fast enough for me, but there's lots of powerful options to tune pt-archiver.

Take backups, test, test, test and you shouldn't need Good Luck :)

Popular Posts