“Windows” is full of bloat

It’s been some time since I’ve had to do some work on a Windows computer, in particular lots of software installs and uninstalls to cleanup a slow computer.

Now before you jump the gun, I’m not talking about Windows the operating system being full of bloat, although it may be. I’m also not talking about the bloat that comes installed on OEM installs.

What I am talking about, is how so much third party free windows software is full of bloat. Just about every installer now wants to install another piece of software with the piece you downloaded, ether a toolbar or another “great” piece of software they recommend to you. And then when you uninstall a piece of software, just about every piece of software then insist on opening a webpage saying how sorry they are that you uninstalled the software and would you please do this 5 minute survey telling them why blah blah blah.

And even my favourite antivirus Avast has started doing it. First, the installer has jumped from 20Mb a few years ago to over 100Mb now. And now when you install it, if you aren’t watching out it’ll try and install Google Drive, and once it is installed it keeps popping up asking you to install the great security extension for chrome, blah blah blah.

What ever happened to nice free software that wasn’t packed full of bloat, and that let you install and uninstall just that software that you asked for without trying to push other software on you? I’ll happily keep my Linux world, where software does what you ask, and nothing more.

And if you live in the Windows world, seriously, I think it’s well past time to get out, but you can still get out now!

WordPress Server Maintenance

Wow. Somehow I missed when this amazing tool was released. wp-cli. Just wow.

I have a bunch of scripts I use to keep all the WordPress installs up to date on my server, it finds all the installs, then it downloads the latest version, checks each install for the installed version and updates them by extracting the files correctly, ensuring user permissions etc. However, that only keeps the core updated, I still need to rely on all my customers to keep their plugins updated. This is ok, except when a plugin has a security issue and needs to be updated ASAP. Recently, we had just that situation with the WP-Super-Cache plugin. So I modified my scripts, and now they update that particular plugin. But it would be a pain to write a script that updated every plugin, so I’m stuck logging into a number of sites I maintain, updating all the plugins regularly, and then hoping for the sites that the customer maintains, they do the same.

Enter, wp-cli. The tool I’ve been dreaming of for a long time. Simply install, then using the basic structure of my script (the part that finds all installs, verifies they are valid installs, and executes commands as the user who owns the install), I can now run any of the awesome wp-cli commands on all the sites I host! Server maintenance just got a whole lot easier (as long as no plugin updates break things)

For the really lazy, here is my code for updating all plugins, using the wp-cli scripts. It uses sudo, so make sure you know what you are doing. You may also have to tweak how locate finds things, as by default it won’t show you files you can’t access.

#!/bin/bash
 
# Just find all installs and try and run tools in them
# find wp-config.php files
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
installs=$(locate -r wp-config.php$)
 
for conffile in $installs
do
# goto root wp dir as user
    wpdir=$(dirname "$conffile");
    pushd $wpdir > /dev/null || exit
 
    echo "Checking $wpdir..."
 
    ## Check we actually have a wordpress install
 
    if [[ ! -f "wp-includes/version.php" || ! -f $conffile ]]
        then echo "$wpdir doesn't appear to be a wordpress install, skipping..."
        continue
    fi
 
    # Get username for tool
    username=$(stat -c %U $conffile)
 
    # run tool commands
    sudo -u $username -- wp plugin update-all
 
    popd > /dev/null
done

Submitting Empty Checkboxes when in Array

In HTML, we often use a nice feature for POSTing a form, or submitting a form, where we have lots of items the “same”. In the Grase Hotspot project, we have groups for example, and we can dynamically add and delete groups, but need a complete form for each group with it’s settings. So we use an “array” to submit the values. Simple put, we have a number of <inputs> with the same name, but with square brackets on the end. i.e. <input type=”text” name=”groupprice[]“/>, and we have multiples of these. Thanks to page ordering, we can have a number of different arrays as well, and can associate all the group values together thanks to their positions in each array.

However, this breaks when we are submitting checkboxes, because unchecked checkboxes won’t submit. When unchecked, they are a form element that isn’t successful, and only successful form elements are submitted. That’s fine, except we are relying on the array behaviour to match elements together, and when a checkbox isn’t submitted, it doesn’t even take up an array position, it’s just absent, which means all following checkboxes on the page are now in the wrong spot in the array and not matched with the other arrays.

So normally when we submit input fields and some items have been left blank, they still consume a spot in the array, like so.

array(4) {
  [0]=>
  string(6) "Item 1"
  [1]=>
  string(0) ""
  [2]=>
  string(0) ""
  [3]=>
  string(6) "Item 4"
}

However, the checkboxes (which have a value on On normally when checked), would appear like so (with Item numbers instead of “On” for clarity)

array(2) {
  [0]=>
  string(6) "Item 1"
  [1]=>
  string(6) "Item 4"
}

There are solutions out there, however none work very well for dynamic adding and creating “groups” of data, including checkboxes. Given that the dynamic creation and deletion of those groups is with Javascript, I’ve turned to a small javascript snippit to get it working. Hopefully I’ll find a nice way to have it fallback when javascript is disabled.

I found the solution I’m using at http://www.dwright.us/?p=472 however couldn’t get it working well. A bit of fiddling and working on the solution in the comments and I came up with the following snippit that works nicely for me.

$(document).ready(function(){
    // do when submit is pressed
    $('form').submit(function() {
 
        $('input:checkbox:not(:checked)').each(function() {
                console.log($(this).attr('name'));
                $(this).before($('<input>')
                .attr('class', '_temp')
                .attr('type', 'hidden')
                .attr('name', $(this).attr('name')));
                // .val('off'));
        });  
    });
 
});

One of the main things I changed (once I made it work) was not having it set the “extra” inputs to the value of “Off”, but leaving them as an empty value. This is because in my code I run the incoming arrays through array_filter which removes all the empty ones (and maintains the array indexes so the order and position is still there).

When did ‘Linux’ start moving away from it’s simplicity?

Today I had the joy of trying to change a users desktop environment from Gnome, to Cinnamon, via ssh. At first I thought it would be easy, change the default desktop environment in lightdm.conf and restart. Fail. Ok, so where is a users desktop environment preference stored? .dmrc, or at least I thought so. I even logged in with a brand new user and confirmed that it saved the users desktop preference in ~/.dmrc. Except, if I changed .dmrc for a user, it just got overwritten with the old contents at their next automatic login. (Remember, ssh, so can’t use the gui to change what was selected in the lightdm login screen).
Wha?!!? Surely not a dconf/gconf setting somewhere. Search search search. Still no luck. Eventually I discover a service called “AccountsService” or something along that name. It stores the users “.dmrc” contents in /var/lib/AccountsService/users/ with a file for each user, which believe it or not, can’t be changed by the user! Arg! (And I believe you need to kill accountsservice to be able to change the contents of the file and have it actually honored, I tried just changing it, but ended up changing it then rebooting to get it to work)

This is stupidity! Have a daemon, that one of it’s tasks is to tell the “display manager” what the users preferred desktop environment preference is, that stores it in a place the user can’t change, without talking to the daemon! Oh, and write the contents of that file out to .dmrc at login, but don’t bother reading from that file.

Doing some more digging, AccountService is for querying and manipulating user account information via D-Bus, essentially replacing the useradd, usermod and userdel commands. I can’t find a “homepage” for it, the homepage listed is it’s source code repo. (http://cgit.freedesktop.org/accountsservice/)  (NB: Some more digging finds this http://www.freedesktop.org/wiki/Software/AccountsService as the homepage)
I personally don’t mind D-Bus, it services its purposes, but here is an instance where it looks like someone has gone to the trouble of writing a piece of code, to complicate some that use to be a simple stat/open of a file, that the user was in total control of.

For the record, lightdm will work without AccountsService, and I believe then it will honor ~/.dmrc
Also, accountsservice isn’t in Debian stable, but is in Debian testing, and gdm3 and gnome depend on it. (So in other words Gnome3 depends on it)

I just don’t understand why we need to reinvent the wheel, take something that is so simple (and fits in the Unix “philosophies” of everything is a file, and that file formats should be simple text based) and turn it into something that user no longer has control over, for something that sets their preference!

Read http://blog.ngas.ch/archives/2011/12/13/the_destructive_desktop__mdash_linux_in_trouble/index.html for some more thoughts on dbus.

Belkin? Rtkit?

While attempting to remotely debug a linux machine today, I was first encountering a strange problem. Any process that took more than about 1/2 second to complete, would freeze. top, ps, lsmod, tail -f, the list goes on. For example, trying to run dmesg and display it’s output would freeze, but dmesg into a file, and it would complete!

After much digging, I eventually found that rtkit (rtkit-daemon) is constantly trying to make pulseaudio operate at realtime. In reality, we don’t need our audio to operate in realtime as most modern computers can keep up with video playback just fine. For the few people we actually want near real time audio (say, people recording multitrack stuff), then they can enable it themselves. Disabling rtkit (actually, uninstalling it as it appears to be started in dbus stuff), seems to have solved that problem.

The next problem was a strange DNS response. A dns request through the Belkin modem, to this server (purewhite.id.au) would return 10.45.41.175 instead of 175.41.45.10. I know what reverse DNS is, but this is reverse IP! Belkin is returning the ip in reverse!! (Or backwards if you desire). A quick check reveals that this relatively new modem, hasn’t got any new firmware for it (and it’s firmware is over 1 year old). Apparently, someone else had this problem and belkin told them to just hard code the ip’s in your hosts file for the hosts that are being returned wrong! I believe it was also a Belkin modem that would return strange results when you did an AAAA request (ipv6).
So if you have a Belkin, maybe force your computer to use your ISP’s DNS servers directly, rather than the routers. (Or take it back to the shop, because after all, it is faulty)

Nginx, PHP-FPM, WordPress, Super Cache

So recently I’ve been exploring the alternative world of Nginx instead of Apache, and PHP-FPM instead of mod_php. There are plenty of tutorials on the net for getting all of this setup, however not that many are up to date anymore for the Super Cache stuff. Hopefully what I present here will be a more up to date config, that is also mostly secure compare to a good number of ones on the net (to do with passing non PHP files to the php interpreter).

Firstly, my Nginx config for this very blog.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
server {
  server_name www.tim.purewhite.id.au;
  rewrite ^/(.*) http://tim.purewhite.id.au/$1 permanent;
}
 
server {
	server_name tim.purewhite.id.au static.tim.purewhite.id.au;
	root /home/tim/domains/tim.purewhite.id.au/public_html;
 
	access_log /var/log/nginx/tim.purewhite.id.au_access_log;
	access_log  /var/log/nginx/default.access.log host_combined;
	#access_log  /var/log/nginx/uri.log host_combined_uri;
	error_log /var/log/nginx/tim.purewhite.id.au_error_log;
 
	index index.php;
 
	location / {
 
		if ($http_cookie ~ "comment_author_|wordpress|wp-postpass_") {
			rewrite ^/(.*) /loggedin$1 last;
		}
		try_files $uri
		/wordpress/wp-content/cache/supercache/$http_host/$uri/index.html
		$uri/
		 /index.php;
	}
 
	location /loggedin {
		internal;
		rewrite ^/loggedin(.*) /$1 break;
		try_files $uri $uri/ /index.php;
	}
 
 
	location ^~ /code {
	        proxy_set_header Host $host;
	        proxy_set_header X-Forwarded-Server $host;
	        proxy_set_header X-Forwarded-Host $host;
	        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_pass http://127.0.0.1:8080/code/;
	}
 
	location ~* \.(ico|css|js|gif|jpe?g|png)$ {
		expires 1w;
		break;
	}
 
 
 
	fastcgi_intercept_errors off;
 
	location ~ \.php {
		try_files $uri =404;
		include fastcgi_params;
		fastcgi_pass   127.0.0.1:9002;
	}
 
	include drop;
}

The first thing to notice is line 1-4. This simply redirects everyone from www.tim.purewhite.id.au to tim.purewhite.id.au. Simple as that.

Next we define the server and the root document path. Still very standard. Then we define access logs, for various reason I’m logging to more than one place, but that’ll change once everything is finished.

Line 15 is boring, we just define the “index index.php” so that if you access a directory it will load index.php or give you a 404 (which it won’t because of things further down).

Now for the fun. Lines 19-21. These catch a logged in user and send them on an internal redirect down to lines 28-32. This is so we don’t serve cached content to logged in users. That little snippit is thanks to a post at http://permalink.gmane.org/gmane.comp.web.nginx.english/15664.

However, there was a problem in the rest of the code. Thanks to a post at http://wordpress.org/support/topic/lack-of-nginx-support-from-wp-super-cache I realised we needed to test if the cache was being used or not. So I added the extra logging and discovered it wasn’t. I quickly worked out what the problem was. The code at lines 22 – 25 had the middle 2 lines swapped around. So “$uri/” was before the supercache line. What this mean was that it would try if the $uri was a directory, and to load a directory it would try index.php (due to the index line) and so would end up loading wordpress through index.php. However, if we try the supercache line first, we find the cache file and so don’t need to load indexes.

And just like that, magic, it works! We use supercache files for normal users, and if a cache file doesn’t exist, we load wordpress like normal!

I’m also looking at how we run Nginx and PHP-FPM. I have heard of a few ways, one being that root runs a Nginx as user nginx or nobody, and each user runs their own Nginx which we proxy to from the main one. (And users run their own PHP-FPM as well). This sounds like a lot of work, very complicated, but yes, it gives you absolute security as only the user can access his web docs and scripts, and everything runs as that user. No one else’s php process can load your config file to discover your database passwords.

Another way of running it is with Nginx as a nginx/nobody/www-data user, and each user run their own php-fpm but give the nginx/nobody/www-data user read only access to the web directory. If done correctly, this can actually be very secure. First, (as root) you chgrp all the files and directories in the users doc root (htdocs, www, public_html etc) to the user nginx will run as. Ideally, you also only allow them read access (so `chmod g+rX,g-w -R public_html` will give them access to read, but not write). You then set the gid bit on the directory; `chmod g+s public_html` (and do this for any directories that already exist underneath). Now any files the user creates underneath the public_html dir will be readable to the nginx user, so nginx can serve static files. Now running php-fpm as each user (I use php-fpm with a pool per user), the php process can read all the files that user can, so only the users own php process can read their config files with the password in it! And it also means that files you upload (i.e. wordpress media files) will be owned by the user, not by www-data or what ever the web user is. This is SO much better than Apache and mod_php, and easier than suExec with mod_php.

Once I have more of my domains moved to Nginx, I’ll do a report on memory and cpu usage.

Munin cgi graph timing out

A problem that has given lots of people problems, caused me issues yesterday. I have munin using munin-cgi-graph to create the graphs on demand due to me not often viewing the graphs. A few days ago I had a server issue that caused apache to lock up (I think a process ran away with my RAM which caused swapping and apache to lock up.) Once I apache running again, I wanted to check the munin graphs to see what the system looked like during the lockup (which killed a number of processes due to out of memory conditions). However, the graphs wouldn’t generate and the cgi was timing out without sending any data.

Timeout waiting for output from CGI script /usr/lib/cgi-bin/munin-cgi-graph
Premature end of script headers: munin-cgi-graph

I’m not alone ether. http://wiki.kartbuilding.net/index.php/Further_issues_upgrading_to_Lenny#munin_with_cgi and http://forum.linode.com/viewtopic.php?t=5171%3E both had issues. More googling still didn’t find an answer so I tried to debug the perl cgi. After using CPAN to get Devel:Trace installed, I discovered the cgi was sitting waiting for a semaphore flag that it uses to ensure no more than a certain number of munin-graphs are running at once. This is great, except when a crash has caused this semaphore to be stuck at the maximum so no more munin-graph processes get started ever!

There are 2 solutions. The first is simple, reboot. The second is also simple, clear the semaphore flags manually. ipcs is the command to show the flags and ipcrm is the command for removing the semaphores. Check the man pages for information on the correct syntax.

Day 13

I think today is Day 13.

So finally I’ve had a chance to sit down and nut this out. First things first, getting hello world to run on the iPhone. Finally I worked out how to self sign a certificate, and get Xcode to build it, then using a custom script I found on the net, sign the code so the iPhone will run it. Yay! Hello World runs on the iPhone!

Next step. Get a GPS app running on the iPhone. After following a pretty good tutorial from http://www.vellios.com/2010/08/16/core-location-gps-tutorial/ I finally got a GPS test app running. (After making some changes so it would run on iOS 3.1). However, this is the end of the good news. So far, as I suspected, the iPhone 3GS isn’t performing well on the GPS. Accuracy of 1km isn’t good enough, that can be achieved with just network location! We need down to about 10m. I’m thinking maybe I will need to upgrade to a newer iOS to see if it’s an issue with the hardware or the software. However I want this app to run on iOS 3.1, so am hesitant to do any upgrades.

The last of the good news is that the Objective-C is starting to make some sense to me, and I’ll now work on learning the major differences, and some more subtle ones, between Objective-C, C++ and C.

Day 6

So not much has happened over the weekend due to it being very busy. Today the mini-DVI to HDMI adapter arrived, so I could finally “dock” the Macbook at my desk. After “docking” it with my monitor, keyboard and mouse, it suddenly feels less like a laptop and more like a desktop! More screen realestate is a big bonus. But being able to just pick it up, pull out a few cables and take it with me, tether it to my mobile via wifi and work on the go, just makes such a difference to having a separate desktop and laptop!

As I already used a password manager under Linux, I wanted an application that was compatible with my existing password database. Thankfully, Password Gorilla runs on OS X, and all the other platforms, and reads the same password database as pwsafe does. So while I was out today, I finally migrated a number of machine passwords into a secure “safe”. The GUI is rather nice on the Mac, so I may considering using it under Linux too.

I’ve also installed Qumana which will allow me to blog without using a browser. This is something I did many moons ago, and so thought I’d try it again for this Project as it assists in making me use the Mac for the whole time, rather than using interfaces/apps/things that I am already comfortable with.

As I have a few uni assignments due this week, I probably won’t spend much time working on the project. And unfortunately, at least for one of the assignments, I will have to use my Linux machine to complete it due to some tools that aren’t available for Mac. I have contemplated installing Linux on the Mac, but for now figure that the time spent doing that is better spent on the assignments.

Interestingly, while setting in a shopping centre working today, some random walked past and said something like “Mac’s are nice aren’t they?” So for all those who want people to stop and talk to you, get a Mac!

I think I have gotten this Mac to the point that it’s now very usable for me. I’ve installed enough apps to get me going, and I’m forcing myself to learn all the Mac shortcuts. Now I’ll use it for all my Assignments this week as much as possible, and I think by the end of the week, I’ll probably be using my Linux machine via VNC on my Mac!

Powered by Qumana

Day 3

I’ve decided to blog this journey of learning and challenge. I am developing a mobile app, initially for iOS and eventually for Android. I have given myself an initial 60 days to get it done, 60 days from getting the equipment.

Being an iOS app, I required an Apple Mac computer to run the required development tools. Tuesday afternoon an iPhone 3GS and a MacBook arrived, curtsey of a friend of Jun’s. First thing I needed to do was download the Lion update, so I could run XCode 4.1. This started Tuesday evening and was finished by Wednesday morning (Day 1). I then prepared a USB Drive to boot the installer from, and went and purchased a new laptop HDD to upgrade the MacBook. By lunch time I’d fitted the new 500Gb HDD and the OS X Lion install was finished. Now started the 6-8hr download of XCode. Meanwhile I spent some time getting aquainted with this MacBook and OS X. No, I’m not converted like some people have been asking, however it probably will be my main computer for the next 60 odd days.

Yesterday (day 2) I wrote a Hello World app in XCode for iOS. It was copied straight from a tutorial on the net, and I quickly discovered that Apple wants $99 from me to be able to even test it on my iPhone! Come on Apple, you already get the money for the MacBook, the iPhone, and now you want more money just so we can test apps on our OWN devices!?!? Eventually we’ll need to fork out this money so we can sign our apps and put them on the App Store. For now though, I think I’ll just jailbreak for testing purposes.
I’ve continued getting acquainted with the MacBook. It now runs Firefox and Thunderbird, iCal is 2 way syncing my calendar. Clementine has become my music player (iTunes, no thanks), LibreOffice is installed. TextWrangler may become my other code editor if it is good enough. Dropbox is setup for syncing between my MacBook and certain folders on my main computer. So far, it’s been nice using this computer, however it’s certainly not better than similar hardware running Linux. For a start, it took me a long time to get network shares with our NAS working well, even though it supports AFP. Eventually it was easier to just setup NFS. I’m getting used to all the keyboard shortcuts, which once learned certainly speed up some navigation. Annoyingly, there have been some things where a keyboard doesn’t work and you have to use a mouse.

Regarding code, it looks like I’ll probably use Git for the SCM as its built in to XCode.

Thats all for Day 3 so far. I need to get some uni work done and we have a busy weekend. I look forward to next week when I’ll have to make this iPhone run my apps!