tag:blogger.com,1999:blog-226770695118841002024-03-13T05:59:19.149-07:00Igregious<a href="http://www.igregious.com/2011/01/start-here.html">About Igregious</a>Greg Fawcetthttp://www.blogger.com/profile/16659789053200729994noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-22677069511884100.post-53825098511947769712023-03-30T17:33:00.001-07:002023-03-30T17:35:51.764-07:00GMail is Breaking Email<h2 style="text-align: left;"><span style="font-family: arial;">Email is an open system, right? Anyone can send a message to anyone... unless they are on Gmail!</span></h2><div><a href="https://www.schoolinterviews.co.nz" style="font-family: arial;" target="_blank">School Interviews</a><span style="font-family: arial;"> uses two email servers to send confirmation emails to parents when they make a booking:</span></div><p></p><ul style="text-align: left;"><li><span style="font-family: arial;">vmailer1.vig.co.nz 69.48.142.156</span></li><li><span style="font-family: arial;">vmailer3.vig.co.nz 190.92.153.150</span></li></ul><p></p><p><span style="font-family: arial;">It goes without saying that our messages are not spam, and we do everything right to prevent them ending up in spam folders. Both email servers </span><span style="font-family: arial;">have PTR records set up, and SPF records only allow email emanating from these two servers. Every message is cryptographically signed with DKIM, and DMARC records tell receiving servers to apply all these restrictions strictly.</span></p><p><span style="font-family: arial;">Neither server is a open relay, and both send only our messages - no-one else is sending spam from these servers. </span></p><p><span style="font-family: arial;">All this effort pays off. W</span><span style="font-family: arial;">e check our servers aren't on spam blacklists every day, and our</span><span style="font-family: arial;"> Google Postmaster Tools account shows practically perfect scores on all metrics:</span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUflYrFFUWM6H_kpNghgyfxLup7kOcvSFBbfy0Zgl2lR0bVbhzS8S_z3jPNM_g-T_ab3vUxn-1gJ681VN-hhkviy2RtpAgB-LkibTX6uTYu1KabZJm_eXD5h339x9YjgWa46Jc8aAE5nisXKLkSKsbwdFy580PiIR0VTOC9JxoV8G54dIVhKIUOA/s904/PMT_DomainReputation.png" style="font-family: arial; margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="742" data-original-width="904" height="263" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUflYrFFUWM6H_kpNghgyfxLup7kOcvSFBbfy0Zgl2lR0bVbhzS8S_z3jPNM_g-T_ab3vUxn-1gJ681VN-hhkviy2RtpAgB-LkibTX6uTYu1KabZJm_eXD5h339x9YjgWa46Jc8aAE5nisXKLkSKsbwdFy580PiIR0VTOC9JxoV8G54dIVhKIUOA/s320/PMT_DomainReputation.png" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWoatPU-aw84xgwZQD6-uwAq63rpbbf5BYdOWKEiAfb2LnxP-tZkMW5tDAKYsxRnoUSpkSMb8GndLVjFSvUEpv_sgYAx1LdIHhCl2OaiOpyz7vycc1QI5ZPQU6ODwF1hORG5L8gWpPuv0VPhZGHhu4NRDsw6o3o2Xksfc-_gLtc0UkII5VmDWtWw/s918/PMT_Feedback.png" style="font-family: arial; margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="744" data-original-width="918" height="259" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWoatPU-aw84xgwZQD6-uwAq63rpbbf5BYdOWKEiAfb2LnxP-tZkMW5tDAKYsxRnoUSpkSMb8GndLVjFSvUEpv_sgYAx1LdIHhCl2OaiOpyz7vycc1QI5ZPQU6ODwF1hORG5L8gWpPuv0VPhZGHhu4NRDsw6o3o2Xksfc-_gLtc0UkII5VmDWtWw/s320/PMT_Feedback.png" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGLSbdsOJFZ7-2UQitYUqIrn97n445lqqZlSHB22uOyfFvQgbygx0llP8nWaaCsSqFm0xCCCOM33JzLdrbh-UwKpBi_dtXi_khiWQyw2ldRJzgfwkiU7ovkab_lWxuBbH941qH836efBkpAD_VCdLkw1RM5NOLb3RCqKLRjy2ZuZGonRmCaxilEQ/s906/PMT_Spam.png" style="font-family: arial; margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="734" data-original-width="906" height="259" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGLSbdsOJFZ7-2UQitYUqIrn97n445lqqZlSHB22uOyfFvQgbygx0llP8nWaaCsSqFm0xCCCOM33JzLdrbh-UwKpBi_dtXi_khiWQyw2ldRJzgfwkiU7ovkab_lWxuBbH941qH836efBkpAD_VCdLkw1RM5NOLb3RCqKLRjy2ZuZGonRmCaxilEQ/s320/PMT_Spam.png" width="320" /></a></div><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBqVfqe_nebzobBlZPKHAK_R7SVppCo_rB0-Qa9u0yIfy2x04NkuCDRwJr7lovYzWrUuam_A7MfB1YyxRRmEG-0c1AVw1wXy_Nj99LK8HeqWOPpGJGqkFrRIATyYqYACviPHLnutOalis0-c79QZ04vDIusEmLQ7idzFRVNTN8MTGmLZG_4MlL1g/s1030/PMT_IPreputation.png" style="font-family: arial; margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="1030" data-original-width="917" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBqVfqe_nebzobBlZPKHAK_R7SVppCo_rB0-Qa9u0yIfy2x04NkuCDRwJr7lovYzWrUuam_A7MfB1YyxRRmEG-0c1AVw1wXy_Nj99LK8HeqWOPpGJGqkFrRIATyYqYACviPHLnutOalis0-c79QZ04vDIusEmLQ7idzFRVNTN8MTGmLZG_4MlL1g/s320/PMT_IPreputation.png" width="285" /></a></div><p></p><p><span style="font-family: arial;"></span></p><div class="separator" style="clear: both; text-align: center;"><p style="text-align: left;"><span style="font-family: arial;">Can anyone hear a "but" coming?</span></p><p style="text-align: left;"><span style="font-family: arial;"><br /></span></p><h2 style="text-align: left;"><span style="font-size: large;"><span style="font-family: arial;"><b>But... </b></span><span style="font-family: arial;"><b>Gmail is rate-limiting our messages daily.</b></span></span></h2><div><span style="font-family: arial;">Vmailer1:</span></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg6VU_Dq42SgtTjmHTlEi2EMLQufUf2l1ZVaYETtxHZwLFCc6uFx9LlIRopyKEBHf5_YuUSasQlIQQPJEUwb7fO-hWCWTBbBBx0zBCg8BXtvDPILW_ktUQ98IDz3jYwaw_m4B1jusPrzIjxwCaE5RbjAtIhfuXho8F9yr4Rz1EupnhNgwYVBBBoA/s828/VM1_Mar31.png" style="font-family: arial; margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="465" data-original-width="828" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg6VU_Dq42SgtTjmHTlEi2EMLQufUf2l1ZVaYETtxHZwLFCc6uFx9LlIRopyKEBHf5_YuUSasQlIQQPJEUwb7fO-hWCWTBbBBx0zBCg8BXtvDPILW_ktUQ98IDz3jYwaw_m4B1jusPrzIjxwCaE5RbjAtIhfuXho8F9yr4Rz1EupnhNgwYVBBBoA/s320/VM1_Mar31.png" width="320" /></a></div></div><span style="font-family: arial;"><br /><div class="separator" style="clear: both; text-align: center;">Vmailer2:</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhzq_PzuXgTRmKhjyJqun8a_uqS4ZJfuJ1bOVvkP7ozbtn8jptw_Ey2WylUvtjqqFn1jL4hmP-xH4QswYKU5ONDWFUQ4i6DP8MhDMwSeHHVu9QvnFYTLD_V1aChKV2rzWR6jvcoUSa-ewxe_HhsH6sVyUUOxngPaG1noWESvu90nv3GjHm-zQnqg/s833/VM3_Mar31.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="464" data-original-width="833" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhzq_PzuXgTRmKhjyJqun8a_uqS4ZJfuJ1bOVvkP7ozbtn8jptw_Ey2WylUvtjqqFn1jL4hmP-xH4QswYKU5ONDWFUQ4i6DP8MhDMwSeHHVu9QvnFYTLD_V1aChKV2rzWR6jvcoUSa-ewxe_HhsH6sVyUUOxngPaG1noWESvu90nv3GjHm-zQnqg/s320/VM3_Mar31.png" width="320" /></a></div><br /></span><p><span style="font-family: arial;">Those blue dots show over 3,500 Gmail customers having the booking confirmation email they asked for delayed by up to 12 hours. </span><span style="font-family: arial;">Our support people get several calls a day asking about missing confirmation emails, and they wearily explain that Gmail is delaying delivery for no good reason. </span></p><div><span style="font-family: arial;">The actual error is:</span></div><p><span style="font-family: arial;"></span></p><blockquote><p><span style="font-family: courier; font-size: x-small;">Our system has detected an unusual rate of 421-4.7.28 unsolicited mail originating from your IP address. To protect our 421-4.7.28 users from spam, mail sent from your IP address has been temporarily 421-4.7.28 rate limited.</span></p><p></p></blockquote><p><span style="font-family: arial;">"An unusual rate of unsolicited mail"? We've established we don't send spam, and Google's own Postmaster Tools can't detect any spam, so it seems to me that Gmail's detection algorithm is broken.</span></p><p><span style="font-family: arial;">And it has been broken for a while, because we've been rate-limited </span><span style="font-family: arial;">for months. We've done everything we can to alert Google to the problem, and this blog post is a last-ditch attempt to get the message across. <i>If you know anyone at Google, please send them a link.</i></span></p><p><span style="font-family: arial;">Wait, I think I hear another "but"!</span></p><p><span style="font-family: arial;"><br /></span></p><h2 style="text-align: left;"><span style="font-size: large;"><b style="font-family: arial;">BUT... </b><span style="font-family: arial;"><b>Problems like this are killing independant email.</b></span></span></h2><p><span style="font-family: arial;">This rate-limiting issue is not the only thorn in the side of email server admins. </span><span style="font-family: arial;">If you search the forums, you'll come across hundreds of reports of the big three email providers (Google, Microsoft and Yahoo) making email delivery so difficult that independant email servers are becoming untenable - keeping messages flowing is just too hard. </span></p><p><span style="font-family: arial;"><i>And this is happening after SPF, DKIM and DMARC provided a solution to the spam problem.</i> </span></p><p><span style="font-family: arial;">Any mail system can remove practically all spam by insisting messages conform to those three standards, so Gmail could - and should - be accepting our squeaky-clean messages without restriction.</span></p><p><span style="font-family: arial;">Instead, </span><span style="font-family: arial;">the big three are making independant email delivery so difficult that we give up and move our inboxes to their services, and use one of their partners (like </span><span style="font-family: arial;">MailChimp or SendGrid) to deliver our system-generated messages.</span></p><p><span style="font-family: arial;">I really hope I can write a follow-up saying that fixing the rate-limiting problem proves that Gmail is still committed to the open email standard. The alternative is that email will be subsumed into giant corporation's proprietary systems, and the wonderfully open and extensible message service we've enjoyed since the dawn of the internet will be gone.</span></p>Greg Fawcetthttp://www.blogger.com/profile/16659789053200729994noreply@blogger.com20tag:blogger.com,1999:blog-22677069511884100.post-8239572898593446492019-10-12T21:27:00.003-07:002019-10-12T21:27:48.792-07:00Ubuntu Filechooser Dialog Size<pre style="border: 0px; direction: ltr; overflow: auto; padding: 0px; width: 1015.59px;"><h2>
<code style="display: inline-block; padding: 0.5em;"><span style="color: #222222;">Gnome 3 Filechooser Dialog Size</span></code></h2>
</pre>
I love GNU/Linux. I've used Ubuntu for well over ten years, and it's stable, powerful and "just works". And if anything doesn't work quite the way you like, you can almost always fix it. Here's a great example.<div>
<br /></div>
<div>
A couple of years back, Ubuntu switched to the Gnome 3 graphics system. When I upgraded to this recently, I noticed that the file chooser dialog now filled the entire window. I found this disconcerting, because without the context of the original program in the background, it was easy to forget what you were trying to accomplish.</div>
<div>
<br /></div>
<div>
With Windows or MacOS, you'd just have to put up with the decisions of the UI design. But with GNU/Linux, a few second's googling brought up the following command: </div>
<div>
<br /></div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">gsettings set org.gtk.Settings.FileChooser window-size '(800, 600)'</span></blockquote>
<div>
Hope that helps anyone else wanting smaller file choosers in Gnome 3.</div>
<div>
<br /></div>
Greg Fawcetthttp://www.blogger.com/profile/16659789053200729994noreply@blogger.com2tag:blogger.com,1999:blog-22677069511884100.post-57205828461297044262018-12-12T20:27:00.000-08:002019-08-10T23:23:04.158-07:00RocketBot 1: Brains, Brawn, Barometers and Batteries<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="" style="clear: both; text-align: left;">
<a href="https://4.bp.blogspot.com/-HPb8njw5zdM/XA86MP_708I/AAAAAAAANVA/YHO0o9Ia7DkXaU6KuPLIS7N8sbAD6Zr5QCPcBGAYYCw/s1600/OI000214.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="150" src="https://4.bp.blogspot.com/-HPb8njw5zdM/XA86MP_708I/AAAAAAAANVA/YHO0o9Ia7DkXaU6KuPLIS7N8sbAD6Zr5QCPcBGAYYCw/s200/OI000214.jpg" width="200" /></a>What is RocketBot made of? We'll go through each part, explaining what it does and A tiny computer, a servo motor, a barometer chip and a battery. This article explains each of the parts that make it tick - the , and also where to get them.</div>
<a name='more'></a><br />
<h3 style="clear: both; text-align: left;">
Brains</h3>
<div class="separator" style="clear: both; text-align: left;">
First of all, we need some brains - but not in a zombie way. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-4ilejO8Bwf8/XAhOFmzFH5I/AAAAAAAANQg/y79g7etxo10JurCZcej830ILWrAz26QYgCLcBGAs/s1600/Wemos%2BD1%2BMini.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="458" data-original-width="458" height="320" src="https://4.bp.blogspot.com/-4ilejO8Bwf8/XAhOFmzFH5I/AAAAAAAANQg/y79g7etxo10JurCZcej830ILWrAz26QYgCLcBGAs/s320/Wemos%2BD1%2BMini.jpeg" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
<div>
This tiny bundle of goodness is the Wemos D1 Mini. You're looking at a 32-bit microprocessor, 4MB of flash memory, plenty of digital inputs and outputs, and a USB port for programming it. But the really cool bit is that gold wiggly track, which is the aerial for <b>built-in WiFi networking.</b><br />
<br />
WiFi means you can connect to it from a smartphone, so you don't need to mess about mounting switches and displays on your rocket. And you can control your rocket from a safe distance, and see how high it went before it even returns to earth.<br />
<br />
OK, WiFi is cool, but now for the fact that <i>really</i> blows my mind. This thing is made out incredibly pure crystals of silicon which are then blasted with X-rays in a vacuum to etch circuits 90 nanometers wide. You could fit 8,000 of those circuits across one human hair, and they can switch on and off 100 billion times a second - it would take you 2,000 years to switch a light switch on and off that many times. Then they connect tiny gold wires from the silicon to the chip pins, and cast the whole thing in a tiny black plastic package. Now multiply all that by the 4 integrated circuits on the Wemos board. Then those chips, and 18 other components, a reset switch and the USB socket are all soldered onto a fibreglass circuit board that is itself a marvel of intricate engineering.<br />
<br />
And think about all the design and programming work, all the millions of dollars spent on incredibly high-tech factories, and the robots that assemble things way to small for humans to manipulate. And at a human level, the person who advertises them on the internet, packages them up and posts them, and adds their profit margin to make a living. And the trucks, ships and planes that move them from the factory to warehouse to shop and then halfway round the world to you.<br />
<br />
<div style="text-align: center;">
<span style="color: red; font-size: x-large;">ALL THIS COSTS $2.50</span></div>
</div>
<div>
<br />
Wow. Just wow.<br />
<br /></div>
<h3>
Brawn</h3>
<div>
If the RocketBot is going to do something to make your rocket come down slower than it went up, it needs some muscle - some way to release a catch or a rubber band, which can then pop out a parachute or streamer. Here is RocketBot's muscle:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-0Ung0EKQZcE/XAh2r--OYZI/AAAAAAAANRI/0aIE8XWMR2IXhMuYUAZXX6I7rVQs_55TQCLcBGAs/s1600/Servo.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="400" data-original-width="400" height="320" src="https://3.bp.blogspot.com/-0Ung0EKQZcE/XAh2r--OYZI/AAAAAAAANRI/0aIE8XWMR2IXhMuYUAZXX6I7rVQs_55TQCLcBGAs/s320/Servo.jpeg" width="320" /></a></div>
<br />
This is called a servo. It contains a tiny electric motor and lots of gears, which moves it's arm to any angle between 0 and 270 degrees. It also has a way to measure the angle, so it can return to the same position pretty accurately.<br />
<br />
We're going to use a 3.7G servo, for two reasons. It's very small and light (3.7 grams), which hurts less if it happens to fall on you from a disintegrating water rocket. It also works fine with the voltage from a single lithium cell, whereas bigger servos need higher voltages. Higher voltage means a heavier battery which means more hurting, and we don't want that!<br />
<br /></div>
<h3>
Barometer</h3>
<div>
RocketBot's brain needs senses, specifically a sense for how high it is. You know how your ears pop when you climb a hill? That's because the air pressure goes down as you get higher. So by measuring air pressure, you can work out how high you are. RocketBot uses this nifty little electronic barometer:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-eNAa9z3Mg0o/XAh1WWpJKEI/AAAAAAAANQ8/xtZj2BKxsa4etX8RkuVTj7e0C-sqBkACQCLcBGAs/s1600/bmp280.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="700" data-original-width="700" height="320" src="https://4.bp.blogspot.com/-eNAa9z3Mg0o/XAh1WWpJKEI/AAAAAAAANQ8/xtZj2BKxsa4etX8RkuVTj7e0C-sqBkACQCLcBGAs/s320/bmp280.jpeg" width="320" /></a></div>
<div>
<br /></div>
<div>
This chip is called a BMP280, and as well as a barometer made out of silicon, it contains a thermometer and it's own processor to let it talk to other digital devices. It's amazingly accurate, to about 25cm of height.<br />
<br />
And here's another "wow" moment: the barometer, thermometer and communications processor are all inside that 2.5mm square metal component between the big (actually only 3mm across) gold holes.<br />
<br /></div>
<h3>
Battery</h3>
<div>
All these electronic components need electricity, so we need a battery. We've carefully chosen parts that run happily on 3.7 Volts so we can use a small, light (less hurting, remember?) lithium polymer battery:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-fJJWOcC6gKg/XJMeKSAGgNI/AAAAAAAAOgA/aBz8j_cFfSMATocPahvYWNKeC44k9iwFACLcBGAs/s1600/LipoBattery.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="426" data-original-width="458" height="297" src="https://4.bp.blogspot.com/-fJJWOcC6gKg/XJMeKSAGgNI/AAAAAAAAOgA/aBz8j_cFfSMATocPahvYWNKeC44k9iwFACLcBGAs/s320/LipoBattery.jpg" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
LiPo (lithium polymer) batteries come in different capacities, measured in milli-Amp-hours (mAh). A 260mAh battery is about right - it can supply 260mA for an hour, but because RocketBot only draws 50mA it will last for about five hours. The capacity isn't critical, so if you find a larger or smaller capacity, go for it. Bear in mind that larger capacities will be physically bigger and heavier (more hurting).<br />
<br />
You will need to connect the battery to the circuit - and also to the charger. There are several kinds of connectors, but the best one is called a micro JST PH connector. Just to make life interesting there are two kinds of JST connector, but luckily they are different colours (red and white). We want the white ones:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-KOZIpgBpKRg/XJMVrh4RI4I/AAAAAAAAOf0/0XLGGZrHN9UUFFQwmIoN0yHFEC2C-YU7gCLcBGAs/s1600/MicroJSTPH.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="800" data-original-width="800" height="320" src="https://2.bp.blogspot.com/-KOZIpgBpKRg/XJMVrh4RI4I/AAAAAAAAOf0/0XLGGZrHN9UUFFQwmIoN0yHFEC2C-YU7gCLcBGAs/s320/MicroJSTPH.jpg" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
So to sum up, you are looking for a 3.7V 260mAh lithium polymer (lipo) battery with a white micro JST PH connector.<br />
<br />
<h3>
Other Stuff</h3>
As well as the components mentioned above, you will need tools for soldering, a hot glue gun, a laptop or desktop computer, a USB cable, and a smartphone.<br />
<br />
<h3>
Let's Go Shopping!</h3>
Unless you are lucky enough to live in Shenzhen in China, you probably want to order all the parts online. There are several websites you can use:<br />
<ul>
<li><a href="https://www.aliexpress.com/" target="_blank">AliExpress</a></li>
<li><a href="https://www.banggood.com/" target="_blank">BangGood</a></li>
<li><a href="https://www.ebay.com/" target="_blank">EBay</a></li>
<li><a href="https://www.amazon.com/" target="_blank">Amazon</a></li>
</ul>
I happen to like AliExpress, so I've included their search links below. You can usually choose different shipping options, and you may want to pay a little more to reduce the number of days you'll have to wait for things to arrive.<br />
<ul>
<li><b>Wemos D1 Mini</b>: Search for "<a href="https://www.aliexpress.com/wholesale?SearchText=wemos+d1+mini" target="_blank">Wemos D1 Mini</a>" with a price below $3. You don't need the Pro (16MB) version, you do need the strips of header pins they almost all come with.</li>
</ul>
<ul>
<li><b>Servo</b>: Search for "<a href="https://www.aliexpress.com/wholesale?SortType=price_asc&SearchText=3.7g+servo" target="_blank">3.7g Servo</a>" with a price around $2.</li>
</ul>
<ul>
<li><b>BMP280</b>: Search for "<a href="https://www.aliexpress.com/wholesale?SortType=price_asc&SearchText=BMP280+3.3" target="_blank">BMP280 3.3</a>" with a price around $1. You need the 3.3V version, which comes with 6 pins. <i>Do not get the 5V version (usually with 4 pins).</i></li>
</ul>
<ul>
<li><b>Battery and charger</b>: Search for "<a href="https://www.aliexpress.com/wholesale?SortType=price_asc&SearchText=3.7v+lipo+battery" target="_blank">3.7V lipo battery</a>", which will bring up batteries and chargers. For the battery, look for something around 260mAh, a price below $2, the white micro JST PH connector, and don't worry about the shape. For the charger, look for a price below $1, and make sure it has the white micro JST PH connector. <i>Some of the charger images show a battery, but don't come with it.</i> </li>
</ul>
<ul>
<li><b>Battery cable</b>: Search for "<a href="https://www.aliexpress.com/wholesale?SortType=price_asc&SearchText=micro+jst+ph+2.0+male+female+cable" target="_blank">micro JST PH 2.0 male female cable</a>" with a target price under $1. You only need one female cable, but it seems you can only get them in sets with ten male and ten female cables. </li>
</ul>
<div>
All up, you should be able to get everything for comfortably under $10, including postage. And if some of your friends want to make a RocketBot too, you can save even more by buying in bulk.</div>
</div>
<div>
<br /></div>
<h3>
What Now?</h3>
<div>
Now you wait... and wait... and wait. I've made probably fifty orders from AliExpress, and they all arrived eventually.</div>
<div>
<br /></div>
<div>
In the meantime, it's worth looking at the next step. Beg, borrow or buy the other stuff mentioned above, and if you haven't done any soldering before, you might want to do some practising.</div>
<div>
<ul>
<li>RocketBot 2: Soldering On</li>
<li>RocketBot 3: Get With The Program</li>
<li>RocketBot 4: (Optional) Making Sense of Madness</li>
</ul>
</div>
Greg Fawcetthttp://www.blogger.com/profile/16659789053200729994noreply@blogger.com1tag:blogger.com,1999:blog-22677069511884100.post-32262977155126686752018-12-10T20:17:00.001-08:002019-08-10T23:12:26.448-07:00RocketBot DIY Water Rocket Altimeter<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-HPb8njw5zdM/XA86MP_708I/AAAAAAAANU8/ESPMhi0VozULezV6TJI5gwIYI2AItZfCACKgBGAs/s1600/OI000214.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="150" src="https://3.bp.blogspot.com/-HPb8njw5zdM/XA86MP_708I/AAAAAAAANU8/ESPMhi0VozULezV6TJI5gwIYI2AItZfCACKgBGAs/s200/OI000214.jpg" width="200" /></a></div>
Welcome to RocketBot, a water-rocket altimeter that you can build yourself for less than $10 (USD).<br />
<br />
Why would you want to build one? And how hard would it be? Read on to find out how RocketBot lets you take water rockets to the next level (hah!) and how easy it is to get started.<br />
<br />
<a name='more'></a><br />
<h3>
Why?</h3>
Because water rockets are <a href="https://www.youtube.com/results?search_query=water+rocket" target="_blank">AWESOME</a>! But as soon as you launch your first one, the questions come bursting out of you:<br />
<ul>
<li>If I use fizzy pop instead of water, will it go higher?</li>
<li>If I add more water - or less water - will it go higher?</li>
<li>If I strap a rock onto the nose, will it go higher?</li>
<li>If I make a sharp nosecone, will it go higher?</li>
<li>If I add fins, will it go higher?</li>
</ul>
<div>
So you start trying out these ideas, but it's hard to tell exactly how high your rocket went each time. And after spending hours making a beautiful nosecone and fins, you see your beautiful rocket now streaks smoothly into the sky! But seconds later, you find that it also streaks smoothly into the ground, destroying all your hard work in a spectacular high speed crash. </div>
<div>
<br /></div>
<div>
And so another budding engineer goes and plays games on their phone, instead of <a href="https://en.wikipedia.org/wiki/Peter_Beck_(engineer)" target="_blank">changing the world</a>.</div>
<div>
<br /></div>
<div>
But what if there was something that showed exactly how high your water rocket went? And maybe that something could also trigger a parachute - or wings or something - so it didn't destroy itself every launch? And if it only cost ten bucks, it wouldn't make too much of a dent in your pocket. </div>
<div>
<br /></div>
<div>
Best of all, as well learning about rockets, you'd learn how to make micro-controllers do really cool things! Maybe water rockets CAN be more fun than Candy Crush...<br />
<br /></div>
<h3>
How?</h3>
<div>
Easy! First, you order a handful of bits and pieces on the internet - and wait patiently for them to arrive. Then you need a steady hand to solder about 15 connections, and finally you download some free software so you can program the micro-controller. </div>
<div>
<br />
You will also need access to a soldering iron, a computer, and a few common tools. If you can't borrow a soldering iron, you can buy your own for under $10, and most households have everything else you need.<br />
<br /></div>
<div>
I'll lead you through each step, explaining how everything works and how they go together. If you are new to electronics and programming, don't worry - making a RocketBot is a great way to dip your toe in the water.<br />
<br />
Sounds fun? Then let's get started!</div>
<div>
<ul>
<li>RocketBot<a href="http://www.igregious.com/2018/12/gespalt-1-brains-brawn-barometers-and.html"> 1: Brains, Brawn and Barometers</a></li>
<li>RocketBot 2: Soldering On</li>
<li>RocketBot 3: Get With The Program</li>
<li>RocketBot 4: (Optional) Making Sense of Madness</li>
</ul>
</div>
Greg Fawcetthttp://www.blogger.com/profile/16659789053200729994noreply@blogger.com0tag:blogger.com,1999:blog-22677069511884100.post-37619678208695457612017-10-09T22:01:00.001-07:002018-01-04T16:28:42.280-08:00Eleksmaker A3X Software<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="" style="clear: both; text-align: left;">
<a href="https://3.bp.blogspot.com/-WWRk5GpO3GU/WbSRt61Zn7I/AAAAAAAAJIY/LEsyUSvgjPo4rWjg9ne41S2rLP2PIyVrwCPcBGAYYCw/s1600/EleksmakerA3X.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="235" data-original-width="235" src="https://3.bp.blogspot.com/-WWRk5GpO3GU/WbSRt61Zn7I/AAAAAAAAJIY/LEsyUSvgjPo4rWjg9ne41S2rLP2PIyVrwCPcBGAYYCw/s1600/EleksmakerA3X.jpg" /></a>I switched to Linux over ten years ago, and never looked back - I love the freedom, stability and security that it provides. There are great equivalents for almost all Windows apps, but I found nothing for generating laser engraver gcode files.<br />
<br /></div>
So I either had to install Windows on an old laptop, or write my own cross-platform app. Guess which I did?<br />
<div class="" style="clear: both; text-align: left;">
</div>
<a name='more'></a><br />
Most laser engravers, 3D printers and other low-end CNC machines use an Arduino-based controller running an excellent open-source package called Grbl. It's ironic that the laser engraver vendors exploit the open-source movement by using Grbl, but provide only closed-source software to drive it.<br />
<br />
Grbl expects to receive gcode instructions via a USB port. Gcode instructions for laser engravers are are fairly simple - they tell the laser to move to given coordinates at a given speed with the laser at a given power. To engrave something, you need a way to create the gcode instructions and a way to send those instructions to the controller. The sending part is easy - there are several solutions like <a href="https://github.com/winder/Universal-G-Code-Sender">Universal Gcode Sender</a> that work on Linux (and other platforms). But how do we generate the gcode in the first place?<br />
<br />
<h4>
Vector and Raster Graphics</h4>
Your engraver is a <i>vector</i> device. That means it is good at drawing lines (vectors) from one position to another, and those positions can be anywhere on the work surface - you can draw a line directly from the bottom left to the top right, for instance. This is ideal for drawing vector graphics like SVG (Scalable Vector Graphic) images.<br />
<br />
You could make it behave like a <i>raster</i> device, too. By making it draw a series of horizontal lines very close together down the work surface, and varying the laser power as you do it, you could engrave a pixellated image. This mode is good for reproducing images contained in PNG or JPEG files.<br />
<br />
<br />
<h4>
Introducing Lasergcode</h4>
Given that the engraver is a vector device, it makes sense to use a vector graphics app to create the gcode. The most popular open-source vector graphics app is <a href="https://inkscape.org/">Inkscape</a>, a very capable program which runs on all operating systems (Windows, MacOS and Linux). And it has a brilliantly simple way to extend it's capabilities.<br />
<br />
Inkscape already has an extension to generate gcode, called Gcodetools, but I was very disappointed with it. Its horrible user interface makes it very difficult to use, and the way it works is far from ideal. I thought I could do better, so I wrote <a href="https://bitbucket.org/vig/lasergcode">lasergcode</a>.<br />
<br />
Lasergcode is designed purely for laser engravers. It's simple to install, and simple to use:<br />
<ul>
<li>Open your Inkscape document.</li>
<li>Set the document width and height to your engraver's dimensions (<i>File/Document Properties</i>).</li>
<li>Convert all objects (including text) to paths (<i>Path/Object to Path</i>).</li>
<li><span style="font-family: inherit;">Choose <em>File/Save a Copy...</em>, select <em>Laser gcode file (*.gcode)</em> and click <em>Save</em>.</span></li>
<li>Set the burn speed and repeat number, and click <em>OK</em>.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-XMYi5EovtcY/WdsBMCjGimI/AAAAAAAAJS0/edOaQulq9WkayiYnm51zyEBZ_9J1T1cTQCLcBGAs/s1600/Selection_037.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="249" data-original-width="329" height="242" src="https://4.bp.blogspot.com/-XMYi5EovtcY/WdsBMCjGimI/AAAAAAAAJS0/edOaQulq9WkayiYnm51zyEBZ_9J1T1cTQCLcBGAs/s320/Selection_037.png" width="320" /></a></div>
<br />
Lasergcode uses the path's opacity (alpha channel) to set the laser power. So if you want to burn a line at half power, set the stroke colour alpha to 128 (half of the maximum 256). The included <span style="font-family: "courier new" , "courier" , monospace;">intensity_chart.gcode</span> file is useful for working out what opacity (power level) you need to use for a new material. Here is the result on chipboard:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Dnk9msKfUdw/We1Mgr8oMZI/AAAAAAAAJbM/LPXrF3B7eRkzyrjFxyYoc4c-e5Jel5YGgCLcBGAs/s1600/IMG_20171023_144636.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://1.bp.blogspot.com/-Dnk9msKfUdw/We1Mgr8oMZI/AAAAAAAAJbM/LPXrF3B7eRkzyrjFxyYoc4c-e5Jel5YGgCLcBGAs/s320/IMG_20171023_144636.jpg" width="240" /></a></div>
<br />
<br />
<br /><br />Lasergcode also ignores any points outside the document boundary, so if you set the document width and height as suggested, it will never try to move outside those limits. This is an important safety measure, preventing damage from driving against the frame and fires from the laser burning one spot while trapped against the frame.<br /><br />You can <a href="https://bitbucket.org/vig/lasergcode">download lasergcode here</a>. Enjoy!<br /><br /><div>
<h4>
What About Raster?</h4>
<div>
I haven't found a cross-platform solution for laser engraving raster images. It wouldn't be too difficult to write a python app that uses the PIL image library to resize an image and generates the Grbl 1.1 gcode to burn it, but I don't have the time to do this right now. If you know of something that does this, let me know in the comments below.<br />
<br /></div>
</div>
Greg Fawcetthttp://www.blogger.com/profile/16659789053200729994noreply@blogger.com4tag:blogger.com,1999:blog-22677069511884100.post-72661332879747247232017-10-01T18:07:00.001-07:002017-10-02T02:24:58.352-07:00Eleksmaker A3X Build Notes<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-WWRk5GpO3GU/WbSRt61Zn7I/AAAAAAAAJIY/LEsyUSvgjPo4rWjg9ne41S2rLP2PIyVrwCPcBGAYYCw/s1600/EleksmakerA3X.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="235" data-original-width="235" src="https://3.bp.blogspot.com/-WWRk5GpO3GU/WbSRt61Zn7I/AAAAAAAAJIY/LEsyUSvgjPo4rWjg9ne41S2rLP2PIyVrwCPcBGAYYCw/s1600/EleksmakerA3X.jpg" /></a></div>
I've completed my Eleksmaker A3X laser engraver and I'm having lots of fun burning strange things. But the building instructions are extremely minimal and there are several things you need to know for a successful build.<br />
<br />
In particular, there is <span style="color: red;">one absolutely vital modification</span> <span style="color: red;">that everyone should make before powering-up their laser</span>. Without this, it's only a matter of time before someone blinds themselves while building their machine.<br />
<br />
<a name='more'></a><h3>
Laser Pull-down Resistor</h3>
<div>
<br /></div>
<div>
When I first connected the power to my A3X, the laser fired up on full power for about a second. I wasn't wearing the protective glasses and if the laser had been pointing towards my eyes, <span style="color: red;">I would now be blind</span>.</div>
<div>
<br /></div>
<div>
Luckily the laser was already mounted pointing down, and I had a protective board over the dining room table. Otherwise there would now be a small black hole in the walnut, and my wife would not be happy. Thanks, Eleksmaker.</div>
<div>
<br /></div>
<div>
This happened again and again, when powering up or connecting and disconnecting the USB cable. Checking the <a href="http://forum.eleksmaker.com/">Eleksmaker forum</a> showed that this was an issue for many people, and also that there was a simple solution.</div>
<div>
<br /></div>
<div>
The laser is controlled by the voltage on a pin on the controller, which usually holds the voltage down to zero (ground). But when the controller is powered up, it takes about a second to get round to initialising the pin output, and meanwhile the voltage "floats". The laser sees a non-zero voltage, and does what it is supposed to - turns on.</div>
<div>
<br /></div>
<div>
This is actually a very common issue in electronics, and the solution is simple. You add a pull-down resistor to ground, which holds the pin voltage at zero until the controller initialises the pin. You need a 1K or 2K resistor soldered to the back of the controller board like this:</div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-TIc2-GChXYk/WdFrZXjgn7I/AAAAAAAAJQM/Awgx2kmt2sktcO6aihEXXYLV0bM4uT9GQCKgBGAs/s1600/IMG_20170912_194505.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="874" data-original-width="1600" height="174" src="https://1.bp.blogspot.com/-TIc2-GChXYk/WdFrZXjgn7I/AAAAAAAAJQM/Awgx2kmt2sktcO6aihEXXYLV0bM4uT9GQCKgBGAs/s320/IMG_20170912_194505.jpg" width="320" /></a></div>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Given the danger involved, I think this is a serious defect. I strongly encourage Eleksmaker to add the pull-down resistor before shipping their controller board, but until they do so, please make sure you do this before powering up your A3X.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div>
</div>
<h3>
Laser Carriage Wheels</h3>
The laser carriage consists of two plates that you bolt together with wheels between them. When I assembled my plates, the upper and lower wheels were slightly too close together, so the carriage didn't run smoothly along the beam. Even worse, after they sat still on the beam for some time they developed indents. I could feel a jerk as the carriage rolled along the beam, every time the wheels made one revolution and the indents hit the beam again. This would result in small X-axis errors when burning.<br />
<br />
My solution is to file away a small amount of the top side of the top two holes of each plate <i>slightly, </i>so the top bolts sit slightly further away from the bottom bolts. They only need to be a fraction of a millimetre higher, so do this in small increments and test each time to see how the carriage moves. I used a small round file.<br />
<br />
<h3>
Drive Belt Tension</h3>
You can read my separate post about my problems with <a href="http://www.igregious.com/2017/09/eleksmaker-laser-belt-fastenings.html">fastening the drive belts</a>, so I won't duplicate that here. But the drive belt tension is important - if it is too tight the servos will have difficulty driving the carriages; if it is too loose the carriages will slop about, making your positioning much less precise.<br />
<br />
<h3>
Cabling</h3>
<div>
The instructions don't mention cabling. My kit came with three lengths of plastic trunking that clip into the beams, but I only used two of them (for the Y-axis cables). You need to put the cable into the trunking first, then push both into the beam. If you put the cable into the beam then try and clip the trunking in, you'll be there all day.</div>
<div>
<br /></div>
<div>
The Y-axis cables should go along the top of the rear beam (under the trunking); under the side beams and out; cable-tied to the top of the corner plate; over and cable-tied to the Y-axis carriages:</div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-cwfKVqn0xPY/WdGPfmKtqDI/AAAAAAAAJQ0/mHL4h5HVIuQdvKVyIEKl6dqJ9VPcaQnEgCKgBGAs/s1600/IMG_20171002_135601.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="954" data-original-width="1600" height="236" src="https://4.bp.blogspot.com/-cwfKVqn0xPY/WdGPfmKtqDI/AAAAAAAAJQ0/mHL4h5HVIuQdvKVyIEKl6dqJ9VPcaQnEgCKgBGAs/s400/IMG_20171002_135601.jpg" width="400" /></a></div>
<br />
I drilled an extra hole in the carriage plates to cable-tie the Y-axis cables so any strain did not work against the servo plug. This should prevent these wires unplugging or breaking due to repetitive flexing.<br />
<br />
<h3>
Controller Mounting</h3>
The back plate for the controller board did not fit with the new layout with the back rail under the side rails (the old A3 Pro had the back rail over the side rail). I drilled and tapped a couple of holes in the rear rail so I could mount it higher up.<br />
<br />
<br />
I hope these tips helped with your build. Please leave a comment below if you have other suggestions, or better ways to solve the problems I came across.</div>
Greg Fawcetthttp://www.blogger.com/profile/16659789053200729994noreply@blogger.com6tag:blogger.com,1999:blog-22677069511884100.post-2372829446287001672017-09-09T19:13:00.000-07:002018-01-07T15:22:01.009-08:00Eleksmaker Laser Belt Fastenings<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-WWRk5GpO3GU/WbSRt61Zn7I/AAAAAAAAJG4/gx0brgJBukI9wPTDLku9EGtX68j4sNFIACLcBGAs/s1600/EleksmakerA3X.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="235" data-original-width="235" height="200" src="https://4.bp.blogspot.com/-WWRk5GpO3GU/WbSRt61Zn7I/AAAAAAAAJG4/gx0brgJBukI9wPTDLku9EGtX68j4sNFIACLcBGAs/s200/EleksmakerA3X.jpg" width="200" /></a></div>
My latest toy is the Eleksmaker A3X laser engraver. While building it, I found the provided drive belt clamps didn't work, and I came up a better method of fastening them.<br />
<a name='more'></a><h3>
Original Clamp</h3>
<br />
<a href="https://4.bp.blogspot.com/-oXHF5yAMhWY/WbSUVtumiiI/AAAAAAAAJHM/71M1PLQ99S8-FqdXafNWEMcOpiiOzIWygCLcBGAs/s1600/Original%2Bclamp.jpg" imageanchor="1" style="clear: right; display: inline !important; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;"><img border="0" data-original-height="1600" data-original-width="1415" height="320" src="https://4.bp.blogspot.com/-oXHF5yAMhWY/WbSUVtumiiI/AAAAAAAAJHM/71M1PLQ99S8-FqdXafNWEMcOpiiOzIWygCLcBGAs/s320/Original%2Bclamp.jpg" width="282" /></a>The A3X kit comes with six belt clamps where the belt comes up through a slot in the white plastic block. This is trapped by the aluminium fitting via a bolt that should screw into a captive nut in the rail.<br />
<br />
Unfortunately the bolt is not long enough to engage the captive nut, so I had to figure out another way to fasten the belt. I was happy to do this because I didn't like the way the belt was bent 90 degrees around sharp corners, and also that this arrangement was quite long which limits the movement of the laser carriage.<br />
<br />
<h3>
Fixed End</h3>
To tension it, you only need to adjust one end of the drive belt so the other end can be fixed. To do this, I drilled two 3mm holes and inserted two 3mm pins (actually a sawn-off nail) and a captive nut like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-TfwgjCni4fs/WbSUTh91eaI/AAAAAAAAJHA/fXR3rn4Kqowh25hNdfnWok_6UNOz-SIqQCEwYBhgL/s1600/Fixed%2BEnd1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1064" data-original-width="1600" height="212" src="https://2.bp.blogspot.com/-TfwgjCni4fs/WbSUTh91eaI/AAAAAAAAJHA/fXR3rn4Kqowh25hNdfnWok_6UNOz-SIqQCEwYBhgL/s320/Fixed%2BEnd1.jpg" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-XfWAUpJedDE/WbSUUbChX3I/AAAAAAAAJHE/ot3mezjLi6w0HBib5wOvf5Sf8v49si4KACEwYBhgL/s1600/Fixed%2BEnd%2B2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="634" data-original-width="1600" height="253" src="https://3.bp.blogspot.com/-XfWAUpJedDE/WbSUUbChX3I/AAAAAAAAJHE/ot3mezjLi6w0HBib5wOvf5Sf8v49si4KACEwYBhgL/s640/Fixed%2BEnd%2B2.jpg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-c3vO1snCQHI/WbSUUqBzBHI/AAAAAAAAJHI/T0hVU9KBw5A3BAPhGBst_eSgV8gZOD9tQCEwYBhgL/s1600/Fixed%2BEnd3.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1025" data-original-width="1600" height="205" src="https://4.bp.blogspot.com/-c3vO1snCQHI/WbSUUqBzBHI/AAAAAAAAJHI/T0hVU9KBw5A3BAPhGBst_eSgV8gZOD9tQCEwYBhgL/s320/Fixed%2BEnd3.jpg" width="320" /></a></div>
<br />
The drive belt goes under the inner pin, over the captive nut, and under the outer pin. It then returns over itself and back under the inner pin. Because the belt teeth engage in each other over the captive nut, the belt cannot slip. The blue bits are slices of a wall anchor to keep the pins from dropping out, although the belt prevents this once it is tensioned.<br />
<br />
<span style="color: purple;"><i>UPDATE: I've now changed the fixed end to use cable ties. I wrap the belt back on itself so the teeth mesh, and use two cable ties. This only needs one hole/pin.</i></span><br />
<br />
<h3>
Clamped End</h3>
<div>
I drilled another 3mm hole about 12mm from the end of the rail, and put a 3mm pin in it. I then cut down the white plastic block and drilled and tapped two threaded holes for the bolts. I cut off the end of the aluminium clamp fitting and filed two semicircles into it like this:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-MWpD3YkCrYQ/WbSatbPdwDI/AAAAAAAAJHc/RMmJlGKPdEY7m9x7Sd3FwC7XPjoVwHMzgCLcBGAs/s1600/IMG_20170908_143619.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1474" data-original-width="1600" height="294" src="https://3.bp.blogspot.com/-MWpD3YkCrYQ/WbSatbPdwDI/AAAAAAAAJHc/RMmJlGKPdEY7m9x7Sd3FwC7XPjoVwHMzgCLcBGAs/s320/IMG_20170908_143619.jpg" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-PMZzgjN78k4/WbSas4SNsSI/AAAAAAAAJHY/lM99aG42cyE0NitNwgDsmldS-X3To_WDgCLcBGAs/s1600/IMG_20170908_143819.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1435" height="320" src="https://1.bp.blogspot.com/-PMZzgjN78k4/WbSas4SNsSI/AAAAAAAAJHY/lM99aG42cyE0NitNwgDsmldS-X3To_WDgCLcBGAs/s320/IMG_20170908_143819.jpg" width="286" /></a></div>
<br />
<br />
<span style="color: purple;"><i>UPDATE: The clamp above is not ideal because it's hard to tighten the belt. To access the bolts, you have to removing the rail from the endplate. </i></span><br />
<span style="color: purple;"><i><br /></i></span>
<span style="color: purple;"><i>I now have a 3D printer, so I printed up these:</i></span><br />
<span style="color: purple;"><i><br /></i></span>
<br />
<div style="text-align: center;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-AcsScBSughM/WlKnYnqEa8I/AAAAAAAAKbQ/Sw0Q8RhBKMcWT2pXvLDwgA1FqIEEDsmAQCK4BGAYYCw/s1600/IMG_20180107_141855.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="https://3.bp.blogspot.com/-AcsScBSughM/WlKnYnqEa8I/AAAAAAAAKbQ/Sw0Q8RhBKMcWT2pXvLDwgA1FqIEEDsmAQCK4BGAYYCw/s400/IMG_20180107_141855.jpg" width="400" /></a></div>
<span style="color: purple;"><i></i></span></div>
<div style="text-align: center;">
<span style="color: purple;"><br /></span></div>
<div style="text-align: center;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-2-t9w_9YJf0/WlKnX43XvGI/AAAAAAAAKbI/nGF1gIebwUUnxU5oZu3DtNpPLEovs6ptQCK4BGAYYCw/s1600/IMG_20180107_144724.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="https://3.bp.blogspot.com/-2-t9w_9YJf0/WlKnX43XvGI/AAAAAAAAKbI/nGF1gIebwUUnxU5oZu3DtNpPLEovs6ptQCK4BGAYYCw/s400/IMG_20180107_144724.jpg" width="400" /></a></div>
<span style="color: purple;"><i></i></span></div>
<br />
<br />
<i style="color: purple;">The block has internal teeth, and the wedge holds the belt teeth against them for a positive grip. To tighten the belt, pull out the wedge, pull the belt up another tooth, and reinsert the wedge. No bolts to fiddle with! </i><br />
<i style="color: purple;"><br /></i>
<i style="color: purple;">It also only takes only 12mm of the end of your extrusion, so increasing the travel of your laser head.</i><br />
<i style="color: purple;"><br /></i>
<i style="color: purple;">You can download the STL file for 3D printers from <a href="https://www.thingiverse.com/thing:2749751" target="_blank">Thingiverse</a>.</i><br />
<br />
So now I have:<br />
<ul>
<li>Positive fixings for both ends of the drive belt.</li>
<li>The belt bends round a 3mm pin instead of a sharp corner, reducing wear and tear.</li>
<li>A couple of centimetres more travel in both X and Y dimensions.</li>
</ul>
<h3>
</h3>
<h3>
</h3>
Greg Fawcetthttp://www.blogger.com/profile/16659789053200729994noreply@blogger.com1tag:blogger.com,1999:blog-22677069511884100.post-61527651538529500792013-05-16T14:45:00.001-07:002017-09-09T16:18:21.930-07:00Korean IPS Monitors and Nvidia under Linux<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-wBgDr4uC0pE/WbR2nyyxo8I/AAAAAAAAJGo/Y1jxSioNDDgOOcBHk4Bx302S5mDjr4JJACLcBGAs/s1600/nvidia.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="427" data-original-width="450" height="189" src="https://4.bp.blogspot.com/-wBgDr4uC0pE/WbR2nyyxo8I/AAAAAAAAJGo/Y1jxSioNDDgOOcBHk4Bx302S5mDjr4JJACLcBGAs/s200/nvidia.jpg" width="200" /></a></div>
The business is doing well, so I thought I'd upgrade my computer. Bye bye old laptop plus extra 23" monitor; hello new desktop system, SSD, a meaty Nvidia graphics card and best of all, a pair of awesome Korean 26" IPS monitors. Cool!<br />
<br />
Actually, not cool - the displays wouldn't work due to what turned out to be a stupid bug in the Nvidia drivers for Linux. Read on for the story and a solution...<br />
<a name='more'></a><br />
At first, I used the open-source Nouveau drivers, and the displays worked, but were very slow - slower than my old laptop, which made a bit of a mockery of the Nvidia graphics card. I tried the proprietary Nvidia drivers, but both screens blanked as soon as I logged in so I switched back.<br />
<br />
But when I upgraded to the latest version of Ubuntu (from 12.10 to 13.04), it became obvious that the new Nouveau drivers were unstable. The machine would randomly crash several times a day, displaying a minced version of the desktop and requiring a hard reset. So I tried the proprietary Nvidia drivers again.<br />
<br />
Same result... booting fine to the login page, then two blank screens as soon as I logged in. I managed to get a terminal session going by hitting <b>Ctrl-Alt-F1</b> at the login page, and went searching for clues to what was going wrong. X-windows creates a log file <span style="background-color: white; font-family: "courier new" , "courier" , monospace;">/var/log/Xorg.0.log</span>, and it contained the following entries:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 4.380] (**) NVIDIA(0): Depth 24, (--) framebuffer bpp 32</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 4.380] (==) NVIDIA(0): RGB weight 888</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 4.380] (==) NVIDIA(0): Default visual is TrueColor</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 4.380] (==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 4.380] (**) NVIDIA(0): Enabling 2D acceleration</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.140] (WW) NVIDIA(GPU-0): <span style="background-color: #f4cccc;">The EDID read for display device DFP-0 is invalid:</span> the</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.140] (WW) NVIDIA(GPU-0): checksum for EDID version 1 extension is invalid.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.140] (--) NVIDIA(GPU-0): </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.140] (--) NVIDIA(GPU-0): Raw EDID bytes:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.140] (--) NVIDIA(GPU-0): </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.140] (--) NVIDIA(GPU-0): 00 ff ff ff ff ff ff 00 04 62 9b 04 00 00 00 00</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">...</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.140] (--) NVIDIA(GPU-0): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.140] (--) NVIDIA(GPU-0): </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.159] (WW) NVIDIA(GPU-0): <span style="background-color: #f4cccc;">The EDID read for display device DFP-2 is invalid:</span> the</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.159] (WW) NVIDIA(GPU-0): checksum for EDID version 1 extension is invalid.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.159] (--) NVIDIA(GPU-0): </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.159] (--) NVIDIA(GPU-0): Raw EDID bytes:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.159] (--) NVIDIA(GPU-0): </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.159] (--) NVIDIA(GPU-0): 00 ff ff ff ff ff ff 00 5d 34 fa 00 00 00 00 00</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">...</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[ 5.159] (--) NVIDIA(GPU-0): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00</span><br />
<br />
After hitting Google, I found that the EDID is a block of data that a monitor passes to the graphics card, containing it's operational parameters (name, frequencies, resolution etc.). The data is protected by a checksum because corrupted data could physically damage the card or the monitor, and the Nvidia card was getting the wrong checksum so it was ignoring the data - and therefore the monitor. Fair enough, if the EDID checksum actually was wrong...<br />
<br />
But then why did they work with other graphics cards, and under Windows and IOS? I wrote a small python program to grab the EDID data from the log, and fed that into <a href="http://www.polypux.org/projects/read-edid/">a useful utility for reading and parsing EDID data</a>. According to this, the checksum was correct!<br />
<br />
More research found that there was an X-windows option to tell the driver to ignore the checksum when reading the EDID, although this came with ominous warnings about how using the option could damage your hardware. To use it, you first need to create an xorg.conf file using the following command:<br />
<br />
<b id="docs-internal-guid-5e9e4ff0-ac55-fcfa-8478-8c03a4c2cad8" style="font-weight: normal;"><span style="font-family: "courier new"; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">sudo nvidia-xconfig</span></b><br />
<b style="font-weight: normal;"><span style="font-family: "courier new"; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b>
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Now you need to edit the new xorg.conf file:</span></span></b><br />
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></b>
<b id="docs-internal-guid-5e9e4ff0-ac56-b29d-69ea-c836695ddbb4" style="font-weight: normal;"><span style="font-family: "courier new"; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">sudo vi /etc/X11/xorg.conf</span></b><br />
<b style="font-weight: normal;"><span style="font-family: "courier new"; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b>
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Find the section below, and add the highlighted line:</span></span></b><br />
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></b>
<b id="docs-internal-guid-5e9e4ff0-ac57-e253-f5bd-c54d307257a8" style="font-weight: normal;"></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b id="docs-internal-guid-5e9e4ff0-ac57-e253-f5bd-c54d307257a8" style="font-weight: normal;"><span style="font-family: "courier new"; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Section "Device"<br class="kix-line-break" /> Identifier "Device0"<br class="kix-line-break" /> Driver "nvidia"<br class="kix-line-break" /> VendorName "NVIDIA Corporation"</span></b></div>
<b id="docs-internal-guid-5e9e4ff0-ac57-e253-f5bd-c54d307257a8" style="font-weight: normal;"><span style="background-color: #ffe599; font-family: "courier new"; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> Option "IgnoreEDIDChecksum" "DFP"</span></b><br />
<b style="font-weight: normal;"><span style="font-family: "courier new"; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">EndSection</span></b><br />
<b style="font-weight: normal;"><span style="font-family: "courier new"; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b>
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Save the file, and reboot.</span></span></b><br />
<b style="font-weight: normal;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></b>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;">After doing this, the monitors fired up beautifully after logging in. There are some warnings about ignoring the EDID in the log, but the Nvidia card is obviously reading the data correctly. So this means there is a bug in the Nvidia driver that incorrectly calculates the checksum for some monitors - particularly these Korean IPS screens, it seems.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;"><br /></span></span>
<br />
<div style="text-align: center;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><span style="color: red; white-space: pre-wrap;">Open Source the Drivers, Nvidia!</span></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;">What happens next is a classic example of the advantages of open-sourcing your code. </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;">What <i>should</i> happen is that I download the source for the driver, figure out why it's getting the wrong checksum, and let the maintainer know exactly what the problem is. I then feel great that I've solved this problem for lots of other people, Nvidia has a better driver, and lot of other people don't have this problem. <b>Awesome - everyone loves Nvidia!</b></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;"><i>But Nvidia have kept their driver proprietary, for whatever reason.</i> So what will actually happen is that I will write blog post criticising Nvidia, rather than submitting a bug report. (Actually I will report the bug, but I'm much less motivated to do this because their development process is so opaque I'm not at all confident it will be acted on.) </span></span><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: 15.199999809265137px; white-space: pre-wrap;">So the end result is I feel frustrated with Nvidia, they still have a buggy driver, and a lot of other people still have this problem. </span><b style="font-family: Arial, Helvetica, sans-serif; font-size: 15.199999809265137px; white-space: pre-wrap;">Everyone thinks Nvidia stinks and will buy a different graphics card next time!</b><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;"><br /></span></span><i>
Do the right thing, Nvidia.</i><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: 15.199999809265137px; white-space: pre-wrap;"><br /></span></span>Greg Fawcetthttp://www.blogger.com/profile/16659789053200729994noreply@blogger.com5tag:blogger.com,1999:blog-22677069511884100.post-42267357141100511842011-10-18T20:57:00.000-07:002011-10-19T20:18:11.416-07:00Migrating a Subversion Project to Git on BitBucket<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-TdYa4mCkUdk/Tpz1yBEuaFI/AAAAAAAARR0/pJVa9ie9tEM/s1600/BitBucket.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-TdYa4mCkUdk/Tpz1yBEuaFI/AAAAAAAARR0/pJVa9ie9tEM/s1600/BitBucket.png" /></a></div>
<span class="Apple-style-span" style="font-family: inherit;">Subversion is still the gold standard for version control (a.k.a. source control) systems, but the new kid on the block - Git - has undeniable advantages for many development models. Even for solo or centralised hierarchical development organisations, Git's agile branching and merging make it a joy to use.</span><br />
<span class="Apple-style-span" style="font-family: inherit;"><br /></span><br />
<span class="Apple-style-span" style="font-family: inherit;">But if you're going to go to the effort of switching version control systems, you probably also want to consider moving your repository to the cloud. </span>This increases your options for off-site developers, and reduces the risks and maintenance costs of internally managed version control servers.<br />
<br />
This article describes how to migrate a Subversion repository to Bitbucket, one of several online repositories.<br />
<a name='more'></a><span class="Apple-style-span" style="font-family: inherit;">First of all, why </span><a href="http://bitbucket.org/" style="font-family: inherit;">Bitbucket</a><span class="Apple-style-span" style="font-family: inherit;">? </span><a href="http://github.com/" style="font-family: inherit;">GitHub</a><span class="Apple-style-span" style="font-family: inherit;"> and </span><a href="http://code.google.com/" style="font-family: inherit;">Google Code</a><span class="Apple-style-span" style="font-family: inherit;"> are worthy alternatives, but both have restrictions that make them less desirable for private repositories. </span><span class="Apple-style-span" style="font-family: inherit;">Google Code doesn't allow them at all, and GitHub currently charges a small but significant amount per private repository. Bitbucket charges by developer instead, with the first five being free. Another important consideration is that a</span><span class="Apple-style-span" style="font-family: inherit;">ll three are backed by large commercial companies that are unlikely to blink out of </span>existence<span class="Apple-style-span" style="font-family: inherit;"> overnight.</span><br />
<br />
<span class="Apple-style-span" style="font-family: inherit; font-size: large;">1. Preparation</span><br />
OK, enough of the justifications, lets get migrating! In the Subversion world, the convention is that you structure your projects as follows:<br />
<pre style="background-color: #e8e8ff;"><span class="Apple-style-span" style="font-size: x-small;">foobar
- Trunk
- Branches
- Europe
- US
- Tags
- Version 3.11
- Version 3.12</span>
</pre>
If you use different conventions (like "trunk" instead of "Trunk"), take extra care to translate these in the code below. Our example Subversion project URL is <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">svn://myserver/myrepo/foobar</span> so you will need to translate this too.<br />
<br />
Next, we need to fix a small discontinuity in user handling between Subversion and Git. Subversion users are simple usernames (like <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">gregf</span>), whereas Git requires a user's name and their email address. Create a file called <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">authors.txt</span> containing an entry for each of your subversion users (usually listed in <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">svn/myrepo/conf/passwd</span>):<br />
<pre style="background-color: #e8e8ff;">lisas = Lisa Simpson <lisa@duff.com>
barts = Bart Simpson <bart@duff.com>
</pre>
Finally. make sure you have <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">git</span> and <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">git-svn</span> installed. Google to find out how to do this on your operating system.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">2. Convert the Subversion Project to a Local Git Repository</span><br />
At the command line, enter:<br />
<pre style="background-color: #e8e8ff;"><span class="Apple-style-span" style="font-size: x-small;">git svn clone svn://myserver/myrepo/foobar --no-metadata -A authors.txt -b Branches -T Trunk -t Tags foobar_local</span>
</pre>
Make yourself some tea, coffee or martini - this will take a wee while, as it works through the entire history of changes in the Subversion project. Assuming all goes well, you should end up with a directory called <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">foobar_local</span> containing both your project, and more importantly a hidden directory called <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">.git</span>. To check, change into the directory and view the log:<br />
<pre style="background-color: #e8e8ff;"><span class="Apple-style-span" style="font-size: x-small;">cd foobar_local
git log
git branch</span></pre>
That last command doesn't show your branches! Don't panic, this will come right later.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">3. Convert Ignore Filemasks</span><br />
If you have set up filemasks for Subversion to ignore, we can easily convert these into the Git equivalent:<br />
<pre style="background-color: #e8e8ff;"><span class="Apple-style-span" style="font-size: x-small;">cd foobar_local
git svn show-ignore -i trunk > .gitignore
git add .gitignore
git commit -m 'Convert svn:ignore properties to .gitignore.'</span></pre>
Note that <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">Trunk</span> in Subversion has been lower-cased by Git to <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">trunk</span>.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">4. Create a Bare Git Repository</span><br />
So far you have a local git repository, but you need a bare repository to clone from. As part of this process, we set Git's symbolic <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">HEAD</span> to what was Subversion's (lower-cased) <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">trunk</span>, tell Git about any branches, and rename <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">trunk</span> to <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">master</span> (the convention in Git).<br />
<pre style="background-color: #e8e8ff;"><span class="Apple-style-span" style="font-size: x-small;">cd ..
git init --bare foobar_bare
cd foobar_bare
git symbolic-ref HEAD refs/heads/trunk
cd ../foobar_local
git remote add bare ../foobar_bare
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ../foobar_bare
git branch -m trunk master</span></pre>
After the push, Git will list the new branches. I told you not to panic!
<br />
<br />
<span class="Apple-style-span" style="font-size: large;">5. Convert Tags</span><br />
Actually, if you use tags you might still be worried that all your tags seem to be branches. Git handles tags very differently from Subversion - in Git tags are simply aliases to a particular commit, whilst Subversion treats tags as a dead-end branch. Git-svn simply renames these branches, so we still need to convert these branch-tags to real Git tags:<br />
<pre style="background-color: #e8e8ff;"><span class="Apple-style-span" style="font-size: x-small;">cd foobar_bare
git for-each-ref --format='%(refname)' refs/heads/tags |
cut -d / -f 4 |
while read ref
do
git tag "$ref" "refs/heads/tags/$ref";
git branch -D "tags/$ref";
done</span></pre>
<br />
<span class="Apple-style-span" style="font-size: large;">6. Push the Bare Repository to Bitbucket</span><br />
Don't worry, we're nearly there now. The last step is to push all the branches up to Bitbucket. Log in to Bitbucket, and create a new repository called <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">foobar</span>. It will then show you the command required to clone this repository, which will include the HTTPS URL. The command will be something like this:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ git clone <span class="Apple-style-span" style="background-color: #ffe599;">https://myname@bitbucket.org/myname/foobar.git</span></span><br />
<br />
The URL is the yellow section above. Now on your local machine, set the orgin and push the master (old trunk) and each branch to Bitbucket:<br />
<pre style="background-color: #e8e8ff;"><span class="Apple-style-span" style="font-size: x-small;">git remote add origin https://myname@bitbucket.org/myname/foobar.git
git push origin master
git push origin europe
git push origin us</span></pre>
Your project is now on Bitbucket.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">7. Tidy Up</span><br />
The only thing left now is to tidy up the local and bare repositories left on your machine.
<br />
<pre style="background-color: #e8e8ff;"><span class="Apple-style-span" style="font-size: x-small;">cd ..
rm -rf foobar_local
rm -rf foobar_bare</span></pre>
And... you're done. Time for another tea, coffee, or martini to celebrate your success!<br />
<br />
<br />
<div style="background-color: #e0f8f8; border: 1px dashed rgb(0, 191, 230); padding: 5px;">
<span class="Apple-style-span" style="background-color: #e0f8f8; font-size: x-small;">If this article has helped you, you can show your appreciation by telling any schools you are involved with about <a href="http://www.schoolconferences.com/">www.SchoolConferences.com</a>. This simple online app has already revolutionised parent-teacher conference evenings for over a thousand schools. See how easy it is to book conferences with the demonstration event code <b>HIGH4</b> for high schools, and <b>ELEM4</b> for elementary schools. Thanks!</span></div>
<br />Unknownnoreply@blogger.com9tag:blogger.com,1999:blog-22677069511884100.post-14339025402938377172011-06-22T19:33:00.000-07:002011-06-23T15:53:53.166-07:00High-Replication Migration Lessons<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-a0PwWYA3eEs/Tbd-79mrObI/AAAAAAAARO0/M3XbF28Nu0Q/s1600/appengine_lowres.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://2.bp.blogspot.com/-a0PwWYA3eEs/Tbd-79mrObI/AAAAAAAARO0/M3XbF28Nu0Q/s1600/appengine_lowres.png" /></a></div>It's about a month since the migration, so how's it gone, what did we learn? What's it done to latency, error rates and CPU consumption? Most of all, was it worth the pain?<br />
<br />
<a name='more'></a><b>Lesson 1: Understand how eventual consistency will affect your application.<br />
</b><br />
<br />
Our biggest fear was that there would be some inconsistencies between master-slave (MS) and high replication (HR) that would cause problems with our apps. And that fear was borne out when users started reporting that new objects were not showing up after they had created them. Some debugging found that the new objects <i>did</i> exist, and that we had been bitten by HR's <b>eventual-consistency</b>.<br />
<br />
The datastore is much more efficient getting objects by ID than via a query, so I tend to give an object an array of IDs referencing child objects. When I added a new child, I ran a query to rebuild this ID array, which guaranteed it's consistency. With HR, the new child didn't show up in the query results for some seconds, so because I was rebuilding the array immediately after adding the new child, it disappeared. And when another new child was added, the <i>previous</i> one appeared - but the new one didn't. Users hate that sort of thing!<br />
<br />
It was easily fixed - get_by_id() works as soon as an object is added, but queries do not. So we dispensed with the query, and simply added the ID to the array when we added a child (and removed it on deletion).<br />
<br />
<b>Lesson 2: Writes take longer and reads take about the same - but both are more consistent.</b><br />
<br />
Probably the next biggest issue is performance. Writes take longer, probably about 50% on average, and reads are just about even. However, performance is far more consistent - our milliseconds/request graphs are much flatter.<br />
<br />
With the MS datastore, we got occasional operations that took <i>much</i> longer than normal, and these seem to have largely disappeared with the HR.<br />
<br />
<b>Lesson 3: An order of magnitude less datastore timeouts.</b><br />
<br />
The biggest difference has been on our error rate. With MS, we got a daily crop of datastore timeouts, and with HR they are extremely rare. With a non-relational database datastore errors lead to inconsistent data, because one entity is updated but the next may not. Users hate inconsistencies, so reducing errors by an order of magnitude is a huge win.<br />
<br />
<b>Lesson 4: HR costs more in CPU time.</b><br />
<br />
Ah, but what about the cost? Google have equalised the storage costs between MS and HR, but you also pay for the CPU time to store and access your data. We're seeing a 30% increase in CPU after migrating, so HR will cost you more. How much depends on your application, but this may be also be moot when Google roll out their new instance-based billing.<br />
<br />
<br />
<br />
So was it worth the time and cost involved in migrating from MS to HR? <b>Yes.</b> The reduced error rate has made our service far more robust, and reduced support calls. Also, we no longer need to manage user's expectations around Google's planned maintenance periods, which reduces a lot of customer irritation. These advantages easily outweigh the extra performance and CPU costs.<br />
<br />
Finally, please leave a comment below about how your migration went, or if you have any questions about the process. If you'd like dedicated support with your migration, please contact me (greg at vig dot co dot nz).<br />
<br />
<br />
<div style="background-color: #e0f8f8; border: 1px dashed rgb(0, 191, 230); padding: 5px;"><span class="Apple-style-span" style="font-size: x-small;">If this article has helped you, you can show your appreciation by telling any schools you are involved with about <a href="http://www.schoolconferences.com/">www.SchoolConferences.com</a>. This simple online app (built on Appengine of course!) has already revolutionised parent-teacher conference evenings for over a thousand schools. See how easy it is to book conferences with the demonstration event code <b>HIGH4</b> for high schools, and <b>ELEM4</b> for elementary schools. Thanks!</span></div>Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-22677069511884100.post-33736936287754718532011-04-26T19:20:00.000-07:002011-12-14T14:38:25.410-08:00Migrating an App to High-Replication Datastore on Appengine<div style="text-align: left;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-a0PwWYA3eEs/Tbd-79mrObI/AAAAAAAARO0/M3XbF28Nu0Q/s1600/appengine_lowres.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://2.bp.blogspot.com/-a0PwWYA3eEs/Tbd-79mrObI/AAAAAAAARO0/M3XbF28Nu0Q/s1600/appengine_lowres.png" /></a></div>
<span class="Apple-style-span" style="font-family: inherit;">Google's Appengine now gives you the choice between the original master-slave datastore, and the new high-replication datastore. In a nutshell, high-replication provides more reliable storage (in terms of data security and latency) but has slower write times and costs considerably more.</span><br />
<br />
<span class="Apple-style-span" style="font-family: inherit;">The owners of almost all serious applications will choose the high-replication datastore. The extra cost is a small price to pay (literally) to escape the maintenance outages, datastore timeouts and high-latency requests that plague the master-slave datastore.</span></div>
<br />
<span style="font-family: inherit;">The following article explains</span><span class="Apple-style-span" style="font-family: inherit;"> how to switch a production application from master-slave to high-replication with the minimum impact on your users.</span><br />
<div style="text-align: left;">
<a name='more'></a></div>
<br />
<span class="Apple-style-span" style="font-family: inherit;">Lets assume the app is called <b>myapp</b>, and it is currently available at the URL <b>www.mydomain.com</b> through Google Apps for your Domain. If you use <b>myapp.appspot.com</b> instead, you become reliant on Google to create an alias from the old app to the new one, and so have less control over the timing and length of the cut-over when your app isn't available to users. I'm also assuming you have billing enabled, because transferring data is likely to exceed the free quotas (particularly CPU time).</span><br />
<span class="Apple-style-span" style="font-family: inherit;"> </span><br />
<br />
Many of the steps below involve switching between different Google accounts, in Appengine and Google Apps. Ideally you would log in to each account/service in a different tab before you started, but you can only have one account open in each browser. This means you either have to log in and out several times, or you can try running different browsers (Firefox, Chrome, IE) or different virtual machines for each account you need. In particular, the Datastore Admin page in Appengine admin may have a blank content section - signing out and in again usually cures this.<br />
<div>
</div>
<div>
<br />
The biggest challenge in the migration is transferring all the data from myapp to myapp-hr. You probably don't want to create new entities with the high-level datastore API on myapp-hr because they will have different keys, and if you use keys or ids to link entities you'll need to map between the myapp keys and the myapp-hr keys. There are several possible approaches to this:</div>
<div>
<ol>
<li>Use the appcfg.py utility to download data from myapp, and then upload it again to myapp-hr. This preserves keys, but is very slow because it is a single thread and all data has to be transferred across the internet to your machine and then back again - a two-stage serial transfer across many networks.</li>
<li>Write your own transfer utility with the Remote API so you can select which data to move at which time. This might allow you to limit the cut-over data transfer to active data only, and you can transfer the rest at your leisure during the preparation stage. However, this entails some fairly complex development and testing - you need to preserve keys (or map between old and new) and ideally have several threads running.</li>
<li>Write your own transfer utility using the Urlfetch API on myapp calling a handler on myapp-hr. As well as limiting the cut-over data transfer to active data only, it happens across Google's local network and both read and write occur simultaneously. This will be the fastest option, but again requires substantial development and testing.</li>
<li>Use the little-known <a href="http://code.google.com/appengine/docs/adminconsole/datastoreadmin.html">Datastore Admin</a> tool. This has all the advantages - across Google's local network, multiple threads and simultaneous read/write - except it transfers <i>all</i> data (you <i>can</i> select by entity kind, but usually each kind will have active and non-active entities so this isn't useful).</li>
<li><i>[UPDATE] There is now a built-in migration tool in the admin console. I haven't used this, but this would now be my first choice.</i></li>
</ol>
<ul></ul>
<div>
My attempts to use appcfg.py to do the transfer ended when it took over an hour to transfer a fairly small dataset. I then looked at developing my own transfer process, before a kind soul on the Appengine group pointed me to the datastore admin documentation. This transferred 140,000 entities (110MB) in 20 minutes - and chewed through 5 hours of CPU time!</div>
<div>
</div>
<br />
If you have massive amounts of data, you may choose option 3 to minimise the cut-over transfer time (and the cost of the associated CPU time), but otherwise I recommend the datastore admin tool - it's simple, fast and safe.<i> [UPDATE]The built-in migration tool in the admin console would now be my first choice.</i></div>
<div>
</div>
<div>
<br />
OK, let's get started! Each instruction is colour-coded for <span class="Apple-style-span" style="color: #073763;">Appengine admin</span>, <span class="Apple-style-span" style="color: #660000;">Google Apps admin</span>, <span class="Apple-style-span" style="color: #4c1130;">Google Apps mail,</span> and <span class="Apple-style-span" style="color: #274e13;">shell/terminal.</span></div>
<div>
</div>
<div>
<div style="margin: 0px;">
<br />
<span class="Apple-style-span" style="font-family: inherit;">The migration process is broken into three stages:</span></div>
<ul>
<li>Preparation, which can be carried out at any time before the cut-over.</li>
<li>Cut-over, when your app is read-only for users.</li>
<li>Tidying up, done after the cut-over.</li>
</ul>
<div style="font-size: 120%; padding: 10px 0px;">
<i><b>Preparation</b></i></div>
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">First of all, we set up the new application, enable the datastore admin tool on the old and new apps, and test the data transfer process.</span></div>
<div>
<ul>
<li><span class="Apple-style-span" style="color: #073763;">Create a new app called myapp-hr. Edit the storage options and select High Replication.</span></li>
<li>If your app sends emails, <span class="Apple-style-span" style="color: #073763;">invite the sending address(es) to become developers for myapp-hr.</span> <span class="Apple-style-span" style="color: #4c1130;">Accept the invitation(s) in Google Apps for mydomain.com.</span></li>
<li><span class="Apple-style-span" style="color: #073763;">Enable billing for myapp-hr, and allocate resources as required. A large allocation for CPU time is a good idea - the transfer may require more than you expect.</span></li>
<li><span class="Apple-style-span" style="color: #274e13;">Create <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">myapp-hr</span> source directory, and copy everything from <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">myapp</span> source directory.</span></li>
<li><span class="Apple-style-span" style="color: #274e13;">Edit the new myapp-hr <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">app.yaml</span>:</span></li>
<ul>
<li><span class="Apple-style-span" style="color: #274e13;"><span class="Apple-style-span" style="font-family: Arial, sans-serif;">Change </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">application:</span><span class="Apple-style-span" style="font-family: Arial, sans-serif;"> from </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">myapp</span><span class="Apple-style-span" style="font-family: Arial, sans-serif;"> to </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">myapp-hr</span><span class="Apple-style-span" style="font-family: Arial, sans-serif;"><br />
</span></span></li>
<li><span class="Apple-style-span" style="color: #274e13;"><span class="Apple-style-span" style="font-family: Arial, sans-serif;">Add builtins section:</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> builtins:<br />
- remote_api: on<br />
- datastore_admin: on</span></span></li>
</ul>
<li><span class="Apple-style-span" style="color: #274e13;">Create file <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">appengine_config.py</span> containing (as a single line):</span><br />
<span class="Apple-style-span" style="color: #274e13;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> remoteapi_CUSTOM_ENVIRONMENT_AUTHENTICATION</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> ('HTTP_X_APPENGINE_INBOUND_APPID',['myapp'])</span></span></li>
<li><span class="Apple-style-span" style="color: #274e13;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"></span></span><span class="Apple-style-span" style="color: #274e13;">Upload the new myapp-hr application:</span></li>
<li> <span class="Apple-style-span" style="color: #274e13; font-family: 'Courier New', Courier, monospace;"> appcfg.py upload myapp-hr</span></li>
<li>Enable datastore admin in myapp. <span class="Apple-style-span" style="color: #274e13;">Edit the original myapp <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">app.yaml</span><span class="Apple-style-span" style="font-family: inherit;"> to add builtins section:</span></span><br />
<span class="Apple-style-span" style="color: #274e13;"> <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> builtins:<br />
- remote_api: on</span></span></li>
<li><span class="Apple-style-span" style="color: #274e13;">Upload the myapp application:</span><br />
<span class="Apple-style-span" style="color: #274e13; font-family: 'Courier New', Courier, monospace;"> appcfg.py upload myapp</span></li>
<li>Test the application is available by going to http://myapp-hr.appspot.com.</li>
<li><span style="color: #073763;">Test the data transfer process to find out how much real and CPU time it takes.</span><span class="Apple-style-span" style="color: #073763;">Switch to myapp (<span class="Apple-style-span" style="color: #073763;"><i>not myapp-hr</i></span><span class="Apple-style-span" style="color: #073763;">)</span> application</span><span class="Apple-style-span" style="color: #073763;">.</span></li>
<li><span class="Apple-style-span" style="color: #073763;">On the Datastore Admin page, select all entity kinds, and click "Copy to Other App".</span></li>
<li><span class="Apple-style-span" style="color: #073763;">Change <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">{TARGET_APPID}</span> to <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">myapp-hr</span> in target URL, leave extra header as is.</span></li>
<li><span class="Apple-style-span" style="color: #073763;">Note start time, go to the Task Queues page, refresh until default task queue is empty, and note end time.</span></li>
<li><span class="Apple-style-span" style="color: #073763;">Switch to myapp-hr application, and go to the Dashboard page to see how much CPU time the data transfer required.</span></li>
<li><span class="Apple-style-span" style="color: #073763;">Still in myapp-hr (<i>double-check it's <span class="Apple-style-span" style="color: #073763;">not myapp!</span></i><span class="Apple-style-span" style="color: #073763;">)</span>, go to the Datastore Admin page, </span><span class="Apple-style-span" style="color: #073763;">select all entity kinds, and click "Delete Entities" to clear all the data you just transferred.</span></li>
<li>Change your local copy of myapp so it shows a maintenance message.</li>
</ul>
<div style="font-size: 120%; padding: 10px 0px;">
<b><i>Cut-Over</i></b></div>
<div>
Now you know how long the data transfer takes, you can warn your users when and for how long the system will be under maintenance. I usually announce a pessimistic figure of at least twice what I think it will take, to cover any problems that might arise. If data transfer takes 10 minutes, you should be able to complete the cut-over in 15 - but I told users maintenance would take an hour. If everything goes well, users will be happy the outage was shorter than announced. Otherwise you have more time to resolve the issue or revert to the old application.<br />
<br />
Here we go!<br />
<ul>
<li><span class="Apple-style-span" style="color: #274e13;">Upload the myapp application to show the maintenance message:</span><br />
<span class="Apple-style-span" style="color: #274e13; font-family: 'Courier New', Courier, monospace;"> appcfg.py upload myapp</span><br />
</li>
<li><span class="Apple-style-span" style="color: #073763;">Switch to myapp application.</span></li>
<li><span class="Apple-style-span" style="color: #073763;">Set myapp to read-only mode by clicking "Disable writes..." on the Application Settings page.</span></li>
<li><span class="Apple-style-span" style="color: #274e13;"></span><span class="Apple-style-span" style="color: #073763;">On the Datastore Admin page, select all entity kinds, and click "Copy to Other App".</span></li>
<li><span class="Apple-style-span" style="color: #073763;"></span><span class="Apple-style-span" style="color: #073763;">Change <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">{TARGET_APPID}</span> to <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">myapp-hr</span> in target URL, leave extra header as is.</span></li>
<li><span class="Apple-style-span" style="color: #073763;"></span><span class="Apple-style-span" style="color: #073763;">Go to Task Queues page and wait until default task queue is empty.</span></li>
<li><span class="Apple-style-span" style="color: #660000;">Log in as admin, click on myapp in the Google App Engine section, and then delete www.mydomain.com under Web address.</span></li>
<li><span class="Apple-style-span" style="color: #073763;">Switch to myapp-hr application, go to the Application Settings page and click "Add Domain".</span></li>
<li><span class="Apple-style-span" style="color: #660000;">Accept agreement, click "Activate this service", add new URL "www" and click "I've completed these steps".</span></li>
<li>Go to www.mydomain.com and wait for the maintenance message to disappear, which means the domain is now pointing to myapp-hr. This should happen within a minute.</li>
<li>Test the application. On one occasion I found issues that disappeared in a few minutes (before I could properly diagnose them), so don't panic if odd things happen at first.</li>
</ul>
<div style="font-size: 120%; padding: 10px 0px;">
<b><i>Tidy Up</i></b></div>
<div>
All that's left to do is remove the datastore admin capability from your application, clear all the data from the old application, and remove it so it doesn't count against your application limit.</div>
<div>
<ul>
<li><span style="color: #274e13;">Delete</span><span class="Apple-style-span" style="color: #274e13;"> file <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">appengine_config.py</span> from myapp-hr.</span></li>
<li><span class="Apple-style-span" style="color: #274e13;">Remove <span style="font-family: 'Courier New', Courier, monospace;">datastore_admin</span> and </span><span class="Apple-style-span" style="color: #274e13;"><span style="font-family: 'Courier New', Courier, monospace;">remote_api</span></span><span class="Apple-style-span" style="color: #274e13;"> from builtins section of myapp-hr <span style="font-family: 'Courier New', Courier, monospace;">app.yaml</span>.</span></li>
<li> <span class="Apple-style-span" style="color: #073763;">Change the CPU time allocation to it's normal level on the myapp-hr Billing Settings page.</span></li>
<li><span class="Apple-style-span" style="color: #073763;">Switch to myapp applicatio<span style="color: #073763; font-size: small;">n, and disable billing on the Billing Settings page.</span></span><span style="color: #073763; font-size: small;"> </span><span style="color: #073763; font-size: small;"><span style="font-family: Arial, sans-serif;">Also cancel recurring charge authorization. This will take several days to complete.</span></span></li>
<li><span style="color: #073763; font-size: small;"><span style="font-family: Arial, sans-serif;">Go to the Application Settings page, and disable the application. You can then request permanent deletion, which will take 72 hours to occur. </span></span></li>
</ul>
</div>
<a href="http://neogregious.blogspot.com/2011/06/high-replication-migration-lessons.html">Read our migration post-mortem.</a><br />
<br />
Finally, please leave a comment below about how your migration went, or if you have any questions about the process. If you'd like dedicated support with your migration, please contact me (greg at vig dot co dot nz).<br />
<br />
<br />
<div style="background-color: #e0f8f8; border: 1px dashed rgb(0, 191, 230); padding: 5px;">
<span class="Apple-style-span" style="font-size: x-small;">If this article has helped you, you can show your appreciation by telling any schools you are involved with about <a href="http://www.schoolconferences.com/">www.SchoolConferences.com</a>. This simple online app (built on Appengine of course!) has already revolutionised parent-teacher conference evenings for over a thousand schools. See how easy it is to book conferences with the demonstration event code <b>HIGH4</b> for high schools, and <b>ELEM4</b> for elementary schools. Thanks!</span></div>
</div>
</div>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-22677069511884100.post-65331673104249454332011-04-02T20:26:00.000-07:002011-11-05T21:58:18.525-07:00Installing Multiple Versions of Python on LinuxThe Google Appengine SDK requires at least version 2.5 of python, or else it spits the dummy. To compound matters, Google has announced that version 1.4.3 of the SDK is the last to support python 2.5, and all newer versions will require python 2.6. <i>[Update: and now you need python 2.7 for multi-threaded apps.]</i><br />
<br />
So how do you install later versions of python without impacting the system default version - which is required for all sorts of vital system services?<br />
<br />
<a name='more'></a><br />
The trick is to use the <span style="font-family: 'Courier New', Courier, monospace;">prefix</span> and <span style="font-family: 'Courier New', Courier, monospace;">altinstall</span> options when building the new version of python. Make sure you have libsqlite3-dev and libssl-dev installed or Sqlite3 and SSL support won't be compiled in.<br />
<br />
<code><span class="Apple-style-span" style="font-size: x-small;">cd ~/src<br />
wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2<br />
tar -jxvf Python-2.7.2.tar.bz2 <br />
rm Python-2.7.2.tar.bz2<br />
cd Python-2.7.2/<br />
./configure <b>--prefix=/opt/python2.7</b><br />
make<br />
sudo make <b>altinstall</b><br />
sudo <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">ln -s /opt/python2.7/bin/python2.7 /usr/bin/python2.7</span></span></code><br />
<br />
You (and the system) will continue to use the old version by calling <span style="font-family: 'Courier New', Courier, monospace;">python</span>, and you can use the new version by calling <span style="font-family: 'Courier New', Courier, monospace;">python2.7</span>.<br />
<br />
So to install sqlite3 support for python2.7 for instance:<br />
<code><span class="Apple-style-span" style="font-size: x-small;">cd ~/src <br />
wget http://pysqlite.googlecode.com/files/pysqlite-2.6.3.tar.gz<br />
tar -xzf pysqlite-2.6.3.tar.gz <br />
rm pysqlite-2.6.3.tar.gz<br />
cd pysqlite-2.6.3/<br />
sudo <b>python2.7</b> setup.py build_static install</span></code>Unknownnoreply@blogger.com6tag:blogger.com,1999:blog-22677069511884100.post-53025334832071666032011-02-12T01:19:00.000-08:002011-02-28T00:01:03.575-08:00Cascade Saddle 2010<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://picasaweb.google.com/g.fawcett/CascadeSaddle2011" target="_default"><img border="0" height="99" src="http://3.bp.blogspot.com/-AGfag1RIxO4/TVY5gRgzhOI/AAAAAAAAQxQ/3mUWtIgOn5k/s200/map.JPG" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Click map to go to photo gallery</td></tr>
</tbody></table>The Cascade Saddle track has a reputation of being one of New Zealand's harder walks, compensated with awe-inspiring views over the Matukituki valley, Mount Aspiring and the Dart Glacier. Who could resist?<br />
<br />
<a name='more'></a>My 16-year old sons Olly and Jasper are Venture Scouts, and part of their Chief Scout Award involves planning and carrying out a journey. They organised to take three days off their holiday jobs, but as the time came closer the weather forecast narrowed down our options. Luckily the Cascade Saddle was a possibility - I had wanted to do this walk for years, and the boys eventually succumbed to my arguments.<br />
<br />
So as soon as Olly got off work on Wednesday 26th January, we loaded up the truck with our gear and set off - in pouring rain. It's about two hours drive from our home in Arrowtown to Raspberry Creek, and Nicole ensured our undying love by dropping us off - then driving all the way back again. After having my shorts licked by a cow as I did up my boots (a good omen?), we set off in showers to walk the two easy hours up to Aspiring hut. Arriving just before dark, we made some hot chocolate in this prince of mountain huts, chatted to a group of Dunedin Sea Scouts, and bunked down ready for the big day.<br />
<br />
<span style="font-size: large;">Day One</span> <br />
We were woken at 6:30 by raucous keas, only to find they had taken to Jasper's orthotic insoles. Luckily they were still mostly intact, and we were glad to see the weather had cleared according to plan. After a quick breakfast we set off at about 7:30.<br />
<br />
The day was broken into four main sections: a steep climb from Aspiring Hut to the pylon; across Cascade Creek to the viewpoint; descending alongside the Dart Glacier to the valley floor; and a flat walk along the valley to Dart Hut.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-Qh6lnNQjZuQ/TVZBWEsRiII/AAAAAAAAQyo/oQeyjvN0flo/s1600/IMG_0033.JPG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="150" src="http://2.bp.blogspot.com/-Qh6lnNQjZuQ/TVZBWEsRiII/AAAAAAAAQyo/oQeyjvN0flo/s200/IMG_0033.JPG" width="200" /></a></div>The first section was a killer. The track was very steep, with frequent large steps that soon had my legs screaming. The boys... not so much. It became obvious that they were far fitter than I, stopping every few minutes to let their decrepit father catch up. We broke out of the bush after about two hours, getting stunning views of the Matukituki Valley and Mount Aspiring. Unfortunately the track got even steeper, and the tales of injuries and deaths on this section seemed only too credible. Steep bluffs were everywhere, and a stumble could easily be disastrous.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-8MLoecfDAlY/TVZJiiNQZTI/AAAAAAAAQzE/cEXgBlG8JnM/s1600/IMG_0050.JPG" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="150" src="http://3.bp.blogspot.com/-8MLoecfDAlY/TVZJiiNQZTI/AAAAAAAAQzE/cEXgBlG8JnM/s200/IMG_0050.JPG" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The pylon</td></tr>
</tbody></table>The boys scampered on like mountain goats, and I toiled after them. The GPS was useful, mainly for the altitude readout - we paused for a few minutes after every hundred metres we gained. Eventually we crested the last bluff, the track mercifully levelled off, and we reached the pylon. At just over 1800m, we'd climbed 1400m in just over four hours. We munched some scroggin, and took in the incredible views - Raspberry creek was still in view, far below us, as was Aspiring Hut. Across the valley we could just make out French Ridge Hut, and we started plotting a visit. Above the hut stood Mount Aspiring cloaked in glaciers.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-cC7BBRCh_WI/TVZNteX-YUI/AAAAAAAAQzo/HuvCZpi3AoE/s1600/IMG_0063.JPG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="150" src="http://2.bp.blogspot.com/-cC7BBRCh_WI/TVZNteX-YUI/AAAAAAAAQzo/HuvCZpi3AoE/s200/IMG_0063.JPG" width="200" /></a></div>We then started to walk down a far more reasonable gradient to Cascade Creek. This runs through a delightful valley with snow grass meadows - maybe it was the euphoria of reaching the top, but it seemed to me to be absolutely idyllic. We reached the viewpoint after an hour, and settled in a small sheltered spot to wolf our lunch - fruit loaf with camembert! We took an hour to enjoy views from the precipitous edge, and laze in the sun under the watchful eye of a Kea.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-gPa4vcqLXyA/TVZVBiGDfKI/AAAAAAAAQ0A/-lkqPEDK94w/s1600/IMG_0077.JPG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="150" src="http://1.bp.blogspot.com/-gPa4vcqLXyA/TVZVBiGDfKI/AAAAAAAAQ0A/-lkqPEDK94w/s200/IMG_0077.JPG" width="200" /></a></div>After recharging our batteries, we set off down to the Dart Valley. Jasper's toes started to get painful in his cheap boots, but truly magnificent views of the Dart Glacier helped distract us. "Awesome" is overused these days, but that's what the glacier was. From the unblemished snow around the peaks, it ran down until the ice started cracking across the glacier as it rode over a lip of rock far beneath. Then longitudinal crevices appear as it is forced to turn a right angle by the Dart Valley, all the while picking up more and more shattered rocks from the valley sides. The tail is so encrusted with debris that it doesn't seem to be ice at all. We reached the tail after about an hour's walk down fairly loose rock on the south side of the valley. <br />
<br />
As often happens, the last section was the hardest, despite being pretty much flat. We had put on sun-screen when we left the bushline that morning, but because we hadn't brought enough, we didn't apply more. The sun was hot, there was no breeze and the raw rock we were walking over reflected the heat straight back at us. Jasper's toes were very sore by this stage, so none of us really enjoyed the experience at this stage.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-PhnC4MhePcM/TVZX4sn1e7I/AAAAAAAAQ0k/nH0C8jjsa0s/s1600/IMG_0088.JPG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="150" src="http://2.bp.blogspot.com/-PhnC4MhePcM/TVZX4sn1e7I/AAAAAAAAQ0k/nH0C8jjsa0s/s200/IMG_0088.JPG" width="200" /></a></div>Which was a shame - it was almost like walking through time. At the glacier tail, no life existed and the valley was bare raw rock. But as we travelled down, mosses and small spruts of grass started appearing; then grass became thicker, and bush started appearing; and when we reached the hut 2.5 hours later, we were in mature beech forest.<br />
<br />
We were all relieved - Jasper's boots came off in a flash, and we stumbled down to the river for a wash. The cold water was heavenly on our necks, arms and legs, and we realised how burnt we all were. All up we'd taken ten hours and ten minutes to get from Aspiring Hut to Dart Hut, including a good three-quarters of an hour idling at the viewpoint.<br />
<br />
That evening was a bit of a blur - we rustled up some macaroni cheese with sliced cabanossi sticks, wolfed a pottle of mango on top of creamed rice, and hit the sack. <br />
<br />
<span style="font-size: large;">Day Two</span><br />
After a solid night's sleep, we cooked porridge on the trusty coke-can cooker, and packed ready for the rain that was forecast. It was overcast as we set off, and after a hour it started drizzling. We marched on, through alternating beech forest and grassy flats, and the rain got stronger. My gaiters had kept my boots dry through all the streams the day before, but I stupidly didn't put them on when we started out. My boots got soaked just from brushing past the grass on the flats, and putting them on halfway along closed the stable door after the horse had bolted.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://lh4.googleusercontent.com/-903u8UM0f9c/TVZZ5WUks0I/AAAAAAAAQ1A/jEVh6p0fO-k/s1600/IMG_0112.JPG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="150" src="https://lh4.googleusercontent.com/-903u8UM0f9c/TVZZ5WUks0I/AAAAAAAAQ1A/jEVh6p0fO-k/s200/IMG_0112.JPG" width="200" /></a></div>We didn't feel like stopping in the rain for lunch and so pushed on to Daley's Flat, arriving about 1:30pm. Despite the rain, the warden was busy with earthworks remodelling the path around the hut. It was a mud bath, and the voracious sandflies made it even less appealing to spend any time outside.<br />
<br />
We had a leisurely lunch of soup, ryveta, cream cheese and sun-dried tomatoes. Then an afternoon snooze as the rain poured down, not waking till late afternoon. Dinner was dehydrated something (who can tell!) and a bar of chocolate. Jasper and Olly played werewolves (a version of mafia) with the rest of the hut inhabitants, then we went off for another good night's sleep.<br />
<br />
<span style="font-size: large;">Day Three</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://lh3.googleusercontent.com/-FokALD-6kmg/TVZZ6bQvz7I/AAAAAAAAQ1Q/vWbBGT9XImY/s1600/IMG_0120.JPG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="150" src="https://lh3.googleusercontent.com/-FokALD-6kmg/TVZZ6bQvz7I/AAAAAAAAQ1Q/vWbBGT9XImY/s200/IMG_0120.JPG" width="200" /></a></div>We woke to find the rain had stopped, and after porridge packed for the last time and rushed out the door to avoid the swarms of sandflies in the porch. The lunch we knew Nicole was bringing in was a constant driver keeping us moving, and we covered the ground at a good pace. Although it had stopped raining, the low cloud obscured Mount Earnslaw to our left. The track was similar to the previous day, with bush sections around bluffs followed by grassy flats.<br />
<br />
And before we know it we were skirting Chinaman's Bluff, seeing parked cars through the trees, and sure enough there was Nicole waiting patiently for us. It was lovely to see her, and even lovelier to see the fine feed she had brought with her - bacon and egg pie and a potato salad!<br />
<br />
It was a good trip. Apart from Jasper's toes and sunburn all round, we had no injuries, and my knees handled the climb and descent without a murmur. We got the weather right, doing the potentially dangerous Cascade Saddle on the one fine day we got. We learned not to leave orthotic insoles anywhere keas can get at them, and apart from Olly's mug cracking, our gear performed well. The boys did a sterling job of organising and executing the walk, and I was proud to have such capable companions for this unforgettable adventure.Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-22677069511884100.post-63821207548711682152011-01-15T20:07:00.000-08:002011-01-15T20:49:09.875-08:00DKIM for Google Apps (but not Appengine)In the ongoing war against spam, DKIM (Domain Key Identified Mail) seems to be the best candidate so far - if all legitimate email users started it, it would be the end of anonymous spam. This article explains DKIM, why you should implement it <i>now</i>, and how to do this with Google Apps.<br />
<a name='more'></a><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/_bkHz6v1q_nI/TTJoW4BthvI/AAAAAAAAQxA/Ey4IN8r1lqo/s1600/nospam.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="166" src="http://3.bp.blogspot.com/_bkHz6v1q_nI/TTJoW4BthvI/AAAAAAAAQxA/Ey4IN8r1lqo/s200/nospam.jpg" width="200" /></a></div>DKIM allows receiving mail servers to reliably check an email is actually from the domain it says it's from, using a cryptographic key from your domain name service (DNS). This doesn't stop spam, but it makes senders responsible. If you get an unwanted DKIM signed message, you know exactly who sent it so you can take measures against them - blocking their domain perhaps, or suing them.<br />
<br />
More and more spam filters are checking for DKIM signatures, and if your messages are signed they usually bypass other spam checks and are delivered immediately. So DKIM will improve your delivery statistics <i>right now</i>. And as more and more organisations implement DKIM, unsigned messages will become increasingly likely to be classed as spam, and so more and more organisations will implement DKIM to ensure their messages get through.<br />
<br />
The faster DKIM spreads, the sooner anonymous spam will become history. We'll still need to manage spam because legitimate organisations will continue to do it, but after a few go bust from CAN SPAM fines even this should dry up. <b>Bottom line - unless you are a spammer, it is very much in your interests to implement DKIM as soon as possible.</b><br />
<br />
If you use Google Apps for your email, the good news is that DKIM is very easy to set up. Here's how...<br />
<ol><li>Log in as the administrator to your Google Apps dashboard. </li>
<li>Click on the <b>Advanced tools</b> tab and then the link to <b>Set up email authentication (DKIM)</b>.</li>
<li>Click on <b>Generate new record</b>.</li>
<li>Change the prefix selector from <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">google</span> to your organisation (if you want to - this isn't important at all).</li>
<li>This should result in a DNS Host name <span class="Apple-style-span" style="background-color: yellow; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><i>something</i>._domainkey</span> and a TXT record value <span class="Apple-style-span" style="background-color: yellow;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">v=DKIM1; k=rsa; t=y; p=lotsOfWeirdCharacters</span></span></li>
<li>Do not click the <b>Start Authentication</b> button yet!</li>
<li>Now go to your registrar's DNS control panel. I can't give exact instructions here because every control panel is different, but you want to find where you can set the A, CNAME, MX and TXT records for your domain name.</li>
<li>Add a TXT record with the host name and TXT record values from step 5. (Some control panels don't provide a </li>
<li>Wait a day or so to let the new DNS records percolate through the global name servers. If you're impatient and have dig, try <span class="Apple-style-span" style="background-color: yellow; font-family: 'Courier New', Courier, monospace; font-size: x-small;">dig txt <i>something</i>._domainkey.<i>your.domain.name</i></span> until it returns the key.</li>
<li>Log back in to your Google Apps dashboard, and go to the <b>Advanced tools</b> tab and the link to <b>Set up email authentication (DKIM)</b>, and <b>c</b>lick the <b>Start Authentication</b> button.</li>
</ol><div>Once you've done this, messages from your Google Apps domain* should contain a DKIM signature header. To test this, log in to your Google Apps account and send an email to another Gmail account. When it arrives, click the <b>show details</b> link - if all is well, there will be a new line saying <b>Signed by</b> giving your domain name.</div><div><br />
</div><div>* But not messages sent from Appengine, sadly. Please <a href="http://code.google.com/p/googleappengine/issues/detail?id=3161">star this issue</a> if this affects you. I'll update this post if the situation changes.</div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-22677069511884100.post-32180889972224039992011-01-10T16:44:00.000-08:002017-09-09T17:16:41.313-07:00About Igregious<h3>
Why?</h3>
Bloated though the blogosphere already is, I occasionally feel the need to make odd thoughts public. I hope I can add more signal than noise, and that you find something interesting here.<br />
<a name='more'></a><h3>
What?</h3>
"Igregious" is just a nice juxtaposition of my name, the egregious use of "i" to prefix everything, and an available domain name.<br />
<h3>
Who?</h3>
I'm Greg Fawcett, and I'm lucky enough to live in one of the most beautiful places in the world (Arrowtown in the heart of the Southern Alps of New Zealand), with the most gorgeous woman in the world (Nicole, a primary school teacher).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/_bkHz6v1q_nI/TSvBCprfTlI/AAAAAAAAQw8/N8uVCvYkt0k/s1600/IMG_5120.JPG" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="209" src="https://4.bp.blogspot.com/_bkHz6v1q_nI/TSvBCprfTlI/AAAAAAAAQw8/N8uVCvYkt0k/s320/IMG_5120.JPG" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Samoa has more palm trees than Arrowtown</td></tr>
</tbody></table>
We have two sons Olly and Jasper, who seem to be turning into healthy, pleasant and intelligent young men. I play the tin whistle and ukulele (badly), love tramping, and help run the Arrowtown Scout Group.<br />
<br />
After knocking off a Physics B.Sc. and an Electronics M.Sc. in the United Kingdom, I got my most important education when I took off round the world for two years. I worked on a bulk carrier from Greece to India; enjoyed the Buddhists in Dharamsala; developed noise calculation software in Singapore; sprayed for cockroaches in Broome; fixed Commodore 64s in Darwin; organised hundreds of cables for a SCADA system in Sydney; played catch-me-if-you-can with police launches around nuclear-powered warships in Sydney Harbour; and discovered that New Zealand was where I wanted to spend the rest of my life.<br />
<br />
I also found that software development suited me, and luckily I suited software development too. Starting with C on DOS , working through C++ on Windows, back to C on Linux, PHP on LAMP and now... Python on everything!<br />
<br />
I started a company (Virtual Industries Group) in 2001 to exploit the shift from PC-based to online applications. Our first major application <a href="https://www.schoolinterviews.co.nz/" target="_blank">School Interviews</a> makes online bookings for parent-teacher interviews easy, and it hit a chord with schools. We currently provide our service to over 4,000 schools and two million parents in 30 countries. That was followed by <a href="http://www.carebookings.co.nz/" target="_blank">Carebookings</a> for before- and after-school care, and <a href="https://www.messagemyway.com/" target="_blank">MessageMyWay</a> making parent communications easy.Unknownnoreply@blogger.com