Skip to content
Categories:

How to be on-call

A few years ago at Arctic Wolf I put together a talk titled “How to be on-call”, in response to the rapid growth of the organization and increasing number of on-call schedules. The talk turned out to be very popular and the recording became part of the onboarding process for new employees. While some of the talk was company-specific, much of it is applicable to the broader software industry, and I’ll share some of the highlights here.

I have been on-call for most of my career and led teams with on-call rotations, and have a lot of experience with the negative impact of on-call to my personal life and the lives of my colleagues. I’ve missed Christmas dinner (years later my Mom still brings it up), worked through weekends and nights, missed many kids’ events, and once juggled a fussy baby and an incident call at the same time. My goal is to make being on-call as sane as possible, balancing what the business needs with our collective personal lives. 

“#oncallsucks” was trending circa 2020, and in response Charity Majors summarized the on-call responsibilities in On Call Shouldn’t Suck: A Guide for Managers “It is engineering’s responsibility to be on call and own their code. It is management’s responsibility to make sure that on call does not suck. This is a handshake, it goes both ways, and if you do not hold up your end they should quit and leave you.”

A few ways that #oncallsucks:

  • Alert Fatigue – there are still too many horror stories of alerts constantly going off that are not actionable. Dan Ravenstone had a great talk at Monitorama “Thinking Critically About Alerting” about how to deal with this.
  • Onblame Oncall – quoted from a colleague – quite the opposite of blameless, where the on-call responder is blamed for whatever broke.
  • Always On – this was the culture at a former company, and you were always on-call even when on vacation. This was horrible in so many ways and led to quick burnout and high turnover.
  • There Can Only Be One: you are the only person on-call, there’s no escalation process to get help, and no team to back you up.

On-call does not have to suck, but it’s hard work for everyone to remove the suck and even harder to continually keep it from sucking.

My original talk was focussed heavily on the individual, but I’m going to focus here more on how a team can support each other, and the responsibilities of the leaders.

Why on-call?

The first question to ask especially as a leader is: do we need an on-call rotation? If your service has an actual SLA (with penalties etc) or is a true global service, or is supporting say a 24×7 manufacturing process, the answer is likely yes. There are many cases where the easy answer is yes, but under scrutiny that may not hold up. I have two examples were a 24 hour schedule could be reduced:

  • If the AWS Trust and Safety team contacts you about potential abuse on your account, you must respond within twenty four hours. If that happens on say early Saturday morning, a response cannot wait until Monday, but it also doesn’t mean you need to wake someone up. In that particular scenario we configured the paging software to only callout during daytime hours, seven days a week.
  • In the case of office network infrastructure, the temptation is to setup a 24×7 alerting process. In many cases a blip in the night is acceptable as long as the network is functioning by the time the first individuals arrive on site. Some creativity with scheduling means the IT staff can sleep most of the night before getting called out.

Don’t take it as gospel truth that you actually need a 24×7 schedule, be creative to reduce the impact to the on-call staff if at all possible.

So you are on-call…

Before you start your on-call rotation, it should be obvious that you should know how you’re going to be called, what you might get called for, and expectations on how to respond and escalate. I say it should be obvious, but it’s shocking how frequently this is all taken for granted.

At a minimum you should know:

  • How am I going to be called – PagerDuty or VictorOps or X? Any setup required?
  • Who gets to wake me up?
  • Am I on the escalation path for anything?
  • What are the expectations on response time when I do get called?
  • Where are the automated alerts defined, and where are the runbooks? (There are runbooks, right?)
  • How do I escalate to get help?

When you do get called, Don’t Panic! In my first few calls in the late 90’s I fought panic, because I didn’t have answers to the questions above and had no real escalation path. If you are the first responder, your job is to triage the alert and determine next steps. Those steps are highly contextual to your situation, but should always include the question “was it worth waking up for this alert”, and if the answer is no, tune the alert the next day.

Take care of yourself. Waking up in the middle of the night for a call always wrecks me – the adrenaline boost of the call means I won’t go back to sleep – anecdotally others struggle with the same. Do not hesitate to ask the team to cover the next night so you can get a night’s sleep.

Your team is on-call…

The best way for an on-call shift to be less stressful for me is to know that my team has my back, that I’m not in it alone. When I’m not on-call, the responsibility of supporting my colleagues falls on me. What does that actually mean in practice? A few suggestions:

  • Write good runbooks. At Arctic Wolf alerts were defined in a GitHub repository, and runbooks were attached and enforced at commit time – you could not commit an alert without a runbook. This brought some discipline to the process of creating alerts, encouraged careful thought on how to debug situations, and meant that when called in the middle of the night you had a really good starting point. Put a slightly different way – if you can’t write a runbook which contains actions on how to resolve the alert, the alert shouldn’t exist. There are a lot of great resources on how to write runbooks, this article is a great example.
  • Keep an eye on the on-call team member’s work load. If they’ve been up all night, organize a swap so they can take the night off. Some individuals (like me) have a tough time asking for help, but recognizing that in others I’ve had to almost forcibly take someone off a schedule to give them a break. The team’s leader can do this but it’s best if the team organically takes care of each other. 
  • Fostering a blameless culture means creating a psychological safety net, which is incredibly important when responding to incidents, especially when there’s significant pressure and a legitimate fear of “what if I make it worse”. I wrote an entire article on that: A few words about blameless culture.
  • Create a good on-call onboarding process that answers the questions above in the “So you are on-call…” section. The first few calls can be extremely stressful. I’ve seen successful onboarding including call shadowing, where a new team member was on the same rotation as a more senior member, and they answered calls together for a few weeks.

The short summary: take care of each other, treat your team members as you want to be treated.

You are a leader with an on-call team…

The best, albeit most controversial, way to support your on-call team is to pay them for the extra time. You are asking your team to work 168 hours per week, impacting their personal lives and their nights and weekends, and you are adding significant stress to their jobs. Paying an employee for that impact sends a strong clear message that you recognize and appreciate the extra hours. There are many ways to do this and the actual amount may be affected by the specific country’s legislation, but my favorite was eight hours of pay for a 24×7 shift and a minimum two hours of overtime for a callout.

The added benefit of paying for on-call is that you will bring visibility of the cost of running service to the business, with a budget line item containing the cost. The rest of the business understandably doesn’t know what it takes to be on-call but they certainly grok budgets, and having a common language to discuss the actual cost becomes very important when planning for hiring, discussing adding more services or products that will require 24×7 support, and setting SLA’s for products.

A leader must measure the number of callouts, and take immediate action if the number starts to increase. I have seen three common reasons why the number of callouts can increase:

  1. Scale – Both of my previous two organizations scaled more than 2x per year for multiple years in a row, and at some point a service would naturally start hitting scaling limits. While we tracked a myriad of metrics showing the four golden signals, an increased callout rate was definitive proof that the service was suffering, and new feature development needed to stop so the team could address scaling.
  2. Services change over time, and alerts need tuning. Alert fatigue, where an alert firing frequently is seen as “normal” and then ignored, is a real thing and should be squashed as fast as possible. Tune those alerts!
  3. Along with rapid development comes the possibility of a service accidentally becoming less resilient. This can easily happen even to a service that isn’t scaling and taking on extra load, and needs to be addressed.

How you measure and track this will depend on your org, but I highly recommend your team has visibility into your tracking and decision making process so they can see that you are looking out for them, and the actions you take to slow the callout rate will speak volumes.

A team needs to know that their leader has their backs and supports them. Again that’s pretty obvious, but a few ways you can do this:

  • Run interference for them. In a previous job I had an awesome manager who would stand in the path leading to our cluster of cubicles, preventing irate users and other leaders from interrupting us while we were resolving an incident. Modern equivalents to someone rushing your desk are out-of-band questions in Slack/Teams/Zoom/etc directly to your team members, and your team needs to feel comfortable to escalate those directly to you instead of trying to answer them.
  • Create a good escalation process. PagerDuty’s internal policy, documented here, is “Never Hesitate to Escalate” – which first means your team needs someone to escalate to, and then a Blameless way to do so without any fear of retribution regardless of the time of day.
  • I strongly recommend not scheduling project work for the on-call team member for the duration of their on-call shift, and build that into your project planning and estimation process. The individual can spend their time on bug hunts or research or annoyance-driven-development, and be ready for interrupt-driven work. A few of my teams implemented very successfully, it takes a lot of pressure off the individual and removes timeline risk from any planned project work.
  • If a response team is physically in the office, send in food for lunch or dinner, or if the team is remote, hand out coupons to food delivery services where possible. This seems like a small gesture, but I fondly remember running large incidents out of a conference room over the dinner hour, and a stack of pizzas appearing without warning, sending a strong signal that our leaders were watching out for us.

One of the harder things I’ve had to do with on-call teams is to formalize response times and expectations. With smaller teams it’s easy to build trust and have an informal “best effort” policy but as teams grow so does the expectation of a more formal policy. I know first hand the impact that on-call has on personal lives and the difficulty of adhering to any policy. That said these were the guidelines I’ve recommended in the past:

  • Be able to acknowledge an alert within 15 minutes.
  • Best effort to get online within 30 minutes. If, when you acknowledge the alert, you know you can’t hit the 30 minute response time, escalate. I had a long commute and I frequently would either find a local coffee shop or escalate and join the incident call when I arrived at home.
  • This can be a thorny one fraught with legal issues – be sober enough to be able to answer the call and work the problem to resolution. My personal rule of thumb is when on-call I should be legally able to drive, but that of course will depend on your local laws.
  • You cannot be on vacation (PTO) and on-call. I was surprised I had to even state this but ran into it a few times. Taking time off is incompatible with being on-call, that’s not negotiable.

The last way to make sure your on-call rotation is healthy is to have enough people on it. The absolute minimum for a 24×7 on-call rotation, assuming weekly shifts, is four individuals. Any less than that and you will burn out your team, and I’m hesitant to even give the number four because in most cases it’s too low. Being on-call one week out of four puts a significant burden on a team member and will cause them to leave. Having six people is better, and eight is good. If your team has less than four or even six, join forces with another team and share a rotation until you can grow the team. A few of my teams implemented that as they grew and split, and the positive side effect was the teams wrote really good runbooks to support each other. 

I’m convinced that on-call doesn’t have to suck, and while getting called in the middle of the night will never be fun, a good on-call culture can make it bearable.

Resources:

Posted in Tech.

Tagged with , .

By mike – October 18, 2023

A few words about Blameless culture

The concept of blameless culture has been around for a long time in other industries, and while the history isn’t clear, you could argue that it became an “official” part of the tech industry with the publication of the definitive book Site Reliability Engineering in 2016. 

My summary of blameless culture is: when there is an outage, incident, or escaped bug in your service, assume the individuals involved had the best of intentions, and either they did not have the correct information to make a better decision, or the tools allowed them to make a mistake.

During my tenure at Arctic Wolf I embraced and evangelized blameless culture, and brought it into every part of our organization that I could, including of course the post-mortem process. That culture took on a life of its own, as they do, and in some cases it grew in directions I hadn’t expected. Before I expand on what my definition of blameless is, I’ll share a personal story that’s somewhat embarrassing but frames my journey.

Once upon a time, when I was young and cocky, I was a Unix system administrator at a large manufacturing company. I responded to a call one night where the power for a data center was cut during maintenance of a fire suppression system in the building, causing immediate and expensive downtime to the manufacturing plant. At the beginning of the subsequent root cause meeting (the modern post-mortem language wasn’t common in the industry yet) I blurted out something to the effect that the meeting was a waste of time because we knew that the fire department tripped the power. Awkward silence ensued, and I discovered the gentleman sitting beside me was the fire chief. He had every right to express his anger at my blameful comment, but instead he quietly explained the process of determining root cause without assigning blame. As it turned out I was wildly wrong and the cause was, as is so often the case, cascading failure with a side dish of unexpected behavior, and no one could have predicted the end result.

That episode stuck with me for a few reasons, in large part because it was an embarrassing moment, but resulted in a few key lessons which took some time to percolate. That moment was a clear example of how to be blameful, not blameless. A more important reason this sticks in my memory is the fantastic leadership example the fire chief set for me, where a senior leader took the opportunity to mentor a newer employee through a potentially bad situation, and I’ve tried to emulate that wherever I can. He very likely has no idea how much impact he had on my career, but I’m very thankful for his patience with me that day.

My favorite public example of blameless behavior is when AWS S3 went down in the us-east-1 region on February 28, 2017, taking down a good portion of the Internet. AWS wrote in the public post mortem “an authorized S3 team member using an established playbook” … performed some maintenance, and the team member inadvertently introduced a typo which resulted in the outage. AWS’s response could have been “an employee made a mistake and we fired them” but instead my brief summary of the response was: the tool did not have enough safeguards and the service’s recovery was not fast enough. AWS addressed the situation by building in more safeguards, and implemented improvements to the service recovery time.

One unexpected direction the blameless culture took, which in hindsight I’m happy about, is to not name individuals beyond the boundaries of the team. This can result in the perception by senior leadership, who may not fully understand blameless culture, that their teams were not taking responsibility for their work. Talented developers in small teams are typically quick to point out their own mistakes, but I insist on not stopping post-mortems when someone points out their own mistake, and continuing investigating how to improve tools and information. The individual’s name is not important, almost immaterial to the postmortem, but the steps needed to improve the systems are the key parts to communicate to company leaders.

Finding the balance between taking responsibility for an issue and not naming names is difficult, but increasingly important as an organization grows. While I fully expect my team to own up to any mistakes, typos, assumptions etc during a post-mortem or in private, their names should be kept out of any docs published to the broader organization. Removing names provides psychological safety for team members, and prevents the temptation by leadership or others to blame individuals and take action. When presented with names I’ve seen leaders take immediate vindictive action, like termination, but also delayed action affecting compensation or promotion decisions later. In my ideal world the team as a whole, and their respective leadership chain takes responsibility for the events.

In my definition of blameless I “assume … best of intentions”. But. I’ve always made it clear to my teams there is an unwritten exception clause: if an incident/outage/bug was caused by either deliberate action, or by mistake and then covered up, there would be immediate consequences that most likely would result in the individuals finding new employment. Deliberate action causing damage is easy to understand and will likely involve law enforcement, but covering up mistakes deserves a bit more attention. I have two examples, one negative and one positive:

  1. I was part of the response team in situation where an individual made a mistake, and then attempted to cover their tracks. That resulted in a spectacular 48-hour outage and shortly thereafter termination of their employment. The org’s leadership was clear that the individual was not fired for making the mistake, but for the attempted coverup.
  2. Back in my sysadmin days a colleague accidentally deleted a production database. They immediately took their hands off their keyboard and said “hey team, I just messed up and need help”. We rallied together and came up with a solution without causing an expensive outage. My colleague could have covered it up, and I’m not sure we would ever have found why the database had disappeared, but because we had a blameless culture and great team, the individual had the psychological safety net to immediately ask for help.

Early in my career a leader told me, well before chaos engineering was a thing, “If you’re not breaking things once in a while you’re not working hard enough.” Anyone who runs production services lives this every day and knows there is a risk of making a mistake that will cause an outage. The best possible action when making a mistake is immediately escalate to your team and leadership and ask for help to help remediate the issue. At Arctic Wolf we had a cultural tenet “Bad News Fast” which we applied to projects and incidents alike, and contributed heavily to the company’s success. Team members were rewarded and praised for raising bad news fast, whether it was an issue in production, or an upcoming capacity planning threshold that would cause months of rework, or finding an issue in a project that would significantly delay the delivery.

Blameless culture cannot be reserved only for postmortems, it needs to be lived and promoted every day. Rob Zuber of CircleCI summarized it really well in The value of blameless culture: “Incidents are the microscope under which we tend to examine blameless culture” and “… the culture that shows up in your incident response is going to be a direct reflection of the culture that you build every day, with every action, under the most mundane circumstances.” 

Building a blameless culture is a complex topic, but one well worth investing a lot of time in, and will result in a healthy and high performing organization. Lots of great books and articles have been written on how to implement it, with a few key ones here: 

Want to learn more? Ping me via LinkedIn, I’d be happy to chat.

Posted in Tech.

By mike – September 11, 2023

A very long overdue update on rebuilding the Richardson

All those ribs finally done!

It has been five years since my last update. Replacing those ribs was extremely time consuming, each one was a three hour commitment needing helpers, and life just gets in the way. But it’s finally done, and my daughters will never willingly crawl under a boat again. On the right is a picture of the ribs installed with the new inner keel in place. The side keels (unsure what they’re called) are original and are mated to a newly milled matching one on the bottom. The dashboard is the original one as well, trying valiantly to hold the shape together.

The big question for most cedar strip rebuilds is whether or not to fibreglass the bottom. The Wooden Boat forums are full of discussion about that topic. The previous owner told me they’d put it in the water in April to let the wood swell, a day later they’d pump out the water and be good for the season. In its new life the boat live on a trailer, dry most of the year, and with the state of the planks fibreglass made sense to me.

I do love the look of the glass before epoxy, like a wedding dress.

The finished glass and epoxy before sanding.

The only possible color to paint a cedar strip runabout is red. Anything else is heresy. So I painted it red.

A very red bottom

Varnishing the inside was far more work than I had imagined, especially when bending over the boat too long left me unable to stand up straight again. But with some help we put on four coats of varnish before losing patience, and it looks quite shiny. The older wood really stands out with a much darker color adding a lot of character.

Four coats of varnish shining in the sun

Currently I’m rebuilding the seats and deck supports. My dad milled up cedar for the deck, but it’ll be the last thing to be installed after the electrical is done. Where possible I’m reusing the existing pieces, drilling out rusted holes and then filling them with epoxy, or filling larger cracks with epoxy. The seat bottoms and backs were made of cedar and both front and middle seats were too far gone, but all the supports are of white oak and I have resurrected almost all of them.

Next comes the motor question. I sold the original motor for parts since it needed far more work than it was worth. Showing the power of Kijiji, the buyer had the same motor sitting in his garage needing parts. I had the impression the boat was overpowered and just recently found out that the 33hp motor had been bored out and was likely closer to a 40 or 45hp motor. I am leaning towards a newer 20hp motor so no one will be tempted to try water skiing or pulling a tube.

It’s good to remember where the project started, so I’ll end with a picture of what it looked like years before I picked it up. If you look closely you can see the twist in the boat, that’s not an artifact of the camera. When it dried out, either the boat wasn’t supported properly or it just twisted as it dried out. In the process of replacing the ribs I managed to get most of the twist out.

The Richardson about ten years ago looking a bit tired.

Posted in General.

Tagged with , .

By mike – July 12, 2020

Slow progress

After an extended break due to other priorities, I’m back at slowly installing more ribs. IMG_20141230_170542On the right is the back section of the boat with seven new ribs in a row. It’s exciting to see a small section that’s completely held together with new ribs. We broke three ribs trying to get the last one in (on the left). Unsure if it was just three bad pieces or because that’s where the curve is the biggest. For now I’ve put that one on hold and working on the rest.

IMG_20141129_153536In other news, my driveway is no longer a big sinkhole so now I can roll the boat out onto the driveway and do the sanding out there, which is a huge improvement over sanding inside, with the dust cloud that inevitably arises.

Posted in General.

By mike – January 1, 2015

Rib Fest!

With my Dad’s help, and that of his well equipped shop, we manufactured almost 60 ribs for the boat, close to 500′ worth of ribs! I first ran the 1″ planks through the bandsaw and ended up with (mostly) straight strips of 1″ x 1/2″ by 8′ long ribs. One pass on each side through the planer took out any humps, and then twice through the router table with a one quarter round over bit. It took a bit of finagling to get the exact dimensions, as the ribs need to fit through the holes in the keel that’s already installed. I had a leftover piece of keel to use as a test piece, basically if the ribs slide through and are mostly round, they’re good. Business end of the steamer

To steam the ribs, the original steam generation a-la-camp-stove wasn’t going to hack it, so I ended up buying a wall paper steamer Steam generator and ribsfrom Home Depot. It claims 75 minutes of steam, and has long hose. I concocted up a series of adapters (think me standing in the plumbing aisle at Home Depot putting random pieces together) to bring the 4″ PVC pipe down to 1″ diameter, and then used some water proof duct tape to hold the hose inside the PVC pipe. It worked marvellously, and the condensation managed to find a way to drip through the tape. After wrapping the whole thing in old towels, the temperature at the far end was around 96C, good enough for what I need. On the left is the steamer putting out a cloud of steam (hard to see in the picture) and on the right is the generator plus a pile of ribs. Ignore the mess, my boat shop doubles as storage for roller blades and newspaper delivery bags.

Finally came the nerve wracking part, trying to bend a rib into the boat. After steaming it for 30 minutes (which is probably overkill), Jodie and I bent a rib into the boat. Minus the bleeding fingers it worked perfectly, although I did manage to forget that steamed wood expands, and getting it through the keel was pretty tight. I’ll have to sand the rest a bit more aggressively. Later that day my brothers and I figured out a decent system for getting the nails clinched in, which is good because we estimate there to be about 3,000 nails. And here’s a picture of the first rib neatly bent into place. Only fifty more to go!IMG_20140503_115113

Posted in General.

By mike – May 5, 2014

Stem and Transom

Finally some new wood went into this boat. I took a day off last week and installed the new stem and keel. I don’t have a good picture but it certainly feels like we’re making some progress. The keel and stem are nailed from the bottom, and every hole has to be drilled for the copper screws as they are too soft to break into the oak.

IMG_20140328_174502I removed the transom, as the boards nailed to it had been nailed a few too many times and rather than trying to fix the cheese grater appearance I just cut the boat 2″ shorter. This picture shows the missing transom, along with the freshly installed keel. The bottom of the boat is hogged quite badly, the supports are in an effort to keep the keel straight, and to prevent the boat from opening up too much.

The transom is old, the top is curved a bit, and there are some big checks in it, but it’s still structurally sound. After some thought and sanding I decided to reinstall it rather than build a new one. I’d much rather keep what I can, else I may as well be building a new boat.

IMG_20140405_152345

And this afternoon the transom was reinstalled again. Because the boat is a wee bit shorter, the gunwales sit a bit higher than the transom instead of flush, but that will be sorted out later. All that’s left on the transom is to cut the boards flush and clean up the sealant.

I sold the old trailer, which wasn’t worth the effort (to me) to rebuild. The father of the purchaser owned a few cedar strip boats many years ago and had a few seats left in his garage. He was so interested in this project that he brought them over in the off chance that I could use them, or at least reuse the wood for some of the interior. I love it, I’d much rather use wood that came from a cousin boat than new wood from the lumber shop.

My brother has the motor apart in his garage, it’s a 1965 33HP Evinrude. Surprisingly all the parts we needed are still available, although some have manufacturing dates from the 80’s. They’ve all arrived so the rebuild can begin.

Next up is removing the ribs and replacing them, half of them at a time. Removing them means splitting them with a chisel, and then prying out the nails. The copper nails were clinch nailed, meaning they were nailed from the outside all the way through the rib and bent to make a “J” shape, and they need to be cut to get them out. This is a very large task, but I have some willing and not-so-willing volunteers to help out.

Posted in General.

By mike – April 5, 2014

First bend

First bend was successful, here’s a gratuitous picture of the wood on the form. It bent quite easily after fifteeen minutes of steaming, 1/4″ thick. You can never own enough clamps!

Laminate bend

Posted in General.

By mike – March 7, 2014

Bending a new stem

The past few weeks have not been great in terms of boat restoration. Every project has its highs and lows, and last weekend saw the project in a state of despair. Lighting a match under the boat seemed like a viable option.

Keel in the jigBefore I get into details, I purchased some quarter-sawn white oak for the keel, stem and ribs. With my Dad’s help we cut the 11′ section for the keel, and made up two blanks for the stem. The keel worked out well, and after drilling out the holes for the ribs, it is ready for installation. Here’s a picture of the keel with my drilling jig.

The despair part of the project came when attempting to bend the stem. The stem is 5′ long and  1.75″ x 1.75″, that’s a lot of wood to bend. I put together a steam box of PVC pipe, wrapped it in blankets to keep it warm, and with this ugly rig managed to keep the temperature at around 95C at the end. The stem was cooked for two hours, and we then attempted to bend it over the form. Unfortunately both blanks broke during the attempt.

Steam box

In hindsight, the mistake I made was to attempt to bend a thick piece of kiln-dried wood. The kiln-drying process damages the cell structure of the wood, which is fine for almost all applications except for steam bending. It’s an expensive lesson to learn, but learn it I have.

IMG_20140217_115813What now? Finding air dried or green quarter sawn white oak is very difficult. Instead with the advice of my local wood supplier I’ve decided to laminate the stem instead, with 1/4″ pieces. They are cut, just need a good sanding and epoxying. I may steam the pieces just to make them a bit more flexible, but I can almost bend them over the form without steaming, so hopefully this will work better. To the left is a picture of the form with the old rotten stem over it. The form has large holes drilled into it for clamping purposes, which were drilled after the picture was taken.

Once the stem is laminated it needs a lot of shaping (my favourite activity, turning wood into shaving) and the installation of both stem and keel. And the move onto the ribs. I’ve decided to replace all the ribs, as there are only ten or so of the fifty that are not in rough shape, so while I’m at it I may as well replace them all.

 

Posted in General.

By mike – March 2, 2014

Rotten stem and keel

stripped hullI finished stripping the bottom a few weeks ago, which uncovered some “interesting” prior repairs, some of which need to be redone. The hole is quite obvious, and now that the outer keel and outer stem are gone, the rot to stem and inner keel are really obvious.

I managed to carefully pry out the stem, first by taking out any nails that were slight raised, and then by very carefully prying the boards away from the stem and cutting any nails I could reach. Finally with only the bottom boards holding the stem I gently pulled it out. It split in half, the rot was far worse than I imagined.

IMG_20130921

Here’s a closeup of where the stem simply gave up and broke, that clearly shows the rot. I guess that’s not bad for a 50 year old chunk of wood that’s taken a beating for so long.

The front part of the keel is also in rough shape, so I need to pull that out without further damaging the boat. Because the stem overlaps the keel, I’m going to replace the keel first and then put in a new stem.

Next up is to purchase some quarter sawn white oak and setup a jig to bend it. I have most of what I need to steam bend a new stem, but a snow storm put a stop to visiting the lumber store. Bending wood is a very interesting and somewhat stressful process, but one I’ve done before and am quite excited to do again.

I counted rotten ribs, and from what I can tell 36 of them, about half, need replacing. All of them need to be steam bent as well.

Posted in General.

By mike – January 26, 2014

Stripping

image

Here’s the stripper hard at work. I really dislike using chemicals but this stuff is amazing. Put it in, wait 15 minutes, and 49 years if paint scrapes neatly off (with a bit if muscle). Scraping with a sprained wrist really sucks though.

Posted in General.

By mike – January 3, 2014

The mess that is the bow

image

After taking the remaining bits of metal off that someone had used to repair the bow, I found a mess. And a hole. Both were expected. The inner and outer stem, as well as the keel are all rotten and need to be replaced. This is turning into a rather large project.

Posted in General.

By mike – January 3, 2014

My new (to me) boat

I recently came into ownership of a 1965 Richardson Avalon 14′ cedarstrip boat. The history of the manufacturer is somewhat cloudy, but it appears in 1962 the former General Manager of Peterborough Canoe Company, the company that built most of the cedar strip runabouts, bought Lakefield Boat Co and renamed it to Richardson Aquacraft, also Rilco Industries. [source] The boat has the Richardson name on the side, but the Rilco name on the manufacturer’s tag. It was purchased either new or very new by a friend’s grandfather and was used at the family cottage until the late 1990’s, after which it was stored in a garage.

Richardson Avalon

The boat is more or less in reasonable shape for its age. It comes with the original 1965 Evinrude 33HP motor, which looks to be in excellent shape. My brother has graciously volunteered to have a go at bringing it back to life. After pulling out the seats and hardware, the real work starts to become apparent. While most of the ribs are intact, there is a lot of rot in the inner stem and the ribs in the front. Most of the ribs at the front will have to be replaced, as will the solid oak inner stem, and possibly the keel as well. There is a bit of hogging in the keel, which can be sorted out, and there is a slight twist in the boat which can also be straightened.

Here’s a picture of the boat sitting on its new cradle.

Boat on its cradle

And another random one showing what the inside looks like. The planks and seats and deck are all cedar, and anything structural looks to be either oak or ash. The screw driver is laying on a seat support, and the ugly block is where the throttle assembly hung off of. Pretty much every screw and bolt is original and completely rusted, and each one takes a lot of work to get it out.

inside

 

I ordered a DVD from Donald Husack on how to restore a boat like this, and it was well worth the money even after watching only the first thirty minutes.

I’ll try keep this page updated as I go along.

Posted in General.

By mike – January 2, 2014

Popup Trailer Roof Repair Part 2

Fixing a rotten roof is an incredible amount of work. After taking the rotten parts off, the horizontal part of the roof is solid, as is the vertical side on the driver-side. The passenger side is entirely rotted out, see this picture for how bad it is.

IMG_20130384This is a picture looking at the top of the side, it’s hard to see but the inside is full of mold. Remember that’s supposed to be a solid board. I’m going to replace the side entirely. A friend scrounged up some aluminum for me, and some hardwood plywood will be the new core.

The next step was to scrape off the remaining plywood from the vinyl (is it vinyl?) roof material without puncturing it. Unfortunately I did go through it a few times, I’m not entirely sure how to fix that just yet.

IMG_20130382The pic on the right is from the inside of the trailer looking back, you can see some of the roof material and some plywood. I managed to scrape off most of the remaining plywood and glue today using a combination of a sharpened paint scraper, and a really long chisel-type tool. I’m not sure what is, but it works really well once it’s sharp.

I still need to clean off the old caulking in various places, then time to start rebuilding. August 1st is looming quickly.

 

Posted in General.

By mike – June 28, 2013

Popup Trailer Roof Repair

After seeing some odd spots on the roof of my tent trailer, some exploratory surgery showed that the back, front and side of the roof are completely rotten. A new roof will take more than eight weeks to order, and cost aside that puts a damper on the planned East Coast trip on August. So the only option short of buying a tent is to repair it. Thanks to the wonderful people at www.popupexplorer.com I feel reasonably confident that this can be done. Further demolition showed the rot to be so bad that I’m not entirely sure how the lift mechanism was still lifting the roof, as the brackets were in completely rotten wood. Two pictures that show the extent of the damage.

This is from the inside looking at the back panel. The white is mould. This panel is so bad that I can just grab the wood and gently nudge it apart.

IMG_20130356

And this picture is of the side, looking forward. That piece that’s got the nice curve to it is supposed to be a solid piece of side that holds the roof together.

IMG_20130376

Should be fun.

 

Posted in General.

By mike – June 20, 2013

BB10 – Google Accounts

Setting up Google Mail works well, but the calendar doesn’t always sync. This is an intermittent issue reported on this thread, and reported in KB33458.

About the fifth entry down in the thread is a workaround that worked very well for me, steps repeated here for convenience:

  1. Remove the previous gmail accounts.
  2. Click on the Add Account button.
  3. Click on the Advanced button.
  4. Select “Microsoft Exchange ActiveSync”.
  5. Enter a description.
  6. Enter “m.google.com” in the Domain field.
  7. Entered my full email address in the “Username” field.
  8. Entered by full email address again in the “Email Address” field.
  9. Entered my password in the “Password” field.
  10. Entered “m.google.com” in the “Server Address” field.
  11. Left the port set to the default (443).
  12. Left “Use SSL” set to “On”.
  13. Left “User VPN” set to “Off”.
  14. Left “Push” set to “On”.
  15. Click on the “Next” button at the top.
  16. Make sure email, calendar and contacts are set to “On”.

I see some wonderful irony in using a Microsoft protocol to connect a BlackBerry with a Google service.

Posted in General.

By mike – February 6, 2013

BlackBerry 10 Tips – Sorting Facebook Feeds

I picked up a new BlackBerry Z10 yesterday. The device is very nice, but a completely different beast than the old BlackBerry. Because of that, there are a lot of questions on how to tweak various settings, and not many answers.

Question: The Facebook apps appears to be sorting by “Top Stories” instead of “Most Recent”. How do I change that?
Answer: In the device browser, go to “touch.facebook.com”, and click on the “Sort” button, select the sort type you want. Open the Facebook app and presto, it sorts as it should.

Source

Posted in General.

Tagged with .

By mike – February 6, 2013

Canoe update – June

A few more pictures available. I’ve got the fibreglass and epoxy on the inside, it only needs a final sanding and then installation of gunwales, seats and decks.

I did have some help sanding it…

From Canoe – Stage 2
From Canoe – Stage 2

Here it is. I love the dark accents, they came out very nicely.

From Canoe – Stage 2

The deadline to get this done is August 2, 2010, so the push is on to get it done.

Posted in General.

By mike – May 29, 2010

Art website

My favourite artist now has a website! Have a look at www.jodiehart.ca.

Posted in General.

By mike – May 24, 2010

WordPress on Mac

I’ve been banging my head against the wall for the past day, trying to figure out why my wordpress install on my Mac doesn’t talk to the database. Setup is simple, enable the built-in apache web server, install the Mac Mysql server and setup as per the wordpress instructions.
. But

When you’re on Linux or any other platform I’ve worked on, one specifies the hostname in wp-config.php as ‘localhost’, or whatever hostname the DB is running on.

Seems that when you’re on Mac, you specify the hostname as ‘localhost:/tmp/mysql.sock’. This is not documented anywhere, but works like magic. Hope this saves someone else a headache in the future.

Posted in General.

By mike – February 15, 2010

Canoe update

The canoe has come a long way since the last update. Over the past few weeks we’ve laid on the fiberglass and four coats of epoxy, and taken it off the molds. Normally you lay on three coats of epoxy, but the third coat was a bit messy so I sanded it off and added a fourth coat.

Here’s my favourite picture, showing the accent strips really well.

From Canoe – Stage 2

And here’s one of the canoe turned over. It’s starting to look like a boat now!

From Canoe – Stage 2

The full album is available here. Unfortunately the canoe is going to sit for a while, as I’m getting sidetracked on a harp.

Posted in General.

By mike – January 3, 2010