Building September - A Communication Assistant

September is a communication assistant for people living with neurodegenerative conditions like ALS (Amyotrophic Lateral Sclerosis, also known as Lou Gehrig's disease), MND (Motor Neuron Disease), or other speech & motor difficulties.

Background

I was diagnosed with ALS in 2019, and have gradually lost the ability to speak and type. I relied on a combination of dictation, one-handed typing, and a mouse. However, as my speech became more slurred and my arms weakened, I continued working, coding, and using my computer. I adapted to various combinations of one-handed typing, dictation, and on-screen keyboard. I currently use an on-screen keyboard and a head-mounted mouse (a device that tracks head movement to control a computer cursor).

Tools I've Used

GitHub Copilot: With the launch of GitHub Copilot in 2021 (an AI assistant that helps write code and text), I was able to continue coding and writing everything in VSCode (a popular text editor used by programmers). The smart text suggestions that learn from your writing were a game-changer. I was able to write e-mails, blog posts, long messages, work documents, and all within VSCode. I would copy-paste text to-and-from different platforms like Slack, Google Docs, GMail, etc. While cumbersome, it was an effective way to type faster. I hoped that the Apple Mac's on-screen keyboard would somehow provide a similar experience.

Voice Banking Challenges: As my speech slurred, I tried several voice banking apps. All of them required long recordings in quiet environments. It was hard to get a good recording with a 3-year-old in the house during lockdown.

ElevenLabs Voice Cloning: In 2023, ElevenLabs launched their voice cloning technology. This was the first easy way to clone my voice without jumping through hoops. But by then, my speech was already slurred. So I dug through my old hard drives and videos to find old recordings of me speaking. I used these recordings to create a long enough recording to clone my voice. It wasn't perfect, but it was better than everything else.

With the rapid evolution of AI technologies in coding, writing, and speech, I set out to build an application that combines all these different advancements to make communication easier.

Daily Communication Needs

All these smart AI tools changed the way I wrote code and worked remotely, but daily communication remained challenging.

The most common thing I do is talking to people around me. It could be for simple things like asking for water, or telling what I'd like to eat for lunch. Sometimes, it could be video calls with my family and friends. Or just telling my wife about how my day went.

Most often I would compose messages in notes or WhatsApp and have people read them while talking to me. Even with voice cloning services, I still need to type out full sentences every time. These tools don't offer auto-complete or suggestions based on previous conversations. Nor could I use my cloned voice with FaceTime, Zoom, or other video calling apps.

While using head-mouse or eye-tracking devices, clicks are precious. Especially when the vocabulary and sentences used every day are the same, I should not have to type out full phrases and sentences every time.

Beyond daily conversations, I also needed to share longer thoughts and stories. Whether explaining technical concepts to my team, recounting childhood memories to my son, or documenting ideas that might help others facing similar challenges, I found myself writing longer pieces of text. These weren't just messages—they were narratives that I wanted to share with my voice, not just text on a screen.

The Product: September

These experiences inspired me to build September—a communication tool that combines the AI technologies I've been using to make daily communication easier and more natural.

Smart Text Editor

I wanted September to learn from how I actually talk, not just provide generic suggestions. The smart text editor uses my message history to provide instant auto-complete suggestions, just like GitHub Copilot did for my coding. It also uses AI (Gemini, Claude, etc.) to provide contextually relevant sentences, phrases, and words. The more I use it, the better it gets at understanding my writing style and providing more accurate suggestions.

I also wanted to be able to share my thoughts and stories with proper context. In every conversation, I can provide additional context in the form of notes, documents, images, videos, or links. September indexes all this information to help me "speak my mind" in conversations, whether I'm explaining technical concepts to my team or sharing childhood memories with my son.

Text-to-Speech

September integrates with multiple text-to-speech providers, giving me a choice of voices that suit my language, style, and personality.

Voice Cloning

The ElevenLabs voice cloning technology was a breakthrough for me. September makes it easy to record my own voice or use samples from existing audio or video files. I can also search and choose from a wide range of community and professional voices. This gives me the flexibility to use my own voice when I have good recordings, or find alternatives that work well for me.

Transcription

When others are talking to me, September needs to understand what they're saying quickly. September uses AI to transcribe audio in real-time and provides contextually relevant suggestions based on the conversation. This helps me respond more naturally and keep up with the flow of conversation.

Multiple Keyboard Layouts

Since I use a head-mounted mouse and need to minimize clicks, September provides multiple on-screen keyboard layouts—QWERTY, Circular, Emojis, and more. I can choose the layout that works best for my current input method, whether that's mouse, or eye-tracking devices.

Accessible Design

September is designed to work for people like me who have speech and motor difficulties. It works on any modern web browser across desktop and mobile devices, and it's compatible with various input methods. No complex setup or downloads are required—I can just open it in my browser and start communicating.