[Howto] Using Mycroft - the Open Source AI

Moderators: meveric, mdrjr, odroid

[Howto] Using Mycroft - the Open Source AI

Unread postby mad_ady » Fri Jul 14, 2017 10:34 pm

Ever since KITT made its appearance as a talking car back in the '80s, everybody dreamed about the day when they would have their own personal talking assistant. The good news is - we're almost there. The bad news is - there is a high cost associated with the lack of privacy and data mining done by big companies. Wouldn't it be nice to have a voice controlled assistant that doesn't share its data with the cloud?
Unfortunately, we're probably still 10 years away from that point, but we're making progress in the right direction. Today we are going to install an open-source artificial intelligence platform that runs on your Odroid, called Mycroft (https://mycroft.ai/).

Mycroft consists of several interconnected components that handle speech acquisition, keyword recognition (done locally), speech to text translation (currently done by Google), skills engine and text to speech (done locally by mimic). Mycroft communicates with several entities on the internet - with it's own cloud to store and retrieve settings and API keys for various services and with Google speech to text services. In addition to this, the skills may communicate with online services to retrieve data for the end user.

Image
Figure 1 - Mycroft architecture

From the start, people will compare Mycroft with other personal assistants - such as Amazon Echo, Alexa, Google Assistant. Well, they are quite different. Mycroft is built to be open and non-intrusive, while all the other assistants are in constant contact with your profile data and can respond to requests involving your personal data, such as your contact list or personal calendar. By default Mycroft does not have access to that information and is mostly used to query public information sources, such as the weather, Wikipedia and WolframAlpha. Some may think this is limiting, but if you value your privacy, then you know it's not. A lot of advanced functionality can be enabled by the end user by activating certain skills (such as Google account integration). In terms of AI complexity, Mycroft is simpler than the other AIs, which run completely in the cloud and benefit from complex recurrent neural networks.

Image
Figure 2 - SMBC comics - Listening (https://www.smbc-comics.com/comic/listening)

One major concern is having a device (that you know of) constantly listening around. Analysing what data is leaked about you by your personal assistant should be high on everyone's priority list (https://www.wired.com/2016/12/alexa-and ... our-voice/). The open-source advantage of Mycroft is that network queries are logged and easily consulted by the average user, so you can actually see what the device is doing behind your back.
Mycroft could track what you speak in their cloud, but network logs show that they don't save a transcript of what you speak. The speech data is sent to Google servers to be converted to text, because speech to text is still a difficult problem which generally requires the use of neural networks (https://www.youtube.com/watch?v=g-sndkf7mCs). So, Google can still track what you are saying, and can tie it to your IP address, which compromises privacy. The Mycroft team has started work on an open speech-to-text engine to solve this issue, but it will realistically take a long time before this can be run locally (https://openstt.org/).

So, if you've wanted a voice activated personal assistant, and are ok with the security implications of having one, let's see how we can get Mycroft running on the Odroid.

Installation
If you have an Odroid U3 you can skip everything and install a Mycroft-enabled Debian image created by forum user @nold: viewtopic.php?t=27172. But, I don't like to waste an Odroid for a single task, so we'll install Mycroft on a general-purpose Odroid C2 which runs Hardkernel Ubuntu 16.04. You can use the same steps to install on a different Odroid model, such as C1 or XU4.

The first thing you're going to need, independent of your Odroid model is swap. The installation process needs to compile the text-to-speech component (mimic), and this takes up a lot of memory. You should also do a clean reboot to free up any used memory in case your Odroid has been up for a long time.

To create a temporary 4GB swap file on your SD/EMMC run the following commands:

Code: Select all
$ sudo dd if=/dev/zero of=/swapfile bs=1M count=4096
$ sudo mkswap /swapfile
$ sudo swapon /swapfile


You can test that swap has been activated with
Code: Select all
$ free -m | grep Swap


To do the actual installation, you may want to read the documentation located at https://docs.mycroft.ai/. We will be going with the git clone install method (https://docs.mycroft.ai/installing.and. ... ne.install). Mycroft will be installed inside a python virtual environment inside your user home directory (e.g. /home/odroid). It will also run as a regular user. For the installation and compilation steps you will need about 2.5 GB of free space.

Code: Select all
$ cd ~
$ sudo apt-get install git
$ sudo git clone https://github.com/MycroftAI/mycroft-core.git
$ cd mycroft-core/
$ sudo ./build_host_setup_debian.sh
$ sudo chown -R `whoami`:`whoami` .
$ ./dev_setup.sh


The ./dev_setup.sh step will create the virtual environment, set up the python modules needed and then will compile mimic. Three and a half hours later (on a C2 @1.75GHz), after mimic has been compiled, you may be prompted with this error when compiling pygtk:

Code: Select all
configure: error: cannot guess build type; you must specify one
make: *** No targets specified and no makefile found.  Stop.
make: *** No rule to make target 'install'.  Stop.



Well, this is an error we can live with. There's a desktop integration component that fails to build on the C2, but we're not going to use it anyway, so you're done!

In the future, if you want to update mycroft, you can redo the installation steps (especially the git clone step), but run ./dev_setup.sh -sm to skip mimic build.

If the build went ok, we can do some cleanup and free up some disk space. We can turn off swap and reclaim 4GB of disk space:
Code: Select all
$ sudo swapoff /swapfile
$ sudo rm -f /swapfile


We can also clean-up about 500MB worth of mimic temporary files by running:
Code: Select all
$ cd mimic
$ make clean


Unfortunately mimic is installed among its sources, so it's difficult to separate it and clean-up further.

Startup

Unfortunately, Mycroft does not come with a systemd startup unit, but we can create one around its startup script. Mycroft's startup script is located in ~/mycroft-core/mycroft.sh. It accepts parameters such as start and stop and manages 4 screen instances which in turn run a key Mycroft component each. To create a systemd startup unit, run the following commands (adjust the username and paths if you're not running as odroid):

Code: Select all
$ sudo vi /etc/systemd/system/mycroft.service
[Unit]
Description=Mycroft personal AI
After=pulseaudio.service

[Service]
User=odroid
WorkingDirectory=/home/odroid/mycroft-core
ExecStart=/home/odroid/mycroft-core/mycroft.sh start
ExecStop=/home/odroid/mycroft-core/mycroft.sh stop
Type=forking
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target


To start up Mycroft on every boot, and to manually start it up now, run:
Code: Select all
$ sudo systemctl enable mycroft
$ sudo systemctl start mycroft


If the default sound settings are ok, you should hear Mycroft complaining that it isn't paired with its cloud and it should tell you a pairing code. But let's assume that you get no sound yet, or maybe you didn't understand what Mycroft said. We'll need to get accustomed with the backend processes.

Backend and debugging
When Mycroft starts up, it creates 4 screen instances where it runs processes for voice, skills, command-line interface and services. You can list and connect to these screen instances with:

Code: Select all
$ screen -list
There are screens on:
   5705.mycroft-cli   (07/13/17 13:50:28)   (Detached)
   5690.mycroft-voice   (07/13/17 13:50:28)   (Detached)
   5675.mycroft-skills   (07/13/17 13:50:28)   (Detached)
   5640.mycroft-service   (07/13/17 13:50:28)   (Detached)



If you connect to the mycroft-voice screen you will be able to see if there are any errors with audio input or output (such as - there is no default audio output device). To connect and interact with a screen session you can run:

Code: Select all
$ screen -r 5690.mycroft-voice


Note that the number is the process PID and will change on Mycroft restart. You should identify it with screen -list, as shown above. To detach from a screen session without terminating the screen process, use the key combination CTRL+a d. See the screen manual for more key bindings: https://www.gnu.org/software/screen/manual/screen.html

Regular text logs from these services are also saved in ~/mycroft-core/scripts/logs/ and can be consulted with regular tools such as grep or tail -f.

Mycroft also comes with a debugging mode where it starts only the engine, skills and CLI components, so you can debug various problems. To start Mycroft in debug mode type:
Code: Select all
$ ~/mycroft-core/mycroft.sh start -d


Image
Figure 3 - Debug view

Configuration
Mycroft's primary configuration file is located at ~/mycroft-core/mycroft/configuration/mycroft.conf. You should not edit this, but instead make a copy at ~/.mycroft/mycroft.conf, to avoid the settings being overridden on Mycroft upgrade.

Code: Select all
$ cp ~/mycroft-core/mycroft/configuration/mycroft.conf ~/.mycroft/


The configuration file has general settings, such as the language (I only tested English), units, location, as well as specific skill parameters for default skills. These will likely move to different configuration files in the future. At this point you can edit the file (~/.mycroft/mycroft.conf) and set your preferred settings, including location, so that the answers are relevant to you. For example, if you leave the default location of Lawrence, Kansas, when you ask for the time or weather, you'll get the time and weather from Lawrence, Kansas. Some of these settings can also be set from the Mycroft cloud, after pairing the device.

Audio

If you're running a desktop image with sound output via HDMI and a microphone, chances are that sound will run fine without manual intervention. But, if you're running a server image without a desktop environment, or if you have multiple microphones/sound outputs you may want to manually configure Mycroft to use a specific sound device.

Although Mycroft supports both ALSA and PulseAudio, we're going to use PulseAudio as a sound backend, because it's more flexible - for instance, it's easier for multiple processes to use the microphone at the same time.

Usually, PulseAudio runs as a user process and starts when you log in your desktop environment. This is not compatible with having Mycroft start as a service at startup, since there would be no PulseAudio to connect to. We need to run PulseAudio as a system service instead (more details here: https://possiblelossofprecision.net/?p=1956).

Code: Select all
$ sudo apt-get install pulseaudio
$ sudo vi /etc/systemd/system/pulseaudio.service
[Unit]
Description=PulseAudio Daemon
 
[Install]
WantedBy=multi-user.target
 
[Service]
Type=simple
PrivateTmp=true
ExecStart=/usr/bin/pulseaudio --system --realtime --disallow-exit --no-cpu-limit


In order to allow user processes to communicate with a root PulseAudio, we need to edit the configuration located at /etc/pulse/system.pa. You will need to add the following lines to the configuration:

Code: Select all
#allow localhost connections
load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1


Before we run PulseAudio as a service, we need to identify the microphone (input) and the speakers (sink) that we want to use. For this, we'll need to run PulseAudio as a user:

Code: Select all
$ pulseaudio -D


Next, we can get a list of output devices (sinks) that Mycroft can play to. The idea is - if your Odroid has both HDMI connected and a USB sound card, you can have Mycroft use the sound card for output, so that you can hear it even when the TV is off.

Code: Select all
$ pacmd list-sinks | egrep "index|name:"
   index: 0
        name: <alsa_output.usb-0d8c_C-Media_USB_Headphone_Set-00.analog-stereo>
  * index: 1
        name: <alsa_output.platform-odroid_hdmi.analog-stereo>


In the output above you can see that the HDMI output is the default sound output. We will want to tell Mycroft to use card 0 (or better, we can index it by name) for sound. To do this, edit ~/.mycroft/mycroft.conf and change "play_wav_cmdline": "aplay %1", to "play_wav_cmdline": "paplay -d alsa_output.usb-0d8c_C-Media_USB_Headphone_Set-00.analog-stereo %1",

This will cause all voice messages generated by Mycroft to be played to the USB soundcard instead.

Some skills will play podcasts, or mp3 content like the news, so if you want to hear that over a different sound output, you need to change the following setting as well. Change "play_mp3_cmdline": "mpg123 %1", to "play_mp3_cmdline": "mplayer -ao pulse::alsa_output.usb-0d8c_C-Media_USB_Headphone_Set-00.analog-stereo %1",

The thing is mpg123 doesn't have a switch for selecting a PulseAudio sink, so we changed it with mplayer (which needs to be installed in your system):
Code: Select all
$ sudo apt-get install mplayer


One last thing to do is to set a default microphone. In my case I have three microphone inputs - one from the sound card (not connected), one from my webcam and one from HDMI.

Code: Select all
$ pacmd list-sources | egrep "index|name:"
    index: 1
        name: <alsa_input.usb-Sonix_Technology_Co.__Ltd._USB_2.0_Camera-02.analog-mono>
    index: 2
        name: <alsa_output.usb-0d8c_C-Media_USB_Headphone_Set-00.analog-stereo.monitor>
  * index: 3
        name: <alsa_input.usb-0d8c_C-Media_USB_Headphone_Set-00.analog-mono>
    index: 4
        name: <alsa_output.platform-odroid_hdmi.analog-stereo.monitor>
    index: 5
        name: <alsa_input.platform-odroid_hdmi.analog-stereo>

I want to use my webcam's microphone for Mycroft input, so I'll set it as default in /etc/pulse/system.pa:

Code: Select all
#set default microphone
set-default-source alsa_input.usb-Sonix_Technology_Co.__Ltd._USB_2.0_Camera-02.analog-mono

At this point save and you are ready to start the service:
Code: Select all
$ sudo systemctl enable pulseaudio
$ killall pulseaudio
$ sudo systemctl start pulseaudio


Pairing and interacting with Mycroft

Right now we can restart the whole system and Mycroft should start after PulseAudio has started. The microphone should work as expected and so should audio output. You are now ready to register Mycroft. To have Mycroft process your speech, you should start your questions with Hey Mycroft!. To get a pairing code, ask Mycroft, register my device. Mycroft should then speak its registration code which usually consists of 6 characters. If you can't hear the code or can't understand it, you can find it in the mycroft-skills.log located at ~/mycroft-core/scripts/logs. You will need to register at http://home.mycroft.ai and navigate to Devices -> Add device.

Image
Figure 4: Registering online

You should now be good to go. You can go ahead and ask Mycroft for various information, like:
  • Hey Mycroft, what's the time?
  • Hey Mycroft, what day is today?
  • Hey Mycroft, wiki European Union
  • Hey Mycroft, tell me about Abraham Lincoln
  • Hey Mycroft, tell me a joke
  • Hey Mycroft, why is 6 afraid of 7?
  • Hey Mycroft, sing me a song!
  • Hey Mycroft, what is 2+5?
  • Hey Mycroft, tell me the news
  • Hey Mycroft, set a reminder for 5 minutes
  • Hey Mycroft, what's the weather? Hey Mycroft, what's the forecast for tomorrow?
To know what you can ask for, consult the basic skills documentation page here: https://docs.mycroft.ai/skills.and.features. You can also review the mycroft-skills.log file and you can see what keywords are registered when a skill is loaded. For example, the joke skill registers the following key phrases: joke, make me laugh, brighten my day, tell me joke.

It's all about the skillz
The basic skills are quite useful, but you can do so much more with Mycroft by using third party skills. There's an official repository here: https://github.com/MycroftAI/mycroft-skills
The skill's quality may be good or bad, so you're on your own here.

We'll be adding support for diagnostics (have Mycroft tell you CPU usage, free space) and Home Assistant integration (have Mycroft read values from Home Assistant, or control switches with your voice).

Third party skills can be installed in /opt/mycroft/skills and will be automatically loaded on Mycroft restart. To add the diagnostics skill, simply clone it from its github link:

Code: Select all
$ cd /opt/mycroft/skills
$ git clone https://github.com/the7erm/mycroft-skill-diagnostics.git


The skill can be configured to run a user-defined script and read its output. The path to the user-defined script needs to be configured inside ~/.mycroft/mycroft.conf (note that the location of third-party configuration is bound to change in new Mycroft releases, so consult the official documentation for the correct location and syntax).

Code: Select all
 "DiagnosticSkill": {
    "script": "/home/odroid/bin/usbreset.sh"
  }



After you restart Mycroft (sudo service mycroft restart) you will be able to use queries such as Hey Mycroft, what is the current CPU usage percent?, Hey Mycroft run diagnostics (to run your custom script), Hey Mycroft, what's your uptime?, etc.

Similarly, we can install the Home Assistant skill and integrate Mycroft with a running Home Assistant instance. This skill has some external dependencies which need to be installed as well:

Code: Select all
$ cd /opt/mycroft/skills
$ git clone https://github.com/btotharye/mycroft-homeassistant HomeAssistantSkill
$ workon mycroft
$ cd HomeAssistantSkill
$ pip install -r requirements.txt



Also, you will need to edit ~/.mycroft/mycroft.conf and add:

Code: Select all
 "HomeAssistantSkill": {
    "host": "hass.mylan.net",
    "password": "mysupersecrethasspass",
    "ssl": true|false
  }

After restarting Mycroft, you can now query the state of your Home Assistant entities by asking things like Hey Mycroft, turn on air conditioning power to activate the "Air Conditioning Power" switch, or Hey Mycroft, ask home assistant what is kids room temperature? to query a sensor called "Kids room temperature". The skill will try to match what you say against the names of Home Assistant's entities so that you can control Home Assistant with your voice! I personally think this is the coolest skill :)

You can view some of the things Mycroft can do in this video: https://www.youtube.com/watch?v=mIM81al60TI

Mycroft's future

Mycroft started as a simple AI, but with the support of the community, it is evolving into something bigger. For now it works in a question - answer pattern, without keeping state of a conversation, but work is being done to change that in the future, so you will eventually be able to have a conversation with Mycroft (https://www.youtube.com/watch?v=pyc3wWYoI8E). In terms of performance and used resources, since it was designed to run on a RPI, it has no problem running on Odroids (even on C1). On my C2, with other background processes running, Mycroft uses about 20% CPU on 2 cores with the governor keeping the CPU frequency at 500MHz when idle.

Today Mycroft might be mildly useful and not really a personal assistant, but it has potential. You could compare it to MS-DOS back in the 80's - clunky, but gets the job done. The thing is - all personal assistants make mistakes - sometimes with funny consequences: https://www.instagram.com/p/BOznj6OAwUG/. I think that in 10 - 20 years, the personal assistant will evolve to something that is hard to predict (hopefully not Skynet), but I hope the personal assistant of the future will be community-based and open-sourced.
User avatar
mad_ady
 
Posts: 2035
Joined: Wed Jul 15, 2015 5:00 pm
Location: Bucharest, Romania
languages_spoken: english
ODROIDs: XU4, C1+, C2

Re: [Howto] Using Mycroft - the Open Source AI

Unread postby odroid » Sat Jul 15, 2017 9:41 am

Very nice tutorial.
I'm trying the instruction to install the Skynet on my XU4. ;)
User avatar
odroid
Site Admin
 
Posts: 23641
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: [Howto] Using Mycroft - the Open Source AI

Unread postby odroid » Sun Jul 16, 2017 1:51 pm

It is running on XU4 now. It took around two hours to build on XU4.
I will play with it early next week because I have no microphone at this moment.
User avatar
odroid
Site Admin
 
Posts: 23641
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: [Howto] Using Mycroft - the Open Source AI

Unread postby mad_ady » Sun Jul 16, 2017 6:44 pm

Great news! :)
User avatar
mad_ady
 
Posts: 2035
Joined: Wed Jul 15, 2015 5:00 pm
Location: Bucharest, Romania
languages_spoken: english
ODROIDs: XU4, C1+, C2

Re: [Howto] Using Mycroft - the Open Source AI

Unread postby odroid » Tue Jul 18, 2017 2:44 pm

I turned XU4 on this morning and Mycroft started to complain loudly about the activation.
The pairing has been completed. But it seems not to understand my bad English pronunciation. :D
Anyway, it's time to play with AI.
User avatar
odroid
Site Admin
 
Posts: 23641
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: [Howto] Using Mycroft - the Open Source AI

Unread postby mad_ady » Tue Jul 18, 2017 4:14 pm

To be honest, I installed it mostly for the kids - to give them a challenge to improve their english pronounciation (actiually the two-year old doesnt' fit the bill, but the older one can play with mycroft). It gives them an incentive to learn english, even if they wouldn't use Mycroft to its best ability.

The Mycroft team is working on adding support for other languages: https://github.com/MycroftAI/adapt/issues/3, but I expect it will take a while.
Edit: It seems french is in beta testing: https://github.com/MycroftAI/mycroft-co ... -in-French
... and so are German and Spanish: https://github.com/MycroftAI/mycroft-co ... in-Spanish

Also, if it can't understand your accent, you can still use the cli component. If you connect to the mycroft-cli screen (use screen -list and screen -r 1234.mycroft-cli, you can type in your commands into that window, and it will process them as if it heard them and would still speak the result.
User avatar
mad_ady
 
Posts: 2035
Joined: Wed Jul 15, 2015 5:00 pm
Location: Bucharest, Romania
languages_spoken: english
ODROIDs: XU4, C1+, C2

Re: [Howto] Using Mycroft - the Open Source AI

Unread postby odroid » Tue Jul 18, 2017 4:46 pm

After slightly increased input volume on the pulseaudio settings, it started to understand my ugly English. :o
The microphone in the webcam seemed to have a low volume.
Anyway, talking to an AI was a very fun experience.

I fully agree this is a nice tool for learning English. :)
User avatar
odroid
Site Admin
 
Posts: 23641
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID


Return to Ubuntu (All Linux'es)

Who is online

Users browsing this forum: No registered users and 2 guests